Webfont test suites or validation tools, which ones are there

One comment I picked up multiple times while talking/reading about webfonts, is that people are weary about creating WOFF and WOFF2 fonts because it adds so much work. These fonts need to be prepared, tested and validated, will introduce extra risk of broken fonts and cause more support questions. I also get the impression that local tooling, or on-the-fly font generation on a remote server, isn't to be trusted or complicated in usage.

I was wondering if there are any test suites or validation tools that people use now that could be improved, or what the requirements would be such a tool if it doesn't exist for your specific situation. Or examples of how current (command line) tools fudge up things, and how they should be improved.

Comments

  • I suppose most of the testing in the desktop environment involves actually using the font in applications? Are you doing anything special for web, e.g. testing in multiple browsers/OSes, testing different circumstances? I'm trying to get a feel for which tests are manual, and which ones are (or can be) automated.
  • Dave Crossland
    Dave Crossland Posts: 1,430
    edited December 2017
    I've been commissioning better web font testing tools, in the github.com/googlefonts org there are a bunch of projects. 

    FontBakery is the main one, and wraps various others like OTS, HinTak Fontval, etc
  • FontBakery is pretty extensive, nice job @Dave Crossland and all the contributors. Am I correct in summarising this as a "font linter", but for Google Font's standards?

    If found an interesting thread here on TypeDrawers about testing font outside of Adobe products, and came across web-specific test pages like the Font Testing Page and Webfont Testing. Both tools offer lots of tests that need to be executed by human eyeballs and brains — nothing automatable.


  • Correct. There are a few more forks of the impallari tester. Huerta Tipografia and Cyreal also have some script specific comparison tools, and James Puckett has posted some great InDesign test docs. 

    Automation is possible with machine learning and computer vision, Marc Foley had started to work on that
  • Dave Crossland
    Dave Crossland Posts: 1,430
    edited June 2018
    I am curious if anyone here is using Font Bakery? 

    I am organizing a meeting for font distributors and foundries who are using it, or are interested in using it because they face the problems that it aims to solve, in Portland Oregon (USA) on the Thursday August 2nd right before TypeCon. Would anyone here like to join, in person or remotely?
  • Portland is too far a bike ride for me. But depending on the time I'd love to join remotely!

    Will a recording or summary be posted afterwards?
  • Dave Crossland
    Dave Crossland Posts: 1,430
    Summary for sure, I'll look into recording
  • Thomas Phinney
    Thomas Phinney Posts: 2,888
    I would certainly attend the Font Bakery get-together.
  • John Hudson
    John Hudson Posts: 3,193
    I've not used Font Bakery, but would be interested in attending to learn more.
  • I would attend as well. I've also not used Font Bakery but I would be interested in learning more than I gathered reading about it.
  • AbrahamLee
    AbrahamLee Posts: 262
    I would love to attend remotely or see a recording.
  • Dave Crossland
    Dave Crossland Posts: 1,430
    https://github.com/davelab6/mfdos

    I put this up to explain a bit more about what I'm doing here :)
  • Ah, superb, I had tested fontbakery before but hadn't found the info on how to write your own specs and dismissed it as a Google Fonts compliance check (with useful batteries included). Thanks for mentioning that link in the mfdos @Dave Crossland, might very well integrate some checks via fontbakery into my own workflow!

    It would be great to have a video of that meetup to learn more.
  • Dave Crossland
    Dave Crossland Posts: 1,430
    Lasse tried to make it very light weight to write new checks :) I just went over the documentation, and I think we still can do more to make it a better generic tool. We welcome any suggestions on the issue tracker!
  • María Ramos
    María Ramos Posts: 100
    I haven’t used Font Bakery, but I would like to know more about it and join the meeting remotely if possible.
  • María Ramos
    María Ramos Posts: 100
    I know I am late to the discussion... @Roel Nieskens from what I know there’s nothing that automates or substitute human eyes and brains in things like hinting and visual appearance of webfonts. I always end up testing the fonts in different operating systems and browsers, there are some tools that may help to make this process easier and faster. I haven’t used it yet, but BrowserStack allows you to test the same web in different OS, not sure how faithful is the rendering though. Retina screens don’t help either if you wanna test different screen resolutions.
  • Dave Crossland
    Dave Crossland Posts: 1,430
    María, you are totally right. We developed a visual comparison tool which enables distributors to compare hinting and review visual changes (ignoring glyphs which didn't change) using BrowserStack. 

    1. https://github.com/googlefonts/gfregression

    This uses BrowserStack to produce visual comparison image GIFs that toggle before/after, and we include this in each github Pull Request (eg https://github.com/google/fonts/pull/1475)

    You can try it out at http://gf-regression.com

    2. https://github.com/googlefonts/fontdiffenator

    This is an underlying tool, which uses FontTools to compare fonts "piece by piece". For example, it has been used to ensure the named instances in a VF version of Roboto are all a 1:1 match with the pre-VF files, https://github.com/TypeNetwork/Roboto/issues/4

    Currently this is not integrated into the fontbakery dashboard, but will be soon.
  • @Dave Crossland - looking forward to your Font Bakery presentation at TypeCon!
  • I'll try to write up the meeting soon. 

    Meanwhile, there is another upcoming Font Bakery event at ATypI on 11th September in Antwerp:

    https://www.atypi.org/conferences/antwerp-2018/workshops
  • @Dave Crossland - I am using font-bakery for testing my font building scripts on Calligraphr.com. It helped me to find some bugs and avoid regressions. Thank you for your work.