Webfont test suites or validation tools, which ones are there

Roel Nieskens

One comment I picked up multiple times while talking/reading about webfonts, is that people are weary about creating WOFF and WOFF2 fonts because it adds so much work. These fonts need to be prepared, tested and validated, will introduce extra risk of broken fonts and cause more support questions. I also get the impression that local tooling, or on-the-fly font generation on a remote server, isn't to be trusted or complicated in usage.

I was wondering if there are any test suites or validation tools that people use now that could be improved, or what the requirements would be such a tool if it doesn't exist for your specific situation. Or examples of how current (command line) tools fudge up things, and how they should be improved.

Paul van der Laan

It is a question of making correct TTF/OTF fonts in the first place. If these are tested, tried, and validated then there won't be a problem with the resulting WOFF/WOFF2 files.

We spend a lot of time making sure our desktop fonts are thoroughly tested, and this includes the use of MS Font Validator (among others) and testing on multiple platforms. If they are correct then making web fonts is only a matter of using a converter such as Sfntly.

If people complain about the extra work that comes with making webfonts then they probably do not have good source fonts to starts with.

Roel Nieskens

I suppose most of the testing in the desktop environment involves actually using the font in applications? Are you doing anything special for web, e.g. testing in multiple browsers/OSes, testing different circumstances? I'm trying to get a feel for which tests are manual, and which ones are (or can be) automated.

Dave Crossland

I've been commissioning better web font testing tools, in the github.com/googlefonts org there are a bunch of projects. 

FontBakery is the main one, and wraps various others like OTS, HinTak Fontval, etc

Roel Nieskens

FontBakery is pretty extensive, nice job @Dave Crossland and all the contributors. Am I correct in summarising this as a "font linter", but for Google Font's standards?

If found an interesting thread here on TypeDrawers about testing font outside of Adobe products, and came across web-specific test pages like the Font Testing Page and Webfont Testing. Both tools offer lots of tests that need to be executed by human eyeballs and brains — nothing automatable.

Dave Crossland

Correct. There are a few more forks of the impallari tester. Huerta Tipografia and Cyreal also have some script specific comparison tools, and James Puckett has posted some great InDesign test docs. 

Automation is possible with machine learning and computer vision, Marc Foley had started to work on that

Dave Crossland

I am curious if anyone here is using Font Bakery? 

I am organizing a meeting for font distributors and foundries who are using it, or are interested in using it because they face the problems that it aims to solve, in Portland Oregon (USA) on the Thursday August 2nd right before TypeCon. Would anyone here like to join, in person or remotely?

Roel Nieskens

Portland is too far a bike ride for me. But depending on the time I'd love to join remotely!

Will a recording or summary be posted afterwards?

Dave Crossland

Summary for sure, I'll look into recording

Thomas Phinney

I would certainly attend the Font Bakery get-together.

John Hudson

I've not used Font Bakery, but would be interested in attending to learn more.

Mike Wenzloff

I would attend as well. I've also not used Font Bakery but I would be interested in learning more than I gathered reading about it.

AbrahamLee

I would love to attend remotely or see a recording.

Dave Crossland

https://github.com/davelab6/mfdos

I put this up to explain a bit more about what I'm doing here

Johannes Neumeier

Ah, superb, I had tested fontbakery before but hadn't found the info on how to write your own specs and dismissed it as a Google Fonts compliance check (with useful batteries included). Thanks for mentioning that link in the mfdos @Dave Crossland, might very well integrate some checks via fontbakery into my own workflow!

It would be great to have a video of that meetup to learn more.

Dave Crossland

Lasse tried to make it very light weight to write new checks I just went over the documentation, and I think we still can do more to make it a better generic tool. We welcome any suggestions on the issue tracker!

María Ramos

I haven’t used Font Bakery, but I would like to know more about it and join the meeting remotely if possible.

María Ramos

I know I am late to the discussion... @Roel Nieskens from what I know there’s nothing that automates or substitute human eyes and brains in things like hinting and visual appearance of webfonts. I always end up testing the fonts in different operating systems and browsers, there are some tools that may help to make this process easier and faster. I haven’t used it yet, but BrowserStack allows you to test the same web in different OS, not sure how faithful is the rendering though. Retina screens don’t help either if you wanna test different screen resolutions.

Dave Crossland

María, you are totally right. We developed a visual comparison tool which enables distributors to compare hinting and review visual changes (ignoring glyphs which didn't change) using BrowserStack. 

1. https://github.com/googlefonts/gfregression

This uses BrowserStack to produce visual comparison image GIFs that toggle before/after, and we include this in each github Pull Request (eg https://github.com/google/fonts/pull/1475)

You can try it out at http://gf-regression.com

2. https://github.com/googlefonts/fontdiffenator

This is an underlying tool, which uses FontTools to compare fonts "piece by piece". For example, it has been used to ensure the named instances in a VF version of Roboto are all a 1:1 match with the pre-VF files, https://github.com/TypeNetwork/Roboto/issues/4

Currently this is not integrated into the fontbakery dashboard, but will be soon.

Matt Heximer

@Dave Crossland - looking forward to your Font Bakery presentation at TypeCon!

Dave Crossland

I'll try to write up the meeting soon. 

Meanwhile, there is another upcoming Font Bakery event at ATypI on 11th September in Antwerp:

https://www.atypi.org/conferences/antwerp-2018/workshops

Tobias Reinhardt

@Dave Crossland - I am using font-bakery for testing my font building scripts on Calligraphr.com. It helped me to find some bugs and avoid regressions. Thank you for your work.