Rounding error questions

Cory Maylett

During interpolation, rounding errors seem unavoidable. The obsessive-compulsive part of me wants to meticulously fix them, but is this considerable effort wasted?

First, since almost no one will ever notice a piece of a glyph being a unit off, does it really matter? Anyway, proper hinting will make rounding errors mostly irrelevant at smaller, low-res sizes. Correct? No? Maybe?

Second, it's my understanding that the set stem values in the interpolated fonts will serve to pull, at least, minor stroke rounding errors back into line during end-user rasterization, which mitigates much of the problem. Again, is this correct?

So depending on whether or not all, some or none of my understandings are misguided, am I missing something important? Are these single-unit interpolation errors non-concerns? Are there some instances where they are a concern?

Thomas Phinney

Cory Maylett said:

During interpolation, rounding errors seem unavoidable. The obsessive-compulsive part of me wants to meticulously fix them, but is this considerable effort wasted?

Maybe not “wasted” but at least... a level of fiddly tweaking few people bother with.

That said, FontLab VI can interpolate and use hinting (link) information to maintain stem thicknesses during interpolation, hence eliminating rounding inconsistencies.

First, since almost no one will ever notice a piece of a glyph being a unit off, does it really matter? Anyway, proper hinting will make rounding errors mostly irrelevant at smaller, low-res sizes. Correct? No? Maybe?

Yes, if you have the hinting, and assuming that the hints make the stem thicknesses end up the same.

Second, it's my understanding that the set stem values in the interpolated fonts will serve to pull, at least, minor stroke rounding errors back into line during end-user rasterization, which mitigates much of the problem. Again, is this correct?

Yes, absolutely.

So depending on whether or not all, some or none of my understandings are misguided, am I missing something important? Are these single-unit interpolation errors non-concerns? Are there some instances where they are a concern?

This is a great philosophical question. Some people are more bothered by this issue than others. Most have decided to live with it and don’t much worry about it any more. But if you want to have your interpolated instance fonts not have rounding inconsistencies, you should use FontLab VI.

Adobe used to have a tool they used in font production, that did something quite similar in spinning out instance fonts while using hinting to maintain stem thickness consistency. I don't think it survived the Y2K transition, if I remember correctly.

Hrant Հրանդ Փափազեան Papazian

Set your extremes and choose your interpolation points such that you end up with integers.

Ray Larabie

It's relevant for ultralights or light fonts with little rounded corners.

In UPM 1000, the difference between 9, 10 or 11 unit stems is conspicuous, especially when dealing with diagonals. But the difference between 29,30,31 unit stems is imperceptible. When dealing with black or ultra-black, the counters present a similar problem. If you've got symmetrical inner curves, it's even more of a problem. Let's say you have an ultra-black Bodoni O and the counter is less than 20 wide. You're really going to notice asymmetry on those inner curves and TrueType conversion will probably make it worse.

As for light fonts with rounded ends the biggest issue is diagonals. I manually resnap the bezier handles for every one of them and do some back and forth Postscript/TrueType tests to make sure they'll survive conversion. But even with heavier fonts that have rounded ends, the interpolation wreaks havoc on the diagonals ends.

Ben Blom

Another approach: You may set the UPM of your font between 3000 and 4000, to reduce rounding errors and to increase design space.

Hrant Հրանդ Փափազեան Papazian

But some legacy RIPs will choke.

Cory Maylett

Thank you Thomas, Hrant, Ray and Ben. Your thoughts and advice have mostly confirmed my own tentative thoughts.

Thomas, I do own a copy of FontLab VI. I'm currently finishing up two families started in Glyphs during the rather buggy transition from FontLab V to VI. I probably need to reacquaint myself with FontLab now that it's stabilized. Sometimes it's difficult working on type design in the absence of others around me with more experience. As an art director, I've worked with type for some 30 years, but using type is far different from creating it.

Hrant, so far I've avoided working with non-integer coordinates and always fix the extremes. What I'm having difficulty ignoring are those, otherwise, symmetrical counters in the U or O, for example, that interpolate to, say, an unsymmetrical 451 units. I'm having some difficulty letting those little imperfections go without tracking them down.

Ray, yes, I can see your point, and with the fonts I'm working on, the ultra-lights and ultra-blacks are a concern for the very reasons you mentioned. Then again, those end weights and widths tend to be masters, so interpolation isn't a problem with them. I suppose I should concentrate on getting them right and be a bit more forgiving of the middle weights' rounding flaws that no one but me will ever notice.

Ben, I've been working with a UPM of 2048. Even if I did use a higher UPM, those even smaller rounding errors would still likely nag at me. I'm also a bit concerned, as Hrant mentioned, with software that might not handle those higher UPMs well. How big of a concern it is any longer, I don't really know.

George Thomas

But some legacy RIPs will choke.

Do you have any info about many are still in use? I would think very few.

Hrant Հրանդ Փափազեան Papazian

@George Thomas I have nothing tangible, so you might be right. I do suspect sticking to 1000 is a lesser evil.

Thomas Phinney

Ben Blom said:

Another approach: You may set the UPM of your font between 3000 and 4000, to reduce rounding errors and to increase design space.

Yes, although one then needs to be careful about format limits. IIRC, with PostScript/CFF/CFF2 outlines, all coordinates in each glyph need to be within +/- 4095. (4K, but starting counting at zero.)

(Given your very particular choice of numbers, I am sure you are aware of this—just mentioning it for the benefit of other readers.)

Also, larger coordinate values slightly increase final font file size. I’ve noted going from 1000 to 3000 UPM made about a 5% increase in TTF file size and a 15% increase in OTF file size. This would of course vary depending on the font.