FreeType backend enhancement for Rasterization tests #5

HinTak · 2015-12-01T03:55:31Z

The code for the MS font renderer was removed and not opened. In any case, it (at least the 2003 version) chokes on about 10% of Mac OS X shipped fonts, so the situation is begging for an update and replacement.

Basic interaction is already established with a FreeType backend for HDMX/LTSH/VDMX tests. The MS font renderer is capable of detecting about 60 errors and 14 warnings for glyph byte-code instructions.

FreeType is currently only capable of detecting only a subset of these, and also only under debug conditions. There is further work to make FreeType detecting a bigger proportion of these, and also some API change/addition to obtain that information in a normal release build.

See also the notes on the hybrid branch.

HinTak · 2016-02-10T05:58:17Z

Implementing the whole 60 errors and dozen warnings from the closed source MS renderer seems daunting; so I looked into what is achievable/possible. Last summer I collected the test results of the 2003 binary against the fonts in win 8.1. The errors and warnings shown is actually a very restricted set:

warnings:
3066 Instruction is only valid on the Apple platform
3555 Projection and freedom vectors at or near perpendicular

errors:
4 Instruction already defined by rasterizer
69 Not called from pre-program
885 Point out of range
22788 RP1 and RP2 have the same position on the projection vector
1243 X and Y components of vector are invalid. X^2 + Y^2 != 0x4000^2

"Instruction is only valid on the Apple platform" was determined to be bogus in previous private discussion with Werner. Out of this small list, "RP1 and RP2 have the same position on the projection vector" seems the obvious one to look at, as it is much more frequent by far, 90% of errors by instances, and also by other metrices (fonts involved, etc).

grep'ing for 'rp1' in freetype sources show that back in Feb 2013, a very localized change in Freetype was made to make it cope with such brokenness in fonts to match the MS renderer's exact behavior.
(
http://savannah.nongnu.org/bugs/?38211
https://bugs.freedesktop.org/show_bug.cgi?id=16442
http://savannah.nongnu.org/bugs/?40975
)

So here is a plan: at exactly the same condition, a diagnostic enhanced Freetype should abort and return an error detail, instead of silently carrying on and working around brokenness in fonts.

This goes towards implementing a test on already fairly well-behaved fonts, to make them even better - there are much more serious brokenness, I am sure, but it is a beginning to start something.

HinTak · 2016-07-02T16:34:45Z

Turned out that the code which silently works around the "Projection and freedom vectors at or near perpendicular" condition was in freetype 2 at the initial commit, so it was in Freetype 1 for nearly 20 years.

It is this section in Compute_Funcs:

    /* at small sizes, F_dot_P can become too small, resulting   */
    /* in overflows and `spikes' in a number of glyphs like `w'. */

    if ( FT_ABS( exc->F_dot_P ) < 0x400L )
      exc->F_dot_P = 0x4000L;

but the MS Rasterer seems to only warn when this condition happens inside Direct_Move, which is narrower.

HinTak · 2016-07-03T20:47:59Z

Out of the 7:

The 'Instruction is only valid on the Apple platform' was bogus from discussion almost a year ago;
'RP1 and RP2 have the same position on the projection vector' and 'Projection and freedom vectors at or near perpendicular' are both silently worked around in freetype. (the former with an 'XXX undocumented' comment, the latter without).

the rest:

'X and Y components of vector are invalid. X^2 + Y^2 != 0x4000^2' is also silently worked around - it is roughly the 'XXX undocumented' in Normalize().
'Point out of range' is essentially what freetype does in pedantic mode. The existing pedantic mode is largely just a bound check on points, and abort on error but continue/ignore when not pedantic.
'Instruction already defined by rasterizer' : freetype allows silent re-definition of instructions; relatively easy to add a diagnostic message when re-definition is encounted.
I can't easily see how to detect calling from pre-program or not, within the byte-code interpreter. answers?

So out of the 7, 1 is bogus, 5 freetype already detects and silently copes (2 have an "XXX undocumented), 1 I haven't figure out how to do.

HinTak · 2016-07-03T21:39:59Z

To detect whether the current glyph is composite is just "exc->is_composite", and calling from pre-program is "exc->curRange == tt_coderange_font" .

So I seem to be able to add all 6 diagnostics.

HinTak · 2016-07-11T11:51:36Z

Fonts on fedora 24 at Size 10, show these 18 codes, and took just over 7 hours to run:

$ grep -E '(P60|A60|E60|W60|I60)' -h fedora-test-files/*.txt | cut -f 4 -d '"' | sort | uniq -c
      1 A   | A6000 | An exception occurred during rasterization testing
     11 E   | E6004 | Contour out of range
    320 E   | E6005 | CVT Out of range
    910 E   | E6020 | Function not defined
    862 E   | E6039 | Point out of range
    472 E   | E6043 | RP1 and RP2 have the same position on the projection vector
    106 E   | E6045 | Stack Overflow
     13 E   | E6046 | Stack Underflow
   1274 E   | E6048 | Storage location not written to
  14292 E   | E6056 | X and Y components of vector are invalid. X^2 + Y^2 != 0x4000^2
     40 E   | E6057 | X and Y components of vector are 0
      2 E   | E6059 | Could not perform rasterization
     36 I   | I6059 | Could not perform rasterization
   1171 P   | P6000 | No problems were found during rasterization testing
 695564 W   | W6008 | Value for count is less than 1. Function will not be called
     18 W   | W6009 | Window 95 and Windows 3.1 always return 12
   7664 W   | W6010 | Projection and freedom vectors at or near perpendicular
     12 W   | W6011 | Point not touched
$ ls fedora-test-files/*.txt | wc -l
1731

HinTak · 2016-07-11T11:59:17Z

It took just over 7 hours. So a standard run, all sizes from size 4 to 72 plus a few higher ones, would take about 3-4 weeks on my computer. And only about 2/3 passes cleanly at this level (testing for more sizes would lower it slightly).

davelab6 · 2016-07-11T13:28:26Z

Nice!

HinTak · 2016-07-11T15:51:53Z

The possible rasterization errors shown by Mac OS X 10.9.5 shipped fonts at size 10 are:

     15 A   | A6000 | An exception occurred during rasterization testing
    180 E   | E6001 | Bits 10 and 13 are set, they are mutually exclusive
    180 E   | E6002 | Bits 8 and 11 are set, they are mutually exclusive
    180 E   | E6003 | Bits 9 and 12 are set, they are mutually exclusive
      6 E   | E6004 | Contour out of range
     32 E   | E6016 | FDEF out of range
    114 E   | E6020 | Function not defined
      3 E   | E6026 | Instruction already defined by rasterizer
    260 E   | E6033 | No twilight zone defined. Invalid zone
    158 E   | E6039 | Point out of range
    180 E   | E6042 | At least one reserved bit is set
   1798 E   | E6043 | RP1 and RP2 have the same position on the projection vector
      6 E   | E6045 | Stack Overflow
      3 E   | E6046 | Stack Underflow
 171160 E   | E6049 | Twilight zone point not set
     48 E   | E6056 | X and Y components of vector are invalid. X^2 + Y^2 != 0x4000^2
      1 E   | E6059 | Could not perform rasterization
    255 I   | I6059 | Could not perform rasterization
    777 P   | P6000 | No problems were found during rasterization testing
    153 W   | W6000 | Instruction is only valid on the Apple platform
      2 W   | W6004 | Repeated executions in glyph programs can have unexpected results
      4 W   | W6007 | Loop variable not 1 at end of program. This means it was set but not used
   3016 W   | W6008 | Value for count is less than 1. Function will not be called
   3903 W   | W6010 | Projection and freedom vectors at or near perpendicular

The number of font files which pass the rasterization test at size 10 is 360/695, or a little over half of them. It took just under 3 hours to analyse all of apple shipped fonts at size 10. The actual proportion of number of fonts which pass is lower - many of them cause the 2003 binary to crash, and therefore not generating a report (i.e. outside the 695).

HinTak · 2016-07-12T04:41:45Z

I have always suspected my private stash of (largely CJK) fonts are rather buggy, so after nearly 12 hours, testing just for size 10, I get a few more exotic errors:

      9 A   | A6000 | An exception occurred during rasterization testing
      1 E   | E6005 | CVT Out of range
      2 E   | E6007 | EIF found without IF
     49 E   | E6039 | Point out of range
   4474 E   | E6043 | RP1 and RP2 have the same position on the projection vector
     20 E   | E6044 | Selector invalid
     10 E   | E6045 | Stack Overflow
      2 E   | E6046 | Stack Underflow
  71454 E   | E6048 | Storage location not written to
   9692 E   | E6049 | Twilight zone point not set
      1 E   | E6054 | Value exceeds capacity of 1 byte number
  18358 E   | E6056 | X and Y components of vector are invalid. X^2 + Y^2 != 0x4000^2
    356 I   | I6059 | Could not perform rasterization
   1403 P   | P6000 | No problems were found during rasterization testing
     21 W   | W6004 | Repeated executions in glyph programs can have unexpected results
   3195 W   | W6008 | Value for count is less than 1. Function will not be called
   2085 W   | W6010 | Projection and freedom vectors at or near perpendicular
    371 W   | W6011 | Point not touched

12 hours at size 10 is about 5-6 weeks for a standard run.

The combined list of apple shipped + fedora shipped + my private stash are:

     25 A   | A6000 | An exception occurred during rasterization testing
    180 E   | E6001 | Bits 10 and 13 are set, they are mutually exclusive
    180 E   | E6002 | Bits 8 and 11 are set, they are mutually exclusive
    180 E   | E6003 | Bits 9 and 12 are set, they are mutually exclusive
     17 E   | E6004 | Contour out of range
    321 E   | E6005 | CVT Out of range
      2 E   | E6007 | EIF found without IF
     32 E   | E6016 | FDEF out of range
   1024 E   | E6020 | Function not defined
      3 E   | E6026 | Instruction already defined by rasterizer
    260 E   | E6033 | No twilight zone defined. Invalid zone
   1069 E   | E6039 | Point out of range
    180 E   | E6042 | At least one reserved bit is set
   6744 E   | E6043 | RP1 and RP2 have the same position on the projection vector
     20 E   | E6044 | Selector invalid
    122 E   | E6045 | Stack Overflow
     18 E   | E6046 | Stack Underflow
  72728 E   | E6048 | Storage location not written to
 180852 E   | E6049 | Twilight zone point not set
      1 E   | E6054 | Value exceeds capacity of 1 byte number
  32698 E   | E6056 | X and Y components of vector are invalid. X^2 + Y^2 != 0x4000^2
     40 E   | E6057 | X and Y components of vector are 0
      3 E   | E6059 | Could not perform rasterization
    647 I   | I6059 | Could not perform rasterization
   3351 P   | P6000 | No problems were found during rasterization testing
    153 W   | W6000 | Instruction is only valid on the Apple platform
     23 W   | W6004 | Repeated executions in glyph programs can have unexpected results
      4 W   | W6007 | Loop variable not 1 at end of program. This means it was set but not used
 701775 W   | W6008 | Value for count is less than 1. Function will not be called
     18 W   | W6009 | Window 95 and Windows 3.1 always return 12
  13652 W   | W6010 | Projection and freedom vectors at or near perpendicular
    383 W   | W6011 | Point not touched

HinTak · 2016-07-12T05:17:12Z

So I have 33 error status (the 32 above, plus 'Not called from pre-program').

The 44 which I don't have a test file are:

> E6000 | Reserved value of 3 used for period
> E6006 | Divide by zero
> E6008 | ELSE found without EIF
> E6009 | ELSE found without IF
> E6010 | ENDF beyond 64K of FDEF
> E6011 | ENDF beyond 64K of IDEF
> E6012 | ENDF found without FDEF or IDEF
> E6013 | IDEF exceeds max instruction defs in maxp
> E6014 | FDEF found within FDEF - ENDF pair
> E6015 | FDEF found within IDEF - ENDF pair
> E6017 | Function definition space not defined
> E6018 | FDEF found without ENDF
> E6019 | Funit coordinate out of range must be -16384 .. 16383
> E6021 | IDEF found within FDEF - ENDF pair
> E6022 | IDEF found within IDEF - ENDF pair
> E6023 | IDEF found without ENDF
> E6024 | IF found without corresponding EIF
> E6025 | Instruction OpCode is to large. Must be between 0 and 255
> E6027 | Invalid Instruction flag
> E6028 | Invalid Instruction
> E6029 | Invalid maxZones in maxp table
> E6030 | Attempt to access stack element out of range
> E6031 | Invalid zone
> E6032 | ZP2 in IUP does not point to zone 1
> E6034 | Jump before beginning of program or function
> E6035 | Jump beyond 2 bytes past end of function or program
> E6036 | Math overflow
> E6038 | Overflow Instruction Stream
> E6040 | Zone referenced in pre-program is not the twilight zone
> E6041 | Reference point used but not set
> E6047 | Storage index out of range
> E6050 | Value invalid should be 0 or 1
> E6051 | Value invalid should be 0 or 2
> E6052 | Value out of range
> E6053 | Value exceeds capacity of 2 byte number
> E6055 | Value too small
> E6058 | Zone not 0 nor 1
> W6001 | CALL to zero length function
> W6002 | Call to zero length user defined instruction
> W6003 | DEBUG found which should not be found in production code
> W6005 | High point is less than low point
> W6006 | Jump offset zero
> W6012 | Function no longer needed because of dropped support of AA
> W6013 | The Freetype version is too old

obviously I do not need a test file for 'The Freetype version is too old'. And some of the others are fairly obvious how it should be checked, even without a test file - and are unlikely to be seen in any in-use font, such as zero-length functions, just because font editors don't allow the user to do things that way.
'FDEF found within FDEF - ENDF pair' I think FreeType already worked around and ignore nested FDEF (that's what it is in FreeType lingo), so is 'Maths overflow' and 'divide by zero'.

HinTak · 2016-07-12T06:53:02Z

Besides those, I seem to have 23 from fedora, 4 from apple and 90 from my private collection, that is 117 font files in total , which can crash the the rasterization test part of the older Font Val without it writing a report behind.

HinTak · 2016-07-13T18:49:28Z

"Storage index out of range" (one of the errors without a test file) is the bound check in Write Store. Number 34.

HinTak · 2016-07-13T18:58:38Z

""Zone not 0 nor 1" is the bound check in Shift Zone. Number 35.

HinTak · 2016-07-13T19:54:36Z

"E6050 | Value invalid should be 0 or 1" and "E6051 | Value invalid should be 0 or 2" both correspond to the 2nd pedantic check in INSTCTRL. The description for "E6052 | Value out of range" in the documentation is a bit vague, but judging from the proximty, it has to be either the same place (i.e. <0 or >3) or the first pedantic check in the same routine in freetype.

So that's number 38, and the first time a silent check in FreeType catches 3 different errors that Font Val catches...

HinTak · 2016-07-13T23:19:29Z

"W6012 | Function no longer needed because of dropped support of AA" is the "/* instruction not supported anymore */" comment in the code. Number 39.

"E6030 | Attempt to access stack element out of range" is the bedantic check in MINDEX and CINDEX. Number 40.

"E6034 | Jump before beginning of program or function" and
"E6035 | Jump beyond 2 bytes past end of function or program" are the bad argument check in JMPR. Number 42

"E6012 | ENDF found without FDEF or IDEF" is Freetype's ENDF_In_Exec_Stream error. Number 43.

"E6014 | FDEF found within FDEF - ENDF pair"
"E6015 | FDEF found within IDEF - ENDF pair"
"E6021 | IDEF found within FDEF - ENDF pair"
"E6022 | IDEF found within IDEF - ENDF pair"
are all Freetype's Nested_DEFS error. Number 47.

W6003 | DEBUG found which should not be found in production code
W6006 | Jump offset zero
are both obvious where they should be. Number 49.

HinTak · 2016-07-14T22:23:36Z

Argh, E6013 is easy. Number 50. Missed it in the new release.

E6018/E6023 - Number 52.

HinTak · 2016-07-15T22:37:20Z

The 26 outside b51 are

An exception occurred during rasterization testing
EIF found without IF
ELSE found without IF
ENDF beyond 64K of FDEF
ENDF beyond 64K of IDEF
Function definition space not defined
Funit coordinate out of range must be -16384 .. 16383
Instruction OpCode is to large. Must be between 0 and 255
Invalid maxZones in maxp table
Invalid zone
ZP2 in IUP does not point to zone 1
Math overflow
Zone referenced in pre-program is not the twilight zone
Reference point used but not set
Twilight zone point not set
Value exceeds capacity of 2 byte number
Value too small
Could not perform rasterization
No problems were found during rasterization testing
Instruction is only valid on the Apple platform
CALL to zero length function
Call to zero length user defined instruction
High point is less than low point
Loop variable not 1 at end of program. This means it was set but not used
The Freetype version is too old
Could not perform rasterization

3 have test files:

EIF found without IF
Twilight zone point not set
Loop variable not 1 at end of program. This means it was set but not used

7 looks well-defined and possibly implementable (though may not be easy):

ELSE found without IF
ENDF beyond 64K of FDEF
ENDF beyond 64K of IDEF
Math overflow
CALL to zero length function
Call to zero length user defined instruction
Invalid maxZones in maxp table

5 already happens in/outside the font engine or not need "implementing":

An exception occurred during rasterization testing
Could not perform rasterization
No problems were found during rasterization testing
The Freetype version is too old
Could not perform rasterization

That leaves 11 which are lacking test files as well as descriptions, and contexts of where and how they happen:

Function definition space not defined
Funit coordinate out of range must be -16384 .. 16383
Instruction OpCode is to large. Must be between 0 and 255
Invalid zone
ZP2 in IUP does not point to zone 1
Zone referenced in pre-program is not the twilight zone
Reference point used but not set
Value exceeds capacity of 2 byte number
Value too small
Instruction is only valid on the Apple platform
High point is less than low point

Some of these might be do'able:

ZP2 in IUP does not point to zone 1
Zone referenced in pre-program is not the twilight zone
Instruction is only valid on the Apple platform

So it might go up from 51 to 3 + 7 + 3 = 64, and there are 5 not needing attention & 8 not likely.

HinTak · 2016-07-16T13:31:12Z

The 117 missing is partly due to name collision.

The problem of collision get worse with bulk test sizes. for 691 files, 4 collided; 1753 files, 22; and 2041 files, 32.

That accounts for 58 of the 117, or just half of those. Another 59 is due to something else.

HinTak · 2016-07-18T06:48:48Z

B54 tagged "FontVal 2.0" includes W6000, E6010, E6011 in addition to the b51 details above.

@anthrotype @davelab6 @aaronbell

Am closing this issue now. This is getting too long, and the rest can become new individual issues.

davelab6 · 2016-07-18T12:10:27Z

Sounds good man :)

fval.xsl: fix xsl for Firefox, fix JS for checkboxes. The fixes here look good as far as I can tell.

HinTak mentioned this issue Dec 9, 2015

193/194 build, dev slide, DSIGInfo tool, and some 2015 spec updates for OS/2 and CBDT/CBLC microsoft/Font-Validator#1

Closed

HinTak mentioned this issue Apr 14, 2016

[semi-off-topic] SFNTQA / Porting FontValidator from C# to Python? fonttools/fonttools#416

Closed

HinTak mentioned this issue Jul 16, 2016

temp file name collision for bulk test of a few hundred files. microsoft/Font-Validator#26

Closed

HinTak closed this as completed Jul 18, 2016

HinTak pushed a commit that referenced this issue May 27, 2017

Merge pull request #5 from moyogo/fval

cd1ee1a

fval.xsl: fix xsl for Firefox, fix JS for checkboxes. The fixes here look good as far as I can tell.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FreeType backend enhancement for Rasterization tests #5

FreeType backend enhancement for Rasterization tests #5

HinTak commented Dec 1, 2015

HinTak commented Feb 10, 2016

HinTak commented Jul 2, 2016

HinTak commented Jul 3, 2016

HinTak commented Jul 3, 2016

HinTak commented Jul 11, 2016 •

edited

HinTak commented Jul 11, 2016 •

edited

davelab6 commented Jul 11, 2016

HinTak commented Jul 11, 2016

HinTak commented Jul 12, 2016

HinTak commented Jul 12, 2016

HinTak commented Jul 12, 2016

HinTak commented Jul 13, 2016

HinTak commented Jul 13, 2016

HinTak commented Jul 13, 2016

HinTak commented Jul 13, 2016

HinTak commented Jul 14, 2016

HinTak commented Jul 15, 2016

HinTak commented Jul 16, 2016

HinTak commented Jul 18, 2016

davelab6 commented Jul 18, 2016 via email

FreeType backend enhancement for Rasterization tests #5

FreeType backend enhancement for Rasterization tests #5

Comments

HinTak commented Dec 1, 2015

HinTak commented Feb 10, 2016

HinTak commented Jul 2, 2016

HinTak commented Jul 3, 2016

HinTak commented Jul 3, 2016

HinTak commented Jul 11, 2016 • edited

HinTak commented Jul 11, 2016 • edited

davelab6 commented Jul 11, 2016

HinTak commented Jul 11, 2016

HinTak commented Jul 12, 2016

HinTak commented Jul 12, 2016

HinTak commented Jul 12, 2016

HinTak commented Jul 13, 2016

HinTak commented Jul 13, 2016

HinTak commented Jul 13, 2016

HinTak commented Jul 13, 2016

HinTak commented Jul 14, 2016

HinTak commented Jul 15, 2016

HinTak commented Jul 16, 2016

HinTak commented Jul 18, 2016

davelab6 commented Jul 18, 2016 via email

HinTak commented Jul 11, 2016 •

edited

HinTak commented Jul 11, 2016 •

edited