[semi-off-topic] SFNTQA / Porting FontValidator from C# to Python? #416

twardoch · 2015-12-01T05:03:52Z

Apologies for a semi-off-topic issue.

Microsoft recently opensourced Font Validator (FontVal) under the super-liberal MIT license. Somewhat unfortunately, Font Validator is written in C#, but it seems to me that, generally speaking, that code is relatively self-contained, and seems to be cleanly written. Do you think it would be feasible to port it to Python (to make "PyFontVal" — which shouldn't really become part of fontTools, much more likely a wholly-separate package)? Given that C# is more strongly typed than Python, it should be possible, at least theoretically.

I've just looked at one function, val_JSTF.cs, and tried to automatically convert a bit of it to Python using the VaryCode converter, which, after the free registration, limits the conversion to 2048 characters.

Here's the input:

 using System;

using OTFontFile;
using OTFontFile.OTL;

namespace OTFontFileVal
{
    /// <summary>
    /// Summary description for val_JSTF.
    /// </summary>
    public class val_JSTF : Table_JSTF, ITableValidate
    {
        public val_JSTF(OTTag tag, MBOBuffer buf) : base(tag, buf)
        {
            m_DataOverlapDetector = new DataOverlapDetector();
        }


        public bool Validate(Validator v, OTFontVal fontOwner)
        {
            bool bRet = true;


            // check the version
            if (v.PerformTest(T.JSTF_Version))
            {
                if (Version.GetUint() == 0x00010000)
                {
                    v.Pass(T.JSTF_Version, P.JSTF_P_Version, m_tag);
                }
                else
                {
                    v.Error(T.JSTF_Version, E.JSTF_E_Version, m_tag, "0x" + Version.GetUint().ToString("x8"));
                    bRet = false;
                }
            }

            // check the JstfScriptRecord array length
            if (v.PerformTest(T.JSTF_JstfScriptRecord_length))
            {
                if ((uint)FieldOffsets.JstfScriptRecords + JstfScriptCount*6 > m_bufTable.GetLength())
                {
                    v.Error(T.JSTF_JstfScriptRecord_length, E.JSTF_E_Array_pastEOT, m_tag, "JstfScriptRecord array");
                    bRet = false;
                }
            }

            // check that the JstfScriptRecord array is in alphabetical order
            if (v.PerformTest(T.JSTF_JstfScriptRecord_order))
            {
                if (JstfScriptCount > 1)
                {
                    for (uint i=0; i<JstfScriptCount-1; i++)
                    {
                        JstfScriptRecord jsrCurr = GetJstfScriptRecord_val(i);
                        JstfScriptRecord jsrNext = GetJstfScriptRecord_val(i+1);

                        if (jsrCurr.JstfScriptTag >= jsrNext.JstfScriptTag)
                        {
                            v.Error(T.JSTF_JstfScriptRecord_order, E.JSTF_E_Array_order, m_tag, "JstfScriptRecord array");
                            bRet = false;
                            break;
                        }
                    }
                }
            }

            // check each JstfScriptRecord
            if (v.PerformTest(T.JSTF_JstfScriptRecords))
            {
                for (uint i=0; i<JstfScriptCount; i++)
                {
                    JstfScriptRecord_val jsr = GetJstfScriptRecord_val(i);

                    // check the tag
                    if (!jsr.JstfScriptTag.IsValid())
                    {
                        v.Error(T.JSTF_JstfScriptRecords, E.JSTF_E_tag, m_tag, "JstfScriptRecord[" + i + "]");
                        bRet = false;
                    }

                    // check the offset
                    if (jsr.JstfScriptOffset > m_bufTable.GetLength())
                    {
                        v.Error(T.JSTF_JstfScriptRecords, E.JSTF_E_Offset_PastEOT, m_tag, "JstfScriptRecord[" + i + "]");
                        bRet = false;
                    }

                    // validate the JstfScript table
                    JstfScript_val js = jsr.GetJstfScriptTable_val();
...

and here is the generated Python code:

from System import *
from OTFontFile import *
from OTFontFile.OTL import *

class val_JSTF(Table_JSTF, ITableValidate):
    """ <summary>
     Summary description for val_JSTF.
     </summary>
    """
    def __init__(self, tag, buf):
        self._m_DataOverlapDetector = DataOverlapDetector()

    def Validate(self, v, fontOwner):
        bRet = True
        # check the version
        if v.PerformTest(T.JSTF_Version):
            if Version.GetUint() == 0x00010000:
                v.Pass(T.JSTF_Version, P.JSTF_P_Version, m_tag)
            else:
                v.Error(T.JSTF_Version, E.JSTF_E_Version, m_tag, "0x" + Version.GetUint().ToString("x8"))
                bRet = False
        # check the JstfScriptRecord array length
        if v.PerformTest(T.JSTF_JstfScriptRecord_length):
            if FieldOffsets.JstfScriptRecords + JstfScriptCount * 6 > m_bufTable.GetLength():
                v.Error(T.JSTF_JstfScriptRecord_length, E.JSTF_E_Array_pastEOT, m_tag, "JstfScriptRecord array")
                bRet = False
        # check that the JstfScriptRecord array is in alphabetical order
        if v.PerformTest(T.JSTF_JstfScriptRecord_order):
            if JstfScriptCount > 1:
                i = 0
                while i < JstfScriptCount - 1:
                    jsrCurr = self.GetJstfScriptRecord_val(i)
                    jsrNext = self.GetJstfScriptRecord_val(i + 1)
                    if jsrCurr.JstfScriptTag >= jsrNext.JstfScriptTag:
                        v.Error(T.JSTF_JstfScriptRecord_order, E.JSTF_E_Array_order, m_tag, "JstfScriptRecord array")
                        bRet = False
                        break
                    i += 1
        # check each JstfScriptRecord
        if v.PerformTest(T.JSTF_JstfScriptRecords):
            i = 0
            while i < JstfScriptCount:
                jsr = self.GetJstfScriptRecord_val(i)
                # check the tag
                if not jsr.JstfScriptTag.IsValid():
                    v.Error(T.JSTF_JstfScriptRecords, E.JSTF_E_tag, m_tag, "JstfScriptRecord[" + i + "]")
                    bRet = False
                # check the offset
                if jsr.JstfScriptOffset > m_bufTable.GetLength():
                    v.Error(T.JSTF_JstfScriptRecords, E.JSTF_E_Offset_PastEOT, m_tag, "JstfScriptRecord[" + i + "]")
                    bRet = False
                # validate the JstfScript table
                js = jsr.GetJstfScriptTable_val()
...

At least it does not look bad. Perhaps, after a conversion and some refactoring, this could be used as a basis for a more-generic font validation library which could integrate fontTools, the "PyFontVal" port and a somewhat refactored AFDKO CompareFamily.py ?

twardoch · 2015-12-01T05:30:14Z

For the sake of completion, I’m attaching below my writeup proposing SFNTQA from June 2014 that I circulated among some interested parties.

I'd like to re-warm the idea that I pitched to Behdad in November 2013 under the name "pyftvalidator":
A validation suite for SFNT fonts with fontTools in the middle.

For the time being, I'd like to call the project / tool "SFNTQA". Actual final name may differ.

In short, I'm proposing:

to start a new project codenamed SFNTQA which will result in an extensible, flexible Python framework which can be used for validation and correction of SFNT fonts
SFNTQA should be licensed under permissive open-source licenses, as it's "infrastructure"
SFNTQA should probably be based on fontTools/TTX but should be able to involve other tools in the process
lots of concepts and components are already in place, but currently, they're not integrated in any way; I'm proposing to organize and integrate them better

Below is a more detailed discussion of my proposal. Please feel free to comment, and also to add other relevant people to this thread if needed.

What We Had in the Past

[MSFontVal] Microsoft Font Validator

http://www.microsoft.com/typography/FontValidator.mspx
https://github.com/HinTak/Font-Validator
Requires real Windows (.NET) or Mono
Primarily GUI
Written in C#
(Update 2015) open-source under MIT

Probably the first tool that tackled SFNT validation was Microsoft Font Validator (MSFontVal for short). It was originally very useful as a GUI validator per se (written as a .NET GUI app), but today it's rather dated. It hasn't seen updates in years, so, for example, it complains about "ss01" being an "unregistered tag", doesn't recognize recent additions to the OT standard. I don't think Microsoft is interested in maintaining it.

Also, the particular report style has been pioneered by MSFontVal (XML+CSS) was rather neat. It has established a nice model for generating reports, with four color-coded levels (info, pass, warning, error). It has inspired some other efforts.

Both FontVal and FontQA generate XML+XSL reports. Both reports came as XML+XSL. They don't render well on modern browsers, just MSIE -- because they're really both quite old packages. But they show the principle.

What We Have Now: Python

Two packages that do some QA/validation and are written in Python already exist.

[FontQA] FontQA

http://www.fontqa.com/download/archive/index.html
Written by Andreas Eigendorf
Written in Python
Published FSI FontShop International GmbH
Published under the GPL license but it should be possible to ask them to re-published under a more liberal license.

Pros:

Implements numerous functional tests developed for real-life font publishing scenarios (FontShop International)
Outputs a nice human-readable XML+CSS format inspired by MSFontVal
Has some concepts for modularity, though possibly not using the newest Python coding techniques
Various functional tests could be probably ported to SFNTQA

Cons:

Runs inside of FontLab Studio 5 only
Relies on the legacy FontLab Python API
Does not work on SFNT fonts but instead inside FontLab's object model

[AFDKO] Adobe FDK for OpenType

http://www.adobe.com/devnet/opentype/afdko.html
https://github.com/adobe-type-tools/afdko
Published by Adobe
Majority open-sourced under Apache 2
Has parts written in Python and has commandline apps
Native Mac OS X and Windows, open-source version also compiles on Linux.

The CompareFamily.py tool is interesting in particular (though there are some other tools of interest, e.g. spot).

Pros (CompareFamily):

Implements numerous functional tests developed for real-life font publishing scenarios (Adobe)

Cons (CompareFamily):

Output is somewhat messy
The code is rather messy, evident by the comment in the source code right at the beginning:

#       This is not code to copy . The original developer did not know Python well,
#       and had odd ideas about design. However, it works.

What We Have Now: Commandline / Lib

In addition, there is a number of commandline tools which work on different OSes, can be run externally and return some useful results.

[TTX] fontTools/TTX

https://github.com/behdad/fonttools/ -- current fork by Behdad Esfahbod
written entirely in Python
has some basic formal tests (fails if tables malformed)
could be used to build the entire infrastructure upon it
Behdad's fork has the subset.py module which shows how the code could be written in a modern way

[WOFFT] woffTools

https://github.com/typesupply/woffTools
written entirely in Python
written by Tal Leming
provides a nicely-organized architecture to perform unit testing on fonts
the structure could be ported/re-used/extended/modified for SFNTQA

[OTS] OT Sanitiser

https://github.com/khaledhosny/ots -- more current by Khaled Hosny
Builds on all OSes as commandline
various formal tests
used in some browsers to validate webfonts
"corrects" problems brute-force by removing various tables or rejecting the font entirely

[FTX] OSX Font Tools Release 4, Beta 1

https://developer.apple.com/downloads/download.action?path=Mac_OS_X/os_x_font_tools/osxfonttools.dmg
You need to sign in with an Apple ID to download
Requires real Mac OS X
includes the ftxvalidator commandline tool which performs numerous formal tests and produces an extensive .plist XML report
uses the same tests as OS X Font Book when the user installs a font

[FT2] FreeType 2 Demos

http://sourceforge.net/projects/freetype/files/freetype-demos/
Builds on all OSes
has a small tool ftvalid which performs some layout table formal tests
has other tools which could be used for rasterization validation (?)
could be used for visual validation and aesthetic reporting

[FF] FontForge

https://github.com/fontforge/fontforge/
Builds on most OSes as commandline or Python module
works as a Python module
has a number of validation functions, including some functional tests, and can be used to correct major problems

[HB] HarfBuzz

https://github.com/behdad/harfbuzz
Builds on most OSes
includes some commandline apps
could be used for formal validation
could be used for visual validation and aesthetic reporting

[sfntly] Google sfntly

https://code.google.com/p/sfntly/
not sure how it can be made useful at this point
probably useful for "does it open?" basic formal test
possibly useful as part of the visual validation and aesthetic reporting (generating webfonts)

General Scope Of Validation

Here are several types of mandatory and optional features for validation, in the increasing order of complexity.

Mandatory features

formal validation -- checking the structural integrity of various SFNT tables and some formal conformance to some defined constraints
functional validation (aka "sanity checks") -- checking for values or interdependencies which can be formally valid by itself but may not be functional in a broader context, for example may cause some problems in particular deployment environments (best example: if a font with a CFF table has OS/2.usWeightClass < 250, in Windows GDI apps it'll be artificially emboldened, which is undesired by users)
visual validation -- production of visuals (PDF, HTML) which can be then evaluated by designers or technicians to check for problems that need a human eye [optional]

Optional features

correction -- automatically perform some corrections (fixes) for issues that can be automatically corrected
aesthetic reports -- i.e. in addition to visual validation, producing renderings (PDF, HTML) of glyphmaps, font showings, language coverage or font metadata which can be re-used in user documentation [very optional and perhaps outside the scope]

General Process Of Validation

Generally speaking, a validation tool should be able to perform the following operations:

Input

read supplied input parameters such as a list of tests to be performed, some options
read supplied input font files

Processing

perform a set of formal and functional validation tests defined in the supplied parameters
optionally, if requested, perform correction
optionally, if requested, perform visual validation and prepare aesthetic reports

Output

generate validation reports in an extent of detail defined in the supplied parameters, in some machine-readable and human-readable form (e.g. XML for machine, XML+CSS or HTML for human)
if correction was performed, save the corrected fonts in a specified location
if visual validation or aesthetic reports were done, output them in specified format and location

Proposal For The SFNTQA Package

Python

I believe that we will all agree that Python is probably the best starting point for considering validation of SFNT fonts:

it's modular
it's easy to integrate into existing workflows
it's supported on most platforms
three levels of UX can be easily provided (Python module for library use, commandline tools and GUI tools)
numerous libraries already exist in Python which can aid development there
can be used as a "launchpad" for other tools such as commandline tools, or Python-to-C modules could be created for some of the tools that currently have none

Input

SFNTQA should work on SFNT fonts (OTF/TTF, possibly OTC/TTC) only. Other formats (e.g. UFO etc.) are outside the scope.
SFNTQA should work on the level of a "family", in the sense that it should accept multiple fonts which are meant to work together as a family or set of fonts. This is not limited to fonts that share the same Preferred Family Name or anything like that. Processing a single font should be just a special case of the overall principle of processing or "looking at" multiple fonts at a time.

Architecture

SFNTQA should be modular.
SFNTQA should be set up in a way that allows additional, custom modules to be placed in a special location, so that users don't need to mix the predefined modules with custom ones.

Organization of validation

SFNTQA should allow the users to work on the basis of "profiles" where multiple tests and target values are grouped together. One such profile would be the "OFF/X:2014" profile I suggested, but there could be other profiles such as "GoogleFonts:2014" or "MyFonts:2014" etc, but also simply "OpenType-1.6:2006" etc., where various groups, entities or vendors could produce and deploy such profiles. The profiles should have the ability to partially or fully inherit/extend existing profiles.
SFNTQA should be able to test various aspects of the input (i.e. of multiple or single fonts), also across the fonts, for conformity with some expected values.
SFNTQA should be able to perform the Mandatory and Optional features as described in the "General scope of validation" section. In particular, it should perform validation of formal conformity (such as validity of certain values within a prescribed range) but also aspects of functional conformity (i.e. where an aspect is formally correct but is "wrong" from the functional point of view).
SFNTQA should be able to report certain values in several different levels of verbosity. SFNTQA should be, potentially, able to "fix/correct" certain values, at least for some of the tests. So there could be three "modes of operation": "report" (reporting the values that are being tested but without actual testing, and whether they're testable), "test" (report the existing values, their desired values, whether they conform, and whether they're fixable) and "correct" (report the existing values before corrections, their desired values, whether they conform, whenever they're fixable -- perform the corrections and report of the results).

The idea for this condition is that three "levels" of tests for a particular aspect could exists:
a) a level that reports a certain value only
b) a level that reports a certain value and tests against some desired values
c) a level that reports, tests but also performs corrections

Interoperability / external tools

SFNTQA should have its core based on TTX (fontTools, actually), but should be able to call for additional tools/packages such as the FT2, FF, sfntly, OTS, AFDKO, FTX, HB and other commandline tools. It should deal gracefully with absence of those additional tools, i.e. module developers should have an easy way for writing optional modules that utilize those tools, and define the behavior what should be done if a tool isn't available. Also, the module developers should have the ability to provide fallback scenarios for external tool availability, i.e. e.g. to implement a more extensive version of a test if AFDKO or sfntly is available, but only perform a basic version of the test if these tools are not available.

Documentation

SFNTQA should have good documentation for people who'd like to contribute additional tests or modules.

Technical requirements

SFNTQA should be compatible with Python 2.7 and Python 3.
SFNTQA should probably be based on TTX (fontTools) as the minimal requirement. The effort should be made that a basic set of tests is implemented using TTX only.
SFNTQA could possibly use the Python unittest module:
https://docs.python.org/2/library/unittest.html
to actually implement the various tests. I'm no expert on unit testing at all, [WOFFT] looks convincing.
SFNTQA should probably use the Python logging module:
https://docs.python.org/2/library/logging.html
to gather the tests and then pass on to the output formatters.
SFNTQA should provide an initial way to generate output, and provide ways to implement human- and machine-readable outputs through some formatters.

Examples

Initial examples of tests that could be implemented are:

Character set

a) testing the character set against a defined character set. As test data, the Adobe Latin 1 character set could be used:
http://blogs.adobe.com/typblography/latin_charsets/Adobe_Latin_1.html
http://blogs.adobe.com/typblography/2008/08/extended_latin.html

b) testing whether all input fonts have the same character set

Linespacing

a) testing whether each set of linespacing values (hhea.Asc/Desc/LGap and OS/2.WinAsc/Desc and OS/2.TypoAsc/Desc/LGap) is consistent across all input fonts

b) testing whether the three sets of linespacing values within each font yield the same total value

c) reporting what effecitve line height in relation to the UPM the total linespacing values will yield (in "em" or percentage, e.g. 120% if the UPM is 1000 and the total linespacing is 1200).

SFNT tests

For a set of SFNT tables which are reasonably expected to have the same values, or at least some fields within those tables are expected to have the same values across all fonts, test whether it is true.

Conclusions

These are the initial conditions I can think of for such package.

I think once an initial framework is in-place, we will find developers in the community who will implement additional modules, especially since people from Google, Adobe, the UFO/RoboFab community, but also FontLab and potentially Apple are already familiar with TTX.

As you see, my goal is to have a multi-headed beast of a system. But I think the font community is really ready for this system. The problem so far has been that most of existing tools were written by people who don't really have much of an idea for good software architecture, and if in Python, they were written using old Python paradigms (old-style classes, no inheritance of classes etc.).

The best "worst" example is the Adobe CompareFamily.py tool -- which does its job but is not really done in a "proper" way (as Adobe acknowledges).
With a new framework in place, most of the CompareFamily tests could be easily ported to the new system.

We would need

a project manager, who'd set up the location for the project and manage the general flow
an initial Python technical lead, who would look at what's already there (my notes and the existing software I listed above), write technical concepts and the initial barebones of the framework
a Python developer who would port tests from tools such as CompareFamily or FontQA, write integration code for some external tools (e.g. FTX) and implement an initial formatter
funding for the above.

Someone needs to design the foundation of the framework with extensibility in mind. Someone needs to set up this pluggable architecture and tell the community how to extend it. Then, we'll see contributions. I'm certain that I'll contribute, but I'm not able to do the duties described above. But I would certainly act as a consultant/contributor (unpaid of course), and so would probably some of you.

So I'd like all of you to think about it yourselves but also, if possible, gather additional support from wherever you can.

There are additional players in the industry whom I'd like to involve in this while the project gets some initial traction.

Please let me know what you think about this.

Best,
Adam

twardoch · 2015-12-01T05:39:57Z

@davelab6 ping

davelab6 · 2015-12-01T07:22:21Z

I think this consolidation is a fine idea :) however, I'm not sure who wants to actually put the work in. I wonder if any Googlers would volunteer? :)

behdad · 2015-12-09T09:40:07Z

My current thinking:

If fully-automated / very low-overhead conversion to Python is possible, yes, that would be interesting. I haven't run the validator yet exactly because I don't have time to deal with getting it running on mono,
I don't think integrating with fonttools makes sense per se. If we were to "integrate", that would be a rewrite of fonttools and validator. What's more interesting IMO is to write a lint tool based on checks in the validator,
At any rate, this is huge amount of work (more than writing fonttools from scratch), so unless there's someone who can do the work, discussing is rather moot.

HinTak · 2016-04-01T22:26:10Z

TL;DR... (only recently learned this short-hand, seems approriate to use).

Am a bit disappointed that neither Dave or Behdad mentioned our previous small-group discussion: .net dll's are usable as python classes under IronPython (python interpreter implemented on top of .net - it is shipped as a standard part of Mono). Assuming ttx runs under IronPython, you can sort of seamlessly go between ttx and Font Validator's internals within the same python session.

I did some experiments in that direction (as in playing with Font Validator's intenals under IronPython) rather early on. Didn't not spend more time on it, just because there wasn't any demand/interest.

HinTak · 2016-04-01T22:33:15Z

The discussion happened towards the very beginning around/before even the MS code fell in my hand. I did a bit of looking in that direction, because, you know, the original plan with Font Validator was just so that some its code is released to allow its knowhows to go into ttx/fonttools...

HinTak · 2016-04-01T23:40:25Z

Actually I have documented this possibility: in the Extended TODO's, line 45
( https://github.com/HinTak/Font-Validator/blob/master/README-extra.txt )

"IronPython scripted access to Font Validator's internals."

behdad · 2016-04-13T22:54:56Z

Am a bit disappointed that neither Dave or Behdad mentioned our previous small-group discussion: .net dll's are usable as python classes under IronPython

Humm. Not sure if I was part of that discussion.

HinTak · 2016-04-14T11:31:25Z

@behdad : Argh. sorry. @davelab6 , @anthrotype , as well as of course @aaronbell , was in on that 2-3 e-mail discussion about IronPython.

Here is a quick intro about IronPython:

it is shipped as a standard part of mono (at least for windows and mac mono; I can run native Linux mono with the ipy installed under wine, so it is stand-alone in that sense) - . ipy.exe / ipy64.exe / ipy64.exe / ipyw.exe / ipyw64.exe . The *w variant is console-less (a windows concept, i.e. when runs under windows, it does not try to open a console window).
you can run either in batch mode or interactively just like CPython. I think there is a small difference in interactive mode about tab-completion, you need a special switch to ipy for it to imitate CPython's tab-completion behavior, because by default tab behaves like how it behaves under cmd.exe to make windows people more comfortable.
version-wise, I think it is slightly behind, not CPython 2.7 compatible but more like 2.5/2.6 .
IronPython cannot use shared libraries/compiled dll's like Cpython does; but .net/mono dll's behaves like python classes under IronPython, so you should be able to get at Font Val's internal from python that way.

In terms of which part(s) of Font Val you want to draw from to enhance FontTools, I have come to the conclusion that the main value of Font Val is in the glyf tests and the rasterization tests. The proprietary rasterization test backend was not released by Microsoft, and that part is pending enhancement in FreeType ( HinTak/Font-Validator#5 ) to do that; but once FreeType is up for it, FontTools does not need to go through FontVal to get at that, but can interactive with FreeType directly, so that's outside the current discussion.

If you want to port over the glyf test from Font Val, the bulk of the code is in "GMath" and "Glyph" . Yes, the glyf test in Font Val has two dll's devoted to it, whereas all the rest of the table parsing machinery is in "OTFontFile" , and all the table checking machinery is in "OTFontFileVal" . I reckon FontTools does not need either of those and people looking to put some of Font Val's logic into FontTools should concentrate on "GMath" and "Glyph", but you are free to think otherwise.

HinTak · 2016-04-14T11:36:04Z

I'd be interested to know how much of FontTools works under IronPython, so if there is a testsuite or something that doesn't take too much (human) time to do, I'd be interested to hear about it.

anthrotype · 2016-04-14T12:26:38Z

You just need to
pip install pytest
Then you can run the tests suite with the command
py.test

HinTak · 2016-05-02T19:01:09Z

@behdad @davelab6 @anthrotype @aaronbell @twardoch @miguelsousa @n8willis : I needed a quick and dirty script to get at fonts' creation/modification times, so I wrote one in IronPython - it demonstrates how to use CPython modules with IronPython, and how to access Font Validator's internals from a python script. Edit the two lines after "NOTE:" to your preference.

You need MS Font Validator, mono, ironpython (it comes with the windows version of mono definitely, and possibly also the mac OS version); CPython is optional.

IronPython example to use Microsoft Font Validator's internals to get at font creation and modification times:
https://gist.github.com/HinTak/8c08a4f3ac9d787c382139506c259ffd

HinTak · 2016-05-02T19:11:55Z

I was a bit wrong about some of IronPython bits - current ipy are at 2.7.5 and 3.0, and not too far behind; and should be sufficient to run the pure python part of ttx/FontTools .

HinTak · 2016-05-03T10:43:12Z

@behdad @davelab6 @anthrotype @aaronbell @twardoch @miguelsousa @n8willis : Another IronPython example script to use FontVal's internals via python - this one does the same thing as "ttx -l ", except it recurses through ttc members and also takes multiple font file arguments.

https://gist.github.com/HinTak/3cbc107a4394b0546f4647ecbe49d92c

behdad · 2016-05-03T12:21:28Z

To be honest, I'm not sure what value this provides...

HinTak · 2016-05-03T13:24:20Z

I am not sure either, but there are at least 3 people who said re-writing FontVal into python is a good idea :-).

I am just offering an alternative: the IronPython interpreter is a net executable - in fact it is a small wrapper around about 3 dotnet/mono dll's. So in principle (well, it is not too hard), I can bundle it with FontVal to give it the ability of accessing any native python script or modules, old or new. That includes ttx, and part of afdko which are reasonably standalone. Instead of re-writing FontVal into Python, it can take on an embedded python-interpreter, and be able to access and process some python script/modules. It is about an extra 4MB.

behdad · 2016-05-03T13:51:25Z

Well, porting it to Python is useful to avoid having to install mono. You add IronPython, and now we have two problems. :)

HinTak · 2016-05-03T14:06:01Z

Haven't you heard? Since January, FontVal does not need a separate installation of mono on Mac OS X anymore. (and it shouldn't need mono on windows). Adding IronPython does not change that.

I don't package it for Linux that way, because the system call interfaces on Linux change too often, and executable's just aren't very transferable between Linux boxes of different vintage/distro.

khaledhosny · 2016-05-03T14:30:15Z

because the system call interfaces on Linux change too often

Huh?

HinTak · 2016-05-04T21:03:01Z

More FontVal-based IronPython scripts, to merge/split ttc's.
https://gist.github.com/HinTak/33cdcb8e1558538389cb5e0475674acc
https://gist.github.com/HinTak/96a686572a824d78f8e1967b464821aa

HinTak · 2016-05-06T10:10:04Z

@schriftgestalt @aaronbell - The sheer number of parameters set-able for the rasterization test ( HinTak/Font-Validator#5 ) is hugh. It would actually make sense to just add one new option "-rp-script parameters.py" to set all of them in a heavily annotated python script. That adds an extra 4MB to bundle an embedded IronPython interpreter core, instead of adding a dozen options to set them individually, and adding all the option-parsing and string-to-number conversion code.

The total of rasterization parameters is basically all the public members of the Validator Parameter class ( https://github.com/HinTak/Font-Validator/blob/master/OTFontFileVal/ValidatorParameters.cs ) plus all of the RastTestTransform class ( https://github.com/HinTak/Font-Validator/blob/master/OTFontFileVal/RastTest.cs ).

That's 4 for stretchX, stretchY, skew, rotation, a 3x3 matrix for transform, 3 toggles for BW/Gray/ClearType, the 4 ClearType toggles, x/Res, and a list of point sizes.

HinTak · 2016-05-06T10:12:02Z

There does not need to be any explicit conversion - one can construct the two parameter classes directly on the python side, in a script.

schriftgestalt · 2016-05-06T10:16:57Z

I’m not sure if the raster test needs to expose all that settings.

And do you really need a python environment to parse one settings file?

HinTak · 2016-05-06T10:56:07Z

I’m not sure if the raster test needs to expose all that settings.

@schriftgestalt : At least during the testing phrase, it would be nice to be able to set all the options individually - of course most people don't need to do that, and just not load a script. The script is not compulsory. In that case, it would just be using the internal default settings, just like most people don't change the GUI options for those.

And do you really need a python environment to parse one settings file?

There are already 3 other python scripts to extend FontVal's (not counting the first one; the ttc splitter/mergers are quite useful). Might as well include a script-parsing functionality. It is only 4 MB extra for the binary, and very little extra new code - a lot less than it would be for all the individual options, parsing and converting from strings to numbers.

I don't need a "python" environment - but a way of running simple script (for whatever language variant) is nice. C# is actually also script'able, and the additional code to load a C# script vs a python script is about the same.

HinTak · 2016-05-06T14:30:17Z

@schriftgestalt @aaronbell : Here is the already tested python script to set any of the rasterization test parameters:

https://gist.github.com/HinTak/cf7528519552f0eded236fa8e9f41247

when the rasterization test works (and the embedded python interpreter hooked up), you do "FontValidator -rp-parms editedversion.py ..." in an expert-usage mode - not using "-rp-parms" just let the default does its job.

It could do with a lot more comments and annotations, of course.

In https://github.com/HinTak/Font-Validator/blob/master/FontValidator/CmdLineInterface.cs , line 203, it already does "ValidatorParameters vp = new ValidatorParameters();", only a rather small number of new lines a bit below to over-write vp, with something like this:

if (-rp-parms_is_on_the_command_line) {
// over-write vp
vp = GetItFromIronPython("script_after_-rp-parms");
}

Much cleaner than adding piles of new code just to parse and convert a dozen strings to numbers or booleans.

HinTak · 2016-05-07T01:30:02Z

I have bundled 4MB of IronPython's dll with FontVal. So it just works - you can do 'FontValidator.exe somescript.py ...' now, where you might be doing 'ipy.exe somescript.py ...' earlier.

Announcement on
http://typedrawers.com/discussion/1558/microsoft-font-validator-gains-python-scripting-ability-and-ttc-splitter-merger-scripts

binary:
[1] https://sourceforge.net/projects/hp-pxl-jetready/files/Microsoft Font Validator/FontVal-empy-bin-2016_05_07.zip/download

code:
[2] https://github.com/HinTak/Font-Validator/tree/embedding-ironpython

It addresses #17 in a way too.

twardoch changed the title ~~[semi-off-topic] Porting FontValidator from C# to Python?~~ [semi-off-topic] SFNTQA / Porting FontValidator from C# to Python? Dec 1, 2015

twardoch mentioned this issue Sep 12, 2017

SFNTQA proposal from Dec 2015 unicode-org/text-rendering-tests#40

Closed

behdad closed this as completed Jan 23, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[semi-off-topic] SFNTQA / Porting FontValidator from C# to Python? #416

[semi-off-topic] SFNTQA / Porting FontValidator from C# to Python? #416

twardoch commented Dec 1, 2015

twardoch commented Dec 1, 2015

twardoch commented Dec 1, 2015

davelab6 commented Dec 1, 2015

behdad commented Dec 9, 2015

HinTak commented Apr 1, 2016

HinTak commented Apr 1, 2016

HinTak commented Apr 1, 2016

behdad commented Apr 13, 2016

HinTak commented Apr 14, 2016

HinTak commented Apr 14, 2016

anthrotype commented Apr 14, 2016

HinTak commented May 2, 2016 •

edited

HinTak commented May 2, 2016

HinTak commented May 3, 2016

behdad commented May 3, 2016

HinTak commented May 3, 2016

behdad commented May 3, 2016

HinTak commented May 3, 2016

khaledhosny commented May 3, 2016

HinTak commented May 4, 2016

HinTak commented May 6, 2016

HinTak commented May 6, 2016

schriftgestalt commented May 6, 2016

HinTak commented May 6, 2016

HinTak commented May 6, 2016

HinTak commented May 7, 2016

[semi-off-topic] SFNTQA / Porting FontValidator from C# to Python? #416

[semi-off-topic] SFNTQA / Porting FontValidator from C# to Python? #416

Comments

twardoch commented Dec 1, 2015

twardoch commented Dec 1, 2015

What We Had in the Past

[MSFontVal] Microsoft Font Validator

What We Have Now: Python

[FontQA] FontQA

[AFDKO] Adobe FDK for OpenType

What We Have Now: Commandline / Lib

[TTX] fontTools/TTX

[WOFFT] woffTools

[OTS] OT Sanitiser

[FTX] OSX Font Tools Release 4, Beta 1

[FT2] FreeType 2 Demos

[FF] FontForge

[HB] HarfBuzz

[sfntly] Google sfntly

General Scope Of Validation

Mandatory features

Optional features

General Process Of Validation

Input

Processing

Output

Proposal For The SFNTQA Package

Python

Input

Architecture

Organization of validation

Interoperability / external tools

Documentation

Technical requirements

Examples

Character set

Linespacing

SFNT tests

Conclusions

twardoch commented Dec 1, 2015

davelab6 commented Dec 1, 2015

behdad commented Dec 9, 2015

HinTak commented Apr 1, 2016

HinTak commented Apr 1, 2016

HinTak commented Apr 1, 2016

behdad commented Apr 13, 2016

HinTak commented Apr 14, 2016

HinTak commented Apr 14, 2016

anthrotype commented Apr 14, 2016

HinTak commented May 2, 2016 • edited

HinTak commented May 2, 2016

HinTak commented May 3, 2016

behdad commented May 3, 2016

HinTak commented May 3, 2016

behdad commented May 3, 2016

HinTak commented May 3, 2016

khaledhosny commented May 3, 2016

HinTak commented May 4, 2016

HinTak commented May 6, 2016

HinTak commented May 6, 2016

schriftgestalt commented May 6, 2016

HinTak commented May 6, 2016

HinTak commented May 6, 2016

HinTak commented May 7, 2016

HinTak commented May 2, 2016 •

edited