NeuralType: A Post-OpenType Font Format

Eli Heuer
Eli Heuer Posts: 7
edited July 31 in Font Technology
I'm working on a post-OpenType font format where font files are small neural networks, not confined to the little boxes inherited from metal typesetting, and I already have a working demo!

You can copy, paste, delete, type, move the cursor around, and drag a kashida axis with a slider. It's all open source and built with the simplest possible neural network.

Play with the demo and find the source code on my blog: https://elih.net/blog/neuraltype/

Comments

  • Eli Heuer
    Eli Heuer Posts: 7
    I know this might look like just a gimmick when only using the square kufic style, but I really do see this as an OpenType replacement. Here is a demo showing a Nastaliq font: https://elih.net/blog/nastaliq-distilled/

    https://youtu.be/fUy_h-e9byI


  • John Hudson
    John Hudson Posts: 3,753
    edited July 31
    We certainly need a technology that a) can work as widely as OpenType and b) is capable of handling all the dynamic complexity of nastaliq and other cascading Arabic script styles. In recent years, brute force GPOS contextual lookups have gotten us further than before, thanks to Simon Cozen’s upstream tooling, but it still isn’t as flexible as the script grammar employed by a trained scribe.
    A dozen or so years ago, I was able to sit down with Eliyezer Kohen, one of the original inventors of OpenType Layout, and talk through the difficulties of cascading Arabic. He told me something that neither I nor any of my colleagues working with Arabic to that date had been aware of: he never expected OTL to be able to handle cascading Arabic, which he considered to be outside the scope of the technology. Unfortunately, this information was lost because Eliyezer had left Microsoft before OpenType became widely adopted. As Eliyezer put it to me at the end of our conversation, he had been happy in the mid-1990s to target 98% of the world with OTL and for Tom Milo to have the rest.
    A lot of time and effort has been wasted over the years trying to force cascading Arabic into OTL.
  • This is ingenious and impressive as a demo. 

    Reading the blog post, to someone with little machine learning understanding, this still sounds pretty close to magic.

    Can you explain this bit more?

    The model learns its letterforms from a teacher program: hand-written code that draws each form as an ASCII-art grid in the Rust source. The network is trained until it reproduces the teacher’s output exactly.

  • Simon Cozens
    Simon Cozens Posts: 865
    The inflexibility of OpenType shaping and the existence of a fixed set of alternate glyphs is certainly a limitation which makes certain Arabic styles very difficult. But I'm not terribly convinced that this justifies something like a ML model for every glyph; the principles of efficiency and simplicity suggest we should at least attempt to find the smallest set of primitives which could bridge the gap. An obvious extension of the OpenType model which would do this would be to allow multiple axis positions within a single glyph run, and to allow contextual shaping rules to affect or set per-glyph variable glyph axis positions. This would allow glyphs within a parametrically designed font to stretch, lengthen, twist and turn to match the surrounding glyphs in their contexts.
  • ... An obvious extension of the OpenType model which would do this would be to allow multiple axis positions within a single glyph run, and to allow contextual shaping rules to affect or set per-glyph variable glyph axis positions. This would allow glyphs within a parametrically designed font to stretch, lengthen, twist and turn to match the surrounding glyphs in their contexts.
    This!
  • John Hudson
    John Hudson Posts: 3,753
    And then the mark positioning.

    But think about all the things a scribe is aware of. Trying to manage text at the level of surrounding glyph context while remaining ignorant of line length, page dimensions, and all the other information that the scribe is taking into account is only ever going to get us so far. The little boxes might be able to stretch, lengthen, and twist, but they’re still little boxes in a line, without any input from adjacent lines, let alone from the page as a whole. Of interest to me is less the specifics of a new or extended font format, but how we build a holistic text layout model in which all the parts are aware of each other and able to adapt to conditions elsewhere in the ‘stack’.
  • Simon Cozens
    Simon Cozens Posts: 865
    I implemented this for Harfbuzz last night. It was 350 lines of code - 200 for the changes to the shaping engine and the rest for the changes to the rasteriser and the utilities. It took me three hours. (I mention this because every time someone proposes a change to the shaping model, people will grumble about how much work it's going to be and how all the shaping engine teams are underresourced, and nothing happens. But some stuff is just easy.)

    Here's a font which only has one set of numeral glyphs, but which uses parametric axes to support tabular / lining / old-style / lining old-style OT features. You can see in the hb-shape output at the bottom that it's the same "one"/"two"/"three" glyphs but they are being rendered at different axis positions.


  • Dave Crossland
    Dave Crossland Posts: 1,564
    @Simon Cozens excellent work :) Where is the code?
  • What could make this complicated is that layout (i.e., determining final glyphs and glyph positions) could be implemented without any rasterizer interaction, even that APIs can be designed with layout completely independent of rasterization (rasterization requires a drawing surface/texture, but layout does not). Yet layout depends on glyph metrics. Hence either there needs to be a way to obtain glyph metric deltas independent of a rasterizer, or else APIs might need significant redesign.

    I've long wanted to pursue this, but for DirectWrite at least I anticipate it would require some significant API changes, both to obtain glyph metric deltas during layout and to convey the layout-determined glyph-specific variation parameters into the rasterization operation.
  • Simon Cozens
    Simon Cozens Posts: 865
    I don't understand this. In Harfbuzz layout is indeed implemented completely independently of rasterization. (hb-raster was added very very recently and is a standalone module.) And as a shaper, it needs to (and does) return glyph metrics, including deltas, without rasterizing. How could it not? The shaper tells the rasterizer where to put things, and to do so it needs to know how wide things are. I'm confused by the idea that DirectWrite doesn't already read glyph metrics deltas during layout. Surely it does, right?
  • kunstx
    kunstx Posts: 27
    We certainly need a technology that a) can work as widely as OpenType and b) is capable of handling all the dynamic complexity of nastaliq and other cascading Arabic script styles. In recent years, brute force GPOS contextual lookups have gotten us further than before, thanks to Simon Cozen’s upstream tooling, but it still isn’t as flexible as the script grammar employed by a trained scribe.
    A dozen or so years ago, I was able to sit down with Eliyezer Kohen, one of the original inventors of OpenType Layout, and talk through the difficulties of cascading Arabic. He told me something that neither I nor any of my colleagues working with Arabic to that date had been aware of: he never expected OTL to be able to handle cascading Arabic, which he considered to be outside the scope of the technology. Unfortunately, this information was lost because Eliyezer had left Microsoft before OpenType became widely adopted. As Eliyezer put it to me at the end of our conversation, he had been happy in the mid-1990s to target 98% of the world with OTL and for Tom Milo to have the rest.
    A lot of time and effort has been wasted over the years trying to force cascading Arabic into OTL.
    The technology exists for decades made by Decotype, as well since this year with out solution as well. The only problem is the lack of interest and knowledge. Steve Jobs wanted to implement Decotype's solution, but he died soon after that, and the management later didn't care about the idea, since the "users didn't complain". The same users who until today don't know the difference between quasi-Arabic and real Arabic script.

    As well Decotype's fault is for not keeping the products transparent and publishing more on it, so there is always two sides of the story.

    We in Kunst X developed the multilingual workflow in our Pub Suite beta branch with custom engine. So far it has only lame bitfont example, but the architecture is there ready to implement any world script with all advanced features.

    Eli Hauer is going in the interesting direction. I adviced him. If he listens and learn certain script grammar foundations and implements that, he will produce a good out come as well. If he stays stubborn, he will just prolong the slop of the past in little more advanced shell.

    So far the tech side is covered, only interested and aware group of typeface designers and developers are needed to implement the tech. Producers of engines on which OTF runs can't produce better even when they would want, since they first need to hire linguistics and typography experts to advice them.

    All or most of the current Arabic typeface designers are people who follow ways of amateur orientalists who thought they know about Arabic script, and created this awful structure of Venezian-Arabic/Euro-Arabic, quasi-Arabic script, which is an approximation of Arabic, mutilated and forced into the "western" typesetting enviroment.

    We could pretend that these problems are not influenced by world politics, colonial narratives, Stockholm syndrome of countries using non-Latin scripts, and "western" industry domination, but let's not pretend.

    Changes need time, but not so much time as the overall interest.
  • I'm confused by the idea that DirectWrite doesn't already read glyph metrics deltas during layout. Surely it does, right?
    I haven't thought about this in any detail for a few years...

    Yes DW shaping reads glyph metrics, but set on a font-wide basis using HVAR (or VVAR). As long as the shaper has access to glyph metric data. 

    But then the glyph run structure produced by shaping needs to be updated with per-glyph variation parameters to pass to the rasterizer.