Work in progress.
The Unicode Indic encoding models, being highly contextual, present challenges for text rendering. Various complex shaping expectations of Unicode-encoded Indic texts are met by the cooperation between fonts and text shaping engines according to text shaping models.
Part of the OpenType specification, OpenType Layout (OTL, mainly maintained by Microsoft) is the predominant text shaping model today for supporting Unicode-encoded complex scripts. Alternative shaping models such as Apple’s AAT (Apple Advanced Typography, available in Apple products) and SIL’s Graphite (available in LibreOffice, XeTeX, Firefox, etc.) also have their different advantages. Each model requires a different set of shaping rules to be supplied by a font, in addition to the font’s glyphs and basic character-to-glyph mapping (cmap). A text shaping engine then shapes Unicode character sequences into final glyph sequences, base on the font’s shaping rules.
In terms of the OTL model, there are four groups of shaping engines that matter the most for digital typography developers to understand their targeted environments and users to understand their editing environments. Note that text shaping engines are often built as part of text layout engines, and thus are conventionally referred to with the latter’s names.
The de facto standard cross-platform solution is an open source project:
- HarfBuzz, which also supports the AAT and Graphite models, however not all softwares are built with the AAT and Graphite shapers turned on.
While some platform vendors have their own platform-specific and proprietary shaping engines available in their products:
- Microsoft: where the shaping engine is now part of the DirectWrite layout engine, but is often referred to with its historical name, “Uniscribe”.
- Apple: part of the Core Text layout engine; also supports the AAT model.
- Adobe: fractured and underdocumented, without a well-known name (commonly just referred to as “Adobe shaping engines”); accessible by various paragraph-level “composer/layout” settings in products such as Photoshop, Illustrator, and InDesign.
Table 2-X. Available shaping models in common software products
| product | shaping engine and supported models |
|---|---|
| Microsoft products (Windows, Internet Explorer, Edge, Office…) | DirectWrite (OTL) |
| Apple products (macOS, iOS, Safari, Pages…) | Core Text (OTL, AAT) |
| Android | HarfBuzz (OTL) |
| Chrome | HarfBuzz (OTL, AAT) |
| Adobe products (Photoshop, Illustrator, InDesign…) | Adobe shaping engines (OTL) |
| Firefox | HarfBuzz (OTL), Graphite engine (Graphite) |
| LibreOffice | HarfBuzz (OTL), Graphite engine (Graphite) |
| XeTeX | HarfBuzz (OTL), Core Text (AAT), Graphite engine (Graphite) |
In the OTL model, the exact shaping operations executed by a shaping engine for a script’s text are based on the engine’s knowledge of the script and supplemented by the font’s rules. The engine’s script-specific knowledge is meant to be prescribed in OpenType’s script development specifications on an opt-in basis, such as Developing OpenType Fonts for Devanagari Script. Each specification is implemented as a script-specific shaper in an OTL shaping engine. A general script or a script unknown to a shaping engine is limited to few possible operations supported by the default script shaper.
It is a long-standing problem that the official OTL script development specifications do not accurately reflect what shapers actually do, due to being generally incomplete and outdated. The work-in-progress project n8willis/opentype-shaping-documents is a major community effort to address the need of a complete, developer-oriented documentation for OTL shapers.
On the contrary, an AAT or Graphite engine is “script-agnostic, and places the responsibility and authority for shaping completely in the hands of the font developer” (see Graphite and OpenType).
In order to manipulate glyphs from cmap for complex text shaping, OTL defines two major mechanisms, substitution and positioning, which are leveraged by fonts with shaping rules that are recorded in the GSUB (Glyph Substitution) and GPOS (Glyph Positioning) tables. Supportive properties of glyphs are recorded in the GDEF (Glyph Definition) table.
When a text is being shaped, before an OTL text shaping engine starts to work on it, generally a text layout engine first segments (or itemize) the text into a series of runs according to the text’s content and formats, so that each text run is internally consistent in terms of writing direction, script, language, and font. Then, the shaping engine processes a run according to its script and language value.
A problematic consequence of this architeture is that OTL does not allow any interaction accross a text run boundary, such as between a regular style font and bold (change of font), or between Latin and Devanagari characters (change of script). Even worse, the algorithm for text run segmenation is not standardized, therefore it is difficult to predict what script run a character with a weak Script property value (Common or Inherited) will get segmented into. Also, a text layout engine’s ability to recognize a script is limited by its built-in UCD (the Unicode Character Database), which can often be outdated.
AAT and Graphite are not affected by the script run issue (although a text layout engine still needs to decide font run boundaries), as they are script-agnostic shaping models.
In OTL tables like GSUB and GPOS, a font’s supported scripts are defined the top level with registered script tags to suggest what script-specific shaper can be used. A default script with the tag DFLT is recommended to always be available as a fallback. A script may have more than one tags registered, each corresponding to one of the the script’s multiple versions of script development specifications and thus multiple versions of shapers.
Within each script, explicit language systems can be defined with registerd language system tags in addition to the implied default one, to allow language-specific shaping rules.
Within each language system, features are then defined. Features provide a mechanism for fonts to declare the purpose of a set of rules to shaping engines with predefined semantics, allowing the latter to execute specific sets of rules based on a users’ implied or explicit request. For example, rules organized under the feature liga are understood as rules for typographically optional ligatures, and can be enabled per users’ discretion; while rules under init are understood as required shaping rules for Arabic letters to correctly connect to each other, and are thus always executed by the Arabic shaper.
Actual shaping rules are defined in the form of an ordered list of lookups that are referred to by features. Note that certain features, especially the ones that are critical to complex script shaping, are executed early in a predefined order, and thus limit the actually executed order of lookups.
In particular, OTL shaping engines try to match a text against any of the predefined, restrictive character cluster patterns, then insert dotted circles as placeholder base glyphs wherever a failure of the match leads to stray marks.
FIGURE: example of an OTL Indic cluster pattern and a match failure.
For Indic scripts, in particular, a large set of mandatory features are always applied by shapers, with a largely restricted order that is structured by a couple of shaping stages. This is to ensure Indic scripts’ required shaping behavior is always achieved. Certain features are expected to only contain rules that have a particular structure, to allow shapers to extract necessary glyph properties from a font and execute reordering properly.
The OTL shaping behavior for the nine Unicode ISCII scripts have undergone a major change, thus there are the original version (Old Shaping Behavior, sometimes referred to as Indic 1) and the version 2 (New Shaping Behavior, commonly referred to as Indic 2), requested with different tags.
| script | Indic1 | Indic2 |
|---|---|---|
| Bangla | beng | bng2 |
| Devanagari | deva* | dev2 |
| Gujarati | gujr | gjr2 |
| Gurmukhi | guru | gur2 |
| Kannada | knda | knd2 |
| Malayalam | mlym* | mlm2 |
| Odia | orya* | ory2 |
| Tamil | taml | tml2 |
| Telugu | telu | tel2 |
Font developers do not need to provide Indic1 support unless Adobe products, the most lagging-behind major platform, are among the targeted environments. The asterisked Indic1 shapers are obsolete even for Adobe products, as their corresponding Indic2 shapers have been supported no later than Creative Suite 6, which was released in 2012. The Indic1 shapers mlym and orya, in fact, were pretty much abandoned by Microsoft early and gave way to their Indic2 versions before even getting widely implemented.
Unlike their Indic 1 counterparts, Indic 2 shapers no longer statically assume a consonant letter to have a particular type of conjoining form. Instead, such a property is defined dynamically by fonts in certain features (rphf, pref, blwf, half, and pstf) in the basic shaping forms stage and is retrieved by shapers. Consequently, the reordering process in Indic 2 shapers is divided into the initial reordering before any shaping rules kick in, and the final reordering after the basic shaping forms stage.
The nine Unicode ISCII scripts are specified to be subject to the following features and shaping stages:
FIGURE: table of the nine Unicode ISCII scripts’ specified features and stages.
For some of the nine Unicode ISCII scripts, their Indic 2 shapers are well implemented in major platforms, therefore their Indic 1 tags are mostly obsolete and do not need to be supported by fonts. But some still rely on their Indic 1 shapers, thus their fonts are recommended to support both Indic 1 and Indic 2 shapers.
FIGURE: table of products that still do not support the Indic 2 shapers.
Shaping prioritization:
- Normalize Unicode characters to units that matter to shaping
- Decompose split vowel signs
- Initial reordering
- Orthographical operations to be done before reordering
- Structures that have higher priority than productive signs
- Productive signs
- Features that affect reordering
- Features that do not affect reordering
- Structures that have lower priority than productive signs
- Final reordering
- Orthographical operations to be done after reordering
- Typographical
Utilizing the predefined and preordered Indic features:
- basic shaping forms (GSUB)
- feature akhn
- feature rphf (reordered)
- feature pref (reordered)
- feature half
- feature cjct
- mandatory presentation forms (GSUB)
- feature pres
- positioning features (GPOS)
- feature dist
- feature abvm
- feature blwm
One or more signs can be placed on a base, forming a composite akshara. There is usually a preference for where exactly a nonspacing sign should be placed in an akshara. The exact positioning of a nonspacing sign needs to be specified with glyph positioning (GPOS) on an appropriate glyph (not necessary the base).
Multiple nonspacing signs can also be placed at the position. Instead of just stacking, they often interact graphically, and typically require OTL glyph substitution (GSUB) treatments for variation and ligation.
FIGURE: base त → simple akshara त base and sign <त ं> → composite akshara तं
FIGURE: composite akshara <त े {reph} {anusvara}> → signs interacting र्तें
Both a base and a sign may have complex (encoded as a character sequence) instead of atomic (encoded as a single character) encoding, and would require complex shaping internally.
FIGURE: characters <क ् ष> → complex base क्ष characters <र ् त> → complex sign र्त
FIGURE: a two-by-two chart, ক র্কেং and ক্ষ র্ক্ষেং; how র্ক্ষেং is broken down to a complex base ক্ষ (different from an atomic base ক) and several productive signs, and how the whole structure is a composite akshara (different from a simple akshara ক্ষ).
A base is either atomic or complex in terms of encoding. A complex bases is formed with GSUB rules, either before forming complex signs or after, depending on desired shaping priority and whether the complex base is expected to have its own conjoining forms.
Complex signs, the productive signs that are encoded as a sequence, are usually conjoining forms of simple or complex consonantal bases, as vowel signs are generally encoded atomically. Conjoining signs are systematically named according to both of their graphic relation and phonetic relation with the base:
- graphically, conjoined to the base: top (phonetically initial by default), left (leading by default), bottom (trailing by default), or right (trailing by default)
- phonetically, in the consonant cluster with the base: initial (the first leading consonant), leading, or trailing
Because the Unicode ISCII model does not resolve consonant-cluster shaping priorities on the encoding level, complex signs especially need to be formed in a prioritized way. Typically, conjoining forms are formed in OTL GSUB features in the following order:
rphf: reph form (top/right initial)pref: left-side trailing form (left trailing)blwf: bottom-side forms (bottom trailing)half: half forms (left leading)pstf: right-side forms (right trailing)
When the default prioritization does not shape a desired composite akshara, a ZWJ (U+200D ZERO WIDTH JOINER) is used to override which side of a virama should absorb the virama and become a conjoining form.
If a virama is neither used for forming a complex base, nor used by either of its flanking consonants for forming a conjoining form, it does not exhibit the conjoiner behavior and is simply left as a vowel killer on the preceding consonant.
A special conjoining form of the initial ra that is graphically different from other (if any) leading conjoining forms, is conventionally referred to as repha. Repha is formed in the OTL GSUB feature rphf (reph form), and is subject to reordering in the Unicode ISCII model because it is encoded initially but its attested graphical forms (top and right) all expect a position after the base in a glyph sequence.
FIGURE: characters <र ्> त → repha र्त
A left-side trailing form is formed in the OTL GSUB feature pref (pre-base form), and is subject to reordering.
Bottom-side forms, along with right-side forms, are the predominant consonantal dependent forms in Telugu and Kannada. They are formed in the OTL GSUB features blwf (below-base forms) and pstf (post-base forms), respectively, and are subject to vowel sign reordering in Telugu and Kannada.
Left-side leading forms (half form) are formed in the OTL GSUB feature half (half forms). Half forms are the predominant consonantal dependent forms in Devanagari and Gujarati. Both simple and complex bases can have half forms. Mainly applicable to typical bases that have a vertical stem on the right side, and the vertical stem is lost in their half forms.
FIGURE: characters <त ्> क → half form त्क
Note the consonant letter ra has a true half form (which is also known as an eyelash) besides its more commonly used conjoining form for a phonetically leading/initial position, repha.
FIGURE: characters <र ्> य → repha र्य characters <र ् ZWJ> य → र्य
Nukta is not an akshara-level sign. Instead, it is placed directly on the grapheme it modifies (typically consonant letters and their dependent forms; certain innovative orthographies apply nukta to vowel letters and vowel signs also). As it is a low-level modifier and a nukta-ed grapheme is generally expected to behave like an atomic grapheme (also sometimes with behavior different from their nukta-less counterparts), nukta is generally ligated early to the modified grapheme with GSUB operations.
FIGURE: example of creating a nukta form.