Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Noto Sans Limbu shows dotted circles with U+193A LIMBU SIGN KEMPHRENG #3

Closed
devosb opened this issue Dec 4, 2019 · 8 comments
Closed

Comments

@devosb
Copy link

devosb commented Dec 4, 2019

Font

NotoSansLimbu-Regular.ttf

Where the font came from, and when

Site: https://github.com/googlefonts/noto-fonts/blob/master/phaseIII_only/unhinted/ttf/NotoSansLimbu/NotoSansLimbu-Regular.ttf
Date: 2019-12-02

Font Version

2.000

OS name and version

Ubuntu Bionic amd64
Windows 10 Pro 1909

Application name and version

On Ubuntu - Libre Office Writer 6.0.7
On Windows - Notepad

Issue

Using U+193A LIMBU SIGN KEMPHRENG as a length mark after a vowel sign causes dotted circles with some vowel signs.

  1. Steps to reproduce
    Using the file auto-length.txt format with font in the above applications.
    auto-length.txt
    The file has space separated clusters of (U+190B|U+190F)(VOWEL SIGN x)(U+193A). I suspect the issue would happen with any consonant.
  2. Observed results
    File harfbuzz-noto.png from LibreOffice Writer on Ubuntu
    harfbuzz-noto
    File directwrite-noto.png from Notepad on Windows. Note that DirectWrite produces more dotted circles than HarfBuzz.
    directwrite-noto
  3. Expected results
    File harfbuzz-namdhinggo.png
    harfbuzz-namdhinggo
    This last example was created with the Namdhinggo font with limb script lookups removed, leaving only latn script lookups. I suspect the results would be the same if the latn lookups were replaced with DFLT script. This way, the USE does not get invoked on Ubuntu LibreOffice Writer. On Widows, the USE would have been called, and dotted circles produced.

If I use hb-view, I can have lookups for both limb and latn script, and by passing --script to hb-view I can reproduce the dotted circles with limb and the desired result (no dotted circles) with latn.

  1. Additional information
    The character U+193A LIMBU SIGN KEMPHRENG has Unicode Indic Syllabic Category (UISC) of Vowel_Dependent and a Unicode Indic Positional Category (UIPC) of Top. This causes the character to have a Sigla of VAbv. The USE allows a sequence of multiple characters with VAbv. However, the cluster model (the Standard cluster is what is used for the text that is having rendering issues) says that all VAbv must come before all VBlw and all VPst. As you can see in the table below, HarfBuzz (HB) and DirectWrite (DW) fails if the vowel sign is VBlw or VPst, since the following Kemphreng is VAbv.

If U+193A is classified with UISC = Tone_Mark, this would cause the Sigla to become VMAbv (VOWEL_MOD_ABOVE) and since all vowel modifiers come after vowels, then the cluster validation will pass. Changing the HB source code (locally) to use VMAbv instead of VAbv fixes the issue.

So, can the Noto font be modified to have a lookup that removes the dotted circle that USE inserts? Other options are:

  1. Change the Unicode properties
  2. Add overrides to the USE implementations in HarfBuzz and DirectWrite
  3. Change the encoding order of the charcters

Personally, I suspect option 1 or 2 would be best, but that is just my opinion.

Character data

Attached above.

Screenshot

Attached above.

@devosb
Copy link
Author

devosb commented Feb 7, 2020

If I swap the order of the vowel sign and KEMPHRENG (option 3 in the comment above) the rendering does not show any dotted circles with the same environments as above.

File harfbuzz-noto.png:
harfbuzz-noto

File directwrite-noto.png:
directwrite-noto

@devosb
Copy link
Author

devosb commented Mar 3, 2020

I forgot to add the table referenced in the original post

USV Name (LIMBU …) UGC UISC UIPC Sigla Problem
1920 VOWEL SIGN A Mn Vowel_Dependent Top VAbv  
1921 VOWEL SIGN I Mn Vowel_Dependent Top VAbv  
1922 VOWEL SIGN U Mn Vowel_Dependent Bottom VBlw HB DW
1923 VOWEL SIGN EE Mc Vowel_Dependent Right VPst HB DW
1924 VOWEL SIGN AI Mc Vowel_Dependent Right VPst HB DW
1925 VOWEL SIGN OO Mc Vowel_Dependent Top_And_Right VAbv DW
1926 VOWEL SIGN AU Mc Vowel_Dependent Top_And_Right VAbv DW
1927 VOWEL SIGN E Mn Vowel_Dependent Top VAbv  
1928 VOWEL SIGN O Mn Vowel_Dependent Top VAbv  
1939 SIGN MUKPHRENG Mn Consonant_Final Bottom FBlw  
193A SIGN KEMPHRENG Mn Vowel_Dependent[Tone_Mark] Top VAbv[VMAbv]  
193B SIGN SA-I Mn Syllable_Modifier Bottom FM (DW)FMBlw (HB)  

@NorbertLindenberg
Copy link

Re option 3: This option could certainly be implemented in a smart keyboard. Do you see a major problem with it? Is there a lot of digital text already that has kemphreng after VBlw or VPst? Or a standard that requires that order?

CC @xadxura @behdad

@devosb
Copy link
Author

devosb commented Mar 4, 2020

I don't see a major problem with using a smart keyboard. A minor one though. It makes the keyboard more complicated, especially if a user is editing text with two marks (vowel and kemphreng) and wishes to delete one mark, which mark gets deleted?

There is a lot of digital text with the kemphreng after VBlw or VPst, but most (if not all) of that text was derived from Devanagari using a conversion table.

I don't know of a standard that requires that order. Linguistically it is the most straightforward, and therefore I would guess easier for any user to understand how to type (with a non-smart keyboard).

@mhosken
Copy link

mhosken commented Mar 5, 2020

The choice here is whether we recategorise U+193A from VAbv to VMAbv. Linguistically, functionally, typing orderly, it is a VMAbv. That it is a VAbv is really a bug. The question is whether changing it is too costly and whether there is a willingness to change.

We know of no other Unicode data in Limbu beyond the text we have auto converted from Devanagari script. So redoing that to reorder is not a problem. The technical change within the USE is trivial and alleviates the need for smarts in the keyboard. But the keyboard smarts aren't huge either if your keyboard has the general capability.

In terms of UX with U+193A being left as VAbv; for the most part a smart keyboard can fake the reordering (the user always types it after a vowel, as they are used to). But there are some rough edges to that. For example, dropping a cursor and hitting backspace will almost certainly delete the VPst (or more confusingly the VBlw) before deleting the U+193A when the user would expect the opposite experience.

If there is a willingness in the community to make the change, then we could write a Unicode proposal to change the category. The question is how long it would take to roll out the change, if it is agreed.

@devosb
Copy link
Author

devosb commented Apr 9, 2020

In my original post my description of auto-length.txt as (U+190B|U+190F)(VOWEL SIGN x)(U+193A) is a bit incorrect. There are 9 vowels, but ten clusters in each line. The extra cluster (at the beginning of the line) does not have a vowel sign, just a consonant and the U+193A.

In a later post I swapped the order of the vowel sign and the U+193A. Since there was no vowel sign in the first cluster in the original post to swap, the example would have been the same, so I omitted the first cluster in the second test.

@behdad
Copy link

behdad commented Apr 9, 2020

Sounds obvious candidate for change to me. We can get this in HarfBuzz very quickly. Definitely right after UTC approving and possibly earlier. If there's no huge Android userbase for it I don't see a problem since chrome and Firefox would update rather quickly.

@devosb
Copy link
Author

devosb commented Mar 17, 2021

Fixed in both HB (tested with 2.8.0) and DW (tested with the following version of Windows)

Edition	Windows 10 Pro Insider Preview
Version	Dev
Installed on	‎2021-‎03-‎15
OS build	21332.1010
Experience	Windows Feature Experience Pack 120.2212.551.0

I think DW made the fix first in microsoft/font-tools@e8e20f5 and HB imported this fix for itself in harfbuzz/harfbuzz@06f49fc

Thanks to @xadxura and @dscorbett for doing this.

@devosb devosb closed this as completed Mar 17, 2021
devosb referenced this issue in silnrsi/font-namdhinggo Mar 19, 2021
- move data from most text files to a ftml file
- remove trial encoding of consonant + kemphreng + vowel

The second issue is explained in a bug report
https://github.com/googlefonts/noto-fonts/issues/1623
@simoncozens simoncozens transferred this issue from notofonts/noto-fonts Jul 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants