Precomposed glyphs not defined by Unicode in Illustrator and Photoshop (CC 2020)

Michael Rafailyk

Hello!

Shortly: In Photoshop and Illustrator CC 2020, I cannot paste combined characters (or insert precomposed from Glyphs panel) that not defined by Unicode, such as Jacute, N_dieresiscomb, N_macroncomb, etc.

What I expect to see:
a) precomposed character if it exist in the font file and defined by liga or ccmp;
b) combined letter with accent after that, aligned or not.
What I see: flat letter without diacritical mark, substituted by application (even if the character is precomposed and defined by ccmp).

I check it on my test font file and on other system fonts. The same time everything works correct in TextEdit, InDesign, and web browsers (in a contenteditable div). The problem is only in Illustrator and Photoshop. First what i think is the ccmp feature can be the problem, so I check the test font file with and without ccmp (many to one substitution that looks like sub N dieresiscomb by N_dieresiscomb;).

First test with ccmp:

1. Paste the N̈ (combined N + dieresiscomb from https://en.wikipedia.org/wiki/N-diaeresis) into the text, Illustrator and Photoshop substitute it to just N letter instead of precomposed N_dieresiscomb glyph (that the tested font is contain).

2. Insert dieresiscomb glyph from the Glyphs panel after the N character one time (double click) – nothing happens, but if I insert it one more time – I finally got the N_dieresiscomb (substituted from ccmp).
3. Insert the N_dieresiscomb glyph into the text directly from the Glyphs panel – it substituted with just N (the same result as point 1.).

Second test without the ccmp feature:

1. Nothing changed, I still cannot paste combined N̈.
2. Again, dieresis appears only after second insertion after N, but now it's not precomposed and appears in wrong position (as expected without ccmp).
3. I can finally insert N_dieresiscomb glyph directly from the Glyphs panel.

I also tried move the rules to the liga, but nothing changed and that works exactly the same as ccmp. The same time I can insert any ligatures from Glyphs panel without any problems, and this is strange because substitution rules are the same – many to one. What the difference for the application between N_dieresiscomb and s_t, for example? The both looks and encoded as ligatures, but works not the same way.

I didn't find any problems with combined characters that have unicode value. So, looks like the problem only with some combined characters that not defined by Unicode and how Photoshop and Illustrator implement them. I also check other system fonts and all the fonts on preview show the letter followed by combined mark, but text line show the letter only (example with the Times):



Is it a known issue? And is it related to specific AI and PS version?

Helmut Wollmersdorfer

In my experience with combining characters, you must be sure to catch the combining character by mark & copy. One secure method is to have only codepoint combination in an editor and mark all (<command>+A on Mac). The other is to have an additional character before and after the combination like e.g. _a`_  for à.

If you are sure that you paste the correct sequence of codepoints, then you can say if the receiving program does something wrong. Some programs manipulate the displayed character, e.g. BBedit shows '¶'  U+00B6 PILCROW SIGN for the control character Feed Forward, TextEdit does not.

Next the rendering in programs can be different. The combination ä + combing_e renders in TextEdit correctly (e above ä), in BBedit the e overlaps the diaresis, i.e. is just positioned above the a and not with a vertical distance to the upper edge of the diaresis.

Thomas Phinney

The fact that it works in InDesign but not Illustrator and Photoshop is an interesting clue.

All three apps use the same underlying font engine (CoolType), but Illustrator and Photoshop use a different text layout engine than InDesign.

App version could matter if something has changed in that text engine. (I think it is called something like “Adobe Text Engine” but my memory is fuzzy on this.) But the fact of the text engine difference has been true for well over a decade now.

Michael Rafailyk

The pasted codepoints are: N + combined diaeresis (uni004E + uni0308) and others like a J + combined acute (uni004A + uni0301). In the test font file a both characters are precomposed as N_dieresiscomb and Jacute and defined in ccmp. In the Times, Hoefler Text, and other fonts – are not precomposed. All the fonts show the same result.

Here the comparison between TextEdit, Safari, InDesign, Illustrator, Photoshop, with using the Hoefler Text (without OpenType features) and pasting the N̈ combination.

TextEdit 1.13



Safari 13.1.2



InDesign 2020




Illustrator 2020



Photoshop 2020



Also interesting that TextEdit and Safari does a smart job and set the accent in correct position (perhaps not optical, but mathematical), even without ccmp and/or mark features.

André G. Isaak

If you're using the version of Hoefler Text included with macOS, bear in mind that that contains AAT tables (i.e. QuickDraw GX tables) rather than OpenType tables. AAT isn't widely supported since it's Apple-only.

Alanna Munro

Thanks for experimenting with the ccmp feature.

I didn't find any problems with combined characters that have unicode value. So, looks like the problem only with some combined characters that not defined by Unicode and how Photoshop and Illustrator implement them. 

I am having a similar problem with the comma accent above marks for the Nuu-chah-nulth support in Tofino. They won't display in illustrator but work fine in InDesign, Affinity products, MS word, etc.

I have a client that needs to use these characters for signage templates now so I may just have to create precomposed versions to make them accessible from the glyph palette but I'm not sure that's the best long term solution? 

Does anyone around here have a better idea? Or is this something we need to tell Adobe about and wait for them to fix? 

John Hudson

Hi Alanna. I am not able to reproduce this issue in Illustrator 2021 25.4.1: I can create a document and both create and paste in text that includes the U+0313 combining comma above mark, and it correctly renders when using a font that supports this character. In my tests, I confirmed this with fonts that user ccmp GSUB and others that use mark GPOS.

Can you post here the live text that you are testing, and I will check the encoding and do some more experiments to see if I run into any ways to break it?

Alanna Munro

John, I just downloaded Illustrator (version 26.0.1) to try to reproduce this issue for my client. I'll ask what version my client is using. I was using the sample text from omniglot: 

ʔUyaaƛaḥ hawiiʔaƛii maapt̓ał c̓išaaʔatḥ ʔuukʷił yuułuʔiłʔatḥ ʔaḥʔaaʔaƛsi n̓ačuʔałʔaƛsi hiikʷis. Meʔiƛqacʔissi ʔiiqḥii ʔanaḥʔis. C̓uʔičḥ qaʔuła p̓iip̓inw̓ałiiq ʔeʔiiḥiiq c̓išaaʔatḥ qaʔuła. Huʔanakšiʔaƛ nism̓a hiteʔitap̓aƛ ʔukʷił yuułuʔiłʔatḥ maapt̓ał. ʔUunuuʔaƛ ʔaḥʔaa ḥałḥaqułʔaƛ qaʔuła ʔani ƛułukqa ƛ̓uƛ̓im c̓eʔinwa hiłḥʔaƛ ƛ̓asatis sučicaqimł qaʔuła.

Thanks so much for looking into it!

Michael Rafailyk

@Alanna Munro
Confirm that Illustrator 2020 (version 24.0.1) don't show commaaccent above letter c. I used text copied from your comment.

Apple said my Mac is too old for update the OS and Adobe apps, so if your client be able to upgrade the Illustrator, that should fix the problem.

John Hudson

Working fine in 26.0.1 for me:

Alanna Munro

Okay. So my client is on 25.2.3 and not updated to Monterey, not working for her.

I tried on all the versions I have access to (25.4.1, 26.0, 26.0.1) and am updated to Monterey, none of them work for me. 

@John Hudson, interesting that it works for you maybe I can send you a copy of Tofino directly to see if you can reproduce the problem, now I'm wondering if theres a problem in my features or something? 

John Hudson

I’d be happy to take a look at Tofino (email me), but this may also be an Adobe text engine issue, and may be working for me because of my preference settings for complex scripts and the fact that I am using the Middle East version of Illustrator. [Operating system version should not be an issue—I am still on Big Sur—, because Adobe do not use many system resources, preferring—for better or worse—to roll their own.]

I don’t actually use Illustrator, so mostly have it only for testing purposes, and am not very familiar with it, but I may be able to help diagnose the issue. Adobe recently announced that Illustrator and Photoshop will be moving to a new Unified Text Engine, based on the open source Harfbuzz shaping engine, which should resolve these kinds of issues and ensure compatibility with browsers etc.

Alanna Munro

@John Hudson I don't use Illustrator either. I got a 7 day free trial just to test this issue 😅I'll send you an email! 

That's great news about the Unified Text Engine. Sorry, @Michael Rafailyk, I feel like I hijacked your post a bit. I tested the N diaeresis in Illustrator 26.0.1 and I can't get it to work either, so it seems to be the same issue. 

John Hudson

Thanks for sending the Tofino test fonts, Alanna. They seem to work fine when I copy and paste the Nuu-chah-nulth text sample into my copy of Illustrator 26.0.1:

So I am thinking that the issue is almost certainly with a specific Adobe text engine, and I just need to figure out where the problem is. I will trying pinging an Adobe text engineer about this issue.