How does Unicode work?

student

I am very curious about the way that writing systems get introduced to tech. Beyond the software aspects, I am curious as to whether any type designers working with lesser-utilized writing systems have run into any issues trying to get their characters encoded. What is the process? Is there a representative that people speak to?

Simon Cozens

While anyone can submit a proposal to Unicode for encoding an unencoded script, the Script Encoding Initiative has a lot of expertise in developing proposals; their web site is a good place to start looking.

John Savard

Indeed, there is a process. Unicode is maintained by an organization called the Unicode Consortium. And you can find online proposals submitted to it for the encoding of new scripts that had not yet been included in Unicode.

This

https://www.unicode.org/pending/proposals.html

is the page on the Unicode site that describes the process.

John Hudson

After a proposal is submitted, it first goes to the Script Encoding Working Group, which reviews the proposal. The SEW is made up of Unicode staff and invited experts, and they will often ask questions of the proposers or assign someone to work directly with them to revise the proposal. Sometimes, proposers are invited to meet with the SEW to discuss aspects of the proposal. The goal is to make sure the proposal is ready to be submitted to the Unicode Technical Committee, usually with a recommendation from the SEW. Often there is more than one possible way in which a writing system could be encoded, so the UTC may have questions about why a particular model has been proposed, but the idea is that by the time a proposal gets to the UTC it is very likely to be accepted and added to the pipeline.

Some proposals are for complete scripts, and those often go through multiple iterations. A preliminary script encoding proposal might be submitted that does little more than introduce the writing system, explain how it works, and identify possible questions or issues to be addressed. A preliminary document like this is useful for guaging interest and for finding expert reviewers. It will then be followed up by a more complete and technically detailed proposal. The point here is that an initial submission does not necessarily need to be fully formed and complete.

Some proposals are for individual characters or small sets of characters as additions to scripts already encoded in Unicode. So, for example, Kevin King at Typotheque proposed, and had accepted, a set of additional Canadian Syllabics characters for specific Inuit languages.

It is important to note that the Unicode character set is synchronised with ISO/IEC 10646. This means that UTC decisions are also subject to balloting by national standards bodies that are part of ISO. So although by the time a proposal reaches the UTC it should be ready for acceptance, occasionally there is push-back on a character encoding by a particular national standards body. A persistant problem is that some national bodies identify a script with a particular modern language, and are resistant to adding characters that may have been used within that script to write other languages.

John Savard

John Hudson said:

A persistant problem is that some national bodies identify a script with a particular modern language, and are resistant to adding characters that may have been used within that script to write other languages.

That sounds like discrimination against a minority ethnic group. The Unicode Consortium should, therefore, immediately inform ISO that it has two choices: expel the relevant national standards body, or its relationship with the Unicode Consortium will be ended. Zero tolerance is the only permissible policy towards racism in any form; doesn't everyone know that by now?

John Hudson

I think you are reading a lot into what I wrote, and overlooked the tense: may have been used. Pushback on encoding tends to arise regarding historical characters, more from ignorance than discrimination I would say, except in that way in which the living regularly discriminate against the dead. National standards bodies in many countries have a mandate to protect languages, and tend to be primarily concerned with one or more modern languages. They are sometimes resistant to recognising historical characters as belonging to a script, e.g. nasalisation markers for Sanskrit texts written in South Indian scripts primarily used for Dravidian languagues. Sometimes ‘We don’t use that’ becomes ‘We don’t want that”.

Peter Constable

In 20+ years involved in the ISO side of the process, I can't recall any case I would consider discrimination against an ethnic minority.

There are often real issues of different elements within a language community being in strong disagreement about writing. I don't recall any case in which a national body was taking the side of those opposed to encoding characters, but I do recall cases in which that tension among stakeholders within their country was part of the context they were having to manage. In one case, a national body coordinated with Unicode asking for a pause on some new characters to allow time for other dust to settle; those characters were added in a later version of Unicode without any fuss.

There can be cases in which a national body hasn't participated in the detailed review and discussion that John mentions takes place within SEW, and then they ask to hold off on some characters because they have questions, or they could have stakeholders within their country-internal network that they want to review, or perhaps those experts disagree with some aspect of what was proposed. Up to now, in most cases those eventually got resolved and characters got encoded. At present, there are some cases that are still in progress.