I have been thinking about withdrawing the pull request for a while.

There are many ways Microsoft folks can contribute: of course they can put some dev resources and hours to it, or commission outsiders (not necessarily me, but that would be welcomed) doing some work; two unique ways they can contribute is (1) supplying a less-antique binary than 2003 - HinTak#5 - I know they updated the code as recent as after the 2009 2nd edition opentype spec, (2) help fix visual studio building so that more Window centric devs can contribute -HinTak#8 - . There are about 20 opened issues at the moment, some involves updating the build instruction docs, reading the specs and documenting what needs to be done . Many do not require a lot of special knowledge, just time.There are discussions going on in the Opentype mailing list they can participate, and there are past design decision they can clarify.

And while it isn't useful for practical purposes, providing binaries based on the stock august 11 upload, just to demonstrate that they can cope with using the open mono + GNU make build chain, instead of visual studio which they had been doing for 13 years, would be assuring too.

Hopefully we would see better participation from Microsoft folks.

There exists an implementation of the rasterisation test in my hard disk, and a few other bits on CFF also.

This year's Google Summer of Code is starting to accept student application in a week or two. For those who are unfamiliar with it, GSoC is a scheme where Google sponsors about 1000 students around the world each year - for just over 10 years now - to work on a project of the student's choice, under the guidance of an expert in that area, for about 3 months in the summer. (I think the students get $5000 US, which is not bad for any summer job, especially when it doesn't even require leaving the house!)

The Linux Foundation has been a mentoring organization for almost every year, and is in this year. I mentored a few students under the open-printing initiative for a few years, as well as a couple of students working on various parts of the linux kernel in the past also.

The Microsoft Font Validator can go under Linux Foundation's openprinting. The Mono Project was a mentor organization in the past, so if they are in this year also, student can apply under the mono project too.
It is allowed to apply under two orgs, which can increase the student's chance of being accepted.

The student application windows period is 14 March to 25 March.

So if you work for an educational organization, or know students with suitable skills interested in that, please forward.

Re-reading some of the e-mails, I am reminded by @davelab6 , who is honest but kind in the best possible way, that maybe I have not explained and communicated well to @aaronbell why I really want a more "recent" internal binary, whichever 'more recent' means. If that's the case, apologies.

The issue I am facing from my experience with FreeType vs MS renderer on LTSH/hdmx/VDMX is that there are going to be differences. Some maybe my lack of understanding of freetype, (e.g. I was initially very excited by freetype giving perfect "hdmx" calculations, until I learned that freetype is optimized to just read the table and tell me, unless told not to, so of course the agreement was perfect!), bugs in freetype, bugs in my new code, and bug in the 2003 binary, bug in the specific font, or just "behavior differences". But I need to understand and account for all the differences until I am happy that all the differences are well-understood. If I get a 'close enough' answer, I can move to the next task. If they differ, I 'll dig.

Having a buggy 2003 binary actually means wasting me time debugging the 2003 binary to make sure that the difference is because I am correct with the new freetype code, and the old 2003 binary is wrong. So it takes extra time.

Besides causing me extra time, if microsoft is afraid that I might learn something I shouldn't, denying me a more recent and less buggy binary will result in exactly what they are afraid of: I may have to dig inside the proprietary 2003 binary to understand that it is buggy, and how buggy in exactly what way, after exhausting the other possibilities like modifying my own new code etc. So putting me in a scenario where I need to dig inside the proprietary 2003 binary to understand why and in what way it is buggy, may actually lead me to learn something I should not. If the new implementation agrees with the old, and the old is correct , I have no reason to dig. The worst is a 3-way disagreement: freetype said one thing, the old binary says another, and the built-in LTSH/hdmx/VDMX tables in the font says a 3rd answer (which should be 'correct' according to the last font editing apps that wrote them). There are indeed 3 way disagreements, even now, of those tables, on the common platform fonts. I keep notes about which of the three is correct, and try to tick the disagreements off one by one.

And I do not look forward to a 3-way disagreement in rasterisation, which is far more complex than calculating the tables.

So, I hope this is a better explanation why I really like a less antique binary for private reference.

dig inside the proprietary 2003 binary

What does the binary EULA say about that?

presumably "reverse engineering is illegal", which is likely technically overruled by EU law that protects reverse engineering for public research purposes, but could equally well still lead to lawsuits.

could equally well still lead to lawsuits

Indeed. Even if not, that kind of thing certainly makes rightsholders unhappy. Even just publicly discussing that kind of thing as a possibility is hostile.

Since Hin-Tak's goal is to be given something by a rightsholder, then doing things that make them less than joyful undermines the possibility the goal can be achieved at all.

I hope this is a better explanation why I really like a less antique binary for private reference

It does explain why your life would be improved by MS sharing the binary, but it isn't clear how MS benefits. I can't see anything in your comment about the benefits to MS of releasing the binary. Maybe this is obvious to you and implicit, but it would help me to understand by stating it explicitly.

There is also a wider context to your request: Since you have - so far - not co-operated with MS, by not signing their CLA, then I am not sure anyone at MS is going to think too hard on any suggestion you make.

Eventually if you annoy anyone too much then they will just write off everything about you and not even reply to your messages any more; and if you keep pushing on them, then they might retaliate.

Please be respectful, offer value, and explain why what you are doing/offering is valuable to other people; and please do not threaten, and be willing to accept 'no' as an answer :)

Hmm, okay, I thought the message was meant to make the situation better, not worse. Anyway, my goal is to make the new code covers no less than all the grounds that the old covers; at least judging from some of the reactions of the users over the last few months, 'why isn't X there?' , where X had been the XML report viewers a few times, and the rasterization test at least once as I can recall. (despite both being mentioned as 'not included' both initially and repeated a few times there after).

Perhaps the new code is too similar to old that users' expectation seems a bit high :-(.

I would also like to say it again that, reading lengthy legal document is hard work; reading a lengthy legal document with sufficient attention to details to decide whether to agree with it or not, is even harder work. So being handed a piece of legal document to read, feels a lot like piling on more home work, and unnecessary one too, as a punishment for being hard-working.

And I hope nobody is suggesting one should sign any legal documents without reading every word carefully, and thinking about it.

@davelab6 , @Pomax : ask a Mozilla developer who has ever tried to figure out why the Adobe flash plug-in crashes firefox, for example. He or she might know a thing or two that Adobe does not want people know. The process is just called 'debugging proprietary plug-ins without the source'. It is not a pleasant process, and few people are motivated or have the skills to do it. If the proprietary plug-in is not buggy, there is no need to spend time on it. And as long as there are users who want to watch flash movies, and as long as the Adobe flash plug-in is the only plug-in to provide that functionality, there is a need to go in that direction. Please do not try to call the process a different name.

As for the CLA situation: 5 months ago ( #1 (comment) ) on 24th Nov, I already said, it is not important, and it is waiting for another interesting fork. If somebody comes up with interesting enhancements that I like to include, and he/she is unwilling to sign the CLA. That would polute my fork, then I have to forfeit the CLA too. If Microsoft comes up with interesting enhancement I like to include, and the CLA allows a merge, it would go in the other direction. Until another interesting enhancement happens outside of my fork that I need to merge, it is simply not important for me to read the CLA. And not reading the CLA means not deciding. If you want me to make a decision whether to sign the CLA or not, please come up with an interesting enhancement that I like to have. Keep on asking will not change the situation.

it isn't clear how MS benefits

Please do answer this

I already said, it is not important

This is the problem, Hin-Tak. What is important is not for you to dictate, but emerges out of the shared values of the people involved. Aaron tried to see if the MS CLA could be avoided, and it could not. It is important to him that contributors sign the CLA, and your unwillingness to do so shows that you don't care about his needs; so it is no wonder that he appears uninterested in caring about your needs.

Putting my enhancement out there is already a significant contribution. Extra work reading a lengthy legal document is extra work.

Microsoft folks can help a merge by adding enhancements that are worth merging with. Other contributors may have enhancements I like to incorporate with, and that are in conflict with a CLA. Until another interesting fork happens, both of these are still possible - @davelab6 : I am not dictating, I am leaving the options open. The possibility that others may contribute which are in conflict with my CLA, is slightly higher than Microsoft contributing to the future development, although both are small.

As @Pomax already responded, he cannot contribute while signing a CLA; and I cannot take on his possible change if I have a CLA in place; so this is leaving the option open for him to contribute: I am allowed to take up his change, if the change is interesting enough, and I do not need to worry about the joint branch poluting and violating my CLA.

Please try not to use words like 'unwilling', 'dictate' or other emotionally charged labeling like 'threats' etc. Not deciding is simply that, not deciding, and leaving the options and possibilities open. I have been trying to explain how and when an decision can be made, and it is quite tying.

FWIW, I have been asking for code contributions from the community too, not just Microsoft; and trying to make it easier for people to contribute too, like reviving the support for building with the more common visual studio. I don't personally use MS VS, but there are people who use it daily, far more than mono.

So I am still waiting for an interesting enhancement outside of my fork to happen.

Not deciding is simply that, not deciding

That's not how social reality works.

I certainly know how frustrating it can be to seemingly get ignored while having the best intentions, but one thing to remember is that Microsoft is not people. It's easy to forget when we're discussing on Github in a PR with comments by Microsoft employees, but @aaronbell, the awesome person, doesn't really get to speak as Microsoft, the company who owns the Font Validator code, when it comes the kind of legal issues associated with the process you're talking about, @HinTak.

It would be incredibly valuable if you could have a look at the newer code to see if oddities you're seeing in a three-way parsing difference can be disambiguated, but this is not something Microsoft, the company, can easily do. As a multi billion dollar company, even if Aaron wanted to help, unless he's a Microsoft VP, he probably cant: Microsoft is so large that it has to go with what its lawyers say they can do, because if they don't they can lose legal standing and could open themselves up to all kinds of IP losses. That's where good faith comes in.

When I represent myself, I will not sign a CLA, because I am not willing to operate on the legal level required for this project. In consequence, I made a choice to not contribute any code, and am unlikely to ever do so. However, if I were as driven as you were to fix up the Font Validator to conform to the modern spec, then I probably would sign that CLA because the world as a whole benefits from that far more than I am inconvenienced by it (I need to be pretty damn invested for that to happen, but it has happened before). Of course, when I do, I would also ask that my code be reviewed not just as code but also from a legal standpoint, because I do not have the resources to guarantee everything I wrote was IP unencumbered.

I would, in that position, also love to see Microsoft's newer code to solve the problem you're having, but asking for that, without showing good will first (companies are terrified of the pandora's box that gets opened when a traditionally closed source project goes open source) and making them feel like my contributions are valuable to their company. Not just the people in the team whose repository I'm contributing to, but the actual company itself. They need to see that the work I did, for free, for my benefit, benefits them, too, and companies don't value words much: a CLA, a signature on an NDA, an agreement of collaboration, etc. are the kind of things that open the doors behind which the people can be found that might help.

Is it frustrating that this is how things work? Absolutely. Should we change that? As a bit of an idealist yes, I think we should. Should we force that? As a realist, no. Making demands, forcing hands, without showing willingness to cooperate in a language that companies (not people) understand just gets great ideas shut down, so in this case I'd suggest a brief respite to gather some thoughts, see if for the amount of work you did you might be willing to sign the CLA, and allow me to make that more interesting for you: I would be happy to send another donation if it makes the difference between the development you've put in so far getting cut off over this, and showing Microsoft that not just you, by I as well, want to make an effort to meet them where they are, so that we can find a way forward that is beneficial to everyone, not just "everyone except Microsoft".

That's not how social reality works. 

Not sure what you mean by that. I am, however, quite tired of being the only person making interesting enhancements on this project. If forfeiting the CLA is a direction to encourage contributions from people who are against it to contribute, that's an option and possibility I'd like to leave open.

Again, this is not a "threat". Signing the CLA to merge with a Microsoft enhancement, or merge with somebody else's contribution who is against the CLA, are the two future possibilities. Until another interesting fork happens, still there. Until then, there is no need to spend time reading a lengthy legal document to decide.

Font Validator gains python scripting ability, and ttc-splitter/merger scripts.

Announcement on
http://typedrawers.com/discussion/1558/microsoft-font-validator-gains-python-scripting-ability-and-ttc-splitter-merger-scripts

binary:
[1] https://sourceforge.net/projects/hp-pxl-jetready/files/Microsoft Font Validator/FontVal-empy-bin-2016_05_07.zip/download

code:
[2] https://github.com/HinTak/Font-Validator/tree/embedding-ironpython

I have re-implemented (part of) the last missing piece, the rasterization test, which tests for hinting problems. One announcement is up at -
http://typedrawers.com/discussion/1673/font-validator-with-rasterization-i-e-hinting-test-out-now

code and binaries at the usual places, and please feel free to click the donate button above the download links. I'll prepare a Mac OS X package in a couple of days.

On 10 July 2016 at 23:58, HinTak notifications@github.com wrote:

I'll prepare a Mac OS X package in a couple of days.

Will you take in the commits that are on Georg Seifert's branch?

I realized that the need for a more up-to-date binary is only to create quality analysis reports on fonts of my choice to look at (versus many bogus warnings/errors from the 2003 binary). If I cannot have quality, large quantity of interesting, if many bogus, reports would do too:

http://typedrawers.com/discussion/1658/old-ms-fontval-reports-to-spare-or-computer-time-to-make-new-ones

and even that isn't needed - to get interesting reports, it isn't necessary to analyze on all sizes - the interesting hinting errors are in the smaller sizes, and one (out of size 4 to 72 plus a few above) would do very well.
Thus I have cut possibly 3-4 weeks of test run time of all 2000 fonts on a typical linux system down to 7 hours. Am collecting reports on Apple shipped fonts now, and there are many more interesting reports to look at.

FYI, only about 2/3 of linux shipped fonts pass (expected to be a bit lower than microsoft's own, around 4/5). And quite interestingly, The proportion of Apple shipped fonts are lower.

The current list of interesting errors with accompanying interesting fonts to look at - i.e. potential hinting errors that could be detected - are up at ( HinTak#5 ).

The process of implementing the rasterization test is simply looking at the analysis report from the older binary, which says exactly which byte at which glyph is problematic. Then look at how FreeType behaves around those bytes at those glyphs. Some of the reports are wrong, but so far, most of the analysis are correct, and they turn out to be (silent) workaround for buggy fonts in FreeType's code through out past history - half of it corresponds to bug reports submitted to FreeType's bug tracker e.g. - "I have this buggy font which causes FreeType to choke - can you have a look?"

So nothing like cool reverse-engineering going on there re-implementing the rasterization test. Just looking at interesting reports which say a font is problematic at a particular glyph at a particular byte with a particular problem, and see how FreeType behaves there. Often enough, I saw a comment in FreeType code at exactly that spot, with a comment 'some fonts are so brain-damaged, we try to cope this way'. My diagnostics patch is just a 'tell me about it at this spot'.

I have tagged and uploaded another version, 4 days after the last one - it is mostly about exciting major enhancement that you should have ASAP.

July 14 was b41. I have tagged and signed for the first time b54 as "FontVal-2.0.0".

dotnet/mono binaries, also signed (hurray!)
https://sourceforge.net/projects/hp-pxl-jetready/files/Microsoft%20Font%20Validator/

When I get round to build the mac os x binaries I'd write properly.

Cut-and-paste and repeating my own writing isn't much fun, so I'd just say I have gpg-signed the 2.0 commit tags, (there are two, one on the py branch) , and am serious about it.

I refer people to the sort-of release overview:
http://typedrawers.com/discussion/1717/font-validator-2-0-arrived

and the summary test result on the libre font part of ~5000 fonts
http://typedrawers.com/discussion/1716/libre-font-test-results-and-hall-of-shame-list
and the details up at
http://htl10.users.sourceforge.net/tmp/FontVal-test-results-2016July/

And the usual, please make a donation at https://sourceforge.net/p/hp-pxl-jetready/donate/ , and if you use it for work, please consider asking your employers to.

For @schriftgestalt mostly : you want the "FontVal-2.0.0-py-osxbin.tgz" and extract ".bin/FontValidator" as well as the stand-alone libfreetype (I don't do libpng.dylib any more) from
https://sourceforge.net/projects/hp-pxl-jetready/files/Microsoft%20Font%20Validator/misc/
and presumably you want to update and offer "+raster-tests" up. Other than those two (no more libpng.dylib, and the "+raster-tests") it should work like it did. While you are at it, you want "FontVal-2.0.0-helpsrc.zip" also - there are a dozen additional figures and docs for the glyf test, since the last time you got hold of the unpacked help src.