I agree with all of this, and you seem to be best positioned to propose a specific technical architecture.

@frank-trampe I'm well-positioned in some respects but a) it does seem like being able to pull contributions from their own repos would be nice and b) I only have a vague understanding of the build infrastructure.

I guess there are actually three options:

1. Pull from externally managed repos,
2. Have contributors merge into the main fontforge repo
3. Open a new fontforge/contributions repo for this stuff and give all contributors edit permissions on it.

Option 3 has some attractive aspects.

Any thoughts @jtanx? If we can work out a mechanism I'm happy to help guide curvatura into the next release.

The problem is, as always, #3663. Curvatura works by registering a bunch of menu items. That's fine, and a lot of scripts can work that way, but to come up with contrib conventions that don't solve this problem is a half solution.

For example, you failed to mention at all @glukfonts' project, SVGinOTlayersCompositor. Maybe you don't know about it? It works via Tkinter! Tkinter, in 2019! The first commit was Oct 1.

And you know what? If we write in our conventions that Tkinter is what is to be used, I will even be OK with that. Even though we had a well working GTK integration, I will be quiet about it. But this cannot continue to be ignored if we are to have sane guidelines. I salute the work @glukfonts did but we will break it at any time. We need to make a promise that GTK will work from Python, or that Tkinter will, and that that will not be broken, for me to bother writing scripts.

cc: @jtanx @skef

My proposal, and therefore this issue, is about the bar for including contributed scripts in FontForge releases. The question of what should be possible to do with a script is an important one, but also a separate one.

does a script that includes Tkinter pass that bar

@ctrlcctrlv, the most basic answer to that question is: Not automatically by default. Some thought would have to go into it before it could.

tkinter does have the advantage of not being X-specific. Beyond that I know very little about it on a practical level. (I played around with tcl/tk some ages ago in its heyday.)

One requirement on any such script living in contrib would be acting sensibly when dependencies aren't installed. I doubt FontForge would add a "hard" tkinter dependency. Beyond that, we'd have to consider how the result acts on different systems and how "fragile" the combination is likely to be over time.

@ctrlcctrlv, it says that it is for Python 2.x. Does it work with Python 3?

@skef Well we ought to have those thoughts because whether or not Tkinter is allowed is going to have a tremendous influence on what scripts get in, and if we let in one script with Tkinter that's an enduring promise that such will be allowed longterm.

@frank-trampe It does not but I could just as easily write/find an example that uses Python 3.

It would be fairly typical if the easy consensus cases wind up in a contrib context while scripts that are more contentious or difficult to evaluate don't wind up there.

@ctrlcctrlv To the extent tkinter-like UI extensions are important you might want to focus on pushing for the plugin-type support discussed elsewhere. That way any worries about fragility are beyond the project members' scope.

I'm not really making a big ask here.

Either we promise not to break Tkinter, or we fix GTK. Either one closes #3663 as far as I'm concerned.

We should only accept scripts for contrib that check whether or not they can import what they need as far as C dependencies go, and which bundle any pure Python dependencies.

I second the motion to draw @jtanx into this discussion for perspective on GTK and the build system.

Since someone is already using Tkinter, and since the Python developers are still bundling it by default, we should just promise not to break it. I can write #4066 with Tkinter as a demo.

By the way @frank-trampe, here's a dead simple demo that works in Python 3 if you're curious:

# Python 3 only
from tkinter import messagebox, Tk

window = Tk()
# On my system, lack of the following line causes a superfluous window to appear
window.wm_withdraw()

messagebox.showinfo("I matter", "Tkinter matters")

# Very important, otherwise window will never close
window.destroy()

I really don't understand why @skef doesn't think that once we open the repository to outside Python scripts people aren't going to (rightfully) want to do stuff like this...we need to know if we're going to allow it, or if we're going to merge the GTK fix and force people to use GTK.

@ctrlcctrlv I think two different issues may be getting confused here.

There was discussion at one point of a script repository. If the FontForge project created a such a repository and then decided what could and could not go into it based on, for example, certain dependencies, that would raise the issues you've been discussing.

That's not what the contrib directory is. Scripts put into the shared contrib directory are not a matter of user choice -- each one is installed by default, probably can't be uninstalled without admin access in most cases, and automatically runs and takes up "real estate" in the Tools menu. Only a very small percentage of user-created scripts are likely to wind up in that context. When I wrote "General applicability of the functionality" I meant very general -- only those scripts likely to be useful to a high percentage of users.

If the linked-to SVG font support script had no dependencies at all I doubt I would personally support adding it to contrib, not because I didn't think it was valuable or well-written but simply because SVG fonts are currently a bit niche. People who need that tool support can install it themselves.

The idea that the project would include a small number of key scripts in contrib does not weigh one way or another on additional repositories or distribution mechanisms for a broader set of scripts.

On the other issue:

I'm happy to argue again for what I argued before: The ideal repository for (Python) FontForge extensions would be PyPI. The obstacle in the way of that solution is a discovery mechanism/convention. That's a solved problem for Python. We should probably use the namespace solution. If folks were to say to me "we think this is the top priority" I would probably do it.

With a discovery mechanism we can add hooks that will be called on packages when FontForge starts up, so that they can (e.g.) register menu items. It would also be nice to have a place in 'Prefs' where you can turn a given discovered plugin off. None of this is very difficult to code up.

There's one problem, however, that I would need @jtanx's help with. We need to be able to install real python-y files in addition to our C-API modules. (I'm not going to bother trying to code up a namespace package in C just because we don't have a way to install a python file. We write far too much Python in C as it is.)

I'm not sure PyPI is ideal. On Windows for example the Python FontForge uses is the one we bundle, right? So are we going to start bundling pip as well?

That's already a problem for any "third party" script with python dependencies -- the storage and retrieval mechanism doesn't change that. Windows is likely a problem in that way unless and until we bind against the "native" one. (I believe switching to CMake was supposed to bring that closer to possible, but I don't know what further issues we face in compiling with Visual Studio.)

(In case anyone has envisioned a repository that bundles all the dependencies, that would end up being a much bigger hornets nest in the end.)

My idea which will work fine on Windows and for users not tech savvy enough to touch command lines:

• We make a repo named fontforge/contrib
• Each contrib script is in a single directory, users add more scripts through PRs
• We do some QA to make sure each script bundles all its pure Python dependencies and bails out if it can't find a C dependency
• We pack all contrib scripts by default, but make a UI for enabling/disabling them, with only a select few enabled

This is similar to how the WeeChat project does things.

We do some QA to make sure each script bundles all its pure Python dependencies and bails out if it can't find a C dependency

Most obvious dependency for people looking to do more advanced spline stuff: numpy. Are we going to ask folks going to "bundle" numpy?

I'll be stronger: Dependencies have their own dependencies. Code changes all the time in ways requiring updates. There are potential security problems. On systems where we're using the system python the dependencies we install will clash with the dependencies installed by the user

It's a non-starter, not going to happen.

Aren't we coming up with conventions in this thread? I think that a system that has no plans for being user-friendly or accessible to Windows users (or other platforms where we bundle the interpreter) is a "non-starter".

I don't think it's a big deal to add a well-defined subdirectory to the Python module search path on select platforms. Just tell developers if they can put any Python module they want into a folder named e.g. vendor. If the system Python is in use just don't put these vendor directories into the Python path.

Apologies, I haven't been following this very closely, and I still don't have too much time on my hands right now.

Here's my 2c

contrib had become a place to put "fragile" projects that mixed C and python interfaces

It's also (if history is anything to go by), the place where code goes to die, because no one looks after it; the same too for pycontrib.

Open a new fontforge/contributions repo for this stuff and give all contributors edit permissions on it.

Out of the three options presented, I like this one the best.

The problem is, as always, #3663

I don't think so, I think that's conflating this issue. As to the whole Tk vs. GTK (vs Qt?) argument, I don't see that as being a problem; correct me if I'm wrong but given the existing limitations on how the GUI can be influenced through scripting, it's unlikely that whatever will be done in the future will cause any pre-existing functionality that uses some other toolkit to break. If it's looking like it will, well there's nothing preventing re-evaluating at that stage.

. We need to be able to install real python-y files in addition to our C-API modules

At a guess I'd think this would be a breaking change. I've thought about this before; basically we'd need to install a package like:

site-packages/fontforge/__init__.py # could potentially pull in the C modules by default or somehting
site-packages/fontforge/fontforge.pyd
site-packages/fontforge/psMat.pyd
site-packages/fontforge/other_garbage.py

But I don't know how we'd do that and retain backwards compatibility.

. On Windows for example the Python FontForge uses is the one we bundle, right? So are we going to start bundling pip as well?

We bundle a whole Python distribution, same as the mac bundle. It already has pip installed, iirc, although it's not that usable because we ship the msys2/mingw-w64 Python, which means there aren't binary wheels for things like numpy/pandas.

Finally some further thoughts:

• I'm generally ok with the idea proposed, but I would like to keep it lean/be wary of what's included by default (and as you mention @skef, be careful about what the tertiary dependencies are)
• Prove me wrong, but I'm not convinced there will be a strong uptake in contributions to whatever we end up doing. So I'd be hesitant to spend a bunch of effort and over-engineer this when no one will use it.

Prove me wrong, but I'm not convinced there will be a strong uptake in contributions to whatever we end up doing. So I'd be hesitant to spend a bunch of effort and over-engineer this when no one will use it.

I think there would be. I already mentioned the SVGinOTlayersCompositor thing...as the manager of the @fontforge Twitter account I see a lot of what people in the community are doing and there are a lot of scripts floating out there, people are disappointed there's no way to contribute them. We'd probably have around 10 scripts by mid-year 2020, I'll be contributing some of those of course.

it's unlikely that whatever will be done in the future will cause any pre-existing functionality that uses some other toolkit to break.

Well so long as we commit to not breaking Tkinter, which works right now, in the formal write up of the conventions, I'd be happy to support them.

@skef also wrote:

It would also be nice to have a place in 'Prefs' where you can turn a given discovered plugin off. None of this is very difficult to code up.

If we had this combined with a vendor PYTHONPATH we'd have a very powerful system. :-)

Open a new fontforge/contributions repo for this stuff and give all contributors edit permissions on it.

Out of the three options presented, I like this one the best.

How much of a problem for end-user builds is that likely to cause? Would we need a step like the old build tools download or can CMake just deal with that itself? (Otherwise this is my preference as well.)

At a guess I'd think this would be a breaking change. I've thought about this before; basically we'd need to install a package like ...

Well, wait, @jtanx there are two "levels" to this. For now I'm just talking about shipping a python file under the fontforge namespace. (Or, if absolutely necessary, a different namespace. We just want to be sure that if fontforge is installed with python the discovery plumbing is there too.) I thought it was possible to ship myapp in one package and myapp.subclass in another, but maybe I'm remembering wrong.

If we're talking about the second level we'd start by changing our classes to "fontforge_int" (or whatever) and then have the python files import and inherit from the renamed C-API modules. That might break some really subtle stuff, but with a little care almost everything should be fine. (Famous last words.)

(I'd be curious to play around with that a bit if and when we get a chance. Just make "empty" python files that inherit, change the names around, and see if things work and if not what breaks. Some of the internal plumbing might need to change a bit but I'm guessing it would be doable.)

I'm generally ok with the idea proposed, but I would like to keep it lean/be wary of what's included by default (and as you mention @skef, be careful about what the tertiary dependencies are)

I initiated this discussion because I think the already discussed Curvatura tool is close to meeting a bar for shipping with FontForge and therefore it would be good to have a model for that. But I agree the generality bar for contrib (or I guess pycontrib ...) should be high and the dependencies should be modest at most.

Subjectively I think the best indication is whether the tool seems like something that we would contemplate building into FontForge itself.

Anyway, if our "custom" python implementations have pip, and we have the same trouble compiling C or other non-python sources for them regardless, then there's even fewer arguments against the PyPI route for the broader set of tools. What we need for that are:

1. A "namespace class" to inherit from (which I can write) and a way to get it installed (which @jtanx can help with
2. The code to handle discovery (which I would also prefer to write in python and just invoke with the C-API, but that I can write in C if absolutely necessary.
3. A simple convention to import and call a function on discovered modules. (Basically make the interpreter aware of the module and call a method on it that means "initialize yourself as a plugin, and btw we're running with/without a UI.")
4. Preferably a screen in Preferences with a scrolling list of discovered plugins that allows you to disable them, which the discovery code takes into account.

Not a huge project.

Why not just install to a new directory like /usr/share/fontforge/python and tell FontForge to look there, like it already looks at ~/.config/fontforge/python?

Install which files?

The contents of fontforge/contrib

Oh, where the files end up isn't the problem, we'd presumably put them where pycontrib/excepthook.py ends up now. The only tricky bit is, if we maintain them in a separate repository, getting the plumbing that pulls them in to work well. We presumably want builds made by end-users to have the files too, which means they're build will have to pull the files down.

Cool. I'm glad we're allowing Tkinter. Do you want me to start writing contribution guidelines, or do you have that covered? If I finish #4066 would that meet the bar for inclusion you think?

We already do something to download additional test fonts when the tests are run and/or when making the dist package. We could hook it so that they're bundled as part of the dist package I guess.

OK, I'll think about the mechanics of the new repository and we can go from there. I would prefer having separate directories for each tool, so that each can have a README.md file we might link to, and then maybe some cmake or just make stuff to coalesce the scripts into a single directory.

And I guess it won't be that hard to do the equivalent of

discovered_plugins = {
    name: importlib.import_module(name)
    for finder, name, ispkg
    in iter_namespace(myapp.plugins)
}

from the C-API so I I suppose I can just write that stuff in C.

(Hmm. Let me read up on this properly before doing anything -- I'm going off a pretty brief skim. The important thing is that it's workable.)

Oh, this is super easy.

@jtanx Shipping actual python code remains among my strong desires, but we don't need that for this. I'll make a prototype (probably without prefs-disabling to start with) to play around with.

We already do something to download additional test fonts when the tests are run and/or when making the dist package. We could hook it so that they're bundled as part of the dist package I guess.

You could do the same to bundle non-core Python dependencies for packages that don't use the system's Python.

For each installed script:

pip install -r requirements.txt --prefix=vendor

The proposal is a system that allows FontForge to be extended through normal python packages. How is the independence from the project itself not a feature rather than a bug? I understand the "excitement" point but I don't see how it outweighs all the downsides.

It certainly is a bug. People want their code in the main release. People want end users to be able to use their scripts without a lot of hassle on Windows and OS X, where a suitable system Python/Pip is unlikely to exist.

I'm against such a system if it's our only answer to plugins. Implement it, fine, but I strongly believe we need to promise to distribute scripts if we're going to get developers interested in contributing them. Only enable a few of them by default. Make them pip packages and at the time of building for Windows have Pip install the requirements into vendor.

People want end users to be able to use their scripts without a lot of hassle on Windows and OS X, where a suitable system Python/Pip is unlikely to exist.

@jtanx already clarified:

We bundle a whole Python distribution [on Windows], same as the mac bundle. It already has pip installed, iirc, although it's not that usable because we ship the msys2/mingw-w64 Python, which means there aren't binary wheels for things like numpy/pandas.

So there's already a pip for windows and mac that can do what your alternative could do at most: install pure python packages.

So there's already a pip for windows and mac that can do what your alternative could do at most: install pure python packages.

Why should we expect users to bust out the command line to install the dependencies for a script we distribute? If we're bundling a whole Python distribution, why can't we install Python dependencies we know we need into the Windows/Mac bundles?

I've said my piece.

So scripts accepted into contrib should have their dependencies installed on the Windows/OS X bundles. Glad we agree. If it's anything like Virtualenv it should be enough just to run pip install -r requirements.txt for each module; you can drop the --prefix and the idea of having vendor folders entirely.

Hold up, well no, I likely won't be doing that. Just like system Python, I'm not about to install a bunch of crap out of the box as part of the default bundle. At the least I'd think long and hard about doing that, as it's not without implications of its own (and this has already been pointed out in earlier discussions).

Here's what I don't understand—

Why do we want to distribute Python scripts at all if we're not willing to pack their dependencies into the executables for Windows/OS X?

Do we even want to make a fontforge/contrib repository and start inviting contributions? I just don't get it.

Because I'm not going to make an announcement on Twitter like "we're inviting contributions but if you have any dependencies end users need to run arcane commands with long paths like /Applications/FontForge.app/lib/python-3.8/bin/pip install ... because we don't want to pack them even though we pack an entire Python distribution" 😵

Let someone else explain the rationale for this to people who would contribute scripts...

Agh. There are indeed two separate needs here. There are some things that we would almost certainly want everybody to have that would make sense to bundle. But then there are also tangential things that would not make sense and would count as clutter to most users.

As @ctrlcctrlv points out, casual users on Windows and Macintosh would really not be able to use the optional ones if doing so required installing dependencies, so I see the big value add in finding a good way to bundle the ones of more universal appeal. If tk seems a little too far off the beaten path to bundle, perhaps we could just shoehorn GTK instead, since that's such a light layer on top of GDK? Then we could bundle a core group of tools that use stock Python and GTK without much trouble.

@ctrlcctrlv, my long ago question (by message count) was whether the SVG thing worked with Python 3, not whether tkinter worked.

Calibre is another example of a software that supports Python plugins. They are installed via ZIP files loadable via the GUI. Many plugins are part of the default distribution as well, though not all enabled by default.

There's nothing wrong with this and it makes the plugins actually accessible to a wide audience.

@frank-trampe I just tried my example Tkinter script on Windows and it doesn't work. So the Python interpreter we're bundling doesn't include Tkinter. Jeremy wrote "We bundle a whole Python distribution [on Windows]" but I looked through C:\Program Files (x86)\FontForgeBuilds and couldn't find pip. So I can't figure out how to install dependencies but an end user should be able to somehow. That's assuming tkinter is even installable via Pip, which it might not be, it's an optional part of the default Python distribution.

So @skef said Tkinter scripts could be accepted into fontforge/contrib but neither he nor @jtanx want to bundle Tkinter (and other native code dependencies like Numpy, by installing them into the distribution via pip). They also don't want script authors to vendorize their depenencies.

So…what do they want?

I oppose any conventions that don't have an answer to the question of dependencies that works for casual users. Half-assing this isn't an option.

So…what do they want?

I'm not going to speak for @jtanx. For myself:

1. I did not say that tkinter scripts "could" be accepted into fontforge/contrib. I clarified here that there are two separate issues and that tkinter-depending scripts would likely fall under the second issue.
2. I have also expressed a preference on the second issue that the project add a discovery mechanism so that user-installed packages can extend fontforge, which takes the project members out of the decision-making process.
3. Given that I was the one who wrote and shared the gtkbridge branch (see We need a way to create UI elements from Python scripts—the inability to import GTK from Python is a bug! #3663), which adds python GTK compatibility with the UI, I'm not sure why I'm being singled out in this way.
4. Especially given that I specifically said in that discussion is that I was happy to share it but didn't want to have to argue for it, I especially don't appreciate being dragged into this argument as a rhetorical device.

@skef, did we ever merge that into mainline? If not, is there anything that would keep us from doing so?

There have been some gdkdraw merges that might call for further testing and maybe some tweaks, but broadly speaking it's probably close to being mergable (unless I'm forgetting some larger PR in the mean time).

If anyone wants me to restate/summarize my case for why that approach makes the most sense given the current state of the UI, I can. What I'll say here is that without further development, it can support the use-cases of 1) modal dialogs and 2) persistent windows not tightly tied to FontForge state. It would of course require compiling with GDK rather than X11. (Or, rather, users that did not compile with GDK could not run extensions programmed to use it.) That takes care of a lot of potential needs, but some guidance pushing contributors in those directions would be warranted.

To have more general interactions FontForge needs a better model for them, with signals and events. I outlined in #3489 how the point info dialog isn't properly bound to a specific state, and the problems someone would face trying to maintain, for example, a window related to a specific CharView are similar. So (in my view) people should be warned away from trying.

Still, the most basic throw-up-a-custom-modal-dialog interaction, similar to the "ask" methods of the python API, should work fine. The "bridge" is mostly a very small bit of modality-preserving code on top of @jtanx's massive amount of GDK work.

I just tried my example Tkinter script on Windows and it doesn't work. So the Python interpreter we're bundling doesn't include Tkinter. Jeremy wrote "We bundle a whole Python distribution [on Windows]" but I looked through C:\Program Files (x86)\FontForgeBuilds and couldn't find pip. So I can't figure out how to install dependencies but an end user should be able to somehow. That's assuming tkinter is even installable via Pip, which it might not be, it's an optional part of the default Python distribution.

The lack of tk support is because the tk dll is lazily loaded, so it's not picked up in the script that bundles all the DLLs required. As for pip, well I thought it did, but apparently pip is packaged separately on msys2. But it's not exactly hard to install pip yourself.

One of my personal scripts now uses Tkinter:

https://github.com/ctrlcctrlv/QuickAnchors/blob/main/QuickAnchors.py

So I hope any future change won't break this, as I plan to use it a lot for quickly adding anchors.

It writes a TSV file which I convert into FEA syntax:

#!/usr/bin/env python3
import csv
import sys
import fontforge

with open("top_marks") as f:
    top_marks = f.read().strip().split()
    print("@top_marks = [{}];".format(' '.join(top_marks)))

gdefsimple = list()
f = fontforge.open("FRBAmericanCursive.sfd")
for g in f.glyphs():
    if g.glyphname not in top_marks:
        gdefsimple.append(g.glyphname)

print("@GDEFSimple = [{}];".format(" ".join(gdefsimple)))

print("""
table GDEF {
    GlyphClassDef @GDEFSimple,,@top_marks,;
} GDEF;
""")

print("feature mark {")
print("    markClass @top_marks <anchor 0 0> @top;")

with open(sys.argv[1]) as csvf:
    r = csv.reader(csvf, delimiter="\t")
    for row in r:
        (glyph, x, y) = row
        print("    position base {} <anchor {} {}> mark @top;".format(glyph, x, y))

print("} mark;")