I did some more tests and it seems that indeed it is enough to store one point delta per contour plus the RSB phantom point delta for each glyph. The rest of the points follows along.
I’ll see if this could be an automatic optimisation in varLib, though it is probably a rare case. If the proposed spacing axis is widely used, it could be built into a font without preparing separate TTFs that would have to be optimized again. It would be easier to make a separate script that could apply this kind of spacing modification to any font.