In my short experience I notice kerning too early makes things more complicated as it doesn't force you to space well. In addition, if you change your letter forms or spacing itself, kerning can become inappropriate.
In the newest family I’m working on
I’ve designed the 1st master with UC,LC, SC, figures,symbols, some OT featured glyphs.
spaced it as best as possible
I’m designing now UC and LC only for all the other Masters
I'll space these new masters
when happy with the result, I’ll complete with SC, figures etc etc
I’ll kern then!
I’ll add some liga, dlig, swashes
As James says, it's good to take breaks between letterforms design, spacing, kerning. Kerning lately lets you improve spacing step by step. Of course this roadmap assumes 1 or 2 years before releasing your font family