Somebody asked me this question and found my answer helpful, so I thought I'd share it here. As a reminder, the OpenType spec defines four formats for colour glyph, using the COLR table, the SVG table, the CBDT table, or the 'sbix' table.
In traditional, non-colour text display, a layout engine resolves a string to an array of glyphs that then get rasterized (by a TrueType or CFF/CFF2 rasterizer), and then the resulting bitmap will get a single colour property assigned and passed to a graphics layer to present on some surface. (There may be some font-smoothing filtering somewhere along the way, but we an skip that for this discussion.) So, let's consider what changes when using a colour font: The different colour formats require different implementations with different interactions between text layout and graphics libraries.
The COLR format is in one sense the simplest for an existing text layout / graphics interaction: instead of a single array of (TT or CFF) glyphs that get rasterized and presented in one colour (which could be any colour), there is an array of glyph arrays that each get drawn with different colours and layered on top of each other. So, it's very similar to drawing different single-colour strings.
The CBDT and sbix formats are conceptually simple, but require a little more of a change in the implementation: from a glyph array, you divide this into two different glyph arrays (not unlike splitting into layers for COLR), but then also send these down two different paths: one set of glyphs will be handled normally, using the rasterizer pipeline, but the other will be handled in an entirely different manner -- not sent to the rasterizer but directly to a 2D graphics/imaging library. (Fortunately, because CBDT and sbix use commonly-used bitmap formats, the graphics/imaging library will undoubtedly support those bitmaps.)
The SVG format is even more different: The glyph array is divided into two different arrays as for CBDT/sbix and sent through different pipelines. But now instead of commonly-supported bitmap formats, you need a graphics library capable of displaying SVG documents.
DirectWrite does have support for all of the colour formats, but for CBDT, sbix and SVG, DirectWrite is actually doing very limited work: it can reports which glyphs are handled with normal rasterization versus the glyphs that use an alternate format, and it can provide the separate glyph arrays corresponding to each format, and it can return to the app the CBDT/sbix/SVG data from the font. But the app then needs to call into an appropriate graphics library to display those other data formats. Direct2D does provide support for those other formats, including SVG, but apps would need to be updated specifically to handle processing with those other formats -- it doesn't just happen unilaterally by the platform.
So, if some browser supports one format but not others, that will be because it hasn't explicitly been updated to support those other formats.