I built my 'first' flow matching image generator, here's what I learned [P]

Today I put out my first flow matching image generation model!

This is a toy example trained on a 2024 MPS Macbook Pro using a small sample of images—specifically, the Apple emoji library and their text labels. Because of this, it’s not a massive model (clocking in at ~4.7 million parameters), but it was an incredible learning experience.

My original approach (which failed):

I initially tried taking the emoji images, converting them to grayscale, and using an extremely basic CNN to expand the gray channels into 64 feature maps. I applied a ReLU activation, repeated this three times, and then coalesced back into a single layer to get the final theta prediction of the velocity field. I coupled this with CLIP word vector embeddings (run across the official Apple emoji descriptions) and a simple time encoding for interpolating between the noise vector field (x_0) and the target image vector field (x_1) to find the state at time t.

This approach wasn't expressive enough for the model to actually learn to predict the velocity field, especially since I was using float32 to keep the model lightweight.

The Pivot (What worked):

To fix this, I switched to using full RGB channels instead of grayscale, implemented residual blocks, and added self/cross-attention. I also increased the feature channels to allow the network to retain more information about the emojis themselves.

This worked much better. When predicting a velocity field for emojis, color is an incredibly important heuristic, and having more capacity allowed the text embeddings to form a much more meaningful relationship with the visual features during inference.

The model is completely free to play around with here:

https://emoji-generator-69.web.app/