DINOv2 way worse than SigLIP in k-NN. Is this expected? [R]

Doing a bachelor thesis on fine-grained car classification (telling apart VW Golf generations from listing photos). Simple setup: frozen encoder → embeddings → weighted k-NN.

On my small dataset (175 train / 132 test):

• SigLIP2 SO400M: ~92%
• CLIP ViT-L: ~59%
• DINOv2 Giant: ~41%

I thought maybe it was a cosine vs euclidean thing, but my embeddings are L2-normalized so both give the same ranking. Tried both, DINOv2 stays at 41%.

I get that SigLIP was trained contrastively so its space is basically built for cosine similarity, while DINOv2 is self-supervised and probably needs a trained head to shine. But a 50 point gap still feels huge to me.

Anyone here tried DINOv2 with a linear probe on something fine-grained? Does it actually catch up or is it just not the right tool for retrieval?

Also open to tips if there's some obvious thing I'm missing (wrong layer, wrong pooling, etc).