Loss functions in Instance Representation Learning [R]
Mirrored from r/MachineLearning for archival readability. Support the source by reading on the original site.
| In Wu et. al, the MLE objective is computationally infeasible due to the high number of images in the dataset. With large n, the denominator in (2) is hard to compute. Therefore, they use NCE (Noise-Contrastive Estimation). Essentially, they approximate the difficult loss in (3) with the easier to compute loss in (7). However, we end up estimating the denominator anyways in (8). Why not just approximate the denominator in (2) with (8)? I asked Claude about this and it said something about it being a biased estimator, but I didn't really get that. I'm also a little confused on the connection of the original NCE formulation as being a way to estimate density and the way it is used here; do we do this because NCE loss is easier to compute and as m (the number of noise samples) increases, we get the gradients of NCE loss and gradients of NLL loss to match? [link] [comments] |
More from r/MachineLearning
-
For the people who got reviews back from neurips, cvpr, eccv, etc and also tested their paper through an agentic reviewer like the stanford one, how different were the reviews? [D]
Aug 14
-
Building text to ASCII diffusion model , need advice and guidance [P]
Aug 14
-
A collision-entropy floor for watermark/retrieval AI-text detection. Looking for a sanity check before I take this further [D]
Aug 14
-
Are supervised and unsupervised learning still relevant today? [D]
Aug 14
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.