Would you say capture-time semantic annotation for robot trajectories is a solved problem? [R]

It seems raw teleoperation data (RGB + joint states) structurally lacks affordance, contact intent, and embodiment-specific kinematic context. (information that can't be reliably recovered post-hoc once the demonstration is recorded)

Most current approaches either filter/clean after collection, or rely on simulation to compensate. But neither seems to close the semantic gap for contact-rich tasks in unstructured environments.

Is anyone working on supervision at acquisition time, enriching the stream as it's captured rather than labeling after the fact?

And if not, is this a real bottleneck or am I overestimating the problem?