Looking for the right pipeline to convert academic textbook figures into interactive/editable assets [R]

Hi everyone,

I'm working on a document understanding project and would appreciate some advice on the right technical direction.

The input will be scanned pages or images from academic books. I don't know in advance what kind of figures they'll contain—they could be biology diagrams, anatomy illustrations, chemistry figures, engineering drawings, maps, charts, art/history figures, or other educational illustrations.

My end goal is to convert these figures into a structured digital representation that can be controlled from the frontend.

The workflow I'm aiming for is:

1. Upload a textbook page or image.
2. Detect the figure(s) and their boundaries.
3. Detect the labels/annotations that are already embedded in the figure (letters, numbers, arrows, callouts, etc.).
4. Remove those existing labels while preserving the underlying illustration.
5. Store the figure geometry (bounding boxes, polygons, masks, etc.) so my frontend can render its own labels that can be shown/hidden, translated, restyled, or repositioned.

This doesn't need to be fully automatic. In fact, the workflow will be human-assisted. If the AI detects a figure incorrectly, misses a region, or fails to remove a label cleanly, a human reviewer will correct it before it's finalized.

My priority is reducing manual work rather than eliminating it completely.

So far I've tried several computer vision approaches such as text detection, contour detection, line detection, and geometric heuristics. They work reasonably well for finding candidate regions, but the biggest challenge is cleaning the figures by removing the embedded labels while preserving the artwork underneath.

Another important requirement is cost. Since this could involve processing a large number of textbook pages, I'd like to avoid expensive multimodal LLMs or large vision models if there's a more traditional or lightweight pipeline that works well. I'm happy to use AI where it adds value, but I'd prefer a solution that keeps inference costs low.

Some questions I have:

• Is this primarily a document layout analysis problem, image segmentation, image inpainting, or something else?
• Are there models trained specifically for textbook or scientific illustrations rather than natural images?
• Is there a recommended low-cost pipeline for this kind of task?
• Has anyone built a human-in-the-loop workflow for document/figure annotation like this?
• Are there papers, datasets, or open-source projects that tackle converting textbook figures into editable, structured assets?

I'd really appreciate any suggestions, even if they're just pointers toward the right research area or open-source tools. Thanks!