Hey folks,

I’m working on designing a local, offline document retrieval + LLM pipeline and would love your input on the architecture. Here’s what I’m aiming for:

Storage

• Upload PDF, DOCX, XLSX, CSV, tables
• All data stored locally (no cloud)

Document Ingestion

• Watch folder (e.g., Watchdog) → auto‑ingest on file add/modify/delete
• Nested folder structure → auto‑tagging
• Supported formats: PDF, scanned PDF, DOCX, XLSX, CSV, JPG/PNG
• Version control on re‑upload

Query & Retrieval

• Restrict queries to a single client’s documents (no cross‑client leakage)
• Structured queries (e.g., “Show invoices > ₹1 lakh”)
• Comparative queries (e.g., “Compare FY23 vs FY24 gross profit”)
• Keyword fallback

Highlighting & Rendering

• Annotated PDF served to frontend
• XLSX → colored cell export
• Jump directly to highlighted page
• Multi‑document highlights in one response

Answer Generation

• Local LLM only
• Every claim cited with doc + page reference

My Questions

1. Parsing: I’m considering LlamaIndex LiteParse.
2. → Should I store document IDs + chunk IDs for PDFs to enable highlighting?
3. Vector DB:
   • Do I need one (e.g., Qdrant)?
   • If yes, how do I store doc IDs + chunk IDs alongside embeddings for highlighting?
   • Would pgvector in Postgres be sufficient?
4. GraphRAGs:
   • How effective are systems like Neo4j or Microsoft GraphRAG?
   • Can they run locally/offline, or are they too computationally heavy?
   • Is this GraphRAG pipeline a good starting point?
5. Highlighting UX:
   • I want something like Turnitin/iThenticate reports → exact sentence highlighted + citation.
   • Any open‑source projects that already do this?
   • I found Kotaemon and AnythingLLM, which are close but don’t highlight documents.

TL;DR

Trying to build a local RAG system with:

• Storage + ingestion + tagging
• Query + retrieval + highlighting
• Local LLM answer generation with citations

Looking for advice on:

• Vector DB vs pgvector
• GraphRAG feasibility offline
• Best way to implement document highlighting + citation preview

Would love to hear from anyone who’s built something similar or explored these tools.