Skip to content
All stories

RAG Knowledge System

Citation-backed answers over private documents — no hallucinated facts.

< 1.2s answer latency

Context

LLMs often hallucinate without access to private, domain-specific data. This retrieval-augmented system ingests documents, embeds them into a vector database, retrieves the relevant chunks for each question and has the model answer with citations.

Constraints

  • Answers must be verifiable: every claim traceable to a source chunk.
  • Latency budget under 1.2 seconds per answer.
  • Embedding models must be swappable without re-architecting.
  • 3 months with a team of 2 engineers.

Architecture

  1. Private documents → Ingest & chunk (FastAPI) (upload)
  2. Ingest & chunk (FastAPI) → Embeddings (LiteLLM) (chunks)
  3. Embeddings (LiteLLM) → Pinecone (vectors)
  4. Next.js Q&A → Pinecone (question → top-k)
  5. Pinecone → OpenAI (context chunks)
  6. OpenAI → Next.js Q&A (answer + citations)

Decisions & tradeoffs

Chose
citation-first retrieval
over
fine-tuning
because
the client needed answers they could verify, not a model that sounds confident; citations made wrong answers detectable.

Results

answer latency
< 1.2s
grounded Q&A
Cited
embedding models
Pluggable

Stack & role