RAG Knowledge System
Citation-backed answers over private documents — no hallucinated facts.
< 1.2s answer latency
Context
LLMs often hallucinate without access to private, domain-specific data. This retrieval-augmented system ingests documents, embeds them into a vector database, retrieves the relevant chunks for each question and has the model answer with citations.
Constraints
- Answers must be verifiable: every claim traceable to a source chunk.
- Latency budget under 1.2 seconds per answer.
- Embedding models must be swappable without re-architecting.
- 3 months with a team of 2 engineers.
Architecture
- Private documents → Ingest & chunk (FastAPI) (upload)
- Ingest & chunk (FastAPI) → Embeddings (LiteLLM) (chunks)
- Embeddings (LiteLLM) → Pinecone (vectors)
- Next.js Q&A → Pinecone (question → top-k)
- Pinecone → OpenAI (context chunks)
- OpenAI → Next.js Q&A (answer + citations)
Decisions & tradeoffs
- Chose
- citation-first retrieval
- over
- fine-tuning
- because
- the client needed answers they could verify, not a model that sounds confident; citations made wrong answers detectable.
Results
- answer latency
- < 1.2s
- grounded Q&A
- Cited
- embedding models
- Pluggable
Stack & role
- FastAPI
- LiteLLM
- OpenAI
- Pinecone
- Next.js