ContextForge — Production RAG & Knowledge Retrieval Platform

Hybrid dense and sparse retrieval over technical documentation, benchmarked and served through a production API

What it does#

Indexes a 2,853-page PyTorch documentation PDF and answers questions over it with cited page sources, built up version by version from a naive pipeline to an evaluated, observable service, each stage added to fix a limitation the previous one exposed.

Retrieval#

Dense (sentence-transformers/BGE) and sparse (BM25 via Qdrant's native sparse vectors) search are fused with Qdrant's built-in Reciprocal Rank Fusion, with an optional cross-encoder reranking stage, three retrieval modes selectable per query.

Evaluation#

A 30-question benchmark, chunk-grounded so gold citations are known by construction rather than hand-labeled, scores Recall@K, MRR, nDCG, and LLM-judged faithfulness/relevancy across all three retrieval modes. The evidence-based result: hybrid search wins outright, and reranking, despite being the more sophisticated technique, measurably underperforms on this corpus.

Production API & ops#

A FastAPI service backed by PostgreSQL runs on a Dockerized Qdrant/Postgres stack, with every query traced end to end through Langfuse (per-stage latency and token usage) and a CI pipeline that runs an automated retrieval-quality regression check on every push.