AI & Retrieval-Augmented Generation
Ask TanStack Query
RAG with Evals, Built in Public
I am building a RAG assistant that answers questions about TanStack Query with citations to the official docs, graded against real GitHub questions. Ingestion and retrieval are underway. Deployment and scored evaluation come next.
Status: Paused
Ingestion and retrieval are underway: 270 documentation pages are chunked and embedded. Deployment and scored evaluation against real GitHub questions are not done. I wrote up the build day by day on the blog.
The Problem
Developers learning TanStack Query hit the same walls: docs are comprehensive but scattered across pages, ChatGPT gives answers that sound right but have outdated patterns, and searching for specific API details takes too long.
The real problem runs deeper: I wanted to build a real AI product, measure whether it actually works, and show that measurement in a way hiring managers understand. A website is easy; proving the quality of an AI system is hard.
Most RAG tutorials skip the hard part: evaluation. They show retrieval and generation, but stop before proving the system is grounded in fact. I am building the complete loop: retrieval, generation, evaluation, and a way to see how each change affects the score.
How It Will Work
When someone asks a question, the system will run through these stages and stream the answer as it is written. Ingestion and search are underway, and the later stages are still ahead:
The first stages are the ones in progress: the official docs are ingested and cut into cards, each card has an embedding, and searching those cards is underway. Answering from them, streaming, and the scoreboard are what come next.
What I Am Building
Three things, built in public:
- The assistant website: Questions, streaming answers with citations, and a link to each source page
- The eval suite: Real questions from GitHub, with answers I can check against the docs
- The scoreboard: One row per quiz run, so each change shows whether it helped
The backend is Python (FastAPI) with Postgres and pgvector for storage, and the frontend is Next.js. So far the docs are ingested and embedded, and retrieval is underway. Deployment and scored evaluation come next.
I count tokens from the first script, so I will know what the system costs to run, and why.
Design Decisions So Far
These are the choices I am working from, and what the build has shown so far:
Ground every answer in a source
Every claim in an answer should map back to a documentation card and its page, so a reader can check the evidence.
Keep the quiz answers hidden
If the answers leak into what the system can see, every score looks great and means nothing.
Start with a simple cutter
270 documentation pages became 905 searchable cards. Some are cut in the wrong place, and those stay in the journal until the cutter improves.
Evaluate with real questions
The evaluation is planned around real GitHub questions, so the score reflects what developers actually ask.
Count the cost from day one
The first script prints the tokens each call uses, so the cost of every later change is visible.
Why This Project
It is built to show four things:
- Getting the right information to the AI: Cutting, embedding, and searching the docs so the model sees the right passages
- Measuring quality: A scored evaluation on real questions, because most demos ship without measurement
- Shipping it: Deploying the backend and frontend, which is the next stage
- Knowing what it costs: Token counts from day one, so cost is never a surprise
The scoreboard is the part that matters most. Many RAG demos stop at "it works." This one is meant to show numbers, with evidence for every change, and I will publish them once they exist.
Tech Stack
Ready to explore?
The repository has the source code so far, and every day lands as a commit. The build log on the blog covers each step, including the mistakes.