CASE STUDY · PLAN ASSISTANT

Answers an advisor can send.

A retrieval pipeline that drafts replies to teachers' 403(b) questions from public district documents, cites every claim, and says so when the documents are silent.

01 · The problem

Every district runs its 403(b) differently.

Advisors who work with public-school teachers field the same questions all week: is this vendor approved here, can I borrow from my plan, how is this different from my pension. The answers live in district notices, vendor registries and state handbooks, and they differ from one district to the next.

A general-purpose chatbot answers confidently and generically, which is the worst outcome in a regulated field. The advisor needs a draft built only from that district's documents, with a source for every sentence, and an honest "not in the documents" when the answer isn't there.

02 · Pipeline

Five steps, each visible in the demo.

  1. 01

    Ingest

    12 public documents (IRS, CalSTRS, CalPERS, FINRA, district notices and vendor lists) split into 396 page-level chunks. Scanned pages are OCR'd; re-running embeds nothing that hasn't changed.

  2. 02

    Search by meaning

    The question is embedded (Voyage) and matched with pgvector, fused with Postgres full-text ranking so exact names like a vendor still score.

  3. 03

    Top 5 in territory

    The advisor's districts are resolved on the server and applied inside the SQL query, before either ranker sees a row.

  4. 04

    Claude answers only from sources

    The five chunks go to Claude as search-result blocks. The prompt allows nothing else, and a source note marks template language as generic.

  5. 05

    Answer + citations

    Citations come back as structured references, never parsed from text. No citation means no draft; no answer means NOT IN THE DOCUMENTS.

03 · Territory

The filter runs in the database, not the UI.

Each advisor covers specific districts. The browser sends only a persona name; the server looks up that persona's districts and passes them into the search function, which drops every other district's chunks before ranking. An out-of-territory document can't be retrieved, so it can't be cited.

-- inside match_chunks(): applied before vector or keyword ranking
scope as (
  select c.id, c.embedding, c.fts
  from chunks c
  where c.district is null                -- statewide documents
     or c.district = any (allowed_districts)
)

In the demo the two advisors are simulated, so anyone can pick either. In production the same filter becomes a row-level security policy keyed on the signed-in advisor, and the search runs with that user's token instead of a service key.

04 · Evaluation

Measured before it shipped.

25 questions written in a teacher's voice: 20 the documents answer, each tagged with the document that holds the answer (checked against the ingested text), and 5 they don't, including a vendor's fees, a market forecast and a district outside the advisor's territory. Every model answered from the same retrieved chunks.

Plan Assistant eval results, Claude Opus 5.5
Claude Opus 5.5Result
Expected document retrieved in the top 520/20
Answerable questions answered with citations20/20
Citations pointing at an expected document97% (90/93)
Unanswerable questions refused5/5
Answerable questions wrongly refused0
Retrieved chunks outside the advisor's territory0
Median time to a full draft3.2 s
Cost per answer$0.028

Claude Haiku 5.5 matched Opus on correctness at about 1/27th of the cost, but only with three times the token budget; its drafts ran longer and slower, so the demo answers with Opus.

05 · Privacy

Public documents, separate database.

The demo reads only public documents: federal and state guidance, district notices and vendor registries. It runs on its own database, separate from any client system, and holds no teacher or advisor records.

Typed questions aren't stored. Rate limiting keeps a salted hash of the visitor's IP address, never the address itself. The browser can't read the database at all; it only calls the answer function.

06 · Stack

What it's built with.

  • Claude API (Opus 5.5)drafting with structured citations and server-side refusal fallbacks
  • Supabase Postgres + pgvectorhybrid vector and full-text search with reciprocal-rank fusion
  • Voyage embeddings1,024-dimension document and query vectors
  • Supabase Edge Functionsstreaming answers over server-sent events
  • TypeScript / Reactthe demo UI, with a live pipeline view and citation badges
  • Cloudflare Turnstilebot check, plus a per-IP daily limit on a salted hash and a global cap