LLM initiative redirected to production RAG

Challenge

Growth-stage startup planned a custom fine-tuned LLM for internal knowledge search. Budget and timeline were open-ended; no evaluation harness or governance plan existed.

KPI targets

• Answer accuracy on golden-set eval ≥ 85%
• Cost per query under $0.02 at expected volume
• Human-in-the-loop review for low-confidence answers

Approach

1. 1.Deploy or redirect: fine-tuning rejected in favor of RAG (faster, cheaper, auditable)
2. 2.Built evaluation harness with 200+ golden Q&A pairs before model selection
3. 3.Retrieval pipeline with chunking, reranking, and citation requirements
4. 4.Guardrails for prompt injection and scope-limited tool access

Production artifacts

• Production RAG API with monitoring and cost dashboards
• Evaluation harness integrated into CI/CD
• Model cards and data retention policy documentation
• Runbook for index updates and incident response

Outcomes

• Shipped production retrieval system vs. indefinite fine-tuning research
• Six months of misdirected spend avoided
• CTO cited honest 'don't build this' guidance as differentiator

Stack