The 8 metrics that actually matter
Ragas evaluates 4 aspects of RAG. Generation quality: faithfulness (does the answer stick to the retrieved context, not hallucinate?), answer relevancy (does the answer actually answer the question?). Retrieval quality: context precision (are the retrieved chunks relevant?), context recall (did we retrieve all relevant chunks?). End-to-end: answer correctness (factual accuracy vs ground truth), answer similarity (semantic similarity to ideal answer). The metrics guide which part of your pipeline to fix: low faithfulness = better grounding needed. Low context recall = increase chunks retrieved or improve embeddings.
Testing 4 chunking strategies and finding hidden failures
I built a RAG over 200 saas.pet documents with 4 chunking strategies: fixed 512-token, fixed 1024-token, semantic (split on paragraph boundaries), and recursive (split on headers then paragraphs). Manual testing on 10 questions: all 4 strategies scored 'acceptable.' Ragas on 50 auto-generated questions: semantic chunking scored 0.82 faithfulness, fixed-512 scored 0.68, recursive scored 0.75, fixed-1024 scored 0.71. The 30% difference between semantic (best) and fixed-512 (worst) was invisible in manual testing.
Auto-generating test questions from your documents
Ragas can generate test questions from your documents: `from ragas.testset import TestsetGenerator; generator = TestsetGenerator(); testset = generator.generate(docs, test_size=50)`. It generates simple, reasoning, and multi-context questions. Simple: 'what is X?' Reasoning: 'compare X and Y.' Multi-context: 'how does X relate to Y and Z?' These 3 question types exercise different parts of your RAG. I found that my pipeline scored 0.85 on simple questions but 0.62 on multi-context questions. This revealed that my retrieval was only looking at top-3 chunks, missing information spread across chunks 4-7.
Integrating Ragas into CI/CD
Ragas outputs JSON scores. I added it to the build pipeline: after any change to the RAG config (chunk size, embedding model, retrieval strategy), run Ragas on a fixed test set. If any metric drops more than 5%, block the deploy. This takes 2 minutes in CI and catches regressions that manual testing would miss. The test set is 50 questions that cover simple, reasoning, and multi-context types.
Ragas vs manual testing vs LLM-as-judge
Ragas: structured metrics, auto-generates test questions, CI-friendly. Best for systematic RAG evaluation. Manual testing: catches UX issues (formatting, tone, speed) that metrics miss. Best for qualitative feedback. LLM-as-judge (using GPT-4 to score answers): more flexible than Ragas metrics, can evaluate things like 'tone' and 'helpfulness.' Best for open-ended evaluation. Use all three: Ragas for metrics, LLM-as-judge for tone, manual testing for UX.
