Asked at Microsoft · GenAI EngineerJan 2026
“Your RAG system is hallucinating answers when retrieving from long legal documents. How would you reduce hallucinations?”
What they're testing
System thinking, debugging intuition, depth of RAG knowledge
The framework · answer in order
- 01Diagnose retrieval quality first — are the right chunks even reaching the LLM?
- 02Check chunking strategy — legal docs need semantic + hierarchical chunking
- 03Add reranking (Cohere rerank or LLM-based)
- 04Tighten the prompt: explicit 'only answer from context, else say I don't know'
- 05Add post-generation verification (claim → source check)
Common traps · what 90% of candidates do wrong
- ×Jumping to 'add more documents' — that often makes it worse
- ×Blaming the model — 90% of RAG failures are retrieval failures
Follow-up they'll throw at you
What if the same chunk gives different answers on retry? How would you detect prompt injection in retrieved documents?
↑ Every question in the bundle follows this same structure