UAE Compliance Copilot
A RAG assistant for UAE VAT, corporate tax, and labour law that cites the exact article and page for every claim, and refuses rather than guesses when a question falls outside what it knows.
A grounded chat UI is table stakes. Knowing whether the answers are actually correct is what makes a RAG system trustworthy in a regulatory domain, so the evaluation and observability layer is the point of this project, not the chat demo.
Measured results
Retrieval
Does the right source come back for the question?
Citation guardrail
A faithfulness check verifies, in code, that every citation traces back to a retrieved chunk.
Routing and refusal
Does it refuse when the question falls outside the corpus?
Answer quality
Scored on a written rubric.
A GitHub Actions CI gate re-runs this full eval on every relevant push and fails the build if any of these numbers regress past a set floor. They are checked on every change, not captured once.
The pipeline
The architecture is a LangGraph state machine: analyze the question (single vs multi-part), run hybrid retrieval (vector search plus BM25, fused with reciprocal rank fusion), rerank, run a grounding check, generate a structured answer, then run a faithfulness guardrail that verifies every citation actually traces back to a retrieved chunk, in code, not just in the prompt.
On a guardrail failure it regenerates once with a stricter prompt, then downgrades to low confidence rather than shipping an unverified claim.
How it was measured
Evaluated against a hand-written 70-question golden set: 56 in-corpus questions across VAT, corporate tax, and labour law, and 14 deliberately out-of-corpus to test refusal.
A GitHub Actions CI gate re-runs the full eval on every relevant push and fails the build if any of these numbers regress past a set floor. The metrics below are not a one-time screenshot; they are checked on every change.
Runs offline, swaps in for production
The system runs fully offline by default with a deterministic fallback and no API key required, with Claude as the pluggable production LLM. Every external dependency swaps in through a config change, not a rewrite.