Legal Document RAG
on LegalBench-RAG 2024
Finding the right clause is not the same as answering the question. A small evaluation shows where contract-aware retrieval works and where evidence is lost between search and response.
Hover over charts for values, change the collection filters, and select a question to inspect its scores. Every chart has a data table and a CSV download. These are saved results, not a live legal advice service.
Four collections, very different documents.
The project uses the LegalBench-RAG corpus and reference spans. The local corpus and question counts below describe this project, not every version of the benchmark.
The 644 documents include non-disclosure agreements, commercial contracts, merger agreements and privacy policies. CUAD supplies most documents. MAUD supplies most text because its agreements are much longer.
How the collections contribute to the corpus.
Figure 01View the data table.
Document lengths span several orders of magnitude.
Figure 02View the data table.
Some benchmark questions lack a source document in this corpus.
Figure 03View the data table.
Length changes the retrieval problem.
The shortest document has 219 words and the longest has 156,104. The median is 5,494 words, compared with a mean of 15,248. Passing whole documents to the answer model would obscure the relevant clause and increase the context sent to the API.
Availability limits the evaluation pool.
Of 6,695 extracted questions, 6,033 have an available source document. Ten of those questions were evaluated. PrivacyQA contributes three documents to the corpus but no questions to this evaluated sample.
Identify the contract before searching its clauses.
A question names a contract before a semicolon. That description and the legal question serve different purposes.
- Preserve the legal wording.Cleaning keeps casing and negation. Contact details and separator lines are replaced with spaces, and whitespace is normalised. An offset map preserves the connection to the original source characters. Stop-word removal is used only in exploratory analysis.
- Create two representations of each document.LangChain's recursive text splitter creates windows of up to 1,500 characters with a 200-character overlap. A separate document card contains the file name and first 1,500 characters. This gives 63,914 chunks and 644 cards.
- Build separate vector collections.OpenAI text-embedding-3-small embeds the chunks and cards. Chroma persists them in two local collections. The stored index is reused rather than rebuilt for every question.
- Route and retrieve.The contract description selects the closest document card. The legal question then searches only that contract's chunks. The five highest-ranked chunks become the answer context.
- Generate a constrained answer.OpenAI gpt-4o-mini runs at temperature zero. The prompt asks for exact quotes, source citations and abstention when the evidence is insufficient. If no excerpts are found, the chain abstains without a generation call.
- Evaluate retrieval separately from generation.Reference-span overlap supplies Hit@5 and MRR@5. Ragas uses an LLM judge for faithfulness, answer relevancy, context precision and context recall. These measures answer different questions and should not be combined into one accuracy score.
Merger agreements account for most of the chunk index.
Figure 04View the data table.
| Technical layer | Implementation and purpose |
|---|---|
| Analysis | Python 3.12, pandas and NumPy organise the corpus and evaluation results. NLTK and scikit-learn support token analysis and TF-IDF comparisons. |
| Retrieval orchestration | LangChain connects text splitting, embeddings, Chroma retrieval and OpenAI generation. |
| Evaluation | Ragas scores evidence use and answer quality. The notebook also computes direct character-span overlap metrics. |
| Visual reporting | Matplotlib and seaborn produce saved notebook figures. This report uses local Plotly.js, plain JavaScript, HTML and CSS on GitHub Pages. It needs no application server or external chart CDN. |
Running the notebook sends document text, questions and retrieved excerpts to OpenAI. Reading this report and exporting its saved data make no model calls. The report contains aggregate statistics and question identifiers, not the document corpus.
Retrieval is stronger than the final answer.
The sample contains three ContractNLI, four CUAD and three MAUD questions, selected with seed 42.
Compare retrieval and answering scores.
Figure 05View the data table.
Nine retrieval hits produced only eight non-abstaining responses.The model abstained twice. At least one retrieval hit therefore did not lead to an answer. A partial span overlap can still be insufficient evidence.
The evidence is often present.
Hit@5 is 0.900 and context recall is 0.871. This supports retaining document-aware retrieval as the baseline. It does not establish that the entire benchmark is solved.
Evidence use needs more attention.
Answer relevancy is 0.525, while faithfulness is 0.802. A response can remain close to the supplied excerpts without directly answering the question. High faithfulness alone is not a measure of usefulness.
| Measure | Interpretation |
|---|---|
| Hit@5 | At least one of the five retrieved chunks overlaps a reference text span. |
| MRR@5 | The reciprocal rank of the first overlapping chunk is averaged across questions. Misses score zero. |
| Faithfulness | The judge assesses whether generated claims are supported by the retrieved excerpts. |
| Answer relevancy | The judge assesses whether the response addresses the question. |
| Context precision and recall | The judge assesses the ordering of useful excerpts and their coverage of the reference answer. |
Look beyond the averages.
The ten saved rows expose different failure modes. A single aggregate score would conceal them.
Inspect an individual question.
Figure 06View the data table.
The first overlapping chunk does not always rank first.
Figure 07View the data table.
- ContractNLI 344 is the only retrieval miss. Its faithfulness score is still 1.000, while recall and relevancy are zero. This is why faithful wording must not be confused with successful question answering.
- CUAD 3537 has a first-ranked retrieval hit and context recall of 1.000, but both answer relevancy and faithfulness are zero. Its answer should be inspected before changing the retrieval strategy.
- Four of the nine hits first overlap the reference below rank one. Two first overlap at rank five. This motivates a controlled re-ranking experiment, although the current run does not measure its benefit.
See the additional corpus exploration.
The notebook also examines frequent terms and TF-IDF similarity. The following charts are saved notebook figures rather than interactive measurements.
A useful baseline, not a legal decision system.
The next work should isolate errors before adding more infrastructure.
What the project establishes.
The implementation demonstrates a complete route, retrieve and answer workflow with source offsets and saved evaluation results. On this ten-question sample, it usually retrieves text that overlaps the reference. The weaker answer relevancy shows that retrieval coverage and useful responses are separate goals.
What a project owner should prioritise.
- Classify failures before changing models.Review the selected contract, retrieved span and generated response for each question. Label routing, ranking, incomplete-context and answering failures separately. This avoids treating every low score as an embedding problem.
- Test one intervention at a time.Compare re-ranking, richer document cards and clause-aware chunking against the current baseline. Keep the question set fixed and record both improved and regressed cases. These are proposed experiments, not demonstrated improvements.
- Keep abstention tied to evidence.Do not remove cautious behaviour merely to increase response rate. Review when the retrieved context genuinely supports an answer, then test prompt changes against that judgement.
- Expand evaluation before claiming reliability.Use a larger, balanced sample across collections and question types. Add human review of citations and factual claims, and record latency and API usage. The current report does not measure production cost, throughput or legal accuracy.
Limits that change the interpretation.
- One question changes Hit@5 by ten percentage points. Collection filters contain only three or four cases and should not be used to rank datasets.
- Routing selects a single contract. A routing mistake can exclude the correct evidence entirely.
- Ragas uses an LLM judge. Scores and generated responses can vary between runs despite a fixed sample and temperature zero.
- The system is a research prototype. Its generated answers are not legal advice and should not replace qualified review.
Sources and reproducibility.
- LegalBench-RAG, published in August 2024, supplies the benchmark provenance.
- The saved notebook and metrics file contain the reported results.
- The report exporter reads saved tables without calling a model. The dependency manifest records the Python environment.