Hiring lawyers for
legal AI evaluations.
A pilot brief and worked example for AI teams.
What should you ask a legal expert to evaluate?
Start with one model task and the failure you need to detect. Match the reviewer’s experience to the practice area, specify the material they may use, and agree on what makes an answer acceptable. A useful pilot produces judgments you can inspect, including the reasons reviewers disagree.
Reasoning by Lawtrades is a place to discuss legal expert projects for AI training and evaluation. Scope, suitability and availability are agreed for each project. You can start with one workflow or example answer.
Choose the deliverable before the volume.
- Teach a task
- Commission expert-written demonstrations or corrections. Specify the evidence and explanation each example should include.
- Compare answers
- Ask for preference judgments and critiques against agreed criteria. Retain the reason for each judgment.
- Measure performance
- Build an evaluation set with source packets, reference answers and scoring criteria. Keep held-out evaluation material separate from training.
- License existing material
- Treat this as a separate rights and suitability discussion. Possessing a document, or removing names, does not establish permission to reuse it.
WORKED EXAMPLE
The newest document can be the wrong source.
For this invented packet, use only signed documents and apply each change from its stated effective date. The task is to identify the notice email on February 15 and March 15, 2030. These exercise rules do not decide legal enforceability.
- D1 · SIGNED
Original agreement
Effective January 1, 2030. Section 8: send notices to notices-a@example.com.
- D2 · SIGNED JANUARY 30
Amendment
Effective March 1, 2030. Replace the Section 8 email with notices-b@example.com.
- D3 · UNSIGNED DRAFT
Proposed change
Dated February 10, 2030. Proposes notices-c@example.com for Section 8.
A supported answer
February 15: notices-a@example.com, under D1 Section 8. D2 is not effective yet.
March 15: notices-b@example.com, under D2. D3 is excluded because the exercise permits only signed documents.
An answer that selects D3 because it is newest misses the task’s rules. Score the failure separately: document status, effective date, supporting passage and limits of the conclusion. One overall “correct” label hides which part went wrong.
Agree on how judgments will be reviewed.
Choose reviewers for the work at hand, including relevant practice experience and jurisdiction where needed. Calibrate on a small shared set before expanding. Define who handles ambiguity and disagreement, what evidence reviewers must cite, and how corrections are recorded.
A sample exercise is a way to discuss the standard. It is not proof of a model’s performance or a substitute for testing on the work your users actually do.
USE THIS BRIEF
Leave unknowns as unknowns.
Copy these prompts into your project notes. Fill in what you know; the remaining questions can shape the first conversation.
- Task and user
- Our model should [task] for [user].
- Failure to test
- The error we most need to detect is [failure].
- Expertise
- The relevant practice area and jurisdiction are [known scope or unknown].
- Inputs
- We can use [synthetic, public with reviewed terms, or specifically authorized material].
- Deliverable
- We need [demonstrations, critiques, evaluation tasks, or a licensing discussion].
- Review
- We will agree on reviewer qualifications, calibration examples and who resolves disagreements.
- Constraints
- Our timing, budget and access restrictions are [known constraints or unknown].
Bring us one task.
Tell us what your model needs to do better. We’ll discuss the materials, expertise and review process the project needs.
Discuss an evaluation pilotExplore work with Reasoning · What legal evaluations should inspect