| Skill | Free resource | What it covers | Portfolio evidence |
|---|---|---|---|
| Defining success and test cases | Anthropic evaluation guide | Success criteria; code, LLM and human grading | A written spec with measurable criteria |
| Running evals in code | OpenAI Evals guide; Inspect | Datasets, graders, scorers and logs | A repository someone else can rerun |
| Measuring agreement | scikit-learn Cohen's kappa docs | Agreement between two annotators | A kappa figure with a note on disagreements |
| Risk framing | NIST AI Risk Management Framework | Trustworthiness across design, use and evaluation | A limits section mapped to real risks |
What the work involves
AI evaluation answers one question with evidence: does this model, prompt or product do the job well enough, and how do we know? The work splits into four parts.
1. Building test sets
A test set is a collection of inputs with known good outputs or clear acceptance criteria. Good ones share a few properties:
- Representative: drawn from the real distribution of tasks, including the boring majority.
- Hard cases on purpose: edge cases, ambiguous inputs and the failures users actually report.
- Held out: kept apart from prompt tuning, so you are not grading your own homework.
- Versioned: every change to the set is recorded, so last month's score and this month's are comparable.
- Free of contamination: public benchmarks may have appeared in a model's training data; private sets avoid that.
2. Writing rubrics
A rubric turns "good" into criteria a second person could apply. Anthropic's guide to defining success criteria and building evaluations asks for criteria that are specific and measurable. Each criterion needs a definition, a scale and anchored examples of each score. Write the rubric before you see outputs, or you will drift toward whatever the model happened to produce.
3. Choosing how to grade
Anthropic's guide describes three grading methods. Code-based grading, such as exact or string match, is the fastest and most reliable but lacks nuance. Human grading is the most flexible and highest quality but slow and expensive. LLM-based grading is fast, flexible and scalable for complex judgments, and Anthropic advises testing it for reliability before scaling. OpenAI's Evals guide shows how to set up graders and data programmatically through its Evals API.
Anthropic's evaluation guide, checked on 16 October 2026, describes human grading as the most flexible and highest quality method but slow and expensive.
4. Measuring inter-rater agreement
If two careful people disagree on a score, the rubric, not the model, is the first problem. The standard statistic for two annotators labeling categories is Cohen's kappa, which corrects raw agreement for the agreement expected by chance. The scikit-learn documentation describes it as "a score that expresses the level of agreement between two annotators on a classification problem," with linear and quadratic weighting options for ordered scales. The same check works between a human and an LLM grader: low agreement means the automated grader cannot replace people yet.
A single headline accuracy number without the test set, the rubric and the agreement rate is not an evaluation result.
Skills that matter
- Domain knowledge. The best evaluators know what a correct answer looks like in the field being tested, whether that is tax law, radiology reports or customer support.
- Careful writing. Rubrics, guidelines and reports are written documents; ambiguity becomes noise in the scores.
- Basic statistics. Sampling, confidence intervals, agreement statistics and the difference between false positives and false negatives.
- Python and data handling. Enough to load a dataset, call an API, score outputs and produce a table.
- Version discipline. Recording model identifiers, prompts, settings and dates so a result can be repeated.
- Skepticism. Our guide to reading AI benchmarks covers the questions to ask of any published score.
A learning path with free resources
Work through these in order; each step ends with something for your portfolio.
- Learn to define success. Read Anthropic's evaluation guide and write success criteria for one task you know well.
- Run a first eval in code. Follow OpenAI's Evals guide, or install Inspect, the open-source framework for large language model evaluations developed by the UK AI Security Institute and Meridian Labs. Its documentation covers datasets, solvers, scorers and a log viewer.
- Study how public benchmarks are built. Hugging Face's evaluation guidebook shares practical lessons from running the Open LLM Leaderboard.
- Add agreement checks. Have a second person label 50 items with your rubric and compute Cohen's kappa. Rewrite the rubric where you disagreed.
- Frame risks. Read the NIST AI Risk Management Framework, released on 26 January 2023 for voluntary use, and map your evaluation's limits to the risks it does and does not cover.
- Take a short vendor course. Anthropic Academy lists courses, starting with Claude 101, if you want structured practice with one vendor's tools; check which are free when you enroll.
How to show evidence of the skill
Employers and clients look for proof that you can run an evaluation another person can trust. A strong portfolio project includes:
- A clear problem and acceptance criteria: what task, for whom, and what "good enough" means.
- The test set or a documented sample of it, with how items were chosen and how many there are.
- The rubric, with anchored examples, written before results.
- An agreement measure between two human raters, or between a human and an automated grader.
- Results with failures shown, not only averages. Include examples of wrong outputs and why they failed.
- Version notes: model identifiers, settings, dates and costs.
- Limitations: what the evaluation cannot tell you, and what you would change.
A good first project is small. For example, evaluate two assistants on 60 real customer questions from a public FAQ, with a three-criterion rubric and a second rater. An example of a domain where error types matter is AI detection; our guide to AI detector false positives shows why a fair evaluation reports both kinds of error separately.
Pay and job titles
There is no Bureau of Labor Statistics occupation called AI evaluator. Titles vary: evaluation engineer, model quality analyst, AI rater, data scientist and research engineer all appear. The closest BLS occupation is data scientist. The BLS Occupational Outlook Handbook gives a median annual wage of $120,230 for data scientists in May 2025 and projects 35% employment growth from 2025 to 2035. Rater and annotation roles are not separately measured in that table.
The US Bureau of Labor Statistics reports a $120,230 median annual wage for data scientists in May 2025, the closest official occupation to AI evaluation work.
How we checked this
We read Anthropic's evaluation guide, OpenAI's Evals guide, the Inspect documentation, the Hugging Face guidebook, scikit-learn's kappa documentation, NIST's AI RMF page, the Anthropic Academy catalog and the BLS data scientists page on 16 October 2026. Pay data are national medians for a related occupation and do not describe any specific AI evaluation role. Related reading: the AI workflow design learning path, the careers and skills overview and how we plan to rank tools once tests are published.
Related career paths
People weighing this role often compare it with the machine learning engineer path, which share some of the same skills. All six routes are listed on our AI career paths overview.
Frequently asked questions
Do I need a computer science degree to work in AI evaluation? Not always. Domain experts who can write precise rubrics are valuable, but you will be more employable with basic Python and statistics. BLS says data scientists typically need at least a bachelor's degree.
What is a good Cohen's kappa for a rubric? There is no universal cutoff. Report the value, look at the items raters disagreed on, and tighten the rubric until disagreements are about genuinely hard cases.
Can an LLM grade another LLM's output? Yes, for many tasks, and Anthropic describes LLM-based grading as fast and scalable. Check its agreement with human graders on a sample before trusting it.
How large should my first test set be? Large enough to include the common cases and the known failure types. A portfolio project with 50 to 100 carefully chosen items is more convincing than thousands of unreviewed ones.
Are public benchmarks enough to evaluate a model for my use case? Rarely. They show general ability, but your own task set reveals the failures that matter to your users.
Is AI rating work the same as AI evaluation? Rating is part of it. Evaluation also covers designing the test set, writing the rubric, measuring agreement and interpreting the results.
Which free tool should I learn first? Start with the documentation for the model you already use, then try Inspect for a framework that works across providers.