HumanRank’s first benchmark program is specified but has not been run. The protocol below defines the evidence required before the site publishes measured rankings.
Choose a bounded question
Start with a specific decision: which tool completes a defined coding change, preserves claims in a rewrite or finds sources for a research question. Avoid a universal score assembled from unrelated tasks.
Construct and freeze the task set
Use representative inputs, record provenance and retain difficult cases. Freeze the scored set before examining results. Keep development examples separate from evaluation examples.
Run and retain
Save model versions, prompts, settings, timestamps, raw outputs, errors, cost and latency. Include failed runs in the denominator. Use repeated runs and paired tasks when appropriate.
Review and publish
Blind reviewers to product identity when feasible. Publish the rubric, per-task data, aggregation script and limitations. Report uncertain differences honestly; a reproducible calculation is not proof that the underlying experiment was unbiased.