Revised — October 10, 2026. This page replaces broad, insufficiently sourced model-ranking and AGI claims with a practical evaluation method. It contains no new benchmark results and does not claim we ran the models.
Start with the decision you need to make
“Which model is best?” hides the task. An assistant extracting an invoice, a coding helper changing an existing project and a search assistant answering from documents need different evidence. Write the intended action first, then define what a usable answer must contain.
The MMLU-Pro paper illustrates why a benchmark name matters: it introduces more demanding questions and ten answer options rather than the four in the original MMLU. A score from one should not be presented as a score from the other. Neither benchmark alone measures whether a model can safely perform your particular workflow.
Build a small task set before comparing outputs
| Task | Illustrative acceptance rule | Failure to record |
|---|---|---|
| Extract a record | Required fields match the supplied document; absent values remain missing. | Invented number, swapped field, invalid format. |
| Answer from supplied evidence | Each factual statement is supported by the supplied passages; gaps are acknowledged. | Plausible but unsupported assertion or wrong citation. |
| Change an existing program | Requested behaviour works in the specified environment; relevant regression checks pass. | Unrelated edits, test failure, undeclared dependency. |
These are example rules, not results from tested models. Replace them with criteria for your own workflow. Include an ordinary case, an ambiguous input and a case where the information is unavailable. Do not invent an “ideal answer” after seeing which model wrote it.
Keep a held-out group of tasks for the final comparison. If you repeatedly rewrite a prompt to fit the trial tasks, you are tuning the workflow; record that work and test the final version on inputs it has not already seen.
Record enough information to repeat the comparison
| Field | What to save |
|---|---|
| Run and task IDs | Stable identifiers, task category and input version. |
| Model and environment | Exact model identifier, run date, tools enabled, system prompt and relevant settings. |
| Output and evidence | The actual response plus the supplied material it could use. Avoid putting private records into public logs. |
| Acceptance | Pass, fail or needs review, with the rule and a specific reason. |
| Effort and cost | Attempts, billed units, elapsed time and review effort. |
| Reviewer disagreement | Keep the disagreement visible; resolve against the predefined rule. |
Download the blank CSV worksheet. The file includes field names only; it contains no simulated scores. Use “unknown” where a provider does not expose a setting rather than filling in a guessed value.
Compare useful outputs and examine the failures
For a fictional trial with 20 tasks, suppose model A produces 16 accepted outputs after 25 attempts, while model B produces 18 after 22. That arithmetic is not a benchmark result. It shows why the accepted count, retries and task mix belong together: you cannot conclude that A is cheaper without the billed cost, or that B is reliable on a different category of work.
Read a sample of failed outputs before choosing. A formatting error that a deterministic parser catches has a different operational effect from a confident unsupported answer that reaches a customer. Record how each failure is detected and what a person must do to recover.
Finally, set a review trigger: a model version change, a changed prompt, new tool access or a shift in the input population should prompt a targeted recheck. Reusing a familiar product name does not prove its behaviour stayed the same.
Sources and scope
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark: Supports the distinction between benchmark designs. This page does not report model scores from it.
AI-assisted drafting. The linked material supports the stated facts; the examples and worksheets are Luminesca’s own editorial work. We did not conduct facility measurements, commercial project due diligence or model benchmarking for this article. Send a correction with the specific passage and supporting source.