Model measurement · pilot
Numbers you can recompute.
Leaderboards on the web are snapshots with unknown settings. We measure your model or your endpoint with open tasks, fixed settings and enough samples that a difference is a difference — and publish every number with its run ID.
How it works
The technology behind it, without the marketing fog.
Coding is measured against unit tests, mathematics against exactly comparable final results, instruction following against automatically checkable constraints. Wherever possible a checker program decides, not another model.
Temperature, prompt template, answer length and tools are recorded per run. We only compare two measurements if they were produced under the same conditions.
Every rate carries its sample size and a confidence interval. With a hundred tasks, a gap of a few percentage points is often noise — and we say so.
Quantization, fine-tuning, a new provider: the same suite before and after the change shows what it costs or gains.
The research field: evaluation science
Measuring LLMs has become a research field of its own. Benchmarks saturate as soon as models know them by heart — tasks can leak into training data (contamination), and a high score then says little about real capability.
Models as judges are convenient but biased: they prefer longer answers, their own phrasing and the first option. Reliable evaluation therefore combines verifiable tasks, enough samples and a clear separation between what was measured and what was estimated.
Open questions
- How few tasks are enough to predict a ranking across thousands?
- How do you detect contamination without knowing the training data?
- How do you measure quality where there is no single correct answer?
If the breakthrough succeeds
Outlook, not a promise: what would become possible once the open questions are answered.
Measurement as continuous operation
If a measurement with few, well-chosen tasks succeeded without losing explanatory power, every endpoint could be checked daily. A provider that silently swaps its model or quantizes it more heavily would be noticed the next morning.
Procurement with evidence
Purchasing decisions about models would rest on measurements in your own use case instead of leaderboards — with a number that can be written into the contract.
Where we stand
The same evidence as on the overview — every number with its kind.
- Compact suite (coding, knowledge, maths, instruction following) from a few francs
- Every number with run ID, sample size and settings
- Before/after for quantisation, fine-tuning or a change of provider
Who it is for
- Teams choosing between models or providers
- Providers who want to substantiate a quantization or fine-tune
- Procurement and compliance teams that need a verifiable basis
Work on it with us.
We open pilots one by one. Briefly describe what you would like to compute — no payment, no commitment, an honest assessment.
Request a pilot