Evaluation Methodology

Each evaluation uses a versioned dataset, a declared generation policy, and a deterministic task-specific scorer. The individual evaluation pages document their dataset versions and scoring rules.

The suite covers instruction following, structured output, science reasoning, and competition mathematics.

Evaluation pipeline

The scoring service never learns which model, runtime, or device produced a completion. Scoring depends only on the pinned dataset and submitted text.

1. Select the dataset. Each evaluation uses a fixed dataset with hidden ground truth.
2. Serve prompts. The scoring service sends sample IDs and prompts to the client without exposing answers.
3. Generate completions. A runner generates answers with the evaluation's pinned sampling policy. The raw submission retains model, quantization, runner, runtime, device, and generation provenance.
4. Submit completions. The client sends generated text and its provenance to the collector.
5. Score without provenance. The collector forwards only the evaluation ID, dataset version, sample IDs, and generated text to the scorer.
6. Store results. The collector reattaches generation provenance and stores per-sample verdicts and the aggregate score.

For a fixed dataset version, the same completion text receives the same verdict regardless of its generation provenance. This property applies to scoring. It does not claim that different runtimes or hardware necessarily generate the same completion.

The client-side generation contract is implemented in pipette-clients. The pinned temperature policy is defined in eval_id.rs, and the llama.cpp request path is implemented in execute/eval.rs.

Sampling policy

The current evaluations use temperature 0.6. Repeated evaluations omit a fixed seed, so attempts are separate sampling draws rather than repeated deterministic generations.

All four evaluations use their complete versioned datasets without Pipette down-selection. A future representative subset would need a distinct dataset version and documentation of its construction and validation before its scores could be compared.

These counts are defined by the pinned dataset builders and metadata, not by aggregate evaluation warehouse fields. Relevant source definitions include the IFBench builder, GPQA Diamond builder, MATH-500 builder, and IFStruct metadata.

Use evaluation warehouse rows to audit submitted scores. Use the dataset source to audit dataset size, attempts, upstream revision, and sample expansion.

Device readiness

Evaluations are not gated by Device Conditions. They measure output quality rather than timing, so thermal throttling does not directly change the score. Runtime, device, and generation settings are still retained as provenance and can affect the completion that is scored.

Comparability

• Compare scores only within the same evaluation and dataset version.
• Do not compare numeric scores across evaluations with different tasks or scoring rules.
• When comparing raw generation rows, match model, quantization, runtime, flags, and generation settings.
• Public charts use a pinned authority scope rather than every raw evaluation row. See How Results Are Selected.