Evaluation Methodology

This page explains how Pipette runs and scores quality evaluations. Each evaluation uses a versioned dataset, a declared generation policy, and a deterministic task-specific scorer. The individual evaluation pages document their dataset versions and scoring rules.

The suite covers instruction following, science reasoning, and competition mathematics.

Evaluation pipeline

The scoring service never learns which model, runtime, or device produced a completion. Scoring depends only on the fixed dataset version and submitted text.

1. Select the dataset. Each evaluation uses a fixed dataset with hidden ground truth.
2. Serve prompts. The scoring service sends sample IDs and prompts to the client without exposing answers.
3. Generate completions. A runner generates answers with the evaluation's fixed sampling settings. The raw submission records the model, quantization, runner, runtime, device, and other generation details.
4. Submit completions. The client sends the generated text and its generation details to the collector.
5. Score without generation details. The collector forwards only the evaluation ID, dataset version, sample IDs, and generated text to the scorer.
6. Store results. The collector reattaches the generation details and stores the per-sample verdicts and aggregate score.

For the same evaluation, dataset version, sample ID, and scorer version, the same completion receives the same verdict regardless of where it was generated.

The client generation behavior is implemented in pipette-clients. The fixed temperature setting is defined in eval_id.rs, and the llama.cpp request path is implemented in execute/eval.rs.

Sampling policy

The current evaluations use temperature 0.6. They do not use a fixed seed, so each attempt is a separate sample rather than a repeat of the same deterministic output.

Device readiness

Evaluations do not use the device readiness check. They measure output quality rather than timing, so thermal throttling does not directly change the score. The runtime, device, and generation settings are still recorded because they can affect the answer that is scored.

Comparability

• Compare scores only within the same evaluation and dataset version.
• Do not compare numeric scores across evaluations with different tasks or scoring rules.
• When comparing raw generation rows, match model, quantization, runtime, flags, and generation settings.
• Public charts publish the latest approved result rather than every raw evaluation row. See Result Publication Methodology.