Evaluation Methodology

This page explains how Pipette runs and scores quality evaluations. Each evaluation uses a versioned dataset, a declared generation policy, and a deterministic task-specific scorer. The individual evaluation pages document their dataset versions and scoring rules.

The suite covers instruction following, structured output, science reasoning, and competition mathematics.

Evaluation pipeline

The scoring service never learns which model, runtime, or device produced a completion. The published score depends only on the fixed dataset version and the submitted text.

1. Select the dataset. Each evaluation uses a fixed dataset with hidden ground truth.
2. Serve prompts. The scoring service gives the sample IDs and prompts to the collector without the answers. The collector includes them in the benchmark definition that the client downloads.
3. Generate completions. A runner generates answers with the evaluation's pinned temperature and token limits. The raw submission records the model, quantization, runner, runtime, device, and other generation details.
4. Submit completions. The client sends the generated text and its generation details to the collector.
5. Score without generation details. The collector forwards the evaluation ID, dataset version, sample IDs, and generated text to the scorer. It also forwards how each completion ended (its stop reason) and whether the job requested thinking mode. It does not forward the model, runtime, or device.
6. Store results. The collector reattaches the generation details and stores the per-sample verdicts and aggregate score.

For the same evaluation, dataset version, sample ID, and scorer version, the same completion receives the same verdict regardless of where it was generated.

The scorer uses the stop reason and the thinking request only for a secondary strict count. That count marks a completion as wrong when thinking mode was requested and generation stopped inside an unclosed reasoning block. The published score does not use the strict count. The scoring guide describes both views.

The client generation behavior is implemented in pipette-clients. The evaluation settings are defined in the benchmark catalog, and the llama.cpp request path is implemented in execute/eval.rs.

Sampling policy

The current evaluations use temperature 0.6. They do not use a fixed seed, so each attempt is a separate sample rather than a repeat of the same deterministic output. Other sampling settings, such as top-p and top-k, use the runtime's defaults.

Each sample has a context limit of 32,768 tokens for the prompt and the output together. Generation stops when the sample reaches that limit. The _ctx32768 suffix in each benchmark ID names this limit.

Thinking mode

A model with a reasoning mode is evaluated with thinking on, with thinking off, or both. The thinking mode records the setting that the job sent to the model's chat template. It does not prove how the model behaved. Some chat templates accept the switch but do not apply it, so some models produce reasoning text with thinking off. The scorer removes closed reasoning blocks before it grades the answer in both modes.

The scorer grades a completion on its text, whatever its stop reason. If generation stops inside a reasoning block that the model opened, the scorer removes the unclosed block. The response that remains is usually empty, so it fails. If the chat template opened the reasoning block in the prompt, the completion has no opening tag, and the scorer grades all of its text.

Average intelligence

The dashboard's default quality metric is the equally weighted mean of the IFBench, IFStruct, GPQA Diamond, and MATH-500 scores. It is computed separately for each model, quantization, and thinking mode, and for the full-precision reference. The average is shown only when all four scores exist, so a missing or failed evaluation leaves it blank rather than averaging fewer scores.

Device readiness

Evaluations do not use the device readiness check. They measure output quality rather than timing, so thermal throttling does not directly change the score. The runtime, device, and generation settings are still recorded because they can affect the answer that is scored.

Comparability

• Compare scores only within the same evaluation and dataset version.
• Do not compare numeric scores across evaluations with different tasks or scoring rules.
• Scores are specific to the runtime. The runtime applies the chat template and can constrain the output, so the same model can score differently on two runtimes. When comparing raw generation rows, match model, quantization, runtime, flags, and generation settings.
• Public charts publish the latest approved result rather than every raw evaluation row. See Result Publication Methodology.