Read a Pipette value as the result of a specific deployment and benchmark configuration. A model name alone is not enough to identify a comparable measurement.
Choose the metric for the question
| Question | Metric | Direction |
|---|---|---|
| How quickly is the prompt processed? | Prefill throughput | ↑ Higher is better |
| How quickly are output tokens generated? | Decode throughput | ↑ Higher is better |
| How long does the complete request take? | End-to-end latency | ↓ Lower is better |
| How much memory does inference require? | Peak memory | ↓ Lower is better |
| How well does the model perform a specific task? | IFBench, IFStruct, GPQA Diamond, or MATH-500 | ↑ Higher is better |
The best deployment is usually a trade-off across quality, speed, and memory on the target hardware. A model that leads one metric may be unsuitable once the other constraints are considered.
Match the comparison identity
A performance value is comparable only when these four contexts match:
Runtime and model flags are part of those contexts. A different flash-attention setting, thread count, GPU-layer count, thinking mode, quantization, or token shape represents a different configuration.
For example, consider a chart value for Model A at Q4 on Device X, using
Runtime R version 1.2 with 2,048 input tokens. Only a newer submission
that matches all of those contexts can become the displayed value for that
comparison. A row using runtime version 1.3 or 4,096 input tokens belongs to
a different configuration and is handled separately.
See How Results Are Selected for the complete public cell identity and latest-row policy.
Interpret token shapes
A request has two token counts:
- Input tokens are the prompt length. They primarily drive prefill work.
- Output tokens are generated after prefill. They primarily drive decode work and total latency.
Together they determine the maximum sequence held in memory. Longer sequences grow the KV-cache and usually increase latency and memory demand.
End-to-end latency includes the time spent processing the prompt, generating the response, and handling the request around those phases.
To estimate the time for either phase, divide its token count by its throughput.
In this example, prefill takes
1,024 input tokens / 5,000 tokens/second ≈ 0.2 seconds, and decode takes
256 output tokens / 80 tokens/second = 3.2 seconds. Together, the phase
estimates total about 3.4 seconds. Measured end-to-end latency can differ
because it also includes tokenization, request handling, and other runtime work.
Read uncertainty and memory carefully
Timing results report a mean and sample standard deviation across five measured repetitions within one benchmark run. The standard deviation describes repeat-to-repeat variation on that device; it is not calculated across devices or submissions, even when they share a configuration. Treat a standard deviation greater than 5% of the mean as evidence that the run was noisy or unstable, rather than relying on the rounded mean alone.
Peak memory is the highest memory use recorded during inference, not the device's total memory capacity. Platforms measure that peak with different counters, so use it to compare models and quantizations within the same device and runtime configuration rather than as an identical cross-platform measure. See Peak Memory for the exact counter used by each platform.
Combine performance and accuracy
Accuracy and device performance answer different questions, so they are measured separately. Accuracy runs use the same dataset version, sampling policy, and task-specific scorer for each evaluation. The runtime and hardware used to generate answers may differ for specific model coverage. Performance runs measure latency, throughput, and memory on the selected device.
For example, the dashboard can match the GPQA score for Model A at Q4 with performance results for Model A at Q4 on a Mac or phone. Selecting a different device changes the performance results, while the shared GPQA score provides a consistent quality reference. It does not mean that GPQA ran on the selected device.
See Evaluation Methodology for how accuracy is measured and How Results Are Selected for how the dashboard combines the values.