This page explains how Pipette makes performance and quality results comparable on the dashboard. It covers configuration matching, normalization, deduplication, and how the two types of results are combined.
How results are identified
Different stages identify a result with different fields:
| Layer | Fields |
|---|---|
| Deployment configuration | Model descriptor, quantization, runtime descriptor and version, device, and effective runtime flags |
| Eligible performance submission | Model descriptor, device chip, benchmark ID, and metric, from the device's pinned collector plans |
| Displayed performance value | Model, quantization, displayed device, runtime name and version, projected runtime flags, and normalized metric key |
| Raw submission | Complete submitted record, including the exact descriptors and flags, measured value, standard deviation where reported, client ID and version, result and submission IDs, filename, and submission time |
| Displayed quality score | Model, quantization, canonical evaluation version, and thinking mode |
The eligibility fields determine whether a raw performance row can appear in the public results. For an eligible row, the normalized metric key identifies the benchmark, metric, and token configuration. The descriptor format is documented in the version-specific storage specification.
Performance and quality publication
Performance and quality originate from different benchmark pipelines and follow separate eligibility rules. Performance results are eligible when their model, device, benchmark, and metric match an approved public configuration. Quality results are eligible when both their benchmark ID and source label are approved.
The dashboard backend selects eligible results, removes duplicates, and combines each approved quality score with compatible performance results for the same model and quantization.
Collection and evaluation scoring happen before this publication flow. See Evaluation Methodology for an overview and the version-specific management storage documentation for the exact processing sequence.
Performance selection
A performance row is eligible only when all required fields match the public configuration. These fields are the model descriptor, which fixes the quantization, the device chip, the benchmark ID, and the metric.
A run that used swap is not eligible. When the operating system moves part of a run to swap, the result describes a device that is short of memory, not the model. The collector runs a cell again when its run swapped, up to five times. If every run swaps, the collector marks the cell as failed. The dashboard shows the latest run of each cell that did not swap. If no such run exists, the cell is empty. Only the Android CLI path samples swap. Other paths do not report it, so this rule cannot exclude a swap on those paths.
Rows outside the public configuration remain available in raw history without entering the public charts.
Quality result selection
The database can contain multiple scored rows for the same evaluation from different clients, runtimes, or collection periods. The dashboard publishes the latest approved accuracy result for each model, quantization, evaluation, and thinking mode. Only one canonical version of each evaluation is approved, so results from two versions of the same evaluation are never mixed. The selected score can come from any approved runtime. The Submissions view shows the runtime of each evaluation row. Average Intelligence is the mean of these selected scores.
The dashboard combines the selected quality value with compatible performance rows for every device that runs the same quantization. Changing the performance device changes speed and memory, but it does not select a different quality row. This is a dashboard publication policy, not a claim that generation is independent of runtime or hardware.
The full-precision reference is the selected row for the same model at its native full-precision weights. It can be FP16 or BF16, depending on the model's published weights.