Replication Guide
Pipette publishes versioned releases of the software used to collect, process, and score benchmark results:
Client release 2026.11.1 is benchmark harness v1.1. Use it to replicate
results collected from 27 October 2026 onward. See
Benchmark harness versions.
Benchmark harness versions
Results collected before 27 October 2026 were produced with benchmark harness v1.0. Results collected from 27 October 2026 onward use benchmark harness v1.1.
Harness v1.1 is the
pipette-clients 2026.11.1 release.
The links on this page point to the source at that release. This guide does not
cover the replication of harness v1.0 results.
To replicate a result with harness v1.1, match the model artifact and revision, quantization, runtime artifact, runtime flags, benchmark configuration, client behavior, device and operating system, and documented device conditions.
Run physical benchmarks under the documented lab conditions, including the same power, cooling, readiness, and background-activity controls. Even with a matched configuration and test environment, normal hardware and operating-system variation can produce small differences between runs. Compare results only when their recorded configurations and device conditions match.
Run performance benchmarks
The links below show how to run the harness v1.1 benchmark clients. Results from these clients are comparable to published results collected with harness v1.1 when the recorded configurations and device conditions match.
Start with the Pipette CLI guide, then choose the llama.cpp or MLX runtime path.
The
model and runtime reference
explains the values accepted by --model, --runtime, and --runtime-flags.
Run evaluation scoring
Choose an evaluation dataset and scoring release. The reviewed reference workflow uses the pipette-scores quick start to install the scoring workspace. Its dataset guide documents the evaluation inputs, and the scoring guide documents each scorer.
Run the management service
You do not need a server for local benchmarks. To host the benchmark catalog, collect submissions, and process results, choose a management service release. The reviewed reference workflow is documented in the pipette-mgmt quick start and operations guide.