Performance Methodology

End-to-end latency, prefill throughput, decode throughput share the timing protocol described here. Peak memory uses a separate single-run protocol. Each benchmark page covers benchmark-specific details. Accuracy evaluations follow a separate path; see Evaluation Methodology.

A request runs in two phases: prefill processes the prompt, and decode generates the output. The split happens at the first output token:

Prefill throughput and decode throughput measure the speed of each phase. The dashboard currently labels prefill time as TTFT. This value measures only the time spent processing the prompt. It does not include tokenization, request handling, or delivery of the first generated token. End-to-end latency measures the complete request through the final output token and includes tokenization, prefill, decode, and local request overhead.

Fixed conditions

Every run uses a fixed shape: a specific prompt-token count and, when the benchmark generates text, a specific generated-token count. The standard benchmark catalog uses prompt-token counts of 100, 256, 512, 1024, 2048, 4096, and 8192. The dashboard selects a device-specific subset of these shapes. See Coverage and Selection for the available benchmark shapes by device.

In this table, <P> is the configured prompt-token count. Decode throughput uses 100 output tokens. End-to-end latency uses 256 output tokens.

Generation uses greedy decoding at temperature 0.0, which removes stochastic sampling from the workload. Prompts are synthesized to an exact token count, so comparisons are by token count rather than prompt text. Two results are comparable only when the model, quantization, runtime, flags, and exact token counts all match.

Before the run, the runtime sizes its context window to the benchmark workload. Prefill uses the input-token count, while decode and end-to-end latency use the input plus output counts. A peak-memory cell is identified by its prompt-token count. Its workload also generates one token with llama.cpp and MLX, or 16 tokens with torch-OAI, so decode-path allocations are included. If the model cannot fit the required context window, the run fails with a recorded error instead of submitting a measurement.

What a result records

Each result stores the following fields for identifying and comparing a run:

Each benchmark page lists its additional shape parameters and measured fields under "What it records."

Repetitions and variance

Each timing benchmark runs a discarded warm-up, followed by five measured repetitions. Pipette reports the mean and sample standard deviation (Bessel-corrected, n − 1). Server-based and in-process paths issue an explicit warm-up request. llama.cpp prefill and decode start a fresh llama-bench process for each repetition, but report the tool's phase timing, which excludes model load. End-to-end latency starts the runtime and loads the model before warm-up and measured requests.

Read the standard deviation together with the mean. A small value means the five repetitions agreed and the mean is a reliable point estimate. A large value means the run was noisy or thermally unstable and the mean should be treated with caution. A single slow repetition can pull the mean upward. Variance is reported so an unstable run is visible rather than mistaken for a clean one.

Device readiness

Timing is sensitive to device state, so before each measured repetition a readiness gate waits until the device is cool and idle. A cool, idle device is the expected default starting condition for every measured repetition, and the gate exists to hold runs to that default. If the load, GPU, throttling, or near-critical thermal signals cannot clear before the per-platform deadline, the benchmark fails with a recorded error instead of recording a throttled measurement. The Windows temperature criterion works differently: it tests for a settled temperature rather than a fixed ceiling and carries its own shorter cap, after which the repetition proceeds with a logged warning. See Device Conditions. The warm-up itself is not gated.

Phone cooling is operator-managed and is not recorded in a warehouse field. Power is recorded: device_power_state, device_battery_level, and device_power_save_mode are stored on every row, so mains power and low-power mode can be verified per result, though the charger's rating cannot. Compare phone results only with runs known to use the same rig conditions.

See Device Conditions for the per-platform signals, externally managed conditions, and the on-device iOS cooldown. See Limitations and Future Directions for the current device, runtime-pin, memory-counter, and audit-source caveats.

Measurement boundaries

Runtimes expose timing through a native benchmark tool, a runtime throughput counter, or direct in-process timing. Each path can exclude a different part of the surrounding orchestration. Compare results only within the same runtime and configuration.

Peak-memory exception

Peak memory does not follow this protocol. It measures capacity, not speed, so it is not readiness-gated. It is captured as a single high-water peak per run with no standard deviation. Repeated runs in the same process share allocator state and high-water counters, which would make repeats non-independent.