Performance Methodology

This page explains how Pipette measures latency, throughput, and peak memory on devices.

End-to-end latency, prefill throughput, and decode throughput share the timing protocol described here. Peak memory uses a separate single-run protocol. Each benchmark page covers benchmark-specific details. Accuracy evaluations follow a separate path; see Evaluation Methodology.

This page describes the runtime paths currently selected for the public dashboard. Other supported client runtimes may use different timing and counter sources; see the client methodology for implementation details.

A request runs in two phases: prefill processes the prompt, and decode generates the output. The split happens at the first output token:

Prefill throughput and decode throughput measure the speed of each phase. The dashboard currently labels prefill time as TTFT. This value measures only the time spent processing the prompt. It does not include tokenization, request handling, or delivery of the first generated token. End-to-end latency measures wall-clock time from submitting the prompt to receiving the complete response. Depending on the client path, this includes tokenization, prefill, decode, and local request/response overhead.

Fixed conditions

Every run uses a fixed benchmark configuration: a specific prompt-token count and, when the benchmark generates text, a specific generated-token count. The standard benchmark catalog uses prompt-token counts of 100, 256, 512, 1024, 2048, 4096, and 8192. The dashboard selects a device-specific subset of these configurations. See Coverage and Selection for the available benchmark configurations by device.

In this table, <P> is the configured prompt-token count. Decode throughput uses 100 output tokens. End-to-end latency uses 256 output tokens.

Generation uses greedy decoding at temperature 0.0, so random sampling does not affect the workload. Prompts are synthesized to an exact token count, so comparisons are by token count rather than prompt text. Two results are from the same benchmark configuration only when the model, quantization, runtime, flags, and exact token counts all match.

Before the run, the runtime sizes its context window to the benchmark workload. Prefill uses the input-token count, while decode and end-to-end latency use the input plus output counts. Peak-memory results are grouped by prompt-token count. To include memory allocated during generation, the peak-memory workload generates one output token. If the model cannot fit the required context window, the run fails with a recorded error instead of submitting a measurement.

What a result records

Submitted results include the benchmark identifier, measurement, client identity, descriptor strings, runtime and model flags, and available device metadata. Field availability depends on the client and platform. GPU details and model parameter counts are not present in every result.

For the public dashboard, each result must match the configured model, runtime, device, and benchmark descriptors. From those matches, the dashboard derives standard fields such as model name, quantization, runtime name, and runtime version. These fields identify comparable results, but they are not necessarily stored directly in every database row.

Each benchmark page lists its token-count parameters and measured fields under "What it records."

Repetitions and variance

Each timing benchmark reports the mean and sample standard deviation (Bessel-corrected, n − 1) across five measured repetitions. Runtime startup and warm-up behavior varies by execution path and is excluded from the reported timing. The version-specific client methodology documents the exact sequence for each path.

Read the standard deviation together with the mean. A small value means the five repetitions produced similar results. A large value means the run was noisy or thermally unstable, so treat the mean with caution. A single slow repetition can pull the mean upward. The reported variation makes an unstable run visible.

Device readiness

Timing is sensitive to device state. Before each measured repetition, a readiness check waits until the device is cool and idle. If the load, GPU, throttling, or near-critical thermal signals do not clear before the platform's deadline, the benchmark fails with a recorded error instead of recording a throttled measurement.

Phone cooling is operator-managed and is not recorded in a database field. Where a client reports them, device_power_state, device_battery_level, and device_power_save_mode can provide per-result power context. These fields are not populated consistently across every platform or row, and the charger's rating is not recorded. Compare phone results only with runs known to use the same power and cooling setup.

See Device Conditions for the per-platform signals, externally managed conditions, and the on-device iOS cooldown. See Limitations and Future Directions for the current caveats about devices, required runtime configurations, memory counters, and audit sources.

Measurement boundaries

Runtimes expose timing through a native benchmark tool, a runtime throughput counter, or direct in-process timing. Each path may exclude different setup or request-handling work. Compare results only within the same runtime and configuration.

Peak-memory exception

Peak memory does not follow this protocol. It measures capacity, not speed, so it is not readiness-gated. It is captured as a single high-water peak per run with no standard deviation. Repeated runs in the same process share allocator state and high-water counters, so later runs would depend on state left by earlier ones.