Performance Methodology

This page explains how Pipette measures latency, throughput, and peak memory on devices.

End-to-end latency, prefill throughput, and decode throughput share the timing protocol described here. Peak memory uses a separate protocol with one measured run. Each benchmark page covers benchmark-specific details. Accuracy evaluations follow a separate path; see Evaluation Methodology.

This page describes the runtime paths currently selected for the public dashboard. Other supported client runtimes may use different timing and counter sources; see the client methodology for implementation details.

A request runs in two phases: prefill processes the prompt, and decode generates the output. The split happens at the first output token:

Prefill throughput and decode throughput measure the speed of each phase. The dashboard currently labels prefill time as TTFT. This value measures only the time spent processing the prompt. It does not include tokenization, request handling, or delivery of the first generated token. End-to-end latency measures wall-clock time from submitting the prompt to receiving the complete response. Depending on the client path, this includes tokenization, prefill, decode, and local request/response overhead.

Fixed conditions

Every run uses a fixed benchmark configuration: a specific prompt-token count and, when the benchmark generates text, a specific generated-token count. Vision benchmarks also fix the image size and image count, and audio benchmarks the clip length; for these, the prompt-token count covers only the text prompt. The token counts and benchmark IDs below are for text. The standard benchmark catalog uses prompt-token counts of 100, 256, 512, 1024, 2048, 4096, and 8192. The dashboard selects a device-specific subset of these configurations. See Coverage and Selection for the available benchmark configurations by device.

In this table, <P> is the configured prompt-token count. Decode throughput uses 100 output tokens. End-to-end latency uses 256 output tokens.

Random sampling does not affect the workload. End-to-end latency requests use temperature 0.0 (greedy decoding). Prefill and decode throughput on the llama.cpp benchmark tool and on the iOS llama.cpp path run the model with no sampler, so no temperature applies. Prompts are synthesized to an exact token count, so comparisons are by token count rather than prompt text. Two results are from the same benchmark configuration only when the model, quantization, runtime, flags, and exact token counts all match.

Before the run, the runtime sizes its context window to fit the benchmark workload: the input-token count for prefill, and the input plus output counts for decode and end-to-end latency. A runtime can round the context window up. For example, llama.cpp rounds it up to a multiple of 256 tokens. Text peak-memory results are grouped by prompt-token count, and the text peak-memory workload generates one output token to include memory allocated during generation. Audio peak-memory runs generate their full output-token count. Vision peak-memory runs generate the output-token count when the benchmark sets one. Otherwise they produce a one-token reply, which runs no decode step. If the model cannot fit the required context window, the run fails with a recorded error instead of submitting a measurement.

What a result records

Submitted results include the benchmark identifier, measurement, client identity, descriptor strings, runtime and model flags, and available device metadata. Field availability depends on the client and platform. GPU details and model parameter counts are not present in every result.

For the public dashboard, each result must match the configured model, runtime, device, and benchmark descriptors. From those matches, the dashboard derives standard fields such as model name, quantization, runtime name, and runtime version. These fields identify comparable results, but they are not necessarily stored directly in every database row.

Each benchmark page lists its token-count parameters and measured fields under "What it records."

Repetitions and variance

Each timing benchmark reports the mean and sample standard deviation (Bessel-corrected, n − 1) across five measured repetitions. Runtime startup and warm-up behavior varies by execution path and is excluded from the reported timing. The version-specific client methodology for prefill throughput, decode throughput, and end-to-end latency documents the exact sequence for each path.

Read the standard deviation together with the mean. A small value means the five repetitions produced similar results. A large value means the run was noisy or thermally unstable, so treat the mean with caution. A single slow repetition can pull the mean upward. The reported variation makes an unstable run visible.

Device readiness

Timing is sensitive to device state. Before each measured repetition, a readiness check waits until the device is cool and idle. If any criterion does not pass before the platform's deadline, the benchmark cell fails with a recorded error instead of recording a throttled measurement.

Phone cooling is operator-managed and is not recorded in a database field. Where a client reports them, device_power_state, device_battery_level, and device_power_save_mode can provide per-result power context. These fields are not populated consistently across every platform or row. A device on a charger can still draw power from its battery, and these fields do not show that. Compare phone results only with runs known to use the same power and cooling setup.

See Device Conditions for the per-platform signals, externally managed conditions, and the iOS readiness check. Compare iOS timing results only with other iOS results, because the iOS app uses its own readiness check and runs the model in process. See Limitations and Future Directions for the current caveats about devices, required runtime configurations, memory counters, and audit sources.

Measurement boundaries

Runtimes expose timing through a native benchmark tool, a runtime throughput counter, or direct in-process timing. Each path may exclude different setup or request-handling work. Compare results only within the same runtime and configuration.

Peak-memory exception

Peak memory does not follow this protocol. It measures capacity, not speed, so it is not readiness-gated. It is captured as one high-water peak per counter from one measured run, with no standard deviation. On most runtime paths, including the llama.cpp CLI, the client first runs the same workload once in a separate process and discards that reading. This keeps one-time kernel compilation out of the measured peak. The iOS llama.cpp path runs no warm-up. Repeated runs in the same process share allocator state and high-water counters, so later runs would depend on state left by earlier ones.