End-to-end latency is the wall-clock time for one complete local request, from sending a text prompt to receiving the full generated response. It approximates user-perceived request latency at a fixed input and output length.
Definition
Latency is measured around a single local text-completion request. Because the prompt is sent as text, the timed path includes tokenization, prefill, decode, and local request/response overhead. That covers everything between submitting the prompt and receiving the last token.
Pipette reports latency in milliseconds for a fixed benchmark shape, such as 512 input tokens and 256 output tokens. The dashboard's TTFT-labeled value is not derived from this run. It is the isolated prefill time from the prefill throughput benchmark at the same input length, not submission-to-first-token latency. The measurement excludes installation, model download, model load, server startup, readiness waits, and warm-up. Two results are comparable only when their model, quantization, runtime, flags, and exact token counts match.
Measurement procedure
Latency follows the standard timing protocol: a discarded warm-up, five readiness-gated repetitions, and a mean reported with its sample standard deviation. See Performance Methodology for the shared details. The benchmark-specific steps are:
- The prompt is synthesized to an exact token count: Pipette asks the runtime's own tokenizer how many tokens a candidate prompt produces and grows the text until it tokenizes to exactly the target length.
- Token counts are validated on the warm-up and every measured request. The reported input-token and output-token counts are checked against the target, and a mismatch invalidates the run, so a result never reflects the wrong amount of work.
- Generation is forced to the full requested length. Early stopping is suppressed, so every compared run does the same amount of decode work.
- Server-side prompt caching is disabled, so each measured request pays the full prefill cost rather than reusing a previous one.
What it records
In addition to the common run record, the result stores:
- Shape: target input-token count and target output-token count.
- Measurement: total wall-clock time (
total_time_ms), with the sample standard deviation across the five repetitions.
The dashboard reports this as end-to-end latency in milliseconds.
Caveats
- On-device (iOS) runs execute in-process, so the timed path has no local server request overhead, and they use a device-side thermal cooldown instead of the shared readiness gate. Compare them only against other on-device runs, not against the server-based runtimes.