End-to-end latency measures wall-clock time from submitting the prompt to receiving the complete response. It approximates user-perceived request latency at a fixed input and output length.
Definition
Depending on the client path, the timed operation includes tokenization, prefill, decode, and local request/response overhead. It covers everything between submitting the prompt and receiving the last token.
Pipette reports latency in milliseconds for a fixed benchmark configuration, such as 512 input tokens and 256 output tokens. The dashboard's TTFT-labeled value is not derived from this run. It is the isolated prefill time from the prefill throughput benchmark at the same input length, not submission-to-first-token latency. The measurement excludes installation, model download, model load, server startup, readiness waits, and warm-up. Two results are comparable only when their model, quantization, runtime, flags, and exact token counts match.
Measurement procedure
Latency follows the standard timing protocol: a discarded warm-up, five readiness-gated repetitions, and a mean reported with its sample standard deviation. See Performance Methodology for the shared details. The benchmark-specific steps are:
- The prompt is synthesized to an exact token count. Pipette asks the runtime's own tokenizer how many tokens a candidate prompt produces, then grows the text until the tokenizer reports the exact target count.
- Token counts are validated on the warm-up and every measured request. The reported input-token and output-token counts are checked against the target, and a mismatch invalidates the run, so a result never reflects the wrong amount of work.
- Generation is forced to the full requested length. Early stopping is suppressed, so every compared run does the same amount of decode work.
- Server-side prompt caching is disabled, so each measured request pays the full prefill cost rather than reusing a previous one.
What it records
In addition to the common run record, the result stores:
- Configuration: target input-token count and target output-token count.
- Measurement: total wall-clock time (
total_time_ms), with the sample standard deviation across the five repetitions.
The dashboard reports this as end-to-end latency in milliseconds.
Caveats
- iOS runs execute in the app's process, so the timed path has no local server request overhead. They also use the app's own readiness check instead of the shared check. Compare iOS results only with other iOS results under the same cooling conditions, not with the CLI or server runtime paths.