Prefill throughput measures how quickly a model processes input tokens and builds the context used for generation. It influences when generation can begin, but it is not caller-visible time to first token.
Definition
The underlying measurement, prefill_time_ms, is the time required to process a
fixed number of input tokens. Pipette converts it to a tokens-per-second prefill
rate. The dashboard also displays the same isolated prefill time in columns
labeled TTFT. That value excludes text tokenization, request overhead, and
delivery of the first generated token, so it is not caller-visible TTFT.
End-to-end latency uses real text but measures through
the final output token; it does not provide caller-visible TTFT either.
The comparison unit is the input-token count, not the prompt text. A result is comparable only for the same input-token count, model, quantization, runtime, and relevant flags.
Measurement procedure
Prefill follows the standard timing protocol: a warm-up, five readiness-gated repetitions, and a mean reported with its sample standard deviation. See Performance Methodology for the shared details. The benchmark-specific steps are:
- The work is fixed by the input-token count from the benchmark definition. A runtime may use a native token-count control or a deterministic token sequence to hit it exactly.
- The underlying measurement is kept in milliseconds, so the derived tokens-per-second rate always has a clear denominator.
- Tokenizer latency is outside the measured signal. That cost is included in end-to-end latency, which sends real text.
What it records
In addition to the common run record, the result stores:
- Shape: target input-token count.
- Measurement: prefill time (
prefill_time_ms), with the sample standard deviation across the five repetitions.
Pipette derives prefill throughput in tokens per second during ingestion.
Caveats
- Prefill cost depends on input-token count, model architecture, attention implementation, batch shape, runtime flags, and device state. Compare only at matching shape and configuration.