This page explains how Pipette controls device state before performance measurements.
Timing benchmarks are sensitive to device state. A hot, throttled, busy, or battery-constrained device can produce a result that reflects temporary conditions rather than the model's runtime performance. Pipette applies a readiness check before each measured timing repetition and manages phone power and cooling as external rig conditions.
At present, treat Pipette device measurements as lab-condition results, not as measurements of uncontrolled consumer use. Future iterations may include third-party submissions tagged accordingly; those submissions would not carry a guarantee that the same lab conditions were used.
What the check does
Before each measured repetition, a platform-specific probe waits until the device is both cool enough and idle enough to measure. Two kinds of signal are used:
- A thermal signal: thermal-zone temperatures or the operating system's reported thermal state.
- A load signal: the device must be mostly idle rather than occupied with other work.
Before each measured repetition, the probe checks the available thermal and load signals against platform-specific thresholds. A passing check confirms only that those signals were within their accepted ranges; it does not establish identical physical conditions across devices or platforms.
The probe polls until every applicable signal passes or a per-platform deadline elapses. When the load, GPU, throttling, or near-critical thermal signals do not clear before the deadline, the benchmark fails with a recorded error instead of submitting a throttled measurement. An explicit failure is better than a misleading number.
The temperature check differs by platform. When one fixed limit would not work across the fleet, the probe waits for the temperature to settle instead. A settled reading has stopped changing, which distinguishes a device at rest from one that is still cooling. On Windows, this settling check stops after 60 seconds, before the overall deadline, because further cooling slows to fractions of a degree per minute. If the temperature is still changing after 60 seconds, the repetition proceeds and the client records that it started hotter than a settled device. The load, GPU, throttling, and near-critical thermal checks must still pass.
The reviewed reference implementation for thresholds, polling intervals, deadlines, and sensor readers is the pipette-clients readiness crate.
Per-platform signals
Signals differ by platform because each operating system exposes different sensors:
- macOS: the OS thermal state, the SoC die temperature where the sensors are readable, and a short CPU-busy sample. See the macOS readiness implementation.
- Linux: the hottest kernel thermal-zone temperature plus the 1-minute load average. Generic Linux uses a thermal-zone threshold. See the Linux readiness dispatcher, and the generic Linux gate.
- Android: the OS thermal-status reading, the hottest CPU-cluster die temperature, and an instantaneous CPU-busy ratio. Phones use a stricter temperature threshold than the desktop platforms because their SoCs shed heat more slowly between runs. The current client default is five minutes. See the Android readiness implementation.
- iOS: a device-side cooldown rather than the shared probe. The public iOS thermal state is coarse and can stay nominal while the chip is already throttling. Benchmark runs therefore use an internal, thermal-aware build that reads the real SoC die temperature. Before each repetition, it waits for that temperature to fall back near the idle floor.
macOS
Linux
Android
iOS
If a platform has no usable sensors, Pipette skips the readiness check. Treat results from that platform as advisory.
Externally managed conditions
Operators also control two parts of the phone test setup that the readiness check does not measure:
- Power. Fleet runs are intended to place phones on mains power, never on battery. Each device class is paired with a charger rated above the chip's sustained peak draw.
- Active cooling. Fleet phone runs use powered external cooling on both sides of the device.
Cooling is operator-managed and is not recorded as a database field, so an individual result row cannot verify the setup. Because power and cooling affect absolute performance, compare phone results only with runs known to use the same conditions.
The fleet-level rig policy and measured Galaxy cooling comparison are documented in the version-specific device-conditions methodology.
Where it applies
The timing benchmarks use the readiness check because heat and background work can change the result:
- End-to-end latency, prefill throughput, and decode throughput run the check before every measured repetition.
- Peak memory does not run the check. It measures capacity, not speed, so device temperature does not change the reported peak.
- Quality evaluations do not run the check. They measure output quality rather than speed, so thermal throttling does not directly change the reported score.
On-device iOS runs use the device-side thermal cooldown described above rather than the shared readiness probe.