Device Conditions on Pipette

Timing benchmarks are sensitive to device state. A hot, throttled, busy, or battery-constrained device can produce a result that reflects temporary conditions rather than the model's runtime performance. Pipette applies a readiness check before each measured timing repetition and manages phone power and cooling as external rig conditions.

At present, treat Pipette device measurements as lab-condition results, not as measurements of uncontrolled consumer use. Future iterations may include third-party submissions tagged accordingly; those submissions would not carry a guarantee that the same lab conditions were used.

What the check does

Before each measured repetition, a platform-specific probe waits until the device is both cool enough and idle enough to measure. Two kinds of signal are used:

A thermal signal: thermal-zone temperatures or the operating system's reported thermal state.
A load signal: the device must be mostly idle rather than occupied with other work.

Before each measured repetition, the probe checks the available thermal and load signals against platform-specific thresholds. A passing check confirms only that those signals were within their accepted ranges; it does not establish identical physical conditions across devices or platforms.

The probe polls until every applicable signal passes or a per-platform deadline elapses. When the load, GPU, throttling, or near-critical thermal signals do not clear before the deadline, the benchmark fails with a recorded error instead of submitting a throttled measurement. An explicit failure is better than a misleading number.

The temperature criterion is not the same test on every platform. Where a fixed ceiling is not portable across the fleet, the probe tests instead for a settled temperature: the reading has stopped moving, which is what distinguishes a box at rest from one still shedding heat. On Windows that decay test carries its own 60-second cap, shorter than the overall deadline, because past the knee of the curve further waiting buys fractions of a degree per minute. If the temperature is still drifting when the cap elapses, the repetition proceeds and the client logs that the cell started hotter than a settled box would. The load, GPU, throttle, and near-critical checks stay fail-closed regardless.

The source of truth for thresholds, polling intervals, deadlines, and sensor readers is the pipette-clients readiness crate.

Per-platform signals

Signals differ by platform because each operating system exposes different sensors:

macOS: the OS thermal state, the SoC die temperature where the sensors are readable, and a short CPU-busy sample. See the macOS readiness implementation.
Linux: the hottest kernel thermal-zone temperature plus the 1-minute load average. Generic Linux uses a thermal-zone threshold. Raspberry Pi 5 also checks the firmware throttling signal. See the Linux readiness dispatcher, generic Linux gate, and Raspberry Pi 5 gate.
Android: the OS thermal-status reading, the hottest CPU-cluster die temperature, and an instantaneous CPU-busy ratio. Phones use a stricter temperature threshold than the desktop platforms, because their SoCs shed heat more slowly between runs. The current client default is five minutes. The public dashboard accepts both the earlier ten-minute collection epoch and the newer five-minute epoch; the resolved deadline is recorded in benchmark_flags and should be checked when auditing a row. See the Android readiness implementation.
Windows: CPU load always. When the sensors are present, Pipette also checks GPU compute utilization, thermal-throttle flags, and thermal-zone temperature. Temperature is judged as a decay test rather than a fixed ceiling, because the fleet's boxes rest about 9 °C apart and use very different shares of their thermal range, so no single ceiling serves both. That test has a 60-second cap; past it the repetition proceeds with a logged warning, while load, GPU, throttle, and near-critical checks remain fail-closed. See the Windows readiness implementation.
iOS: a device-side cooldown rather than the shared probe. The public iOS thermal state is coarse and can stay nominal while the chip is already throttling, so benchmark runs use an internal, thermal-aware build that reads the real SoC die temperature and waits for it to fall back near the idle floor before each repetition.

macOS

the OS thermal state, the SoC die temperature where the sensors are readable, and a short CPU-busy sample. See the macOS readiness implementation.

Linux

the hottest kernel thermal-zone temperature plus the 1-minute load average. Generic Linux uses a thermal-zone threshold. Raspberry Pi 5 also checks the firmware throttling signal. See the Linux readiness dispatcher, generic Linux gate, and Raspberry Pi 5 gate.

Android

the OS thermal-status reading, the hottest CPU-cluster die temperature, and an instantaneous CPU-busy ratio. Phones use a stricter temperature threshold than the desktop platforms, because their SoCs shed heat more slowly between runs. The current client default is five minutes. The public dashboard accepts both the earlier ten-minute collection epoch and the newer five-minute epoch; the resolved deadline is recorded in benchmark_flags and should be checked when auditing a row. See the Android readiness implementation.

Windows

CPU load always. When the sensors are present, Pipette also checks GPU compute utilization, thermal-throttle flags, and thermal-zone temperature. Temperature is judged as a decay test rather than a fixed ceiling, because the fleet's boxes rest about 9 °C apart and use very different shares of their thermal range, so no single ceiling serves both. That test has a 60-second cap; past it the repetition proceeds with a logged warning, while load, GPU, throttle, and near-critical checks remain fail-closed. See the Windows readiness implementation.

iOS

a device-side cooldown rather than the shared probe. The public iOS thermal state is coarse and can stay nominal while the chip is already throttling, so benchmark runs use an internal, thermal-aware build that reads the real SoC die temperature and waits for it to fall back near the idle floor before each repetition.

A platform with no usable sensors runs no probe, and results from it should be treated as advisory.

Externally managed conditions

Beyond what the probe reads, the phone fleet policy defines two manually managed parts of the test rig:

Power. Fleet runs are intended to place phones on mains power, never on battery. Each device class is paired with a charger rated above the chip's sustained peak draw.
Active cooling. Fleet policy specifies two powered Neveika 001 coolers, one on the screen side and one on the back. The effectiveness of this setup was measured on Galaxy devices. The same setup is recommended for iPhone, but its effect has not yet been measured on iOS.

Cooling is operator-managed and is not recorded as a warehouse field, so an individual result row cannot verify the setup. Because power and cooling affect absolute performance, compare phone results only with runs known to use the same conditions.

The fleet-level rig policy and measured Galaxy cooling comparison are documented in the pinned device-conditions methodology.

Where it applies

The readiness gate is used for the timing benchmarks, where heat and contention change the result:

End-to-end latency, prefill throughput, and decode throughput gate before every measured repetition.
Peak memory does not gate. It measures capacity, not speed, so device temperature does not change the reported peak.
Quality evaluations do not gate. They measure output quality rather than speed, so thermal throttling does not directly change the score being reported.

On-device iOS runs use the device-side thermal cooldown described above rather than the shared readiness probe. As on the other platforms, the warm-up itself is not gated.