Device Conditions

This page explains how Pipette controls device state before performance measurements, and where that control stops.

Timing benchmarks are sensitive to device state. A hot, throttled, busy, or battery-constrained device can produce a result that reflects temporary conditions rather than the model's runtime performance. Pipette controls device state in two ways:

• Detected conditions. A readiness check reads the device's temperature and load before each measured timing repetition.
• Managed conditions. The test rig sets power, cooling, and operating-system settings. The readiness check does not measure them.

At present, treat Pipette device measurements as lab-condition results, not as measurements of uncontrolled consumer use. Future iterations may include third-party submissions tagged accordingly; those submissions would not carry a guarantee that the same lab conditions were used.

What the check does

Before each measured repetition, a platform-specific probe waits until the device is both cool enough and idle enough to measure. Each probe reads two kinds of signal:

• A thermal signal: thermal-zone or die temperatures, and the operating system's thermal state where the platform reports one.
• A load signal: the share of CPU time that is busy, and GPU load where the platform reports it.

The probe polls until every applicable signal passes or a per-platform deadline elapses. Each criterion is a fixed threshold, and the same thresholds apply to every model. A passing check confirms only that those signals were within their accepted ranges. It does not establish identical physical conditions across devices or platforms.

If any criterion does not pass before the deadline, the probe stops the whole benchmark cell. Pipette does not run the remaining repetitions and stores no partial or throttled measurement. The cell is recorded as an error, so it does not appear in the published results.

The reviewed reference implementation for thresholds, polling intervals, deadlines, and sensor readers is the pipette-clients readiness crate.

Per-platform signals

Signals differ by platform because each operating system exposes different sensors. The table shows the criteria for the platforms in the published results and for generic Linux hosts.

Platform notes:

• macOS. The thermal-pressure level is conservative but coarse. On a MacBook Neo it stays above nominal for about five minutes after the CPU goes quiet. The 7-minute deadline gives that wait enough time. See the macOS readiness implementation.
• Windows. The probe does not wait for temperatures to stop changing. Measurements showed that waiting for flat readings did not change results on the GMKtec EVO-X2 by more than 0.2%, and it made each run take 3.4 times as long. Temperature holds the check only near the hardware's critical trip point, or where a host has a measured ceiling for a named sensor. A host with no temperature counter uses the load criteria only.
• Linux. A host that exposes no thermal zone uses the load criteria only. See the Linux readiness dispatcher and the generic Linux gate.
• Android. The OS thermal status stays nominal while the CPU dies reach 75–80 °C, so the CLI also reads the CPU-cluster temperatures directly. Phones use a lower temperature threshold than desktop platforms because their SoCs shed heat more slowly between runs. See the Android readiness implementation.
• iOS. The app runs its own version of the macOS check. The iOS limitations below explain the two temperature sources.

A platform outside this list has no probe, and its runs start without a readiness check. Treat results from such a platform as advisory.

iOS temperature limitations

Apple's public interfaces do not report the SoC die temperature. The public thermal state is coarse. It can stay nominal while the chip already runs at a lower clock speed.

An app in the App Store must use only interfaces that Apple approves. The App Store build of Pipette therefore cannot read the die temperature. Its readiness check uses the public thermal state and an estimate from the motion sensors, calibrated against the die temperature.

A build from source can turn on private interfaces that read the exact die temperature. Apple does not permit an app that uses private interfaces in the App Store. Such a build runs only on registered development devices.

Published iOS results come from a build with the private interfaces turned on. Anyone can make the same build from the public source with the PIPETTE_PRIVATE_THERMAL=1 setting. See the private-interface build guide. A run from the App Store build uses a less exact readiness check. Its results can differ from the published iOS results.

What the check does not control

The readiness check controls only the device state at the start of each measured repetition. A long repetition can heat the device past the readiness band, even with active cooling. Under a full load, the hottest CPU zone of a cooled Galaxy S26 Ultra reaches its 105 °C trip point within 2–4 seconds. Long cells can therefore include throttled time inside the measured window.

Heavier models heat the device more, so their cells are more likely to reach the deadline and fail. These failures are recorded as errors and are not removed silently, so the failure rate of each cell can be checked.

Externally managed conditions

The rig sets three conditions that the readiness check does not measure. They stay constant for the duration of a run:

• Power. Phones stay connected to a charger, and dedicated hosts are connected to mains power. Laptops use the adapter that ships with them. Connection to a charger does not prevent a device from drawing power from its battery, and Pipette cannot prevent it:
  • The Galaxy S26 Ultra keeps its battery in the power path. Energy measurements showed the Galaxy S26 Ultra and the iPhone drawing from the battery during a repetition while on the charger.
  • The MacBook Neo ships with a 20 W adapter and can draw up to 33 W. Under load, it can also draw from its battery.
• Active cooling. Each phone has one powered cooler clamped to its back. No cooler touches the screen, because a cooler on the screen can cause phantom touches. The MacBook Neo has a passive pad inside the case and an active pad under it. Other machines use their stock cooling. A device with added cooling does not represent the product as sold.
• Operating-system settings. Each device's settings are set to a fixed state before it joins the test fleet.

Each result records the power state that the client read for that result, where the platform reports it:

• device_battery_level: the battery level, 0 to 100.
• device_power_state: charging, plugged_in_not_charging, or not_charging (on battery).
• device_power_save_mode: whether the OS power-save mode is on.

These fields show the power source at the time of the reading. They do not show whether the device also drew from its battery during the repetition. Cooling is not recorded as a field, so an individual result row cannot verify the cooling setup. Because power and cooling affect absolute performance, compare results only with runs known to use the same conditions.

The fleet rig policy and the measured Galaxy S26 Ultra cooling comparison are documented in the version-specific device-conditions methodology.

Where it applies

The timing benchmarks use the readiness check because heat and background work can change the result:

• End-to-end latency, prefill throughput, and decode throughput run the check before every measured repetition.
• Peak memory does not run the check. It measures capacity, not speed, so device temperature does not change the reported peak.
• Quality evaluations do not run the check. They measure output quality rather than speed, so thermal throttling does not directly change the reported score.

On-device iOS runs use the app's own readiness check, described above. Compare iOS timing results only with other iOS results under the same cooling conditions.