These limitations define how current Pipette results should be interpreted. The linked methodology pages contain the operational detail and source references.
Client builds and release records
The public dashboard can contain results from multiple compatible client builds. Most performance publication requirements specify the model, runtime artifact, flags, device, benchmark, and metric, but do not require an exact client release.
Release pages provide supported software for running Pipette. Version-specific methodology links provide stable source references. Neither type of link proves that every displayed row was produced by one release. Historical and internal builds may not have a corresponding public release.
Published iOS measurements come from a source build of the app that reads the exact SoC die temperature through private Apple interfaces. Apple does not permit such an app in the App Store, so the App Store build uses a less exact temperature estimate. To reproduce the published iOS readiness check, make the private-interface build from source. See iOS temperature limitations.
Compare performance within one device
Use Pipette performance results to compare models and quantizations within the same device environment. Each device path represents a complete deployment configuration: hardware, operating system, runtime artifact and backend, effective flags, readiness and cooling conditions, and platform measurement counters. Cross-device views show the results produced by each published deployment. However, they are not controlled hardware-only comparisons, so a difference should not be attributed to the chip alone.
Public performance results currently use stock llama.cpp b11988, except the
large desktop-only text models (stock llama.cpp b11349), the Bonsai models
(PrismML llama.cpp fork), and Voxtral, Qwen2.5-Omni and MiniCPM-o on audio
(Liquid's llama.cpp ASR fork). For the most accurate comparison, we recommend
comparing results collected with the same runtime version and configuration.
The populated paths still use different runtime flavors and execution surfaces:
macOS arm64 CLI, Android arm64 CLI, and the iOS in-app runtime. Their thread
counts, GPU offload, flash-attention settings, readiness metadata, and operating
systems also differ.
Android accelerator coverage
Android results in this dataset use CPU inference. GPU execution is omitted because no single Android GPU backend consistently outperforms the CPU path across Pipette's current workloads and model coverage. Android measurements run through the CLI with flash attention disabled. Their thread count is chosen for the Galaxy S26 Ultra to reduce thermal throttling during long runs.
iOS measurements run inside the app on the Metal backend with runtime settings that have no Android equivalent. This path is substantially faster than the Android CPU path. A difference between Android and iOS therefore reflects the complete device and runtime configuration, not only the hardware.
iOS runtime coverage
Published iOS results come from the llama.cpp path. The app also includes an AFM runtime for Apple's on-device Foundation Models text model. It can report decode throughput, end-to-end latency, and evaluations in the same way as other runtimes, but it cannot report prefill throughput or peak memory. Publishing that runtime alongside llama.cpp remains future work.
Device conditions are controlled but not fully recorded
Treat current device measurements as lab-condition results. Timing runs use platform-specific readiness gates, while phone power and cooling are managed by the operator.
Support for third-party submissions from less controlled, real-world environments is planned as future work. Those submissions may be tagged separately because they would not carry the same lab-condition guarantee. See Device Conditions.
Memory counters are platform-specific
Most unified-memory devices, such as Apple Silicon and Android phones, report whole-process host memory. The Ryzen AI Max+ 395 has unified memory too, but its host counter misses most GPU memory, so it reports GPU memory as well, as devices with separate GPU accounting do. These values are most reliable for comparing models on the same device and should not be interpreted as identical cross-device capacity measurements.
See Peak Memory for counter windows, fields, and platform behavior.