These limitations define how current Pipette results should be interpreted. The linked methodology pages contain the operational detail and source references.
Client builds and release records
The public dashboard can contain results from multiple compatible client builds. Most performance publication requirements specify the model, runtime artifact, flags, device, benchmark, and metric, but do not require an exact client release.
Release pages provide supported software for running Pipette. Version-specific methodology links provide stable source references. Neither type of link proves that every displayed row was produced by one release. Historical and internal builds may not have a corresponding public release.
Published iOS measurements use the internal thermal instrumentation described under Device Conditions. Reproducing the public runtime and benchmark workload does not reproduce that private sensor access exactly.
Compare performance within one device
Use Pipette performance results to compare models and quantizations within the same device environment. Each device path represents a complete deployment configuration: hardware, operating system, runtime artifact and backend, effective flags, readiness and cooling conditions, and platform measurement counters. Cross-device views show the results produced by each published deployment. However, they are not controlled hardware-only comparisons, so a difference should not be attributed to the chip alone.
Public performance results currently require specific llama.cpp builds,
including b10216 and b10516. For the most accurate comparison, we recommend
comparing results collected with the same runtime version and configuration.
The populated paths still use different runtime flavors and execution surfaces:
macOS arm64 CLI, Android arm64 CLI, and the iOS in-app runtime. Their thread
counts, GPU offload, flash-attention settings, readiness metadata, and operating
systems also differ.
Android accelerator coverage
Android results in this dataset use CPU inference. GPU execution is omitted because no single Android GPU backend consistently outperforms the CPU path across Pipette's current workloads and model coverage. Android measurements run through the CLI with flash attention disabled. Their thread count is chosen for the Galaxy S26 Ultra to reduce thermal throttling during long runs.
iOS measurements run inside the app on the Metal backend with runtime settings that have no Android equivalent. This path is substantially faster than the Android CPU path. A difference between Android and iOS therefore reflects the complete device and runtime configuration, not only the hardware.
iOS runtime coverage
Published iOS results come from the llama.cpp path. The app also includes an AFM runtime for Apple's on-device Foundation Models text model. It can report decode throughput, end-to-end latency, and evaluations in the same way as other runtimes, but it cannot report prefill throughput or peak memory. Publishing that runtime alongside llama.cpp remains future work.
Device conditions are controlled but not fully recorded
Treat current device measurements as lab-condition results. Timing runs use platform-specific readiness gates, while phone power and cooling are managed by the operator.
Support for third-party submissions from less controlled, real-world environments is planned as future work. Those submissions may be tagged separately because they would not carry the same lab-condition guarantee. See Device Conditions.
Memory counters are platform-specific
Unified-memory devices report whole-process host memory, while devices with separate GPU accounting may report GPU memory. These values are most reliable for comparing models on the same device and should not be interpreted as identical cross-device capacity measurements.
See Peak Memory for counter windows, fields, and platform behavior.