These limitations define how current Pipette results should be interpreted. The linked methodology pages contain the operational detail and source references.
Compare performance within one device
Use Pipette performance results to compare models and quantizations within the same device environment. Each device path represents a complete deployment configuration: hardware, operating system, runtime artifact and backend, effective flags, readiness and cooling conditions, and platform measurement counters. Cross-device views are useful for understanding the results produced by each published deployment, but they are not controlled hardware-only comparisons and should not be used to attribute a difference to the chip alone.
Current performance warehouse rows align all four public device paths on upstream llama.cpp
b10216, so llama.cpp version drift is not currently a source of cross-device
variation. The paths still use different runtime flavors and execution surfaces:
macOS arm64 CLI, Windows Vulkan CLI, Android arm64 CLI, and the iOS in-app
runtime. Their thread counts, GPU offload, flash-attention settings, readiness
metadata, and operating systems also differ.
Android accelerator coverage
Android results in this dataset use CPU inference. NPU results are not included because current NPU paths do not support enough of the published model set for a consistent comparison. Broader NPU coverage remains future work.
GPU execution is omitted for a similar reason. Under Pipette's current workloads and model coverage, no single Android GPU backend consistently outperforms the CPU path across the full set. The Android llama.cpp path is currently CPU-focused, while the iOS path can use Metal acceleration.
These backend differences reinforce the need to compare performance within one device environment. Android measurements use the CLI with flash attention disabled and an Android-specific thread count. iOS measurements run in an app, use Metal, and have different runtime settings. A difference between the two therefore reflects the complete device and runtime configuration, not only the hardware.
Device conditions are controlled but not fully recorded
Treat current device measurements as lab-condition results. Timing runs use platform-specific readiness gates, while phone power and cooling are managed by the operator.
Support for third-party submissions from less controlled, real-world environments is planned as future work. Those submissions may be tagged separately because they would not carry the same lab-condition guarantee. See Device Conditions.
Memory counters are platform-specific
Unified-memory devices report whole-process host memory, while devices with separate GPU accounting may report GPU memory. The dashboard prefers a submitted GPU peak and otherwise uses the host peak. These values are most reliable for comparing models on the same device and should not be interpreted as identical cross-device capacity measurements.
See Peak Memory for counter windows, fields, and platform behavior.