Pipette at a Glance

Pipette helps teams choose an open-weight model, quantization, and runtime on a selected device by comparing speed, memory use, and accuracy in one place.

Pipette is designed to make model benchmarking transparent and reproducible at the configuration level. Each result includes its model, quantization, runtime, device, and benchmark context, while the methodology links to the public source behind the measurement. The plotted performance measurements and evaluation scores were produced with the linked public Pipette source code snapshots.

The results shown come from verified benchmark runs conducted in Liquid AI's lab-optimized environment. Each run follows the same readiness, power, and cooling protocol defined for its device path, which limits environmental variation within a comparison. Support for third-party submissions from less controlled, real-world environments is planned as future work. Use the Leaderboard for results that meet documented comparison requirements, Results to inspect configuration-level measurements, and Submissions to browse all submitted benchmark rows and their provenance.

Compare Performance and Accuracy Together

Pipette has two measurement paths:

• Device performance measures latency, prefill throughput, decode throughput, and memory on the device running the benchmark.
• Accuracy evaluations score generated outputs according to each benchmark's published dataset and scoring standard. Evaluation Methodology lists the current evaluations and their source definitions.

The dashboard pairs device performance measurements with accuracy scores for the same model and quantization. The accuracy score may come from a separate evaluation run rather than the displayed device. For example, a chart point for LFM2-2.6B at Q4_K_M on the Galaxy S26 Ultra can combine throughput measured on the phone with an IFBench score generated by the separate evaluation system. The dashboard pairs them because the model and quantization match. How Results Are Selected explains how both values are chosen and combined.

What to read next

Start with Compare Pipette Results to learn how to interpret a chart value. Use Performance Methodology and Evaluation Methodology to verify how measurements are produced.

The individual pages under Benchmark Reference define each performance metric and quality evaluation in detail. The Glossary defines the terminology used throughout the dashboard.

To run Pipette yourself, use the Replication Guide for device-specific benchmark clients, evaluation scoring, and management-service setup.