Quantization

Quantization stores model weights at lower precision. It usually reduces model size and can improve runtime performance, with a possible loss in quality. The effect depends on the model, weight format, runtime, hardware, and workload.

Formats in Pipette

Formats below are ordered from highest to lowest bit width. The table is derived from the same quantization registry used by the dashboard.

Quant	Bits	Notes
`fp16`	16	Half precision. Full quality baseline.
`bf16`	16	Brain Floating Point 16. Same bit width as fp16, with a wider dynamic range but less mantissa precision. Often preferred for inference.
`q8_0`	8	Roughly half the size of fp16, minimal quality loss.
`8bit`	8	MLX 8-bit. Comparable to fp16 accuracy at ~half the size; mlx-lm format.
`q6_k`	6	k-quant 6-bit. Close to q8_0 quality; mid-tier size.
`q5_k_m`	5	k-quant medium.
`q4_k_m`	4	k-quant medium. Better quality than q4_0 thanks to mixed precision within blocks.
`4bit`	4	MLX 4-bit. mlx-lm equivalent of GGUF q4 variants; group-wise quantization.
`q4_0`	4	Aggressive baseline 4-bit. Among the smallest standard GGUF-style formats; often fast, but hardware-dependent.

fp16

Bits

Notes

Half precision. Full quality baseline.

bf16

Bits

Notes

Brain Floating Point 16. Same bit width as fp16, with a wider dynamic range but less mantissa precision. Often preferred for inference.

q8_0

Bits

Notes

Roughly half the size of fp16, minimal quality loss.

8bit

Bits

Notes

MLX 8-bit. Comparable to fp16 accuracy at ~half the size; mlx-lm format.

q6_k

Bits

Notes

k-quant 6-bit. Close to q8_0 quality; mid-tier size.

q5_k_m

Bits

Notes

k-quant medium.

q4_k_m

Bits

Notes

k-quant medium. Better quality than q4_0 thanks to mixed precision within blocks.

4bit

Bits

Notes

MLX 4-bit. mlx-lm equivalent of GGUF q4 variants; group-wise quantization.

q4_0

Bits

Notes

Aggressive baseline 4-bit. Among the smallest standard GGUF-style formats; often fast, but hardware-dependent.

Full-precision references can use FP16 or BF16, depending on the native weights published for a model. These are both 16-bit formats, but they allocate their bits differently. They should be read as model-specific reference ceilings, not as one globally identical format.

Compare quantizations

Quantization can affect:

Weight storage and peak memory
Prefill and decode throughput
End-to-end latency
Evaluation accuracy

A model is quant-resilient when it retains most of its quality as precision drops. Quant resilience is model-specific, so a format that works well for one model should not be assumed to behave the same way for another.

Choose a deployment point

The highest-precision model is not always the best deployment choice. A smaller quantization may be preferable if it fits the target device, responds faster, and retains enough quality for the task.

Use Compare Pipette Results to compare quality, speed, and memory together. Use Coverage and Selection to understand which weight sources and quantizations are eligible for public charts.