IFBench

IFBench measures single-turn instruction following across verifiable constraints such as counting, formatting, and sentence manipulation. Pipette evaluates each published quantization rather than reporting only a full-precision reference.

Configuration

Dataset and generation

The IFBench dataset builder defines the exact source revision, content hash, and five-attempt expansion. All 300 prompts are used.

An approved evaluation runner generates the completions. Each prompt is sampled five times without a fixed seed. See Evaluation Methodology for the shared generation and quality-result selection policy.

Scoring

Pipette uses the official allenai/IFBench checks in loose mode. A prompt passes when every constraint passes under at least one of eight allowed text transformations, such as removing surrounding Markdown or introductory lines. This prevents a harmless wrapper from failing an otherwise correct answer.

Before scoring, Pipette removes <think> and [THINK] reasoning blocks. An empty response fails every constraint. The scorer has no repetition filter, and it grades the text as submitted. A sample that falls into a repetition loop runs until the model emits an end-of-sequence token or the sample reaches its context limit. Some earlier client builds stopped runaway repetition on the llama.cpp path, and the scorer graded the shortened text.

Thinking mode explains how the scorer grades a completion that stops inside a reasoning block.

The leaderboard publishes correct / total across all 1,500 attempt rows. This equals the prompt-level pass@1 mean across five attempts. The scorer also reports instruction-level accuracy as the fraction of individual constraints passed.

Implementation details are documented in the version-specific IFBench scorer and dataset builder.