IFBench

IFBench measures single-turn instruction following across verifiable constraints such as counting, formatting, and sentence manipulation. Pipette evaluates each published quantization rather than reporting only a full-precision reference.

Configuration

Dataset and generation

The IFBench dataset builder defines the exact source revision, content hash, and five-attempt expansion. All 300 prompts are used.

An approved evaluation runner generates the completions. Each prompt is sampled five times without a fixed seed. See Evaluation Methodology for the shared generation and quality-result selection policy.

Scoring

Pipette uses the official allenai/IFBench checks in loose mode. A prompt passes when every constraint passes under at least one of eight allowed text transformations, such as removing surrounding Markdown or introductory lines. This prevents a harmless wrapper from failing an otherwise correct answer.

Before scoring, Pipette removes <think> and [THINK] reasoning blocks. An empty response fails every constraint. The client stops and truncates runaway repetition, and the scorer evaluates the truncated text as submitted.

The leaderboard publishes correct / total across all 1,500 attempt rows. This equals the prompt-level pass@1 mean across five attempts. The scorer also reports instruction-level accuracy as the fraction of individual constraints passed.

Implementation details are documented in the version-specific IFBench scorer and dataset builder.