IFBench

IFBench measures single-turn instruction following across verifiable constraints such as counting, formatting, and sentence manipulation. Pipette evaluates each published quantization rather than reporting only a full-precision reference.

Configuration

Dataset and generation

The IFBench dataset builder defines the pinned source revision, content hash, and five-attempt expansion. All 300 prompts are used without Pipette down-selection.

Completions are generated by a runner included in the dashboard's evaluation selection scope. Each prompt is sampled five times without a fixed seed. See Evaluation Methodology for the shared generation and authority-selection policy.

Scoring

Pipette uses the official allenai/IFBench checks in loose mode. A prompt passes when every constraint passes under at least one of eight allowed text transformations, such as removing surrounding Markdown or introductory lines. This prevents a harmless wrapper from failing an otherwise correct answer.

Before scoring, Pipette removes <think> and [THINK] reasoning blocks. An empty response fails every constraint. The client truncates runaway repetition with its generation-side doom-loop detector, and the scorer evaluates the truncated text as submitted.

The leaderboard publishes correct / total across all 1,500 attempt rows. This equals the prompt-level pass@1 mean across five attempts. The scorer also reports instruction-level accuracy as the fraction of individual constraints passed.

Implementation details are pinned in the IFBench scorer and dataset builder.