IFStruct measures whether a model can generate JSON or YAML that satisfies an explicit schema and structural constraints. Pipette evaluates each published quantization with a deterministic validator rather than a judge model.
Configuration
| Field | Value |
|---|---|
| Benchmark ID | eval_ifstruct_release_v1_0_ctx32768 |
| Dataset scope | All 2,000 prompts in the release_v1_0 set |
| Attempts | 1 per prompt; 2,000 completions per model configuration |
| Sampling | Temperature 0.6; no fixed seed |
| Published metric | Per-prompt pass rate |
eval_ifstruct_release_v1_0_ctx32768release_v1_0 set0.6; no fixed seedDataset and generation
IFStruct is a Liquid AI benchmark, introduced in the IFStruct launch post. The dataset metadata defines the release version, sample count, and single attempt per prompt. All 2,000 prompts are used.
An approved evaluation runner generates the completions. Each prompt is sampled once without a fixed seed. See Evaluation Methodology for the shared generation and quality-result selection policy.
Scoring
Liquid's ifstruct validator checks each response deterministically in a fixed
order: code block, JSON or YAML parsing, commentary outside the structured value,
top-level structure and wrapper key, per-field schema rules, and item count. The
code-block and no-commentary checks apply only to prompts that request them.
Schema rules include type, required fields, enum membership, and numeric bounds.
The validator first removes <think> and [THINK] reasoning blocks. A sample
passes only when the validator returns no errors.
The leaderboard publishes the share of the 2,000 prompts that pass. The scorer also reports slices by output format, top-level structure, entity type, and common error category.
The release data and sample definitions are in the IFStruct dataset. Validation is implemented in the version-specific IFStruct scorer, which calls the pinned IFStruct validator.