IFStruct measures whether a model can generate JSON or YAML that satisfies an explicit schema and structural constraints. Pipette evaluates each published quantization with a deterministic validator rather than a judge model.
Configuration
| Field | Value |
|---|---|
| Benchmark ID | eval_ifstruct_release_v1_0 |
| Dataset scope | Full 2,000-prompt release_v1_0 set; no down-selection |
| Attempts | 1 per prompt; 2,000 completions per model configuration |
| Sampling | Temperature 0.6 |
| Published metric | Per-prompt pass rate |
eval_ifstruct_release_v1_0release_v1_0 set; no down-selection0.6Dataset and generation
IFStruct is a Liquid AI benchmark. Pipette uses the full 2,000-prompt release set without down-selection and generates one completion per prompt.
Completions are generated by a runner included in the dashboard's evaluation selection scope. See Evaluation Methodology for the shared generation and authority-selection policy.
Scoring
Liquid's ifstruct validator checks each response deterministically in a fixed
order: required code block, JSON or YAML parsing, commentary outside the
structured value, top-level structure and wrapper key, per-field schema rules,
and item count. Schema rules include type, required fields, enum membership, and
numeric bounds.
Pipette removes <think> and [THINK] reasoning blocks before validation. Other
commentary outside the structured value fails the no-commentary check. A sample
passes only when the validator returns no errors.
The scorer reports the overall pass rate, with additional slices by output format, top-level structure, task type, and common error category.
The release data and sample definitions are in the IFStruct dataset. Validation is implemented in the pinned IFStruct scorer.