GPQA Diamond measures graduate-level multiple-choice reasoning in biology, physics, and chemistry. Pipette evaluates each published quantization rather than reporting only a full-precision reference.
Configuration
| Field | Value |
|---|---|
| Benchmark ID | eval_gpqa_diamond_2026.06.1 |
| Dataset scope | Full 198-question Diamond subset; no Pipette down-selection |
| Attempts | 5 per question; 990 completions per model configuration |
| Sampling | Temperature 0.6; no fixed seed |
| Published metric | pass@1 across all attempt rows |
eval_gpqa_diamond_2026.06.10.6; no fixed seedDataset and generation
The source is the 198-question Diamond subset of Idavidrein/gpqa. Each question has one correct answer and three distractors. The builder shuffles the four options deterministically from a hash of the question text, making option order stable and reproducible.
Completions are generated by a runner included in the dashboard's evaluation selection scope. Each question is sampled five times without a fixed seed. See Evaluation Methodology for the shared generation and authority-selection policy.
Scoring
The model generates a free response ending in Answer: A/B/C/D. The scorer uses
the Artificial Analysis multiple-choice regex, restricted to A-D, to extract the
selected option. This is generation-based scoring, not single-token,
logit-masked decoding. Pipette removes <think> and [THINK] reasoning blocks
before extracting the answer.
The leaderboard publishes correct / total across all 990 attempt rows, which
is pass@1 across five attempts per question.
The pinned GPQA Diamond builder documents the Hugging Face revision, option shuffle, prompt template, content hash, and repeats. Answer extraction is implemented in the multiple-choice scorer.