GPQA Diamond measures graduate-level multiple-choice reasoning in biology, physics, and chemistry. Pipette evaluates each published quantization rather than reporting only a full-precision reference.
Configuration
| Field | Value |
|---|---|
| Benchmark ID | eval_gpqa_diamond_2026.06.1 |
| Dataset scope | All 198 questions in the Diamond subset |
| Attempts | 5 per question; 990 completions per model configuration |
| Sampling | Temperature 0.6; no fixed seed |
| Published metric | pass@1 across all attempt rows |
eval_gpqa_diamond_2026.06.10.6; no fixed seedDataset and generation
The source is the 198-question Diamond subset of Idavidrein/gpqa. Each question has one correct answer and three distractors. The builder shuffles the four options deterministically from a hash of the question text, making option order stable and reproducible.
An approved evaluation runner generates the completions. Each question is sampled five times without a fixed seed. See Evaluation Methodology for the shared generation and quality-result selection policy.
Scoring
The model generates a free response ending in Answer: A/B/C/D. The scorer uses
the Artificial Analysis multiple-choice answer-extraction rules, restricted to
A-D, to identify the selected option. This is generation-based scoring, not
single-token, logit-masked decoding. Pipette removes <think> and [THINK]
reasoning blocks before extracting the answer.
The leaderboard publishes correct / total across all 990 attempt rows, which
is pass@1 across five attempts per question.
The version-specific GPQA Diamond builder documents the Hugging Face revision, option shuffle, prompt template, content hash, and repeats. Answer extraction is implemented in the multiple-choice scorer.