MATH-500 measures multi-step competition mathematics across algebra, geometry, number theory, precalculus, and other subjects. Pipette evaluates each published quantization rather than reporting only a full-precision reference.
Configuration
| Field | Value |
|---|---|
| Benchmark ID | eval_math_500_2026.06.1 |
| Dataset scope | Full named 500-problem subset; no Pipette down-selection |
| Attempts | 5 per problem; 2,500 completions per model configuration |
| Sampling | Temperature 0.6; no fixed seed |
| Published metric | pass@1 across all attempt rows |
eval_math_500_2026.06.10.6; no fixed seedDataset and generation
The source is HuggingFaceH4/MATH-500, a 500-problem subset of the Hendrycks MATH benchmark from OpenAI's PRM800K work. All 500 problems are used from a pinned upstream revision.
Completions are generated by a runner included in the dashboard's evaluation selection scope. Each problem is sampled five times without a fixed seed. See Evaluation Methodology for the shared generation and authority-selection policy.
Scoring
MATH-500 is open-ended rather than multiple choice. The model provides a
step-by-step solution and places its final answer in \boxed{}. The PRM800K
grader checks symbolic equivalence, so equivalent forms such as 1/2 and 0.5
can match. A numeric-normalization fallback handles additional valid forms.
Pipette removes <think> and [THINK] reasoning blocks before grading.
The leaderboard publishes correct / total across all 2,500 attempt rows, which
is pass@1 across five attempts per problem.
The pinned MATH-500 builder documents the Hugging Face revision, prompt template, content hash, repeats, and grader path. Scoring is implemented in the MATH-500 scorer.