MATH-500 measures multi-step competition mathematics across algebra, geometry, number theory, precalculus, and other subjects. Pipette evaluates each published quantization rather than reporting only a full-precision reference.
Configuration
| Field | Value |
|---|---|
| Benchmark ID | eval_math_500_2026.06.1 |
| Dataset scope | All 500 problems in the named subset |
| Attempts | 5 per problem; 2,500 completions per model configuration |
| Sampling | Temperature 0.6; no fixed seed |
| Published metric | pass@1 across all attempt rows |
eval_math_500_2026.06.10.6; no fixed seedDataset and generation
The source is HuggingFaceH4/MATH-500, a 500-problem subset of the Hendrycks MATH benchmark from OpenAI's PRM800K work. All 500 problems are used from an exact upstream revision.
An approved evaluation runner generates the completions. Each problem is sampled five times without a fixed seed. See Evaluation Methodology for the shared generation and quality-result selection policy.
Scoring
MATH-500 is open-ended rather than multiple choice. The scorer extracts the
final answer from boxed or unboxed numeric output and checks it for mathematical
equivalence. Equivalent forms such as 1/2 and 0.5 can match. Pipette removes
<think> and [THINK] reasoning blocks before grading.
The leaderboard publishes correct / total across all 2,500 attempt rows, which
is pass@1 across five attempts per problem.
The version-specific MATH-500 builder documents the Hugging Face revision, prompt template, content hash, repeats, and grader path. Scoring is implemented in the MATH-500 scorer.