Pipette by Liquid AI
About

MATH-500

MATH-500 measures multi-step competition mathematics across algebra, geometry, number theory, precalculus, and other subjects. Pipette evaluates each published quantization rather than reporting only a full-precision reference.

Configuration

Field Value
Benchmark ID eval_math_500_2026.10.1_ctx32768
Dataset scope All 500 problems in the named subset
Attempts 5 per problem; 2,500 completions per model configuration
Sampling Temperature 0.6; no fixed seed
Published metric pass@1 across all attempt rows
Benchmark ID
eval_math_500_2026.10.1_ctx32768
Dataset scope
All 500 problems in the named subset
Attempts
5 per problem; 2,500 completions per model configuration
Sampling
Temperature 0.6; no fixed seed
Published metric
pass@1 across all attempt rows

Dataset and generation

The source is HuggingFaceH4/MATH-500, a 500-problem subset of the Hendrycks MATH benchmark from OpenAI's PRM800K work. All 500 problems are used from an exact upstream revision.

An approved evaluation runner generates the completions. Each problem is sampled five times without a fixed seed. See Evaluation Methodology for the shared generation and quality-result selection policy.

Scoring

MATH-500 is open-ended rather than multiple choice. Pipette removes <think> and [THINK] reasoning blocks before grading. Dataset version 2026.10.1 uses the answer parser and verifier from liquid_evals, the evaluation harness behind the MATH-500 results on Liquid AI's model cards:

  1. The parser takes the last \boxed{} answer. If there is none, it looks for an answer line such as ###, ANSWER:, or "the answer is". If none of these match, it takes the last number.
  2. The verifier accepts an exact match. Otherwise it checks the extracted answer for mathematical equivalence, so forms such as 1/2 and 0.5 can match.
  3. If that fails, the verifier checks the whole response, and then each of the last 150 lines of the response.

The scoring guide describes the grader and its limits in full.

The leaderboard publishes correct / total across all 2,500 attempt rows, which is pass@1 across five attempts per problem.

The version-specific MATH-500 builder documents the Hugging Face revision, prompt template, content hash, repeats, and grader path. Scoring is implemented in the MATH-500 scorer.