GSM1K
GSM1K
Description
GSM1K is an environment for evaluating mathematical reasoning on grade-school level word problems, based on the GSM1K benchmark by Scale AI. It contains 1,205 multi-step arithmetic word problems designed to mirror the difficulty and style of GSM8K while being entirely new problems to detect data contamination. Problems require multi-step reasoning with basic arithmetic operations.
Capabilities
- Grade-school level mathematical reasoning
- Multi-step arithmetic word problem solving
- Numerical computation and answer extraction
Compute Requirements
Agents are given a standard environment with no sandbox or file system access.
License
MIT.
Tasks
There is one split in this environment:
- test: 1,205 tasks
Each task presents a grade-school math word problem requiring multi-step reasoning to solve.
Reward Structure
This is a single-turn environment. The agent submits a numerical answer via the submit_answer tool. Validation uses numerical comparison with floating-point tolerance (handles integer/decimal equivalence like "133" == "133.0"). Reward is binary: 1.0 if correct, 0.0 if incorrect.
Data
Data consists of a Parquet file (gsm1k_data.parquet) sourced from HuggingFace ScaleAI/gsm1k. Each row contains a word problem and numerical answer. Data is stored on the OpenReward platform.
Tools
| Tool | Description |
|---|---|
submit_answer | Submit your numerical answer to the math problem. Ends the episode. |
Time Horizon
Single-turn. The agent reads the word problem and submits one numerical answer.
Environment Difficulty
GSM1K evaluates grade-school mathematical reasoning while testing for data contamination. Selected model accuracies from the original paper:
| Model | GSM8K | GSM1K |
|---|---|---|
| GPT-4 | 91.0% | 91.8% |
| Gemini-1.5 | 89.7% | 87.9% |
| Claude-3-Opus | 80.2% | 82.5% |
| Mistral-Large | 85.3% | 85.3% |
| Mixtral-8x22B-Instruct | 85.9% | 76.0% |
| Phi-3-mini | 78.2% | 68.4% |
Other Environment Requirements
There are no further environment requirements; GSM1K works out of the box with the OpenReward endpoint without any external API keys.
Safety
Agents in GSM1K solve grade-school math problems in a standard environment. The environment does not present direct safety risks.
Citation
@inproceedings{zhang2024careful,
title={A Careful Examination of Large Language Model Performance on Grade School Arithmetic},
author={Zhang, Hugh and Da, Jeff and Lee, Dean and Robinson, Vaughn and Wu, Catherine and Song, Will and Zhao, Tiffany and Raja, Pranav and Zhuang, Charlotte and Slack, Dylan and Lyu, Qin and Hendryx, Sean and Kaplan, Russell and Lunati, Michele and Yue, Summer},
booktitle={Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS)},
year={2024}
}