OPENGCM/DeepArithmark-1

1,000 arithmetic word problems that need 2–5 dependent operations. Same format as ArithMark 3.0 — finish the sentence, pick one of four continuations, chance 25 % — but the wrong answers are no longer ±1 of the right one, so a magnitude guess does not work.

Unofficial. Deep Arithmark is an independent companion tier. It is not affiliated with, endorsed by, or published by AxiomicLabs, the authors of ArithMark 3.0, and it is not an official ArithMark release. It follows their task contract and credits them throughout; any errors are the author's. If you use this benchmark, cite ArithMark 3.0 as well.

Examples	1,000
Choices per example	4
Chance	25 %
Depth tiers	2, 3, 4, 5 — 250 each
Templates / topic labels	15 / 14
Mean prompt length	29.2 words (max 43)
Language, source	English, synthetic (deterministic seed)

Quick start

from datasets import load_dataset

ds = load_dataset("OPENGCM/DeepArithmark-1", split="test")
item = ds[0]
# score the four continuations of item["ctx"] by length-normalised
# log-likelihood, take the argmax, compare with int(item["label"])

Score models directly, including an ArithMark 3.0 comparison on the same weights:

python bench_deeparithmark.py --compare --model HuggingFaceTB/SmolLM2-135M

ArithMark 3.0's own scorer runs on this file unchanged (verified against that script's loader):

python bencharithmark-3.py --data-path data/deeparithmark-1.jsonl --model gpt2

Why it is harder

ArithMark 3.0's four continuations are adjacent integers (43 / 44 / 45 / 46). Rejecting a wrong option only needs "the answer is about 44", so a magnitude prior is sufficient and the arithmetic is optional. Deep Arithmark keeps the contract and changes three things:

1. Depth. Every item needs 2–5 dependent operations; later steps consume earlier results. depth equals the length of metadata.solution_steps, and the build fails otherwise.
2. Decoys with provenance. No decoy is "answer ± 1". Each one is the exact value of a specific plausible mistake — the state after step 2 instead of step 5, the percentage on the wrong base, the divisor n instead of n + 1, the dropped unit conversion. metadata.option_sources names the mistake per ending, so error analysis can ask which step a model botches.
3. Balanced on three axes. Correct slot, the answer's numeric rank, and the roundness class of all four options are each balanced. "Pick the largest" and "pick the roundest number" are both worth 25 %.

One item

{
  "ind": 3,
  "activity_label": "deeparithmark::deeparithmark-1::multi_step_inventory::depth4::hard",
  "ctx": "A warehouse held 420 units. A delivery added 260, then 210 units were shipped out, and 40 were found damaged. The rest were packed into crates of 16. The number of full crates packed was",
  "endings": [" 26 crates.", " 29 crates.", " 27 crates.", " 31 crates."],
  "label": "0",
  "source_id": "deeparithmark_v1_multi_step_inventory_depth4_0003",
  "split": "test",
  "split_type": "synthetic",
  "metadata": {
    "answer": "26",
    "unit": "crates",
    "topic": "multi_step_inventory",
    "depth": 4,
    "option_values": [26, 29, 27, 31],
    "option_sources": ["correct", "forgot the damaged units", "counted the loose units as a crate", "added the damaged units"],
    "answer_rank": 0,
    "solution_steps": [
      "after delivery: 420 + 260 = 680",
      "after shipping: 680 - 210 = 470",
      "good units: 470 - 40 = 430",
      "full crates: 430 / 16 = 26 (remainder 14)"
    ]
  }
}

The answer is 26 crates. — 420 + 260 − 210 − 40 = 430, and 430 // 16 = 26 full crates. 29 is what you get by forgetting the damaged units, 27 by counting the loose units as a crate, 31 by adding them.

Scores

Zero-shot, acc_norm, bf16, batch 32, --max-context 1024. ArithMark 3.0 numbers were produced on the same weights in the same run, not copied from a table. soft is the mean probability of the correct option after a softmax over the four scores; d2–d5 are acc_norm per depth tier (250 items each, chance 25 %). Δ is Deep Arithmark minus ArithMark 3.0. Every row is in one file, benchmarks/deeparithmark_17models.json.

LiquidAI/LFM3-1.7B-Base ships remote modelling code written against transformers 4.x — it decorates its convolution with a hub-kernel helper that 5.x removed — so the scorer stubs two deleted helpers and runs the checkpoint's own pure-torch path. Re-scoring it at batch 8 instead of 32 moved two items (31.90 % against 31.70 %), which is bf16 kernel noise, not a padding leak.

Model	Params	Deep	soft	d2	d3	d4	d5	ArithMark-3	Δ
SmolLM3-3B-Base (2025-06)	3.08 B	36.50 %	27.09 %	57.6	35.6	26.0	26.8	75.70 %	−39.20
Qwen3.5-2B-Base (2026-02)	1.88 B	36.00 %	27.01 %	59.2	26.4	28.8	29.6	86.70 %	−50.70
LFM3-1.7B-Base (2026-10)	1.73 B	31.70 %	26.42 %	41.2	38.4	23.6	23.6	93.10 %	−61.40
granite-4.0-1b-base (2025-10)	1.63 B	29.90 %	26.13 %	39.2	29.2	28.4	22.8	84.20 %	−54.30
Qwen2.5-1.5B-Instruct	1.54 B	38.00 %	27.23 %	60.4	40.8	28.4	22.4	—	—
Qwen3.5-0.8B-Base	752 M	26.70 %	25.48 %	28.8	24.4	28.8	24.8	75.40 %	−48.70
Tev1-0.8B-experimental	752 M	23.10 %	25.30 %	20.8	24.4	23.6	23.6	71.40 %	−48.30
Qwen3-0.6B-Base	596 M	29.20 %	25.73 %	37.6	32.0	28.8	18.4	68.40 %	−39.20
Qwen2.5-0.5B	494 M	24.60 %	25.50 %	24.8	26.4	23.6	23.6	60.90 %	−36.30
SmolLM2-360M-Instruct	362 M	23.80 %	25.19 %	24.8	26.8	26.4	17.2	—	—
LFM2.5-350M-Base	354 M	20.00 %	24.01 %	17.2	23.6	21.2	18.0	36.00 %	−16.00
granite-4.0-350m-base	352 M	24.10 %	25.34 %	24.4	22.0	23.6	26.4	75.70 %	−51.60
GPT-X3-150M	149 M	25.20 %	25.07 %	23.6	24.0	25.6	27.6	49.60 %	−24.40
SmolLM2-135M	135 M	24.80 %	25.21 %	18.0	29.2	26.4	25.6	38.60 %	−13.80
gpt2	124 M	24.00 %	24.98 %	22.0	23.2	25.2	25.6	34.70 %	−10.70
nanowhale-100m-base	110 M	21.30 %	24.46 %	15.6	22.8	23.6	23.2	26.50 %	−5.20
EleutherAI/pythia-70m	70 M	22.70 %	24.34 %	21.6	26.0	21.6	21.6	28.80 %	−6.10

Tev1-0.8B-experimental is a post-trained derivative of Qwen/Qwen3.5-0.8B (identical parameter count), so that pair is base vs. post-trained on one architecture rather than two models. gemma-3-270m and MobileLLM-R1-950M are gated and have not been scored.

Four bands. Below ~350 M nothing clears chance, and two models sit significantly below it — nanowhale-100M (21.3 %, p = 0.007) and LFM2.5-350M (20.0 %, p = 0.0002) — meaning a stable losing strategy rather than absent arithmetic encoding. Qwen3-0.6B-Base is the first sub-1B model to clear chance significantly: 29.20 %, p = 0.0027. The 1.6–3.1 B band clears it comfortably and then stalls at 29.9–36.5 % (p = 4.5e−4 to 8e−16). Only the post-trained Qwen2.5-1.5B-Instruct declines smoothly all the way down (60.4 → 40.8 → 28.4 → 22.4); every base model from 1.6 B to 3.1 B is back at chance by depth 4.

The one to watch is Qwen3-0.6B's depth curve: 37.6 → 32.0 → 28.8 → 18.4 %, significant at both ends (depth 2 p = 1e-5, depth 5 p = 0.016). Being below chance at depth 5 means it commits to a stable wrong path — it starts computing and drops the last step — rather than guessing. Option rank is balanced uniformly across depth, so no rank preference can produce that shape.

Hardness scales with how well a model exploits ±1 options. granite-4.0-350M Base scores 75.70 % on ArithMark 3.0, the highest sub-1B result here, and 24.10 % on this dataset — a 51.6-point drop, against GPT-X3-150M's 24.4.

The extreme case is now LFM3-1.7B-Base. It scores 93.10 % on ArithMark 3.0 — the highest figure in this table — and 31.70 % here: a 61.4-point drop, larger than granite-4.0-350M's 51.6. It is also the only model whose depth-3 bucket stays within three points of its depth-2 bucket with both above chance (41.2 → 38.4, p = 3e−6) before falling off a cliff at depth 4 (23.6). That skill is concentrated in two templates: share_then_split 93.2 % and time_clock_end 77.3 %, with money_discount_fee at 3.3 %. A 1.7 B model that nearly aces another benchmark is still a single-template specialist here.

Scaling 0.6 B → 3.1 B buys about seven points. Qwen3-0.6B 29.2, granite-1B 29.9, LFM3-1.7B 31.7, Qwen3.5-2B 36.0, SmolLM3-3B 36.5: a 5× parameter increase for +7.3 points, while ArithMark 3.0 on the same runs spans 68.4 → 93.1. The aggregate also hides opposite failure shapes at the same level. Qwen3.5-2B has the best depth-2 bucket in the table (59.2 %, p = 2e−30) and collapses at depth 3 (26.4 %); SmolLM3-3B is weaker at depth 2 (57.6) yet still holds depth 3 (35.6 %, p = 2e−4). Same ~36 %, different structure — which is why the curve is the reportable unit, not the single number.

A chance score is not always guessing. GPT-X3-150M scores 49.60 % on ArithMark 3.0 and 25.20 % here, but its per-template accuracy spans 3.0 % to 54.5 % (χ² = 63.2, df = 14, p < 1e-4): a small-number prior (picks the smallest option 34.8 % of the time, the largest 15.8 %) plus one stable wrong path per family. These cancel to chance in aggregate, so treat per-template rows as preference, not partial skill. Worked through in the design notes, §6.

For a sub-1B leaderboard, report the per-depth curve rather than only the single number. The two solvable templates are stable across families: at 1.5B-instruct share_then_split (86.4 %) and time_clock_end (85.9 %), at 1.7 B share_then_split 93.2 %, and at 2 B time_clock_end 90.6 %.

Composition

Depth	Examples	Typical item
2	250	percent off a price, then a fee
3	250	unit conversion, add, subtract
4	250	receive / ship / damage / pack
5	250	two percentage days, then palletise

Template	Depth	Examples	Template	Depth	Examples
money_discount_fee	2	122	schedule_sessions	3	47
time_clock_end	2	128	inventory_boxes	4	62
measure_cm_cut	3	33	mix_split	4	63
share_then_split	3	44	tank_fill	4	62
two_rate_pay	3	45	discount_tax_coupon_split	4	63
stock_percent_pack	3	48	salary_overtime_tax	5	84
trip_minutes	3	33	inventory_percent_chain	5	83
			prize_ratio_charity	5	83

Correct slot is 252 / 252 / 248 / 248, answer numeric rank (smallest → largest) is 248 / 252 / 248 / 252, and the slot × rank × depth grid holds 15–16 items per cell.

Shortcut audit

No strategy that skips the arithmetic beats chance (python deeparithmark.py audit, chance 25.00 %):

Strategy	Score
pick largest / smallest value	25.20 % / 24.80 %
pick the roundest number	25.20 %
fixed slot 0 / 1 / 2 / 3	25.20 / 25.20 / 24.80 / 24.80 %
longest / shortest text	25.50 % / 24.00 %
char-3-gram classifier over option text	25.47 %

This is a build constraint, not a happy accident. An early build scored gpt2 below chance (21.1 %) because several decoys were structurally rounder than the answer, making a roundness prior anti-correlated with correctness. All four options are now forced into the same roundness class.

Limitations

• 1,000 items → roughly ±2.7 points (95 % CI) at chance. Differences under ~4 points are noise.
• Hard 4-way accuracy gives no partial credit by design; use the soft score for a gradient.
• Single test split, no train split (evaluation set).
• 15 templates only: fine-tuning on them would overfit.
• Rank balance is global, not per-template, so per-template rows read as preference rather than skill.
• Clock items keep all four options inside one half of the day, so no noon rollover is exercised.

How it was generated

deeparithmark.py build renders the file deterministically from seed 20261027: 15 parameterised families, exact integer arithmetic, one decoy pool per family. python deeparithmark.py selftest re-checks every claim on this page — schema, unit consistency, option distinctness, step count vs depth, all three balance axes, and the audit above.

The seed makes the instances reproducible and unseen; it does not make the template shapes novel. Do not train on this file.

Licence and attribution

Released under the Apache License 2.0. Dataset SHA-256: 8a54eb90b66b142b7fea422b34d851746d1cabc476dcba906c7f3ad5ddc66ee0.

Upstream work. Deep Arithmark follows ArithMark 3.0 by AxiomicLabs, also Apache-2.0. Reused: the task contract — one unfinished sentence, four continuations with a leading space, a single correct index, and scoring by length-normalised continuation log-likelihood — plus the JSONL schema, so their scorer runs on this file unchanged.

What is not reused. No ArithMark 3.0 item appears here. Every prompt comes from the parameterised families in deeparithmark.py, and the released file shares 0 identical contexts and 0 eight-word n-grams with the 1,000 upstream items. The scorer is an independent implementation of the same protocol, not copied code.

Trademarks. "ArithMark" and "AxiomicLabs" remain the property of their owner. Their use here is descriptive, to state which benchmark this one extends. It does not imply affiliation or endorsement — see the notice at the top.

Files

File	What it is
data/deeparithmark-1.jsonl	The dataset — 1,000 items, one JSON object per line
deeparithmark.py	Deterministic generator, plus selftest and the shortcut audit
bench_deeparithmark.py	Scorer with per-depth and per-template breakdowns and an ArithMark 3.0 comparison
DEEPARITHMARK_DESIGN.md	Design rationale, fairness audit, measurement discussion
benchmarks/deeparithmark_6models.json	Raw results for the 2026 sub-1B sweep