ARTN β€” Adaptive Recursive Ternary Networks

A working language model trained from scratch on a 2-core CPU with 4 GB RAM β€” no GPU, no cluster, no excuses.

Model details

Architecture BitLinear 1.58-bit ternary, one shared block run 6Γ— with a learned halting gate
Parameters 2,904,065 β€” 50.2% fewer than the FP32 baseline
Training 10,000 steps in 14.35 hours, CPU-only
Validation loss 2.0346 (perplexity 7.65)
Fair non-padding NLL 2.904 (FP32 2.423 Β· BitLinear 2.743)
Stability 0 NaN/Inf losses; halting converged to 4.00 expected passes
Hardware Dual-core AMD CPU, 4 GB RAM, no GPU
Dataset TinyStories (synthetic children's-story corpus)
Checkpoint artn_step_10000.pt (34 MB)

The journey

ARTN did not work on the first try. It did not work on the fifth.

8 training attempts Β· 5 failed Β· 211,000 total steps Β· 390+ hours Β· 3.5 months 62 investigation phases Β· 11,000+ lines of documentation

# Date Steps Hours Result
1 Aug 1 5,000 4 Pipeline verified
2 Aug 3–6 50,000 79 COLLAPSED β€” "time time time..."
3 Aug 6 2,000 1 Stopped β€” wrong accumulation
4 Aug 6–10 50,000 83 Completed β€” healthy but gibberish
5 Aug 13–16 50,000 82 Completed β€” healthy but gibberish
6 Aug 19 5,000 β€” FP32 + AdamW β€” first success (val 1.69)
7 Sep 10–11 10,000 β€” BitLinear + AdamW β€” ternary works (val 1.92)
8 Oct 25 10,000 14.35 ARTN β€” the model that works (val 2.03)

244 hours of failed training across three 50K-step runs before the root cause was found: a bug in the optimizer that silently overwrote variance-scaled updates with momentum-only updates β€” making it behave like SGD with momentum instead of Adafactor. The model trained, the loss dropped, the entropy looked healthy. The output was gibberish.

The fix: switch to AdamW. Then build ARTN on top.

Architecture

ARTN stacks three innovations into a single architecture. Each one attacks a different resource bottleneck.

 Input tokens
     β”‚
     β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  Embeddings  β”‚   (token + position, not shared)
β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜
       β”‚
       β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    pass 1
β”‚  BitLinearBlock   │◄── shared weights (ONE block, used 6Γ— max)
β”‚  {-1, 0, 1}       β”‚
β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
       β”‚            β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
       β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Ίβ”‚ Halting Gate  β”‚  "has this position converged?"
       β”‚            β”‚ sigmoid(WΒ·x)  β”‚  learned per-token, per-pass
       β”‚            β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜
       β”‚                   β”‚
      no (p < ΞΈ)           β”‚ yes (p β‰₯ ΞΈ)
       β”‚                   β”‚
       β–Ό                   β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    stop updating this position
β”‚  BitLinearBlock   │◄── same weights, pass 2
β”‚  (same weights)   β”‚
β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
       β”‚              ... repeat up to max_passes = 6
       β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚   ln_f +     β”‚
β”‚   lm_head    β”‚
β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜
       β”‚
       β–Ό
   logits β†’ next token
Constraint Technique Measured effect
Weight memory BitLinear 1.58-bit {-1, 0, 1} with straight-through estimation 2,904,065 params vs 5,828,608 FP32 β€” 50.2% fewer
Depth memory Recursive parameter tying β€” one shared block, 6 passes 789,760 params in the shared block (instead of 6 Γ— 789,760)
Compute cycles Learned adaptive halting gate 3.57 average passes at threshold 0.9 (vs 6.0)
Training stability AdamW + cosine schedule + gradient clipping converged CPU-only, 0 NaN/Inf
Hardware envelope β€” full training in 14.35 h on 4 GB RAM

How the three fit together. Ternary weights cut the model's parameter footprint in half β€” each weight is one of three values instead of a 32-bit float. Parameter tying cuts it further: instead of six separate transformer blocks, ARTN uses one block and runs it six times. The halting gate then makes those six passes optional β€” each token position learns when it has "converged" and stops early. At inference, the model averages 3.57 passes instead of 6, cutting compute by 40% for a 0.08% NLL cost. Training runs all six passes for exact gradients; halting applies only at inference.

Generation examples

ARTN checkpoint step_10000 β€” sign-aware repetition penalty 1.3, greedy decoding, seed 42.

Prompt: once there was a little boy

Output: once there was a little boy who liked to play with his toys. One day, he saw something shiny in the box. He wanted to open it and see what it was. The little boy looked around and saw that it had many colors. It was so big that it could not touch it. But then the boy heard the noise and came out of the room. His mom said, "Please help me find it." She picked up the toy box and put it on the shelf. The little boy was very happy and thanked the boy for helping him. From that day on,

Prompt: a little bird flew high

Output: a little bird flew high in the sky. The bird was very happy to fly and it could fly high up high. But then one day, the bird saw something shiny on it. It had a long tail and it made a loud noise. Suddenly, the bird heard a voice. He looked around and saw a big bird. The bird said, "Hello, bird!" The bird smiled at the bird. The bird felt so happy. From that day on, the bird became friends and played together every day.

Prompt: a dragon lived in a cave

Output: a dragon lived in a cave and he was very happy. One day, he decided to go for a walk with his friends. He walked around the forest and saw a big tree. The bird looked so high that it could not find it. It had many friends like a long tail. They were happy and excited. But then, they heard a loud noise. A man came out of the window and saw the bird flying towards them. The bird said "Hello, bird," said the bird. "Do you want to help me get my friend?" The bird said, "Yes, I can help you." The

The model generates subject-specific vocabulary β€” the bird sample talks about flying and the sky; the dragon sample mentions a cave, a forest, and friends. Coherence degrades toward the end of each 100-word window, which is expected for a 2.9M-parameter model trained for 10k steps on TinyStories.

Model comparison

Model Parameters Val loss Perplexity Fair non-padding NLL
FP32 baseline 5,828,608 1.69 5.44 2.423
BitLinear GPT 6,852,682 1.92 6.81 2.743
ARTN (1 shared block) 2,904,065 2.03 7.65 2.904

ARTN reaches comparable quality with 50.2% fewer parameters than FP32, using one shared block instead of six unique ones.

CPU inference benchmark

500 tokens, greedy decoding, seed 42, RSS measured via psutil:

Configuration Throughput Avg passes Peak RSS
FP32 baseline 22.62 tok/s 6.00 278.4 MB
BitLinear GPT 5.45 tok/s 6.00 354.6 MB
ARTN, threshold 1.0 5.29 tok/s 6.00 282.3 MB
ARTN, threshold 0.9 8.42 tok/s 3.57 276.4 MB

The adaptive halting gate lifts throughput from 5.29 to 8.42 tokens/sec at threshold 0.9 while lowering peak memory β€” 1.59Γ— faster at a cost of +0.0024 NLL.

How to use

import torch
import sys
sys.path.insert(0, '.')  # if using the GitHub repo

from experiments.artn_model import ARTNGPT
from src.tokenizer.bpe_tokenizer import BPETokenizer

# Load tokenizer
tokenizer = BPETokenizer.load('tokenizer.json')

# Initialize ARTN architecture
model = ARTNGPT(
    vocab_size=4000,
    d_model=256,
    n_heads=8,
    d_ff=1024,
    max_seq_len=256,
    n_layers=6,        # maps to max_passes
)

# Load the checkpoint
checkpoint = torch.load('artn_step_10000.pt', map_location='cpu')
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()

# Generate text
input_ids = torch.tensor([tokenizer.encode("once there was a little boy")])

with torch.no_grad():
    for _ in range(100):
        logits = model(input_ids)
        next_token = logits[0, -1].argmax(dim=-1, keepdim=True)
        input_ids = torch.cat([input_ids, next_token.unsqueeze(0)], dim=-1)

print(tokenizer.decode(input_ids[0].tolist()))

Full source code and training scripts: github.com/kishxrx/artn

Evaluation notes

Results are measured on TinyStories β€” a synthetic children's-story corpus β€” after 10,000 steps, and characterise that setting rather than general-domain text.

Generation quality is decoding-sensitive. Greedy decoding without a repetition penalty drifts into repetition; the sign-aware penalty of 1.3 used above removes it and produces the prose shown. Reproducing the example therefore means reproducing the decoding configuration, not just the checkpoint.

Prompt adherence for rare subjects remains an open problem, and is the focus of the ongoing work described below.

Full investigation

Every phase is documented. Every measurement is backed by an evidence file. Every failure is recorded β€” including the dead ends.

Document Contents
ROOT_CAUSE_INVESTIGATION_SUMMARY.md Phases 1–60.29 β€” root-cause analysis, optimizer forensics, architecture development, generation-quality and context-retention studies, diagnostic probes, controlled construction-transfer experiments. 9,600+ lines of measured findings with evidence files for every claim.
TRAINING_HISTORY.md Complete training history β€” configurations, loss trajectories, checkpoints and results for every run.

GitHub repository: kishxrx/artn β€” 44 evaluation scripts, 62 investigation phases, all evidence files included.

License

Code is licensed under GPL-3.0 β€” you may use, modify, and distribute, but derivatives must also be open-sourced under GPL.

Documentation (investigation reports, training history, and analysis files) is licensed under CC-BY-NC 4.0 β€” you may read and learn, but may not use commercially without permission, and must provide attribution.

Citation

@article{artn2026,
  title  = {Adaptive Recursive Ternary Networks: Training Transformer Language Models Under Extreme Hardware Constraints},
  author = {ARTN Project},
  year   = {2026},
  note   = {Active research project},
  url    = {https://huggingface.co/kishxrx/ARTN}
}

Ongoing research

ARTN is an active project and the model continues to improve. Current work focuses on generation quality and prompt adherence β€” reducing repetition, sustaining coherence over longer contexts, and following the prompt more faithfully across a wider range of subjects. Alongside that: further halting-threshold tuning, training-efficiency work, and an expanding evaluation suite.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support