ARTN β Adaptive Recursive Ternary Networks
A working language model trained from scratch on a 2-core CPU with 4 GB RAM β no GPU, no cluster, no excuses.
Model details
| Architecture | BitLinear 1.58-bit ternary, one shared block run 6Γ with a learned halting gate |
| Parameters | 2,904,065 β 50.2% fewer than the FP32 baseline |
| Training | 10,000 steps in 14.35 hours, CPU-only |
| Validation loss | 2.0346 (perplexity 7.65) |
| Fair non-padding NLL | 2.904 (FP32 2.423 Β· BitLinear 2.743) |
| Stability | 0 NaN/Inf losses; halting converged to 4.00 expected passes |
| Hardware | Dual-core AMD CPU, 4 GB RAM, no GPU |
| Dataset | TinyStories (synthetic children's-story corpus) |
| Checkpoint | artn_step_10000.pt (34 MB) |
The journey
ARTN did not work on the first try. It did not work on the fifth.
8 training attempts Β· 5 failed Β· 211,000 total steps Β· 390+ hours Β· 3.5 months 62 investigation phases Β· 11,000+ lines of documentation
| # | Date | Steps | Hours | Result |
|---|---|---|---|---|
| 1 | Aug 1 | 5,000 | 4 | Pipeline verified |
| 2 | Aug 3β6 | 50,000 | 79 | COLLAPSED β "time time time..." |
| 3 | Aug 6 | 2,000 | 1 | Stopped β wrong accumulation |
| 4 | Aug 6β10 | 50,000 | 83 | Completed β healthy but gibberish |
| 5 | Aug 13β16 | 50,000 | 82 | Completed β healthy but gibberish |
| 6 | Aug 19 | 5,000 | β | FP32 + AdamW β first success (val 1.69) |
| 7 | Sep 10β11 | 10,000 | β | BitLinear + AdamW β ternary works (val 1.92) |
| 8 | Oct 25 | 10,000 | 14.35 | ARTN β the model that works (val 2.03) |
244 hours of failed training across three 50K-step runs before the root cause was found: a bug in the optimizer that silently overwrote variance-scaled updates with momentum-only updates β making it behave like SGD with momentum instead of Adafactor. The model trained, the loss dropped, the entropy looked healthy. The output was gibberish.
The fix: switch to AdamW. Then build ARTN on top.
Architecture
ARTN stacks three innovations into a single architecture. Each one attacks a different resource bottleneck.
Input tokens
β
βΌ
βββββββββββββββ
β Embeddings β (token + position, not shared)
ββββββββ¬βββββββ
β
βΌ
ββββββββββββββββββββ pass 1
β BitLinearBlock ββββ shared weights (ONE block, used 6Γ max)
β {-1, 0, 1} β
ββββββββ¬ββββββββββββ
β ββββββββββββββββ
βββββββββββββΊβ Halting Gate β "has this position converged?"
β β sigmoid(WΒ·x) β learned per-token, per-pass
β ββββββββ¬ββββββββ
β β
no (p < ΞΈ) β yes (p β₯ ΞΈ)
β β
βΌ βΌ
ββββββββββββββββββββ stop updating this position
β BitLinearBlock ββββ same weights, pass 2
β (same weights) β
ββββββββ¬ββββββββββββ
β ... repeat up to max_passes = 6
βΌ
βββββββββββββββ
β ln_f + β
β lm_head β
ββββββββ¬βββββββ
β
βΌ
logits β next token
| Constraint | Technique | Measured effect |
|---|---|---|
| Weight memory | BitLinear 1.58-bit {-1, 0, 1} with straight-through estimation |
2,904,065 params vs 5,828,608 FP32 β 50.2% fewer |
| Depth memory | Recursive parameter tying β one shared block, 6 passes | 789,760 params in the shared block (instead of 6 Γ 789,760) |
| Compute cycles | Learned adaptive halting gate | 3.57 average passes at threshold 0.9 (vs 6.0) |
| Training stability | AdamW + cosine schedule + gradient clipping | converged CPU-only, 0 NaN/Inf |
| Hardware envelope | β | full training in 14.35 h on 4 GB RAM |
How the three fit together. Ternary weights cut the model's parameter footprint in half β each weight is one of three values instead of a 32-bit float. Parameter tying cuts it further: instead of six separate transformer blocks, ARTN uses one block and runs it six times. The halting gate then makes those six passes optional β each token position learns when it has "converged" and stops early. At inference, the model averages 3.57 passes instead of 6, cutting compute by 40% for a 0.08% NLL cost. Training runs all six passes for exact gradients; halting applies only at inference.
Generation examples
ARTN checkpoint step_10000 β sign-aware repetition penalty 1.3, greedy decoding, seed 42.
Prompt:
once there was a little boyOutput: once there was a little boy who liked to play with his toys. One day, he saw something shiny in the box. He wanted to open it and see what it was. The little boy looked around and saw that it had many colors. It was so big that it could not touch it. But then the boy heard the noise and came out of the room. His mom said, "Please help me find it." She picked up the toy box and put it on the shelf. The little boy was very happy and thanked the boy for helping him. From that day on,
Prompt:
a little bird flew highOutput: a little bird flew high in the sky. The bird was very happy to fly and it could fly high up high. But then one day, the bird saw something shiny on it. It had a long tail and it made a loud noise. Suddenly, the bird heard a voice. He looked around and saw a big bird. The bird said, "Hello, bird!" The bird smiled at the bird. The bird felt so happy. From that day on, the bird became friends and played together every day.
Prompt:
a dragon lived in a caveOutput: a dragon lived in a cave and he was very happy. One day, he decided to go for a walk with his friends. He walked around the forest and saw a big tree. The bird looked so high that it could not find it. It had many friends like a long tail. They were happy and excited. But then, they heard a loud noise. A man came out of the window and saw the bird flying towards them. The bird said "Hello, bird," said the bird. "Do you want to help me get my friend?" The bird said, "Yes, I can help you." The
The model generates subject-specific vocabulary β the bird sample talks about flying and the sky; the dragon sample mentions a cave, a forest, and friends. Coherence degrades toward the end of each 100-word window, which is expected for a 2.9M-parameter model trained for 10k steps on TinyStories.
Model comparison
| Model | Parameters | Val loss | Perplexity | Fair non-padding NLL |
|---|---|---|---|---|
| FP32 baseline | 5,828,608 | 1.69 | 5.44 | 2.423 |
| BitLinear GPT | 6,852,682 | 1.92 | 6.81 | 2.743 |
| ARTN (1 shared block) | 2,904,065 | 2.03 | 7.65 | 2.904 |
ARTN reaches comparable quality with 50.2% fewer parameters than FP32, using one shared block instead of six unique ones.
CPU inference benchmark
500 tokens, greedy decoding, seed 42, RSS measured via psutil:
| Configuration | Throughput | Avg passes | Peak RSS |
|---|---|---|---|
| FP32 baseline | 22.62 tok/s | 6.00 | 278.4 MB |
| BitLinear GPT | 5.45 tok/s | 6.00 | 354.6 MB |
| ARTN, threshold 1.0 | 5.29 tok/s | 6.00 | 282.3 MB |
| ARTN, threshold 0.9 | 8.42 tok/s | 3.57 | 276.4 MB |
The adaptive halting gate lifts throughput from 5.29 to 8.42 tokens/sec at threshold 0.9 while lowering peak memory β 1.59Γ faster at a cost of +0.0024 NLL.
How to use
import torch
import sys
sys.path.insert(0, '.') # if using the GitHub repo
from experiments.artn_model import ARTNGPT
from src.tokenizer.bpe_tokenizer import BPETokenizer
# Load tokenizer
tokenizer = BPETokenizer.load('tokenizer.json')
# Initialize ARTN architecture
model = ARTNGPT(
vocab_size=4000,
d_model=256,
n_heads=8,
d_ff=1024,
max_seq_len=256,
n_layers=6, # maps to max_passes
)
# Load the checkpoint
checkpoint = torch.load('artn_step_10000.pt', map_location='cpu')
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()
# Generate text
input_ids = torch.tensor([tokenizer.encode("once there was a little boy")])
with torch.no_grad():
for _ in range(100):
logits = model(input_ids)
next_token = logits[0, -1].argmax(dim=-1, keepdim=True)
input_ids = torch.cat([input_ids, next_token.unsqueeze(0)], dim=-1)
print(tokenizer.decode(input_ids[0].tolist()))
Full source code and training scripts: github.com/kishxrx/artn
Evaluation notes
Results are measured on TinyStories β a synthetic children's-story corpus β after 10,000 steps, and characterise that setting rather than general-domain text.
Generation quality is decoding-sensitive. Greedy decoding without a repetition penalty drifts into repetition; the sign-aware penalty of 1.3 used above removes it and produces the prose shown. Reproducing the example therefore means reproducing the decoding configuration, not just the checkpoint.
Prompt adherence for rare subjects remains an open problem, and is the focus of the ongoing work described below.
Full investigation
Every phase is documented. Every measurement is backed by an evidence file. Every failure is recorded β including the dead ends.
| Document | Contents |
|---|---|
| ROOT_CAUSE_INVESTIGATION_SUMMARY.md | Phases 1β60.29 β root-cause analysis, optimizer forensics, architecture development, generation-quality and context-retention studies, diagnostic probes, controlled construction-transfer experiments. 9,600+ lines of measured findings with evidence files for every claim. |
| TRAINING_HISTORY.md | Complete training history β configurations, loss trajectories, checkpoints and results for every run. |
GitHub repository: kishxrx/artn β 44 evaluation scripts, 62 investigation phases, all evidence files included.
License
Code is licensed under GPL-3.0 β you may use, modify, and distribute, but derivatives must also be open-sourced under GPL.
Documentation (investigation reports, training history, and analysis files) is licensed under CC-BY-NC 4.0 β you may read and learn, but may not use commercially without permission, and must provide attribution.
Citation
@article{artn2026,
title = {Adaptive Recursive Ternary Networks: Training Transformer Language Models Under Extreme Hardware Constraints},
author = {ARTN Project},
year = {2026},
note = {Active research project},
url = {https://huggingface.co/kishxrx/ARTN}
}
Ongoing research
ARTN is an active project and the model continues to improve. Current work focuses on generation quality and prompt adherence β reducing repetition, sustaining coherence over longer contexts, and following the prompt more faithfully across a wider range of subjects. Alongside that: further halting-threshold tuning, training-efficiency work, and an expanding evaluation suite.