Local-Axiom-AI commited on
Commit
49e63ec
·
verified ·
1 Parent(s): afad6be

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -3
README.md CHANGED
@@ -14,7 +14,7 @@ tags:
14
 
15
  # Sabaki Preview
16
 
17
- Sabaki is a small, experimental language model from **Local-Axiom-AI**. It was trained from scratch to explore a simple question: how much can a compact model do when recurrent memory, occasional global attention, and sparse experts work together?
18
 
19
  This upload is a research preview, not a polished assistant. It is here for people who want to inspect the design, run local experiments, or continue post-training. Expect rough edges, including uneven instruction following, repetition, factual mistakes, and sensitivity to prompts.
20
 
@@ -50,8 +50,6 @@ Most layers use Kimi Delta Attention (KDA), which carries a fixed-size recurrent
50
 
51
  The feed-forward side is sparse in all but the first block. For each token, the router selects two of eight routed experts and combines them with two shared experts. Persistent routing biases help keep expert use balanced without adding a separate balancing loss.
52
 
53
- ![Diagram of Sabaki's decoder blocks and layer schedule](./sabaki-architecture.png)
54
-
55
  The complete layer schedule is:
56
 
57
  | Blocks | Sequence mixer | Feed-forward path |
 
14
 
15
  # Sabaki Preview
16
 
17
+ Sabaki our first attempt at a small, experimental language model. It was trained from scratch with a custom architecture.
18
 
19
  This upload is a research preview, not a polished assistant. It is here for people who want to inspect the design, run local experiments, or continue post-training. Expect rough edges, including uneven instruction following, repetition, factual mistakes, and sensitivity to prompts.
20
 
 
50
 
51
  The feed-forward side is sparse in all but the first block. For each token, the router selects two of eight routed experts and combines them with two shared experts. Persistent routing biases help keep expert use balanced without adding a separate balancing loss.
52
 
 
 
53
  The complete layer schedule is:
54
 
55
  | Blocks | Sequence mixer | Feed-forward path |