RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding
Paper โข 2610.29434 โข Published โข 14
AI & ML interests
None defined yet.
Recent Activity
View all activity
Aratakoย
authored a
paper 3 days ago
AbstractPhilย
posted an update 3 days ago
Post
65
I approach the next stage with a potential fix for shared global context with local dynamic positional capacity. This involves a fixed position global rotary to represent the local dynamic offset rotary merged from independent components. Experimentally and empirically, the framing system now supports differential framing. This process means the core primary engineering limiter - merging, can properly stabilize, and guarantee a merged functional system does not lose that very learned capacity.
Think of this like, a lora that can be merged, providing that optimized learned structure, will preserve an effective structure post-merge and can continue to learn along a full finetune.
The earlier experiments needed something called a quiet loss, which used KL and micro batching to sample each system meant to be quiet while another learned. This has a large overhead cost and I've been planning a fix for this for some time.
The arms showed this has effective preservation, at a compounding overhead.
The solution here, is allocated aleph addressing space on the rotary planar. This experiment spans further back to a lensing experiment I ran some time ago, which enabled models to rotate away from latent space, modify the latent information in that space, then rotate back to the original space. The concept ran like a slug on a saltbed, so it was mothballed, but it did work.
This new concept is similar to that, except we do not rotate back. The rotary behavior exists as a direct positional offset, and with those offsets we anchor to the attention present within each component's offset.
With this we form a multi-spectrum fixed position offset using the dynamic positioning from the model itself. This will enable deep thought integration of each component without overhead cost.
This system here, this solution, will provide full MOE dynamic extension rather than partial, without the AMOE arm overhead.
Meaning, this engineering solution will solve current limits with beatrix.
Think of this like, a lora that can be merged, providing that optimized learned structure, will preserve an effective structure post-merge and can continue to learn along a full finetune.
The earlier experiments needed something called a quiet loss, which used KL and micro batching to sample each system meant to be quiet while another learned. This has a large overhead cost and I've been planning a fix for this for some time.
The arms showed this has effective preservation, at a compounding overhead.
The solution here, is allocated aleph addressing space on the rotary planar. This experiment spans further back to a lensing experiment I ran some time ago, which enabled models to rotate away from latent space, modify the latent information in that space, then rotate back to the original space. The concept ran like a slug on a saltbed, so it was mothballed, but it did work.
This new concept is similar to that, except we do not rotate back. The rotary behavior exists as a direct positional offset, and with those offsets we anchor to the attention present within each component's offset.
With this we form a multi-spectrum fixed position offset using the dynamic positioning from the model itself. This will enable deep thought integration of each component without overhead cost.
This system here, this solution, will provide full MOE dynamic extension rather than partial, without the AMOE arm overhead.
Meaning, this engineering solution will solve current limits with beatrix.
- 4 replies
Aratakoย
submitted a
paper to Daily Papers 3 days ago
AbstractPhilย
posted an update 6 days ago
Post
96
nano-castor and nano-pollux - (4 variants total) are training complete for the story versions. AbstractPhil/alephllm-nano-castor-pollux
Trained with an identical dataset until framing as the mini-beatrix-2s variants, and finetuned for context and framing as an expected behavior.
The context windows and framing show the models can capture meaningful storage, read, copy, and recall capacity at 4096 byte depth. They aren't particularly smart, but they are functional, and even capable at driving Anima sliders in a meaningfully similar way as the mini-beatrix-3 variant can.
AbstractPhil/alephllm-nano-castor-pollux
Anima sliders often do not have a bi-directional capacity, so the multi-slider arrays were created. Thus allowing the model to rebound naturally as expected when necessary, but this still isn't strong enough.
Up next is the full frame trained variant of the champion; nano-pollux-dual-hub and nano-castor. Both of them are the remaining champions. Neither canon models showed similar capacity as the dual-hub's capacity yielded, so we have our champions.
Before continuing to the micro variant, these two will be retrained and issued a full education in art. Everything from color theory, to structural awareness, to semantic relations, techniques, and everything between.
Our little nano twins here, will be a cooperative pair meant for the first diffusion edit experiments.
There were quite a few experiments, many yielded interesting potentials and information. The most powerful variant being the beatrix 3 as a conditioner, yet the nano variants showed a similarly powerful control.
AbstractPhil/geolip-beatrix-anima
Even just untrained regularization training on Anima shows yield. The model is pretty underfitted, which is both helpful and not helpful for this train.
Trained with an identical dataset until framing as the mini-beatrix-2s variants, and finetuned for context and framing as an expected behavior.
The context windows and framing show the models can capture meaningful storage, read, copy, and recall capacity at 4096 byte depth. They aren't particularly smart, but they are functional, and even capable at driving Anima sliders in a meaningfully similar way as the mini-beatrix-3 variant can.
AbstractPhil/alephllm-nano-castor-pollux
Anima sliders often do not have a bi-directional capacity, so the multi-slider arrays were created. Thus allowing the model to rebound naturally as expected when necessary, but this still isn't strong enough.
Up next is the full frame trained variant of the champion; nano-pollux-dual-hub and nano-castor. Both of them are the remaining champions. Neither canon models showed similar capacity as the dual-hub's capacity yielded, so we have our champions.
Before continuing to the micro variant, these two will be retrained and issued a full education in art. Everything from color theory, to structural awareness, to semantic relations, techniques, and everything between.
Our little nano twins here, will be a cooperative pair meant for the first diffusion edit experiments.
There were quite a few experiments, many yielded interesting potentials and information. The most powerful variant being the beatrix 3 as a conditioner, yet the nano variants showed a similarly powerful control.
AbstractPhil/geolip-beatrix-anima
Even just untrained regularization training on Anima shows yield. The model is pretty underfitted, which is both helpful and not helpful for this train.
- 7 replies
AbstractPhilย
posted an update 10 days ago
Post
117
Been a hectic week since the completion of the mini-beatrix-3 as I tested new formats for the model's v4 state. Two new model formats now exist; Castor and Pollux. They will be trained with the same data as the beatrix-2s model was given, around 30b tokens.
The V4 variant of the AlephLLM dubbed Beatrix is a pair of twins particularly designed with a series of context window curated frame tokens.
Castor, and Pollux. Each housing a curated frame-spaced context window with curated depth embedded frame curations. For example; a divided 4096 context window can be 1024, 512, 256, 256, 64, 64, and the rest padding.
Nano-Castor-V1, the rotary softmax attention variant with around 8m params. Structurally capable of better curated "needle in a haystack" behavior. Stronger at depth and houses a cheaper format of attention than Pollux. Faster, stable by design, and only 6 blocks. Trains at about 900k tok/s.
Pollux, the rotary splat/softmax hybrid trained with a format of Canon LLM inspired by music theory lookup and multiple attention hubs at around 9m params. This model houses a deeper understanding and a more complex relational behavior than simple frame lookup, but the formula is a little more complex so it takes a little longer per step. Trains at around 450k tok/s.
All attributions will be given to the teams for the released papers utilized to create the components.
The experiment is based on training the four models to be capable of deciding and deducting high complexity predictions of frames and the context of those frames within their own frame deduction, creating a pseudo-language. The ultimate goal to make this context window look like gibberish with a full lookup table for debugging, and out comes the correct next unicode byte.
Will it work? Maybe not at this size. The tests show the frames work, the structure lines up, and the system does conform. Even if it doesn't work, the four prototypes will be ready by tomorrow evening and with those results new data
The V4 variant of the AlephLLM dubbed Beatrix is a pair of twins particularly designed with a series of context window curated frame tokens.
Castor, and Pollux. Each housing a curated frame-spaced context window with curated depth embedded frame curations. For example; a divided 4096 context window can be 1024, 512, 256, 256, 64, 64, and the rest padding.
Nano-Castor-V1, the rotary softmax attention variant with around 8m params. Structurally capable of better curated "needle in a haystack" behavior. Stronger at depth and houses a cheaper format of attention than Pollux. Faster, stable by design, and only 6 blocks. Trains at about 900k tok/s.
Pollux, the rotary splat/softmax hybrid trained with a format of Canon LLM inspired by music theory lookup and multiple attention hubs at around 9m params. This model houses a deeper understanding and a more complex relational behavior than simple frame lookup, but the formula is a little more complex so it takes a little longer per step. Trains at around 450k tok/s.
All attributions will be given to the teams for the released papers utilized to create the components.
The experiment is based on training the four models to be capable of deciding and deducting high complexity predictions of frames and the context of those frames within their own frame deduction, creating a pseudo-language. The ultimate goal to make this context window look like gibberish with a full lookup table for debugging, and out comes the correct next unicode byte.
Will it work? Maybe not at this size. The tests show the frames work, the structure lines up, and the system does conform. Even if it doesn't work, the four prototypes will be ready by tomorrow evening and with those results new data
- 7 replies
AbstractPhilย
posted an update 12 days ago
Post
181
Mini-Beatrix-V2-control-softmax is complete. The model is substantially more intelligent than the counterparts. Okay well, maybe not smarter than v3, but that model cooked for weeks. Considerably more stable at >3700 context window, while 3s fails over 1k.
mini-beatrix-2s-control-attn-fp16-r10k
AbstractPhil/alephllm-mini-beatrix-training
---
With that information I have built the tests required to create a smaller and substantially more powerful variant.
Nano-Beatrix-V4 will have hundreds of trains as intelligent as the V3 version within a few days. I will be slowly pushing up the capacity as each will train a hundred thousand steps within a couple hours on a colab card.
Roughly 400k byte/s or so, these shallow models have nearly perfect recon at 4096 context length. HOWEVER, a full matching corpus train that fed Beatrix V3 60 billion bytes - is only about 2 days or less on a colab card vs the near 20 for the v3.
RECON won't guarantee intelligence, however these tiny models here will train so quickly that I can directly test which is eating what space and why, then increase that with minimal cost through direct minimal expansion and then genetic interpolation, until the model hits around 200k token/s or so.
Essentially these lightning quick little models will be capable of attaching to other models, training miniature modular routing systems with the quiet loss structure attached to other models.
They won't control a big model, but that's what the mini version is for.
mini-beatrix-2s-control-attn-fp16-r10k
AbstractPhil/alephllm-mini-beatrix-training
---
With that information I have built the tests required to create a smaller and substantially more powerful variant.
Nano-Beatrix-V4 will have hundreds of trains as intelligent as the V3 version within a few days. I will be slowly pushing up the capacity as each will train a hundred thousand steps within a couple hours on a colab card.
Roughly 400k byte/s or so, these shallow models have nearly perfect recon at 4096 context length. HOWEVER, a full matching corpus train that fed Beatrix V3 60 billion bytes - is only about 2 days or less on a colab card vs the near 20 for the v3.
RECON won't guarantee intelligence, however these tiny models here will train so quickly that I can directly test which is eating what space and why, then increase that with minimal cost through direct minimal expansion and then genetic interpolation, until the model hits around 200k token/s or so.
Essentially these lightning quick little models will be capable of attaching to other models, training miniature modular routing systems with the quiet loss structure attached to other models.
They won't control a big model, but that's what the mini version is for.
- 3 replies
AbstractPhilย
posted an update 13 days ago
Post
108
Now that the Beatrix recall problem will soon be resolved, I can move forward to teaching her useful information related to pragmatism.
Beatrix V3 has a lot of trouble with communication, but the model herself is entirely filled with information. With minimal processing and curation, this information will be extractable. It's not the most robust information, but we can say for certain that the information is most definitely represented usefully in a way that can be resurfaced, purposed, utilized, enhanced, expanded, and reduced as necessary.
With the AMOE directly hosting and supporting the softmax attention behavioral arms, it brought me a new form of understanding and respect for token recall at depth. I knew Beatrix would be able to represent the data, but what I got was very unexpected.
This model when those numbers don't line up, she simply speaks gibberish. Absolute gibberish as shown in the 2s softmax control variant. That model turned into a spinning top that flew off the rails, so if you query that model you will only see gibberish.
With those numbers lined up, she speaks... I don't know how to describe it. It's not gibberish, but it's definitely not, expected. Like an understanding of confusion.
There are multiple behaviors that her arms are meant to train which are currently devoid, and in some aspects the arms work perfectly like the caption arm.
If you use the 3s model with ONLY the caption arm, she will complete a simple caption for you. It may not be very good, and she's not the best model for it, but she most definitely can do it.
Don't need all 9 arms, but it doesn't hurt to have them on. Just takes too long.
SOFTMAX attention however, inferenced at bf16 this should be fine. Lightning quick you could say. TRAINING I have to train the softmax attention at fp32 forward with some rounding black magic for backward. Otherwise the v3 model simply implodes.
There have been a huge series of problems related to her in the coming writeup.
Beatrix V3 has a lot of trouble with communication, but the model herself is entirely filled with information. With minimal processing and curation, this information will be extractable. It's not the most robust information, but we can say for certain that the information is most definitely represented usefully in a way that can be resurfaced, purposed, utilized, enhanced, expanded, and reduced as necessary.
With the AMOE directly hosting and supporting the softmax attention behavioral arms, it brought me a new form of understanding and respect for token recall at depth. I knew Beatrix would be able to represent the data, but what I got was very unexpected.
This model when those numbers don't line up, she simply speaks gibberish. Absolute gibberish as shown in the 2s softmax control variant. That model turned into a spinning top that flew off the rails, so if you query that model you will only see gibberish.
With those numbers lined up, she speaks... I don't know how to describe it. It's not gibberish, but it's definitely not, expected. Like an understanding of confusion.
There are multiple behaviors that her arms are meant to train which are currently devoid, and in some aspects the arms work perfectly like the caption arm.
If you use the 3s model with ONLY the caption arm, she will complete a simple caption for you. It may not be very good, and she's not the best model for it, but she most definitely can do it.
Don't need all 9 arms, but it doesn't hurt to have them on. Just takes too long.
SOFTMAX attention however, inferenced at bf16 this should be fine. Lightning quick you could say. TRAINING I have to train the softmax attention at fp32 forward with some rounding black magic for backward. Otherwise the v3 model simply implodes.
There have been a huge series of problems related to her in the coming writeup.
- 7 replies
AbstractPhilย
posted an update 16 days ago
Post
59
Beatrix v3's attention has problems. The model at high byte counts simply cannot recall very well, which has been an expensive lesson but a necessary one.
To remedy this fault, I will complete the Beatrix 2s softmax control variant until stable. I will train the curriculum stages with QK normalization similar to Gemma, giving us a proper full model MHA softmax control. The answer is likely faulty bf16, needs fp32 and QK norms on the attention, softmax attention in intervals for the primary model.
After stabilizing the v2 softmax variant, I will train the v3 softmax variant as well.
These are required. There are many open questions about stability at depth and stability at context window capacity, all of which can be answered with the high accuracy recall at depth and context window size.
An expensive lesson, but a necessary lesson for progress.
Even with the faults, Beatrix V3's mechanisms have proven to be invaluable in their own rite. Multiple experiments have yielded since, including the tokenizer processing system - giving Beatrix the ability to directly speak as another model's tokenizer language. It's not perfect yet, but the process is likely reusable for other similar architected models.
The quiet loss has been extensively successful. The structure of each arm trained one after the other shows we can keep each arm nice and quiet while being fully expert in their own field. Activation happens when the byte pattern says so, and the arms stay nice and quiet when the gated pattern doesn't say so.
Modularized arms are successful. This is a process that can likely be utilized on other models as well, but the process hasn't been fully completed yet. The model Beatrix was a good catalyst for this because of the byte format of the model and the anchored behavior allows this to be trained more quickly and stable.
In any case, the 2s-control is upcoming and will begin today. I'll try to get it cleaned up by the end of the week to prepare for the 3s control.
To remedy this fault, I will complete the Beatrix 2s softmax control variant until stable. I will train the curriculum stages with QK normalization similar to Gemma, giving us a proper full model MHA softmax control. The answer is likely faulty bf16, needs fp32 and QK norms on the attention, softmax attention in intervals for the primary model.
After stabilizing the v2 softmax variant, I will train the v3 softmax variant as well.
These are required. There are many open questions about stability at depth and stability at context window capacity, all of which can be answered with the high accuracy recall at depth and context window size.
An expensive lesson, but a necessary lesson for progress.
Even with the faults, Beatrix V3's mechanisms have proven to be invaluable in their own rite. Multiple experiments have yielded since, including the tokenizer processing system - giving Beatrix the ability to directly speak as another model's tokenizer language. It's not perfect yet, but the process is likely reusable for other similar architected models.
The quiet loss has been extensively successful. The structure of each arm trained one after the other shows we can keep each arm nice and quiet while being fully expert in their own field. Activation happens when the byte pattern says so, and the arms stay nice and quiet when the gated pattern doesn't say so.
Modularized arms are successful. This is a process that can likely be utilized on other models as well, but the process hasn't been fully completed yet. The model Beatrix was a good catalyst for this because of the byte format of the model and the anchored behavior allows this to be trained more quickly and stable.
In any case, the 2s-control is upcoming and will begin today. I'll try to get it cleaned up by the end of the week to prepare for the 3s control.
- 4 replies
AbstractPhilย
posted an update 20 days ago
Post
124
Mini-Beatrix-3 AbstractPhil/mini-beatrix-3 is released into automodel testing phase.
The arms have been implemented:
1. Trunk only, pure output no extensions.
2. 4 arm mode, the pretrained arms retrained and added.
3. 8 arm mode, 4 arm mode + 4 new arms from scratch.
4. 9 arm mode, 8 arm mode + 1 caption specific arm
Each arm is issued a very specific task to curate behavior, with the primary 8 the model is curated within a series of expected parameters. With this 9th arms inclusion, I'm hoping to bring a bit of stability to the experimental diffusion slider system thanks to photography, cc12m captions, llama captions, and booru image tagging curation.
The model's entire run stayed stable as shown by the tensorboard readouts, the model is now ready for post-train testing.
Additionally, as the 8 arm spectrum updates, the diffusion router is in experimentation phase with Sana's 256 position form, and Anima's CircleStone lab release.
AbstractPhil/geolip-beatrix-anima
AbstractPhil/geolip-beatrix-sana
The earlier tests are building sliders to determine how effective Beatrix is at managing both Sana and Anima's mood scales through various slider techniques.
The results are mixed for now, however there are some very positive and yielding results that ought to allow not only Beatrix to control Anima, but also other models to control Anima in a similar intelligent experimental fashion.
This is the first experimental line of conditioning through sliders, leading hopefully to full edit processing given a bit of time.
So, Beatrix will be editing images.
The arms have been implemented:
1. Trunk only, pure output no extensions.
2. 4 arm mode, the pretrained arms retrained and added.
3. 8 arm mode, 4 arm mode + 4 new arms from scratch.
4. 9 arm mode, 8 arm mode + 1 caption specific arm
Each arm is issued a very specific task to curate behavior, with the primary 8 the model is curated within a series of expected parameters. With this 9th arms inclusion, I'm hoping to bring a bit of stability to the experimental diffusion slider system thanks to photography, cc12m captions, llama captions, and booru image tagging curation.
The model's entire run stayed stable as shown by the tensorboard readouts, the model is now ready for post-train testing.
Additionally, as the 8 arm spectrum updates, the diffusion router is in experimentation phase with Sana's 256 position form, and Anima's CircleStone lab release.
AbstractPhil/geolip-beatrix-anima
AbstractPhil/geolip-beatrix-sana
The earlier tests are building sliders to determine how effective Beatrix is at managing both Sana and Anima's mood scales through various slider techniques.
The results are mixed for now, however there are some very positive and yielding results that ought to allow not only Beatrix to control Anima, but also other models to control Anima in a similar intelligent experimental fashion.
This is the first experimental line of conditioning through sliders, leading hopefully to full edit processing given a bit of time.
So, Beatrix will be editing images.
- 6 replies
Post
120
Hi! We open-source PrunaSuperPoint in collaboration with Eclipse Aidge for the DeepGreen project
It's an optimized version of SuperPoint built for faster keypoint detection on edge devices.
- Up to 2.1ร faster on Jetson Orin Nano, with optimizations applicable to other runtimes.
- The distilled model retains strong keypoint coverage across indoor and outdoor data, with low descriptor differences from the original model.
- We structurally prune the most expensive convolutional layers, recover performance through distillation, and accelerate keypoint selection with hierarchical top-k, all while preserving the original architectureโs core behavior.
Check it here: PrunaAI/PrunaSuperPoint
It's an optimized version of SuperPoint built for faster keypoint detection on edge devices.
- Up to 2.1ร faster on Jetson Orin Nano, with optimizations applicable to other runtimes.
- The distilled model retains strong keypoint coverage across indoor and outdoor data, with low descriptor differences from the original model.
- We structurally prune the most expensive convolutional layers, recover performance through distillation, and accelerate keypoint selection with hierarchical top-k, all while preserving the original architectureโs core behavior.
Check it here: PrunaAI/PrunaSuperPoint
chengkunliย
submitted a
paper to Daily Papers 24 days ago
AbstractPhilย
posted an update 26 days ago
Post
104
GPT, Gemini, Claude, and I have identified a multitude of direct utilities for Beatrix useful for diffusion conditioning in very powerful geometric formats. We have also identified multiple weaknesses to compensate for, multiple strengths to augment, the cause of the final layer's weak erank output state, and an emergent mathematical property of calculation in this format. The final stage directional magnitude is overwhelming and becoming amplitude.
There is a full article brewing for this information, including a massive set of information already learned from Beatrix V3 that could not be extracted from the 2s variant.
As the model trains, the amplitude begins to strengthen over and over. The weak tokenization processing from splat attention, forms the internal state of the model towards a bloating fashion. This is due to the articulation applied by the structure of the aleph addressing.
This creates massive erank geometry naturally, exhausting the space, producing comprehensively complex geometric structures. This internal structure here is weakly bound to the internal bytes, causing recon to weaken over time >2048, producing the output tokenization to be weaker at higher token lengths. Training improves this but is not known to solve it.
Along this chain the final layer has formed a sort of unexpected behavior, an amplitude behavior. I've met amplitude responses before in multiple models, and even attempted to curated magnitude through flow matching to some success, however amplitude in that nature is costly and adds additional overhead to the train so I'll need to come up with something more careful, and potentially something more clever than just attaching a composite or an energy dampener.
Attention will be solved by introducing various MHA layers throughout, ensuring the recon through the depth of the model survives. With that we'll want to ensure large erank composites form as well, allowing those humongous geometric structures to form and contribute.
There is a full article brewing for this information, including a massive set of information already learned from Beatrix V3 that could not be extracted from the 2s variant.
As the model trains, the amplitude begins to strengthen over and over. The weak tokenization processing from splat attention, forms the internal state of the model towards a bloating fashion. This is due to the articulation applied by the structure of the aleph addressing.
This creates massive erank geometry naturally, exhausting the space, producing comprehensively complex geometric structures. This internal structure here is weakly bound to the internal bytes, causing recon to weaken over time >2048, producing the output tokenization to be weaker at higher token lengths. Training improves this but is not known to solve it.
Along this chain the final layer has formed a sort of unexpected behavior, an amplitude behavior. I've met amplitude responses before in multiple models, and even attempted to curated magnitude through flow matching to some success, however amplitude in that nature is costly and adds additional overhead to the train so I'll need to come up with something more careful, and potentially something more clever than just attaching a composite or an energy dampener.
Attention will be solved by introducing various MHA layers throughout, ensuring the recon through the depth of the model survives. With that we'll want to ensure large erank composites form as well, allowing those humongous geometric structures to form and contribute.
- 3 replies
AbstractPhilย
posted an update about 1 month ago
Post
69
My apologies for the incorrect format for the AMOE arms from the experimental branch. They have been saving as torch objects. They are now correctly saving as safetensors format. My apologies for the inconvenience this may cause for you use. I will be modifying the codespaces to use the correct safetensors formats.
After the first 20.9b tokens trained, the real experiments begins. Beatrix V3's first prepped-state modular command structure has been attached for dynamic training. These arms will exist as appendages for Beatrix - trained alongside with the trunk until the end of the run.
These exist for experimental extraction, analysis, distillation experiments, memory experiments, mathematics experiments, and more. Each arm will be built along the chain for specific test cases. Expectation for each is already lined up and the outcomes are tested for, but the model still may face instability and must be monitored.
As her first arm learns tinystories, she builds direct composite semantic structure throughout this system. Think of it like, the first higher-functioning cognition attachment.
She's still very naรฏve and structurally unaware, so attaching new limbs is essentially extending a structure that is not yet finished forming. Nothing but fragments of issued information from an unknown source.
In this case, this structure has been tested hundreds of times to ensure she will not simply collapse during training by having this attached.
After the first 20.9b tokens trained, the real experiments begins. Beatrix V3's first prepped-state modular command structure has been attached for dynamic training. These arms will exist as appendages for Beatrix - trained alongside with the trunk until the end of the run.
These exist for experimental extraction, analysis, distillation experiments, memory experiments, mathematics experiments, and more. Each arm will be built along the chain for specific test cases. Expectation for each is already lined up and the outcomes are tested for, but the model still may face instability and must be monitored.
As her first arm learns tinystories, she builds direct composite semantic structure throughout this system. Think of it like, the first higher-functioning cognition attachment.
She's still very naรฏve and structurally unaware, so attaching new limbs is essentially extending a structure that is not yet finished forming. Nothing but fragments of issued information from an unknown source.
In this case, this structure has been tested hundreds of times to ensure she will not simply collapse during training by having this attached.
- 4 replies
AbstractPhilย
posted an update about 1 month ago
Post
109
12 day cook for mini beatrix v3 begins. This model's byte input is formatted using a method dubbed atlas input.
ETA OCTOBER 2 2026
AbstractPhil/alephllm-mini-beatrix-training
https://github.com/AbstractEyes/geolip-bytelex
https://github.com/AbstractEyes/alephllm
Upgrades:
* 64 billion byte training pipeline up from 16 billion
* 32 block depth 376.0M in v3 up from 20 block 237.1M in 2s.
* Active aleph head, repaired via the 2s faults and a large series of tests.
* Byte atlas gateway router, explained below.
* Guaranteed convergence follow-up AMOE arms on pretrain, fused into the final form, trained together over time to increase the collective capacity.
* Multi-tokenizer oriented post-training arms distilled from multiple experts; E.G. Qwen 3.8 27b multi-layer teacher/student arms, CLIP big_g, Bert Code, and more.
* Special token word implementation via AMOE arms is now tested up to 240 special tokens for routing. Theoretically each can implement it's own sub-arm aka nested commands. E.G; <think><think_symbolic> ... </think_symbolic></think>
* Fused words post-training for faster inference.
# The Atlas
This atlas structure contains the conjoined shape of 12 tokenizers represented in the trigram format. This is used to predict difficulty in the overlaps, as per determined by the average byte overlap measured via corpus text and the compared overlap. This accuracy is only related to difficulty but it provides pre-training difficulty assessment that we will use to test post-training accuracy with it. This will determine if we can precalculate the likelihood of byte difficulty via tokenizer shape in byte form, for the multibyte fusion upcoming arm experiments for v3.
The reason for this, is distillation. We need to train Beatrix to behave with multiple tokenizers, and this theory is showing accuracy with v1 and v2, but the 32 block depth of v3 will answer many questions alongside of the structure.
ETA OCTOBER 2 2026
AbstractPhil/alephllm-mini-beatrix-training
https://github.com/AbstractEyes/geolip-bytelex
https://github.com/AbstractEyes/alephllm
Upgrades:
* 64 billion byte training pipeline up from 16 billion
* 32 block depth 376.0M in v3 up from 20 block 237.1M in 2s.
* Active aleph head, repaired via the 2s faults and a large series of tests.
* Byte atlas gateway router, explained below.
* Guaranteed convergence follow-up AMOE arms on pretrain, fused into the final form, trained together over time to increase the collective capacity.
* Multi-tokenizer oriented post-training arms distilled from multiple experts; E.G. Qwen 3.8 27b multi-layer teacher/student arms, CLIP big_g, Bert Code, and more.
* Special token word implementation via AMOE arms is now tested up to 240 special tokens for routing. Theoretically each can implement it's own sub-arm aka nested commands. E.G; <think><think_symbolic> ... </think_symbolic></think>
* Fused words post-training for faster inference.
# The Atlas
This atlas structure contains the conjoined shape of 12 tokenizers represented in the trigram format. This is used to predict difficulty in the overlaps, as per determined by the average byte overlap measured via corpus text and the compared overlap. This accuracy is only related to difficulty but it provides pre-training difficulty assessment that we will use to test post-training accuracy with it. This will determine if we can precalculate the likelihood of byte difficulty via tokenizer shape in byte form, for the multibyte fusion upcoming arm experiments for v3.
The reason for this, is distillation. We need to train Beatrix to behave with multiple tokenizers, and this theory is showing accuracy with v1 and v2, but the 32 block depth of v3 will answer many questions alongside of the structure.
- 4 replies
AbstractPhilย
posted an update about 1 month ago
Post
50
After a week of failures and invalid hypothesis with bytelex using generic structures, I found a highly successful aleph prototypical structure that conforms to the needs. This structure conforms to standard transformer, FFN, and RNN with some minor tweaks.
You can speak to the model AbstractPhil/alephllm-chat , all of the primary experiments are the listed arms.
AbstractPhil/alephllm-mini-beatrix-training All the weights of the week are stored here and in various nearby directories.
* We've managed to overlap multiple arms to train multiple simultaneous templates.
* Introduce new tokens as composite tokens from multiple teachers.
* Retrain existing tokens into the behavior of one teacher or another.
* Extend new chains and new behaviors from training in combination.
* Properly instantiate and reinforce behavior using Aleph RNN to reinforce training from raw data.
The EMA Relay. The code has been pushed to both beatrix repos.
AbstractPhil/mini-beatrix-2s
The structure itself is built specifically as a solidification unit to extensible arms, allowing more composite structures to build.
EMA structures aren't new, but when applied correctly at just such a methodology, the models begin to behave as though the extension relays are in fact the original model. The chains and behavior form naturally and the substructure begins to conform with the token fragments from much more complex structures like combined token differences of T5, Qwen, and CLIP as unified teachers.
The cross-token noise is mitigated using a series of principles and the blueprints are showing both solidity and failure simultaneously, both proving many new utilizable states and disproving multiple theoretical pathologies utilized in current running modern papers as the methodologies tested in the specific formats.
You can speak to the model AbstractPhil/alephllm-chat , all of the primary experiments are the listed arms.
AbstractPhil/alephllm-mini-beatrix-training All the weights of the week are stored here and in various nearby directories.
* We've managed to overlap multiple arms to train multiple simultaneous templates.
* Introduce new tokens as composite tokens from multiple teachers.
* Retrain existing tokens into the behavior of one teacher or another.
* Extend new chains and new behaviors from training in combination.
* Properly instantiate and reinforce behavior using Aleph RNN to reinforce training from raw data.
The EMA Relay. The code has been pushed to both beatrix repos.
AbstractPhil/mini-beatrix-2s
The structure itself is built specifically as a solidification unit to extensible arms, allowing more composite structures to build.
EMA structures aren't new, but when applied correctly at just such a methodology, the models begin to behave as though the extension relays are in fact the original model. The chains and behavior form naturally and the substructure begins to conform with the token fragments from much more complex structures like combined token differences of T5, Qwen, and CLIP as unified teachers.
The cross-token noise is mitigated using a series of principles and the blueprints are showing both solidity and failure simultaneously, both proving many new utilizable states and disproving multiple theoretical pathologies utilized in current running modern papers as the methodologies tested in the specific formats.
- 1 reply
[Join request] Verification and provenance for AI agents
2
#29 opened about 2 months ago
by 
AiOpsEnabler
AbstractPhilย
posted an update about 2 months ago
Post
96
The post-beatrix-2s and control variant article is finally satisfactory, so the article is now released https://huggingface.co/blog/AbstractPhil/beatrix-ft2
The control variant will need another train with better SDPA stabilization, as the control variant destabilized and collapsed. The primary fault is the lack of QK normalization, which caused the model to simply collapse given enough time. Claude lists the rest of the suspected reasons in the article.
This was a very difficult series of experiments to tune with many fault points. Trying to make heads or tails of Fable 5.1 Claude-speak hasn't been the easiest task either. It seems the model is more likely to create pedantically rigid responses rather than cooperative. Not necessarily insulting, but definitely a sort of refrigerator-magnet behavior - treating my individual contributions as little sketches for the refrigerator. This often completely ignores my larger MD or complex behavioral instructions in favor of my theoretical or hypothetical - likely considering the MD and technical as the model's own, rather than my direct contributions. Right there... right on the refrigerator goes my hypothesis that worked.
https://github.com/AbstractEyes/geolip-bytelex
In any case, this upcoming week will be related entirely to cross-tokenizer distillation research. It may stretch long beyond the next week, but as it stands the geometric vocabulary has evolved into a codebook prediction system.
I would like to give this program linear wings. The Beatrix model supports it, but how well is up for this week to decide.
There are a multitude of potentials based on a series of very recent articles I will be exploring, providing the necessary bytelex complexity to a roughly 60 hour battery of experiments and trainings throughout the geometric systems.
The results will determine the best and worst methodologies of using these models, these shapes, and these structures with more complex byte-level cross tokenization systems
The control variant will need another train with better SDPA stabilization, as the control variant destabilized and collapsed. The primary fault is the lack of QK normalization, which caused the model to simply collapse given enough time. Claude lists the rest of the suspected reasons in the article.
This was a very difficult series of experiments to tune with many fault points. Trying to make heads or tails of Fable 5.1 Claude-speak hasn't been the easiest task either. It seems the model is more likely to create pedantically rigid responses rather than cooperative. Not necessarily insulting, but definitely a sort of refrigerator-magnet behavior - treating my individual contributions as little sketches for the refrigerator. This often completely ignores my larger MD or complex behavioral instructions in favor of my theoretical or hypothetical - likely considering the MD and technical as the model's own, rather than my direct contributions. Right there... right on the refrigerator goes my hypothesis that worked.
https://github.com/AbstractEyes/geolip-bytelex
In any case, this upcoming week will be related entirely to cross-tokenizer distillation research. It may stretch long beyond the next week, but as it stands the geometric vocabulary has evolved into a codebook prediction system.
I would like to give this program linear wings. The Beatrix model supports it, but how well is up for this week to decide.
There are a multitude of potentials based on a series of very recent articles I will be exploring, providing the necessary bytelex complexity to a roughly 60 hour battery of experiments and trainings throughout the geometric systems.
The results will determine the best and worst methodologies of using these models, these shapes, and these structures with more complex byte-level cross tokenization systems
- 2 replies
AbstractPhilย
posted an update about 2 months ago
Post
2,049
Mini-Beatrix-2s pretraining is ready.
AbstractPhil/mini-beatrix-2s
The model passed a great deal of rigor and hardship, trained roughly 16 billion tokens or so. The full writeup for the model including the arms training for the first version arms and the second version arms will be drafted and prepared as soon as the v2 arms are done training and testing.
There are many possibilities present with such a model. The hub itself has been marked capable of potentially operating as similarity comparison, 87% of the capacity retained within a 256 dim structure. Along with this, the multi-dimensional hub attention system shows serious promise with controlling diffusion model inference, which I look forward to see the results of.
Additionally, sentence similarity, next token prediction, and a large array of prediction formats have been heavily improved by introducing the full model with splat attention. The model not only improved, the structure complemented everything measured, along with the more effective training regiment for version 2.
Beatrix 2s is essentially an autoregression decoder, however the attention mechanism houses a dual-stage encoder/decoder structure internally. Each adopting the SVAE as a core component, revamped and fitted to the exact rules of AlephLM. So there are essentially 20 SVAE in this structure, each with their own independent encoders, residually learning from the last.
Upcoming tests will include finetunes to bring out the strengths of all special tokens, presented in the upcoming article. The full experiment battery will be completed within a few days and the findings presented.
Modularization, compartmentalization, secularized behavior, and everything between are to be tested with rigor. This model is a rapid learner, there will likely be byproduct problems with that, and I look forward to solving the corewise problems one at a time until the model is strong enough to be useful for all the tested tasks.
AbstractPhil/mini-beatrix-2s
The model passed a great deal of rigor and hardship, trained roughly 16 billion tokens or so. The full writeup for the model including the arms training for the first version arms and the second version arms will be drafted and prepared as soon as the v2 arms are done training and testing.
There are many possibilities present with such a model. The hub itself has been marked capable of potentially operating as similarity comparison, 87% of the capacity retained within a 256 dim structure. Along with this, the multi-dimensional hub attention system shows serious promise with controlling diffusion model inference, which I look forward to see the results of.
Additionally, sentence similarity, next token prediction, and a large array of prediction formats have been heavily improved by introducing the full model with splat attention. The model not only improved, the structure complemented everything measured, along with the more effective training regiment for version 2.
Beatrix 2s is essentially an autoregression decoder, however the attention mechanism houses a dual-stage encoder/decoder structure internally. Each adopting the SVAE as a core component, revamped and fitted to the exact rules of AlephLM. So there are essentially 20 SVAE in this structure, each with their own independent encoders, residually learning from the last.
Upcoming tests will include finetunes to bring out the strengths of all special tokens, presented in the upcoming article. The full experiment battery will be completed within a few days and the findings presented.
Modularization, compartmentalization, secularized behavior, and everything between are to be tested with rigor. This model is a rapid learner, there will likely be byproduct problems with that, and I look forward to solving the corewise problems one at a time until the model is strong enough to be useful for all the tested tasks.
- 4 replies
[Support] Community Articles
๐ฅ๐ 2
115
#5 opened over 2 years ago
by 
victor
Join request pending for a week, anything else needed?
1
#28 opened 3 months ago
by 
yuncheol-ncdtech