gemini 2.0 flash was peak, went downhill after that.
alkinun
AtAndDev
AI & ML interests
decentralize
Recent Activity
liked a model 27 minutes ago
LiquidAI/LFM3-VL-2B liked a model 29 minutes ago
LiquidAI/LFM3-1.7B repliedto Banaxi-Tech's post 1 day ago
BananaMind is in the google situation before argon right now, right?Organizations
replied to Banaxi-Tech's post 1 day ago
reacted to mithulaartigala's post with 🔥 2 days ago
Post
3,241
We need to make software where people can start using decision models.
reacted to appvoid's post with 🔥 2 days ago
Post
3,923
You know, I can't believe we are actually doing it, literally advancing the frontier at the edge! We've come a long way since the first models on open slm leaderboard, and, as many things that change, performance is one of the main things that has exploded.
7 years ago, the best we had was GPT-2. Today, we got 90m models beating smollm2! I can't barely see what a future with increasingly powerful small models will look like.
All I can say is that the best is yet to come. And will be agentic.
7 years ago, the best we had was GPT-2. Today, we got 90m models beating smollm2! I can't barely see what a future with increasingly powerful small models will look like.
All I can say is that the best is yet to come. And will be agentic.
- 11 replies
replied to Banaxi-Tech's post 2 days ago
im talking about gemini, gemma always cooks. gemini is shit always.
replied to Banaxi-Tech's post 2 days ago
google always has been bad, you arent :)
make a comeback!
reacted to Banaxi-Tech's post with 😔 2 days ago
Post
4,862
BananaMind is in the google situation before argon right now, right?
- 29 replies
reacted to jiteshkprasad's post with 🔥 6 days ago
Post
2,937
Last week we released Gargi, a 384M English language model trained from scratch on about 17B tokens, roughly 1,000× less training data than Qwen2.5-0.5B (~18T tokens, as published by its authors).
What we measured (same harness for every model: lm-evaluation-harness, 0-shot, greedy, each model's start token):
- Base model vs Qwen2.5-0.5B: ARC-Easy 62.8 vs 64.7, ARC-Challenge 32.3 vs 32.4, WinoGrande 54.6 vs 56.1.
- Chat model: thinking on or off; GSM8K 36.0 with thinking on.
- Long inputs: needle-in-a-haystack recall 100% at 8K, 93% at 16K and 90% at 20K tokens, so use it up to about 16K.
Where it falls short: knowledge and everyday common sense still scale with data. MMLU is 25.5 vs 47.3, and it trails on HellaSwag and PIQA. It is not a maths specialist, and it is not safety-tuned: put a moderation layer in front of any public use.
It runs on a laptop: GGUF (Q8_0 at 410 MB, Q4_K_M, f16) and MLX builds.
Apache-2.0 ·
weights: udgamlabs/Gargi
try it: https://udgamlabs.com/models/gargi
code: https://github.com/udgamlabs/gargi
Named after Gargi Vachaknavi, the Vedic philosopher remembered for asking the hardest questions.
What we measured (same harness for every model: lm-evaluation-harness, 0-shot, greedy, each model's start token):
- Base model vs Qwen2.5-0.5B: ARC-Easy 62.8 vs 64.7, ARC-Challenge 32.3 vs 32.4, WinoGrande 54.6 vs 56.1.
- Chat model: thinking on or off; GSM8K 36.0 with thinking on.
- Long inputs: needle-in-a-haystack recall 100% at 8K, 93% at 16K and 90% at 20K tokens, so use it up to about 16K.
Where it falls short: knowledge and everyday common sense still scale with data. MMLU is 25.5 vs 47.3, and it trails on HellaSwag and PIQA. It is not a maths specialist, and it is not safety-tuned: put a moderation layer in front of any public use.
It runs on a laptop: GGUF (Q8_0 at 410 MB, Q4_K_M, f16) and MLX builds.
ollama run hf.co/udgamlabs/Gargi-GGUF:Q8_0Apache-2.0 ·
weights: udgamlabs/Gargi
try it: https://udgamlabs.com/models/gargi
code: https://github.com/udgamlabs/gargi
Named after Gargi Vachaknavi, the Vedic philosopher remembered for asking the hardest questions.
- 21 replies
reacted to danielhanchen's post with 🤯🚀❤️🔥 7 days ago
Post
4,883
Run DeepSeek V4.1-Flash-1017 locally! 🐳
We made 1-bit retain ~88.2% accuracy while shrinking size from 508GB to 250GB (−52%).
Run on a 128GB+ RAM device. V4.1 Flash 1017 is the strongest open model to date
Guide: https://unsloth.ai/docs/models/deepseek-v4.1-flash
GGUF: unsloth/DeepSeek-V4.1-Flash-1017-GGUF
We made 1-bit retain ~88.2% accuracy while shrinking size from 508GB to 250GB (−52%).
Run on a 128GB+ RAM device. V4.1 Flash 1017 is the strongest open model to date
Guide: https://unsloth.ai/docs/models/deepseek-v4.1-flash
GGUF: unsloth/DeepSeek-V4.1-Flash-1017-GGUF
reacted to DedeProGames's post with 🔥 11 days ago
Post
3,457
Meet Supra2.5-Ultra-Preview-001. 230M parameters, ~40.4B training tokens.
SupraLabs has released the first preview of its Supra2.5-Ultra family: a compact, Llama-style base model for text completion and small-model experimentation.
What's inside?
• 230M parameters
• 32 transformer layers
• 2,048-token context window
• Grouped Query Attention: 12 attention heads, 4 KV heads
• 49,152-token vocabulary
• SwiGLU activation, RoPE and tied embeddings
• BF16 weights and Hugging Face Transformers support
• Apache 2.0 license
Training
The model was trained on approximately 40.4B tokens from FineWeb-Edu, DCLM-Baseline and FineMath using a single A100 SXM.
Reported zero-shot results
• HellaSwag: 37.18% normalized accuracy
• PIQA: 65.78% accuracy
• ARC-Easy: 57.11% accuracy
• ARC-Challenge: 26.28% normalized accuracy
Want to experiment?
Code on REPO.
Model, weights and usage example:
SupraLabs/Supra2.5-Ultra-Preview-001
SupraLabs has released the first preview of its Supra2.5-Ultra family: a compact, Llama-style base model for text completion and small-model experimentation.
What's inside?
• 230M parameters
• 32 transformer layers
• 2,048-token context window
• Grouped Query Attention: 12 attention heads, 4 KV heads
• 49,152-token vocabulary
• SwiGLU activation, RoPE and tied embeddings
• BF16 weights and Hugging Face Transformers support
• Apache 2.0 license
Training
The model was trained on approximately 40.4B tokens from FineWeb-Edu, DCLM-Baseline and FineMath using a single A100 SXM.
Reported zero-shot results
• HellaSwag: 37.18% normalized accuracy
• PIQA: 65.78% accuracy
• ARC-Easy: 57.11% accuracy
• ARC-Challenge: 26.28% normalized accuracy
Want to experiment?
Code on REPO.
Model, weights and usage example:
SupraLabs/Supra2.5-Ultra-Preview-001
- 4 replies
reacted to Avifenesh's post with 👍🔥👀 13 days ago
Post
2,181
I added a small gate to the Gated DeltaNet layers of Qwen3.5 and trained it with GRPO under a length penalty.
On Qwen3.5-4B it reasons 48% shorter than LoRA on MATH-500, and anytime accuracy goes from 0.786 to 0.852. A LoRA with about the same parameter count barely moves.
The gate sits on beta, the per-token write strength. One matrix per layer, zero init, so it starts as a no-op.
It started as a negative eigenvalue experiment. The negative range turned out not to matter. The gate did.
Write-up: https://huggingface.co/blog/Avifenesh/beta-gate-qwen35
Adapters: Avifenesh/gate-qwen35-6accb63081800b2f2d583b80
Code: https://github.com/avifenesh/gdn-beta-gatehttps://huggingface.co/collections/Avifenesh/gate-qwen35
On Qwen3.5-4B it reasons 48% shorter than LoRA on MATH-500, and anytime accuracy goes from 0.786 to 0.852. A LoRA with about the same parameter count barely moves.
The gate sits on beta, the per-token write strength. One matrix per layer, zero init, so it starts as a no-op.
It started as a negative eigenvalue experiment. The negative range turned out not to matter. The gate did.
Write-up: https://huggingface.co/blog/Avifenesh/beta-gate-qwen35
Adapters: Avifenesh/gate-qwen35-6accb63081800b2f2d583b80
Code: https://github.com/avifenesh/gdn-beta-gatehttps://huggingface.co/collections/Avifenesh/gate-qwen35
reacted to Banaxi-Tech's post with 🚀👀🔥 13 days ago
Post
2,274
We're announcing KV1. Now.
We're going to take some models, do a bit of continued training on them and QAT them for only a 2 bit KV cache.
And because of what happened with BGA, and this is also kind of some research stuff, here's what we're going to change:
We're going to always, before we release how and make any claims, release our results.
We also would like (idk if we can) to have some external reviewers before releasing them, I'M talking to YOU @AtAndDev and @mithulaartigala
And we're going to properly cite other research.
The models we are going to release is:
KV1 1.2B, based on LFM 2.5 1.2B Instruct
KV1 2.6B, based on LFM 2.5 2.6B Instruct
KV1 4B, based on Qwen3.5 4B.
If we get enough compute, we may also train:
KV1 9B
And if you guys want to a ~27B model, you guys decide which, either on LFM 2.5 24B A2B, Qwen3.8 27B, or Qwen3.6 35B A3B.
Our goal is to get 2 bit, at 4 bit quality.
Here's an image of our current progress, KV1 is at 3 bit, not 4 bit, but it will improve!
We're going to take some models, do a bit of continued training on them and QAT them for only a 2 bit KV cache.
And because of what happened with BGA, and this is also kind of some research stuff, here's what we're going to change:
We're going to always, before we release how and make any claims, release our results.
We also would like (idk if we can) to have some external reviewers before releasing them, I'M talking to YOU @AtAndDev and @mithulaartigala
And we're going to properly cite other research.
The models we are going to release is:
KV1 1.2B, based on LFM 2.5 1.2B Instruct
KV1 2.6B, based on LFM 2.5 2.6B Instruct
KV1 4B, based on Qwen3.5 4B.
If we get enough compute, we may also train:
KV1 9B
And if you guys want to a ~27B model, you guys decide which, either on LFM 2.5 24B A2B, Qwen3.8 27B, or Qwen3.6 35B A3B.
Our goal is to get 2 bit, at 4 bit quality.
Here's an image of our current progress, KV1 is at 3 bit, not 4 bit, but it will improve!
- 43 replies
replied to Banaxi-Tech's post 17 days ago
Based on the model cards I've reviewed, here are the key ways BGA (BananaMind Gate Attention) differs from NSA/DeepSeek (2025):
Core Architecture Differences
BGA (BananaMind Gate Attention):
- Uses a three-tower relay transformer architecture with towers A, B (relay), and C
- Tower B has no output head or loss term - it only passes gradients between A and C
- 256x less attention compute at 1M context windows
- Context windows: 10M, 16K, 25M, 32K, 64K (up to 1M in training)
- ~35M parameters (BananaMind-2.1-Unified)
NSA/DeepSeek-R1:
- Uses MoE (Mixture of Experts) architecture
- 671B total parameters with 37B activated parameters
- 128K context length
- Focuses on reinforcement learning for reasoning rather than architectural efficiency
Design Goals
BGA: Optimized for computational efficiency - can train 1M context models at the cost of ~4K context training
NSA: Optimized for reasoning capabilities - uses RL to develop chain-of-thought behaviorTraining Approach
BGA: Trained on 38B tokens with flat mix, no curriculum
NSA: Uses reinforcement learning without supervised fine-tuning as a preliminary step, then incorporates cold-start dataCurrent Status
BGA: Released (BGA model available), BananaMind 3 "will start training soon"
NSA: Already deployed with strong benchmark performance (outperforming o1-mini on many tasks)The fundamental difference is that BGA is an efficiency innovation for scaling context windows, while NSA is a reasoning breakthrough using RL. They're solving different problems in LLM development.
@KlondikeDev this is just hallucinated btw, not correct.
replied to their post 17 days ago
well he didnt show some decorum when he hid my comments that corrected him... 3 times... i already provided evidence that the thing he was talking about is just a copy of nsa but he chose to hide my comments instead of responding...