Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
NILKNARFGonzo 
posted an update Sep 23
Post
114
just recieved my stack of 10 floppy disks - you know what that means

floppyx4 is canceled, floppyx10 is next

here's the intended specs:
- official tokenizer (the actual tokenizer for gpt-2)
- actual gpu training (barely)
- sharegpt (if i can afford it computationally)
- full thing fitting on 10 floppy disks (not just the safetensors file)
- and if needed different arch (like llama)

also unsloth on a gpu from 2015 is insane

Should i make CDx1? i have a CD and DVD's btw

·

I'd recommend using a very small head_dim, and using the freed parameters for feed-forward width or something.