sol pro???
why was sol pro taken down?
why was sol pro taken down?
trained on train splits way above the limits, like +50
why was sol pro taken down?
trained on train splits way above the limits, like +50
what does that means?
why was sol pro taken down?
trained on train splits way above the limits, like +50
what does that means?
its card literally says "task-adapted checkpoint using public benchmark training splits", and their evaluation.json has the base model at 19.65 before 2500 steps of multiple choice finetuning on the train sets. so basically the whole #1 comes from that finetune,
on stuff it wasnt tuned on (swag, sciq, openbookqa, winogrande) it just ties v3.5, and its worse on lambada
i also tried the same kind of finetune on v3.5 privately to see how much it does. 20 min on one gpu took it from 27.49 to 39.20.
checked train vs test acc:
arc-c: 87.9 train vs 32.0 test (+56)
arc-e: 88.5 train vs 54.1 test (+34)
hellaswag: 58.3 train vs 46.8 val (+12)
piqa: 79.4 train vs 71.0 val (+8.5)
why was sol pro taken down?
trained on train splits way above the limits, like +50
what does that means?
its card literally says "task-adapted checkpoint using public benchmark training splits", and their evaluation.json has the base model at 19.65 before 2500 steps of multiple choice finetuning on the train sets. so basically the whole #1 comes from that finetune,
on stuff it wasnt tuned on (swag, sciq, openbookqa, winogrande) it just ties v3.5, and its worse on lambadai also tried the same kind of finetune on v3.5 privately to see how much it does. 20 min on one gpu took it from 27.49 to 39.20.
checked train vs test acc:
arc-c: 87.9 train vs 32.0 test (+56)
arc-e: 88.5 train vs 54.1 test (+34)
hellaswag: 58.3 train vs 46.8 val (+12)
piqa: 79.4 train vs 71.0 val (+8.5)
Oh ok I understood it now
Yep