https://arxiv.org/pdf/2609.12303 this paper suggests a model that skips models reading tokens that and has the model read raw bytes, basically one letter at a time. what they did: trained small models (~1B params), some on tokens and some on bytes trained them partly by copying a bigger model (llama 3 8b), aka distillation the tricky part is that the teacher thinks in tokens and the student reads bytes, so they had to work out a clean way to translate between the two what they found: with a little training the token models win, not even close with more training the token models kind of flatline while the byte ones keep improving if you extend the trend lines, the byte model ends up beating llama 3.2 1b and gemma 3 1b by a few points the byte models also needed about 6x less text to hit the same level the catch (and there’s a few): the “beats llama and gemma” part is a prediction. they extended the curves past what they actually trained. in their real runs the token model was still ahead reading letter by letter is slower. a token is ~4-5 letters, so it’s roughly 5x more steps for the same text. less data but more compute tokens aren’t really dead, since the teacher model still uses them only tested on small models, no idea yet if it holds at scale still pretty cool imo. for small models on phones/laptops, squeezing more quality out of the same size matters a lot. and getting rid of the giant word list might fix the dumb stuff like miscounting letters in a word anyone know if there’s work on making byte models run as fast as token ones?   submitted by   /u/DotaMate [link]   [comments]