GigaToken: ~1000x faster Language model tokenization

Wait 5 sec.

Article URL: https://github.com/marcelroed/gigatoken/Comments URL: https://news.ycombinator.com/item?id=49010167Points: 28# Comments: 0