ArthurZucker released v1.0.0-rc.2 of huggingface/tokenizers · September 21, 2026 04:28 huggingface / /huggingface/tokenizers Release candidate v1.0.0.rc.2 Making tokenizers the sota tokenization library, same API, same token IDs, same standardsWe are happy to finally bring tokenizers up to speed ⚡ !tokenizers was released 7 years ago, in a time whe… Read more Contributors