Tokenizers v1 first release candidate is out, claiming up to 30x faster tokenization

Lysandre (@LysandreJik) said Tokenizers v1’s first release candidate is out. He described it as a broad release, claiming up to 30x faster tokenization, scaling across threads, lower latency, lower memory use, and a microscopic crate size.

He pointed to a write-up at huggingface.co/blog/tokenizers-v1.