Skip to contentNewskeryx

9 October 2026

ttok 1.0 released, fixing a default tokenizer problem

First reported

Simon Willison ran this on .

  • A new version, ttok 1.0, has been released as a tool for counting text in units the AI model reads.
  • The update was prompted by a bug where the tool defaulted to a tokenizer (the piece that splits text into chunks for the model) from GPT-4, an OpenAI model.
  • The author noticed the problem after upgrading to the earlier ttok 0.4 version.

How it was covered

Simon WillisonDaily notes and links

The ttok 1.0 release and a default-tokenizer problem found in 0.4