2026Author
Cleave
Tokenization is where a lot of subtle model behavior actually lives, and it is almost always a black box you cannot inspect, let alone watch run.
The result
A byte-level BPE tokenizer built from the Python standard library with a provably lossless round-trip on arbitrary Unicode, plus a dependency-free browser export that matches the Python encoder token for token.
Cleave is a byte-level Byte Pair Encoding tokenizer written from scratch using only the Python standard library. Any input, including arbitrary Unicode, encodes to token IDs and decodes back to exactly the original bytes. It also exports a small, dependency-free tokenizer that runs in the browser and produces the same tokens as the Python encoder, so you can watch tokenization happen live.
- Round-trip
- Provably lossless
- Alphabet
- Byte-level
- Browser export
- Token-for-token
- Dependencies
- Stdlib only
Architecture
Arbitrary Unicode text
UTF-8 bytes
Byte-level BPE merges
Token IDs
Decode
IDs to bytes to text
Round-trip identical
provably lossless
Text becomes UTF-8 bytes, which is what makes the round-trip total: working at the byte level means there is no character a merge table can fail to represent. The BPE merges turn byte sequences into token IDs, and decoding runs the whole thing in reverse, IDs back to bytes back to text. The round-trip is checked to be identical, which is the lossless guarantee. That same merge table is compiled into the browser export, so the JavaScript tokenizer is not a reimplementation that drifts: it produces the same tokens as Python, token for token.
Key decisions and trade-offs
Byte-level BPE so the round-trip is provably lossless
Character-level tokenizers have to decide what to do with bytes they have never seen, and that is where silent corruption creeps in. Operating on UTF-8 bytes means every possible input is representable and decode is the exact inverse of encode, so lossless on arbitrary Unicode is a property of the design rather than a hope. It costs some sequence length on non-ASCII text, which is the accepted trade-off for correctness.
A browser export that matches Python token for token
A demo tokenizer that only approximates the real one teaches the wrong thing. Exporting the actual merge table and running it in dependency-free JavaScript means the in-browser view is the same tokenizer, not a lookalike, so anyone can watch real tokenization on their own text with nothing to install.
Standard library only
Building on the Python standard library keeps the implementation transparent and portable and removes any question of a dependency doing the hard part. The tokenizer is small enough to read end to end, which is the point.
Results
- Provably lossless round-trip on arbitrary Unicode input.
- Browser export matches the Python encoder token for token.
- Byte-level BPE implemented from the standard library, no third-party dependencies.
Repository
The in-browser tokenizer is the strongest live-demo candidate of the suite: a text box where you type and watch the tokens split in real time.