tiktoken Tokeniser

What tiktoken Is#

tiktoken is a tokenisation library, not a tokenisation algorithm — under the hood it is byte-level BPE, the same algorithm as GPT-2's. What makes it notable is the implementation: a Rust core that uses careful regex pre-tokenisation, hash-table merge lookups and SIMD-friendly byte handling to achieve very high throughput on long inputs.

It is also the official source of OpenAI's tokeniser vocabularies. If you need to count GPT-4 prompt tokens accurately or replicate GPT-4's exact tokenisation, tiktoken with the cl100k_base encoding is the canonical implementation.

Pre-trained Encodings#

Each successive encoding has been more efficient on common inputs. cl100k_base is roughly 30 per cent more efficient than r50k_base on English-and-code; o200k_base adds another ~20 per cent on multilingual and code-heavy inputs by allocating tokens to common patterns in those domains.

Using tiktoken#

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
# Or by model name:
enc = tiktoken.encoding_for_model("gpt-4")

ids = enc.encode("Hello, world!")            # [9906, 11, 1917, 0]
text = enc.decode(ids)                        # "Hello, world!"
print(len(enc.encode(long_text)))             # token count

# Many threads:
ids_batch = enc.encode_batch(["a", "b", "c"], num_threads=8)

Use Beyond OpenAI Models#

Several open models have adopted tiktoken-compatible or tiktoken-style tokenisers. Llama 3 uses a byte-level BPE built with tiktoken's conventions, with a 128,256-token vocab. Qwen 2 and Qwen 3 use byte-level BPE with vocabs around 150k. DeepSeek-V3 uses ~129k. The interoperability is partial — exact merge tables differ — but the conventions (byte alphabet, regex pre-tokenisation, vocab sizes 100k+) are shared.

Performance#

On a single core, tiktoken encodes roughly 4-6 MB/s of text into tokens, several times faster than the equivalent Python or pure-Python tokenisers and meaningfully faster than HuggingFace tokenizers in most benchmarks. For dataset preprocessing where trillions of tokens must be encoded, that throughput difference translates to hours versus days of CPU time.