alibaba/open-code-review · error

tiktoken encoding file %q is not embedded and cannot be fetc

Error message

tiktoken encoding file %q is not embedded and cannot be fetched offline

What it means

The offline tiktoken BPE loader only serves files compiled into the binary (looked up by their canonical URL in urlToFileMap). When asked for any other encoding file there is no network fetch fallback, so it returns this error. This keeps token counting fully offline and deterministic.

Source

Thrown at internal/llm/embedded_loader.go:43

	tiktoken.SetBpeLoader(loader)
}

// embeddedBpeLoader implements tiktoken.BpeLoader interface.
// It maps known encoding URLs to local embedded files, eliminating network dependency.
type embeddedBpeLoader struct{}

var urlToFileMap = map[string]string{
	"https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken": "cl100k_base.tiktoken",
	"https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken":  "o200k_base.tiktoken",
	"https://openaipublic.blob.core.windows.net/encodings/p50k_base.tiktoken":   "p50k_base.tiktoken",
	"https://openaipublic.blob.core.windows.net/encodings/r50k_base.tiktoken":   "r50k_base.tiktoken",
}

func (l *embeddedBpeLoader) LoadTiktokenBpe(tiktokenBpeFile string) (map[string]int, error) {
	if localName, ok := urlToFileMap[tiktokenBpeFile]; ok {
		return loadFromEmbed(localName)
	}
	return nil, fmt.Errorf("tiktoken encoding file %q is not embedded and cannot be fetched offline", tiktokenBpeFile)
}

func loadFromEmbed(filename string) (map[string]int, error) {
	data, err := embedFS.ReadFile(embedPrefix + filename)
	if err != nil {
		return nil, fmt.Errorf("embedded tiktoken file %q not found: %w", filename, err)
	}
	return parseBpeData(data)
}

// parseBpeData parses the base64-encoded BPE data format.
// Each line: <base64-token> <rank>
func parseBpeData(data []byte) (map[string]int, error) {
	bpeRanks := make(map[string]int)
	for _, line := range strings.Split(string(data), "\n") {
		line = strings.TrimSpace(line)
		if line == "" {
			continue

View on GitHub (pinned to 5cf97d0d15)

Solutions

1. Upgrade the library to a version that embeds the requested encoding file
2. Use one of the supported/embedded encoding URLs exposed by the library
3. If you maintain a fork, add the encoding to urlToFileMap plus the embedded assets and rebuild
4. Precompute token counts elsewhere if the encoding cannot be embedded

Example fix

// before
enc := "https://openaipublic.blob.core.windows.net/encodings/o200k_custom.bpe" // not embedded
// after
enc := "https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken" // embedded

// only pass URLs documented as embedded
if !supportedTiktokenURLs[u] { /* switch to a supported encoding first */ }

m, err := loader.LoadTiktokenBpe(url)
if err != nil {
	// fall back to a default embedded encoding or disable token counting
	m, err = loader.LoadTiktokenBpe(defaultTiktokenURL)
}

When it happens

Trigger: LoadTiktokenBpe called with a tiktokenBpeFile URL that is not one of the pre-embedded encoding URLs (e.g. a newer encoding or a custom .bpe path).

Common situations: Tiktoken updated to an encoding not yet embedded in this library version; custom tokenizer URL configured; typo in the encoding URL.

Related errors

• get tiktoken encoding %q: %w
• embedded tiktoken file %q not found: %w

AI-assisted analysis of alibaba/open-code-review@5cf97d0d15 (2026-09-02). Data as JSON: /api/errors/7d4cf97d167fbaf4. Report an issue: GitHub.