What causes "get tiktoken encoding %q: %w" in alibaba/open-code-review?

Check the encoding name in the model mapping — valid values include cl100k_base, o200k_base, p50k_base, r50k_base

get tiktoken encoding %q: %w

Error message

get tiktoken encoding %q: %w

What it means

modelTokenizerCache.getOrLoad (internal/llm/client.go:444) lazily initializes a tiktoken encoder by name via tiktoken.GetEncoding and caches it. If tiktoken cannot resolve the encoding name, the error is wrapped as "get tiktoken encoding %q: %w". It indicates the requested BPE encoding name is not one tiktoken knows (e.g. a model was mapped to a non-existent encoding).

Source

Thrown at internal/llm/client.go:444

}

func (c *modelTokenizerCache) getOrLoad(encName string) (*tiktoken.Tiktoken, error) {
	c.mu.RLock()
	if tke, ok := c.cache[encName]; ok {
		c.mu.RUnlock()
		return tke, nil
	}
	c.mu.RUnlock()

	c.mu.Lock()
	defer c.mu.Unlock()

	if tke, ok := c.cache[encName]; ok {
		return tke, nil
	}
	enc, err := tiktoken.GetEncoding(encName)
	if err != nil {
		return nil, fmt.Errorf("get tiktoken encoding %q: %w", encName, err)
	}
	c.cache[encName] = enc
	return enc, nil
}

var defaultTokenizer = newModelTokenizerCache()

func countTokensWithEncoding(text string, encName string) int {
	tke, err := defaultTokenizer.getOrLoad(encName)
	if err != nil {
		return len([]byte(text)) / 4
	}
	return len(tke.Encode(text, nil, nil))
}

func CountTokens(text string) int {
	return CountTokensForModel(text, "")
}

View on GitHub (pinned to 5cf97d0d15)

Solutions

Check the encoding name in the model mapping — valid values include cl100k_base, o200k_base, p50k_base, r50k_base
Upgrade the tiktoken library to a version that knows the encoding for your newest model
Note the caller degrades gracefully: countTokensWithEncoding falls back to len(text)/4, so this is non-fatal
Log which encName failed so the mapping table can be corrected

Example fix

// before
encName := modelToEncoding("gpt-5-nano") // returns "gpt5_base" (typo)
tokens := countTokensWithEncoding(text, encName)
// after
encName := modelToEncoding("gpt-5-nano")
if encName != "cl100k_base" && encName != "o200k_base" {
    log.Printf("unknown encoding %q, falling back to byte estimate", encName)
}
tokens := countTokensWithEncoding(text, encName)

Defensive patterns

Strategy: validation

Validate before calling

validEncodings := map[string]bool{"cl100k_base": true, "o200k_base": true, "p50k_base": true, "r50k_base": true}
if !validEncodings[encName] {
    log.Printf("encoding %q unknown; byte/4 fallback will be used", encName)
}

Try / catch

if _, err := defaultTokenizer.getOrLoad(encName); err != nil {
    var unknown *tiktoken.ErrUnknownEncoding
    if errors.As(err, &unknown) {
        log.Printf("tiktoken does not know %q; using estimate", encName)
    }
}

Prevention

Keep the tiktoken dependency updated when adding newer model support
Centralize the model→encoding mapping in one tested table
Only allow encoding names from a known whitelist in config
Remember the caller falls back to len(bytes)/4 — treat this as degraded accuracy, not failure

When it happens

Trigger: countTokensWithEncoding is called with an encName that tiktoken.GetEncoding rejects — a typo, an empty string, or an encoding name tied to a newer model that the vendored tiktoken version does not know yet.

Common situations: A new OpenAI model ships with a fresh encoding name but the project's tiktoken-go version predates it; a config file or model-mapping table holds an invalid encoding string; a config typo like "o200_base" instead of "o200k_base".

Related errors

AI-assisted analysis of alibaba/open-code-review@5cf97d0d15 (2026-09-02). Data as JSON: /api/errors/f05bc0f45ac84f27. Report an issue: GitHub.