alibaba/open-code-review · error

embedded tiktoken file %q not found: %w

Error message

embedded tiktoken file %q not found: %w

What it means

The embedded loader found a mapping for the requested tiktoken URL but could not read the embedded file from the embedded filesystem (embedFS.ReadFile failed). The read error is wrapped, preserving the underlying cause — missing asset, embed path/prefix mismatch, or a build without the embedded data.

Source

Thrown at internal/llm/embedded_loader.go:49

var urlToFileMap = map[string]string{
	"https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken": "cl100k_base.tiktoken",
	"https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken":  "o200k_base.tiktoken",
	"https://openaipublic.blob.core.windows.net/encodings/p50k_base.tiktoken":   "p50k_base.tiktoken",
	"https://openaipublic.blob.core.windows.net/encodings/r50k_base.tiktoken":   "r50k_base.tiktoken",
}

func (l *embeddedBpeLoader) LoadTiktokenBpe(tiktokenBpeFile string) (map[string]int, error) {
	if localName, ok := urlToFileMap[tiktokenBpeFile]; ok {
		return loadFromEmbed(localName)
	}
	return nil, fmt.Errorf("tiktoken encoding file %q is not embedded and cannot be fetched offline", tiktokenBpeFile)
}

func loadFromEmbed(filename string) (map[string]int, error) {
	data, err := embedFS.ReadFile(embedPrefix + filename)
	if err != nil {
		return nil, fmt.Errorf("embedded tiktoken file %q not found: %w", filename, err)
	}
	return parseBpeData(data)
}

// parseBpeData parses the base64-encoded BPE data format.
// Each line: <base64-token> <rank>
func parseBpeData(data []byte) (map[string]int, error) {
	bpeRanks := make(map[string]int)
	for _, line := range strings.Split(string(data), "\n") {
		line = strings.TrimSpace(line)
		if line == "" {
			continue
		}
		parts := strings.SplitN(line, " ", 2)
		if len(parts) != 2 {
			return nil, fmt.Errorf("invalid bpe data line: %q", line)
		}
		token, err := base64.StdEncoding.DecodeString(parts[0])

View on GitHub (pinned to 5cf97d0d15)

Solutions

1. Check the wrapped (%w) error for the exact ReadFile failure
2. Verify the embedded tiktoken data files exist in the source tree under the expected embed prefix
3. Rebuild the binary from a clean checkout so go:embed picks up the assets
4. If packaging, confirm embedding is compile-time and do not strip asset files before go build

m, err := loader.LoadTiktokenBpe(url)
if err != nil {
	var pathErr *fs.PathError
	if errors.As(err, &pathErr) {
		// embedded asset missing -> broken build; rebuild from a clean checkout
	}
	return err
}

When it happens

Trigger: loadFromEmbed fails reading embedPrefix+filename — typically the .bpe/.tiktoken asset was excluded from the build, the embed prefix is wrong, or a partial source tree omits the data file.

Common situations: go:embed pattern changed or data directory not committed; build tags excluding assets; stripped-down packaging that dropped the embedded files before build.

Related errors

• read embedded system_rules.json: %w
• get tiktoken encoding %q: %w
• tiktoken encoding file %q is not embedded and cannot be fetc
• unmarshal default system rules: %w
• read default rule file %q: %w

AI-assisted analysis of alibaba/open-code-review@5cf97d0d15 (2026-09-02). Data as JSON: /api/errors/2d91fbf130ccaacb. Report an issue: GitHub.