{"record":{"id":"2d91fbf130ccaacb","repo":"alibaba/open-code-review","slug":"embedded-tiktoken-file-q-not-found-w","errorCode":null,"errorMessage":"embedded tiktoken file %q not found: %w","messagePattern":"embedded tiktoken file %q not found: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/llm/embedded_loader.go","lineNumber":49,"sourceCode":"\nvar urlToFileMap = map[string]string{\n\t\"https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken\": \"cl100k_base.tiktoken\",\n\t\"https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken\":  \"o200k_base.tiktoken\",\n\t\"https://openaipublic.blob.core.windows.net/encodings/p50k_base.tiktoken\":   \"p50k_base.tiktoken\",\n\t\"https://openaipublic.blob.core.windows.net/encodings/r50k_base.tiktoken\":   \"r50k_base.tiktoken\",\n}\n\nfunc (l *embeddedBpeLoader) LoadTiktokenBpe(tiktokenBpeFile string) (map[string]int, error) {\n\tif localName, ok := urlToFileMap[tiktokenBpeFile]; ok {\n\t\treturn loadFromEmbed(localName)\n\t}\n\treturn nil, fmt.Errorf(\"tiktoken encoding file %q is not embedded and cannot be fetched offline\", tiktokenBpeFile)\n}\n\nfunc loadFromEmbed(filename string) (map[string]int, error) {\n\tdata, err := embedFS.ReadFile(embedPrefix + filename)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"embedded tiktoken file %q not found: %w\", filename, err)\n\t}\n\treturn parseBpeData(data)\n}\n\n// parseBpeData parses the base64-encoded BPE data format.\n// Each line: <base64-token> <rank>\nfunc parseBpeData(data []byte) (map[string]int, error) {\n\tbpeRanks := make(map[string]int)\n\tfor _, line := range strings.Split(string(data), \"\\n\") {\n\t\tline = strings.TrimSpace(line)\n\t\tif line == \"\" {\n\t\t\tcontinue\n\t\t}\n\t\tparts := strings.SplitN(line, \" \", 2)\n\t\tif len(parts) != 2 {\n\t\t\treturn nil, fmt.Errorf(\"invalid bpe data line: %q\", line)\n\t\t}\n\t\ttoken, err := base64.StdEncoding.DecodeString(parts[0])","sourceCodeStart":31,"sourceCodeEnd":67,"githubUrl":"https://github.com/alibaba/open-code-review/blob/5cf97d0d15cbd41b602513c4be3bfec3cee5bf7f/internal/llm/embedded_loader.go#L31-L67","documentation":"The embedded loader found a mapping for the requested tiktoken URL but could not read the embedded file from the embedded filesystem (embedFS.ReadFile failed). The read error is wrapped, preserving the underlying cause — missing asset, embed path/prefix mismatch, or a build without the embedded data.","triggerScenarios":"loadFromEmbed fails reading embedPrefix+filename — typically the .bpe/.tiktoken asset was excluded from the build, the embed prefix is wrong, or a partial source tree omits the data file.","commonSituations":"go:embed pattern changed or data directory not committed; build tags excluding assets; stripped-down packaging that dropped the embedded files before build.","solutions":["Check the wrapped (%w) error for the exact ReadFile failure","Verify the embedded tiktoken data files exist in the source tree under the expected embed prefix","Rebuild the binary from a clean checkout so go:embed picks up the assets","If packaging, confirm embedding is compile-time and do not strip asset files before go build"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"m, err := loader.LoadTiktokenBpe(url)\nif err != nil {\n\tvar pathErr *fs.PathError\n\tif errors.As(err, &pathErr) {\n\t\t// embedded asset missing -> broken build; rebuild from a clean checkout\n\t}\n\treturn err\n}","preventionTips":["Build from a clean checkout so go:embed sees all asset files","Keep the embedded tiktoken data files committed to the repo","Avoid packaging steps that strip data directories before go build"],"tags":["tiktoken","embed","build","tokenizer"],"backgroundTag":"embedded-asset-missing","analyzedSha":"5cf97d0d15cbd41b602513c4be3bfec3cee5bf7f","analyzedAt":"2026-09-02T02:08:09.116Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}