{"record":{"id":"7d4cf97d167fbaf4","repo":"alibaba/open-code-review","slug":"tiktoken-encoding-file-q-is-not-embedded-and-cann","errorCode":null,"errorMessage":"tiktoken encoding file %q is not embedded and cannot be fetched offline","messagePattern":"tiktoken encoding file %q is not embedded and cannot be fetched offline","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/llm/embedded_loader.go","lineNumber":43,"sourceCode":"\ttiktoken.SetBpeLoader(loader)\n}\n\n// embeddedBpeLoader implements tiktoken.BpeLoader interface.\n// It maps known encoding URLs to local embedded files, eliminating network dependency.\ntype embeddedBpeLoader struct{}\n\nvar urlToFileMap = map[string]string{\n\t\"https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken\": \"cl100k_base.tiktoken\",\n\t\"https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken\":  \"o200k_base.tiktoken\",\n\t\"https://openaipublic.blob.core.windows.net/encodings/p50k_base.tiktoken\":   \"p50k_base.tiktoken\",\n\t\"https://openaipublic.blob.core.windows.net/encodings/r50k_base.tiktoken\":   \"r50k_base.tiktoken\",\n}\n\nfunc (l *embeddedBpeLoader) LoadTiktokenBpe(tiktokenBpeFile string) (map[string]int, error) {\n\tif localName, ok := urlToFileMap[tiktokenBpeFile]; ok {\n\t\treturn loadFromEmbed(localName)\n\t}\n\treturn nil, fmt.Errorf(\"tiktoken encoding file %q is not embedded and cannot be fetched offline\", tiktokenBpeFile)\n}\n\nfunc loadFromEmbed(filename string) (map[string]int, error) {\n\tdata, err := embedFS.ReadFile(embedPrefix + filename)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"embedded tiktoken file %q not found: %w\", filename, err)\n\t}\n\treturn parseBpeData(data)\n}\n\n// parseBpeData parses the base64-encoded BPE data format.\n// Each line: <base64-token> <rank>\nfunc parseBpeData(data []byte) (map[string]int, error) {\n\tbpeRanks := make(map[string]int)\n\tfor _, line := range strings.Split(string(data), \"\\n\") {\n\t\tline = strings.TrimSpace(line)\n\t\tif line == \"\" {\n\t\t\tcontinue","sourceCodeStart":25,"sourceCodeEnd":61,"githubUrl":"https://github.com/alibaba/open-code-review/blob/5cf97d0d15cbd41b602513c4be3bfec3cee5bf7f/internal/llm/embedded_loader.go#L25-L61","documentation":"The offline tiktoken BPE loader only serves files compiled into the binary (looked up by their canonical URL in urlToFileMap). When asked for any other encoding file there is no network fetch fallback, so it returns this error. This keeps token counting fully offline and deterministic.","triggerScenarios":"LoadTiktokenBpe called with a tiktokenBpeFile URL that is not one of the pre-embedded encoding URLs (e.g. a newer encoding or a custom .bpe path).","commonSituations":"Tiktoken updated to an encoding not yet embedded in this library version; custom tokenizer URL configured; typo in the encoding URL.","solutions":["Upgrade the library to a version that embeds the requested encoding file","Use one of the supported/embedded encoding URLs exposed by the library","If you maintain a fork, add the encoding to urlToFileMap plus the embedded assets and rebuild","Precompute token counts elsewhere if the encoding cannot be embedded"],"exampleFix":"// before\nenc := \"https://openaipublic.blob.core.windows.net/encodings/o200k_custom.bpe\" // not embedded\n// after\nenc := \"https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken\" // embedded","handlingStrategy":"fallback","validationCode":"// only pass URLs documented as embedded\nif !supportedTiktokenURLs[u] { /* switch to a supported encoding first */ }","typeGuard":null,"tryCatchPattern":"m, err := loader.LoadTiktokenBpe(url)\nif err != nil {\n\t// fall back to a default embedded encoding or disable token counting\n\tm, err = loader.LoadTiktokenBpe(defaultTiktokenURL)\n}","preventionTips":["Use only the canonical encoding URLs listed in the library docs","Pin library versions known to embed your encodings","Check release notes for newly embedded encodings before upgrading tokenizers"],"tags":["tiktoken","offline","tokenizer"],"backgroundTag":"resource-not-embedded","analyzedSha":"5cf97d0d15cbd41b602513c4be3bfec3cee5bf7f","analyzedAt":"2026-09-02T02:08:09.116Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}