{"record":{"id":"f05bc0f45ac84f27","repo":"alibaba/open-code-review","slug":"get-tiktoken-encoding-q-w","errorCode":null,"errorMessage":"get tiktoken encoding %q: %w","messagePattern":"get tiktoken encoding %q: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"internal/llm/client.go","lineNumber":444,"sourceCode":"}\n\nfunc (c *modelTokenizerCache) getOrLoad(encName string) (*tiktoken.Tiktoken, error) {\n\tc.mu.RLock()\n\tif tke, ok := c.cache[encName]; ok {\n\t\tc.mu.RUnlock()\n\t\treturn tke, nil\n\t}\n\tc.mu.RUnlock()\n\n\tc.mu.Lock()\n\tdefer c.mu.Unlock()\n\n\tif tke, ok := c.cache[encName]; ok {\n\t\treturn tke, nil\n\t}\n\tenc, err := tiktoken.GetEncoding(encName)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"get tiktoken encoding %q: %w\", encName, err)\n\t}\n\tc.cache[encName] = enc\n\treturn enc, nil\n}\n\nvar defaultTokenizer = newModelTokenizerCache()\n\nfunc countTokensWithEncoding(text string, encName string) int {\n\ttke, err := defaultTokenizer.getOrLoad(encName)\n\tif err != nil {\n\t\treturn len([]byte(text)) / 4\n\t}\n\treturn len(tke.Encode(text, nil, nil))\n}\n\nfunc CountTokens(text string) int {\n\treturn CountTokensForModel(text, \"\")\n}","sourceCodeStart":426,"sourceCodeEnd":462,"githubUrl":"https://github.com/alibaba/open-code-review/blob/5cf97d0d15cbd41b602513c4be3bfec3cee5bf7f/internal/llm/client.go#L426-L462","documentation":"modelTokenizerCache.getOrLoad (internal/llm/client.go:444) lazily initializes a tiktoken encoder by name via tiktoken.GetEncoding and caches it. If tiktoken cannot resolve the encoding name, the error is wrapped as \"get tiktoken encoding %q: %w\". It indicates the requested BPE encoding name is not one tiktoken knows (e.g. a model was mapped to a non-existent encoding).","triggerScenarios":"countTokensWithEncoding is called with an encName that tiktoken.GetEncoding rejects — a typo, an empty string, or an encoding name tied to a newer model that the vendored tiktoken version does not know yet.","commonSituations":"A new OpenAI model ships with a fresh encoding name but the project's tiktoken-go version predates it; a config file or model-mapping table holds an invalid encoding string; a config typo like \"o200_base\" instead of \"o200k_base\".","solutions":["Check the encoding name in the model mapping — valid values include cl100k_base, o200k_base, p50k_base, r50k_base","Upgrade the tiktoken library to a version that knows the encoding for your newest model","Note the caller degrades gracefully: countTokensWithEncoding falls back to len(text)/4, so this is non-fatal","Log which encName failed so the mapping table can be corrected"],"exampleFix":"// before\nencName := modelToEncoding(\"gpt-5-nano\") // returns \"gpt5_base\" (typo)\ntokens := countTokensWithEncoding(text, encName)\n// after\nencName := modelToEncoding(\"gpt-5-nano\")\nif encName != \"cl100k_base\" && encName != \"o200k_base\" {\n    log.Printf(\"unknown encoding %q, falling back to byte estimate\", encName)\n}\ntokens := countTokensWithEncoding(text, encName)","handlingStrategy":"validation","validationCode":"validEncodings := map[string]bool{\"cl100k_base\": true, \"o200k_base\": true, \"p50k_base\": true, \"r50k_base\": true}\nif !validEncodings[encName] {\n    log.Printf(\"encoding %q unknown; byte/4 fallback will be used\", encName)\n}","typeGuard":null,"tryCatchPattern":"if _, err := defaultTokenizer.getOrLoad(encName); err != nil {\n    var unknown *tiktoken.ErrUnknownEncoding\n    if errors.As(err, &unknown) {\n        log.Printf(\"tiktoken does not know %q; using estimate\", encName)\n    }\n}","preventionTips":["Keep the tiktoken dependency updated when adding newer model support","Centralize the model→encoding mapping in one tested table","Only allow encoding names from a known whitelist in config","Remember the caller falls back to len(bytes)/4 — treat this as degraded accuracy, not failure"],"tags":["tiktoken","tokenizer","configuration","go"],"backgroundTag":"unknown-encoding-name","analyzedSha":"5cf97d0d15cbd41b602513c4be3bfec3cee5bf7f","analyzedAt":"2026-09-02T02:08:09.116Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}