{"record":{"id":"9024545f06cdbe88","repo":"yamadashy/repomix","slug":"failed-to-count-tokens-path-filepath-error","errorCode":null,"errorMessage":"Failed to count tokens. path: ${filePath}, error: ${message}","messagePattern":"Failed to count tokens\\. path: (.+?), error: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/core/metrics/TokenCounter.ts","lineNumber":84,"sourceCode":"  public countTokens(content: string, filePath?: string): number {\n    if (!this.countFn) {\n      throw new Error('TokenCounter not initialized. Call init() first.');\n    }\n\n    try {\n      // Use PLAIN_TEXT_OPTIONS to treat all content as ordinary text,\n      // skipping gpt-tokenizer's default regex scan for special tokens.\n      return this.countFn(content, PLAIN_TEXT_OPTIONS);\n    } catch (error) {\n      let message = '';\n      if (error instanceof Error) {\n        message = error.message;\n      } else {\n        message = String(error);\n      }\n\n      if (filePath) {\n        logger.warn(`Failed to count tokens. path: ${filePath}, error: ${message}`);\n      } else {\n        logger.warn(`Failed to count tokens. error: ${message}`);\n      }\n\n      return 0;\n    }\n  }\n\n  // No-op: gpt-tokenizer is pure JS, no WASM resources to free\n  public free(): void {}\n}\n","sourceCodeStart":66,"sourceCodeEnd":96,"githubUrl":"https://github.com/yamadashy/repomix/blob/f465ad909315a22120636baf03fa5e28701a50cb/src/core/metrics/TokenCounter.ts#L66-L96","documentation":"TokenCounter.countTokens wraps its tiktoken/GPT-tokenizer counting in try/catch and logs this warning when counting fails for a specific file path, returning 0 tokens instead of throwing. The metrics output will show 0 for that file while the rest of the pack proceeds.","triggerScenarios":"The tokenizer throws on the content of the file at <filePath> — typically content containing characters or sequences the tokenizer cannot encode, or invalid input passed to count() with a filePath argument.","commonSituations":"Packing repos with unusual encodings, surrogate-pair-heavy or malformed strings, or very large single-line files; count() called on content read with a mismatched encoding.","solutions":["Inspect the logged `error: ${message}` to identify the offending file and re-encode/clean its content (e.g. strip invalid UTF-8).","Exclude the problematic file via config exclude patterns if its token count is not needed.","Verify the content passed to countTokens is a string (not Buffer/undefined) at the call site.","Treat the 0 result as a metrics-only degradation; packing still succeeds, so no pack-level change is required."],"exampleFix":"// before: Buffer passed straight through\nconst tokens = tokenCounter.count(fs.readFileSync(p), p);\n// after: decode to string first\nconst tokens = tokenCounter.count(fs.readFileSync(p, 'utf8'), p);","handlingStrategy":"type-guard","validationCode":"if (typeof content !== 'string' || content.length === 0) throw new TypeError('countTokens expects a non-empty string');","typeGuard":"const isCountable = (c) => typeof c === 'string' && !/\\uD800/.test(c.replace(/[\\uD800-\\uDBFF](?![\\uDC00-\\uDFFF])/g, ''));","tryCatchPattern":"let tokens = 0;\ntry { tokens = tokenCounter.count(content, filePath); } catch (e) { logger.warn(`token count failed for ${filePath}: ${e.message}`); }","preventionTips":["Always pass decoded UTF-8 strings, never Buffers or undefined.","Pass filePath so warnings identify the source file.","Strip lone surrogates / invalid sequences from content before counting."],"tags":["token-counting","tokenizer","metrics","encoding"],"backgroundTag":"token-count-failed","analyzedSha":"f465ad909315a22120636baf03fa5e28701a50cb","analyzedAt":"2026-08-29T01:27:42.024Z","schemaVersion":2},"datasetVersion":"2026-08-29T02:17:18.158Z"}