{"record":{"id":"4acc5e34b07dbe3d","repo":"yamadashy/repomix","slug":"failed-to-count-tokens-error-message","errorCode":null,"errorMessage":"Failed to count tokens. error: ${message}","messagePattern":"Failed to count tokens\\. error: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/core/metrics/TokenCounter.ts","lineNumber":86,"sourceCode":"      throw new Error('TokenCounter not initialized. Call init() first.');\n    }\n\n    try {\n      // Use PLAIN_TEXT_OPTIONS to treat all content as ordinary text,\n      // skipping gpt-tokenizer's default regex scan for special tokens.\n      return this.countFn(content, PLAIN_TEXT_OPTIONS);\n    } catch (error) {\n      let message = '';\n      if (error instanceof Error) {\n        message = error.message;\n      } else {\n        message = String(error);\n      }\n\n      if (filePath) {\n        logger.warn(`Failed to count tokens. path: ${filePath}, error: ${message}`);\n      } else {\n        logger.warn(`Failed to count tokens. error: ${message}`);\n      }\n\n      return 0;\n    }\n  }\n\n  // No-op: gpt-tokenizer is pure JS, no WASM resources to free\n  public free(): void {}\n}\n","sourceCodeStart":68,"sourceCodeEnd":96,"githubUrl":"https://github.com/yamadashy/repomix/blob/f465ad909315a22120636baf03fa5e28701a50cb/src/core/metrics/TokenCounter.ts#L68-L96","documentation":"The no-filePath variant of the same TokenCounter.countTokens catch block in src/core/metrics/TokenCounter.ts: when the failed count call did not supply a filePath, the warning omits the path and just logs the underlying error message. Returns 0 without throwing.","triggerScenarios":"countTokens is invoked without a filePath argument and the tokenizer throws — e.g. counting raw content snippets, CLI --token-count-tree aggregates, or direct library use of count() with undefined path.","commonSituations":"Programmatic use of the TokenCounter on arbitrary strings; internal metrics calls that pass only content; tokenizer failing on special characters in dynamically generated text.","solutions":["Check the logged error message for the root cause and sanitize the input string (remove lone surrogates / invalid sequences).","Pass a filePath when counting file content so future warnings identify the source.","Confirm the content argument is a defined non-empty string at the call site.","Ignore if purely cosmetic — the call returns 0 and the pipeline continues."],"exampleFix":"// before\ntokenCounter.count(maybeUndefined);\n// after\nif (typeof content === 'string') tokenCounter.count(content);","handlingStrategy":"type-guard","validationCode":"if (typeof content !== 'string') throw new TypeError('content must be a string before counting tokens');","typeGuard":"const isString = (v) => typeof v === 'string';","tryCatchPattern":"let tokens = 0;\ntry { tokens = tokenCounter.count(content); } catch (e) { logger.warn(`token count failed: ${e.message}`); }","preventionTips":["Validate the content is a defined string before counting.","Pass a filePath whenever one exists so failures are traceable.","Sanitize dynamically generated text for exotic characters."],"tags":["token-counting","tokenizer","metrics"],"backgroundTag":"token-count-failed","analyzedSha":"f465ad909315a22120636baf03fa5e28701a50cb","analyzedAt":"2026-08-29T01:27:42.024Z","schemaVersion":2},"datasetVersion":"2026-08-29T02:17:18.158Z"}