yamadashy/repomix · warning

Failed to count tokens. path: ${filePath}, error: ${message}

Error message

Failed to count tokens. path: ${filePath}, error: ${message}

What it means

TokenCounter.countTokens wraps its tiktoken/GPT-tokenizer counting in try/catch and logs this warning when counting fails for a specific file path, returning 0 tokens instead of throwing. The metrics output will show 0 for that file while the rest of the pack proceeds.

Source

Thrown at src/core/metrics/TokenCounter.ts:84

  public countTokens(content: string, filePath?: string): number {
    if (!this.countFn) {
      throw new Error('TokenCounter not initialized. Call init() first.');
    }

    try {
      // Use PLAIN_TEXT_OPTIONS to treat all content as ordinary text,
      // skipping gpt-tokenizer's default regex scan for special tokens.
      return this.countFn(content, PLAIN_TEXT_OPTIONS);
    } catch (error) {
      let message = '';
      if (error instanceof Error) {
        message = error.message;
      } else {
        message = String(error);
      }

      if (filePath) {
        logger.warn(`Failed to count tokens. path: ${filePath}, error: ${message}`);
      } else {
        logger.warn(`Failed to count tokens. error: ${message}`);
      }

      return 0;
    }
  }

  // No-op: gpt-tokenizer is pure JS, no WASM resources to free
  public free(): void {}
}

View on GitHub (pinned to f465ad9093)

Solutions

1. Inspect the logged `error: ${message}` to identify the offending file and re-encode/clean its content (e.g. strip invalid UTF-8).
2. Exclude the problematic file via config exclude patterns if its token count is not needed.
3. Verify the content passed to countTokens is a string (not Buffer/undefined) at the call site.
4. Treat the 0 result as a metrics-only degradation; packing still succeeds, so no pack-level change is required.

Example fix

// before: Buffer passed straight through
const tokens = tokenCounter.count(fs.readFileSync(p), p);
// after: decode to string first
const tokens = tokenCounter.count(fs.readFileSync(p, 'utf8'), p);

if (typeof content !== 'string' || content.length === 0) throw new TypeError('countTokens expects a non-empty string');

const isCountable = (c) => typeof c === 'string' && !/\uD800/.test(c.replace(/[\uD800-\uDBFF](?![\uDC00-\uDFFF])/g, ''));

let tokens = 0;
try { tokens = tokenCounter.count(content, filePath); } catch (e) { logger.warn(`token count failed for ${filePath}: ${e.message}`); }

When it happens

Trigger: The tokenizer throws on the content of the file at <filePath> — typically content containing characters or sequences the tokenizer cannot encode, or invalid input passed to count() with a filePath argument.

Common situations: Packing repos with unusual encodings, surrogate-pair-heavy or malformed strings, or very large single-line files; count() called on content read with a mismatched encoding.

Related errors

• Failed to count tokens. error: ${message}

AI-assisted analysis of yamadashy/repomix@f465ad9093 (2026-08-29). Data as JSON: /api/errors/9024545f06cdbe88. Report an issue: GitHub.