yamadashy/repomix · warning

Failed to count tokens. error: ${message}

Error message

Failed to count tokens. error: ${message}

What it means

The no-filePath variant of the same TokenCounter.countTokens catch block in src/core/metrics/TokenCounter.ts: when the failed count call did not supply a filePath, the warning omits the path and just logs the underlying error message. Returns 0 without throwing.

Source

Thrown at src/core/metrics/TokenCounter.ts:86

      throw new Error('TokenCounter not initialized. Call init() first.');
    }

    try {
      // Use PLAIN_TEXT_OPTIONS to treat all content as ordinary text,
      // skipping gpt-tokenizer's default regex scan for special tokens.
      return this.countFn(content, PLAIN_TEXT_OPTIONS);
    } catch (error) {
      let message = '';
      if (error instanceof Error) {
        message = error.message;
      } else {
        message = String(error);
      }

      if (filePath) {
        logger.warn(`Failed to count tokens. path: ${filePath}, error: ${message}`);
      } else {
        logger.warn(`Failed to count tokens. error: ${message}`);
      }

      return 0;
    }
  }

  // No-op: gpt-tokenizer is pure JS, no WASM resources to free
  public free(): void {}
}

View on GitHub (pinned to f465ad9093)

Solutions

1. Check the logged error message for the root cause and sanitize the input string (remove lone surrogates / invalid sequences).
2. Pass a filePath when counting file content so future warnings identify the source.
3. Confirm the content argument is a defined non-empty string at the call site.
4. Ignore if purely cosmetic — the call returns 0 and the pipeline continues.

Example fix

// before
tokenCounter.count(maybeUndefined);
// after
if (typeof content === 'string') tokenCounter.count(content);

if (typeof content !== 'string') throw new TypeError('content must be a string before counting tokens');

const isString = (v) => typeof v === 'string';

let tokens = 0;
try { tokens = tokenCounter.count(content); } catch (e) { logger.warn(`token count failed: ${e.message}`); }

When it happens

Trigger: countTokens is invoked without a filePath argument and the tokenizer throws — e.g. counting raw content snippets, CLI --token-count-tree aggregates, or direct library use of count() with undefined path.

Common situations: Programmatic use of the TokenCounter on arbitrary strings; internal metrics calls that pass only content; tokenizer failing on special characters in dynamically generated text.

Related errors

• Failed to count tokens. path: ${filePath}, error: ${message}

AI-assisted analysis of yamadashy/repomix@f465ad9093 (2026-08-29). Data as JSON: /api/errors/4acc5e34b07dbe3d. Report an issue: GitHub.