Mintplex-Labs/anything-llm · error · Error

Not a supported tokenized format.

Error message

Not a supported tokenized format.

What it means

Thrown by TokenManager.statsFrom() when the input is neither a string nor an array. statsFrom counts tokens for a string directly, or estimates chat-message tokens for an array of {content} objects (OpenAI cookbook heuristic); any other JS type (number, object, null, undefined that slipped past the default param) has no tokenizer path. This is the type-narrowing terminal of the method.

Source

Thrown at server/utils/helpers/tiktoken.js:102

   */
  statsFrom(input) {
    if (typeof input === "string") return this.countFromString(input);

    // What is going on here?
    // https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb Item 6.
    // The only option is to estimate. From repeated testing using the static values in the code we are always 2 off,
    // which means as of Nov 1, 2023 the additional factor on ln: 476 changed from 3 to 5.
    if (Array.isArray(input)) {
      const perMessageFactorTokens = input.length * 3;
      const tokensFromContent = input.reduce(
        (a, b) => a + this.countFromString(b.content),
        0
      );
      const diffCoefficient = 5;
      return perMessageFactorTokens + tokensFromContent + diffCoefficient;
    }

    throw new Error("Not a supported tokenized format.");
  }
}

module.exports = {
  TokenManager,
};

View on GitHub (pinned to 526360e320)

Solutions

1. Ensure the argument is a string or an array of {content: string} objects.
2. Normalize at the call site: wrap a single message in an array, stringify non-strings.
3. Add a type guard (isString || isArray) before calling statsFrom and log the offending type.
4. Write a unit test covering string, array, and rejected-input cases.

Example fix

// before
const n = tokenManager.statsFrom(message); // message is a single object -> throws

// after
const n = tokenManager.statsFrom(
  Array.isArray(message) ? message : String(message)
);

function countTokens(tm, input) {
  if (typeof input === 'string') return tm.statsFrom(input);
  if (Array.isArray(input) && input.every(m => m && typeof m.content === 'string'))
    return tm.statsFrom(input);
  throw new TypeError('input must be a string or array of {content:string}');
}

function isTokenizable(v): v is string | Array<{content:string}> {
  if (typeof v === 'string') return true;
  return Array.isArray(v) && v.every(m => !!m && typeof m.content === 'string');
}

try {
  const n = tokenManager.statsFrom(input);
} catch (e) {
  if (e.message === 'Not a supported tokenized format.')
    return fallbackEstimate(input);
  throw e;
}

When it happens

Trigger: Calling tokenManager.statsFrom(123), statsFrom({content:'x'}), statsFrom(null), or statsFrom([{role:'user'}]) where elements lack a .content property is fine for the array branch, but a non-array object hits the throw. Most commonly a caller passes a single message object instead of an array of messages, or a number token id.

Common situations: A prompt-building helper that sometimes passes a single message object and sometimes an array; an LLM provider adapter that hands the raw message without wrapping; a refactor that changed the input shape but missed one call site.

Related errors

• Type "${type}" is not a valid type to sync.
• Invalid link provided
• Invalid source property provided
• No base URL was set.
• FFMPEG candidate path is not valid ffmpeg binary.

AI-assisted analysis of Mintplex-Labs/anything-llm@526360e320 (2026-08-13). Data as JSON: /api/errors/6b46a9d4c2b21d59. Report an issue: GitHub.