{"record":{"id":"6b46a9d4c2b21d59","repo":"Mintplex-Labs/anything-llm","slug":"not-a-supported-tokenized-format","errorCode":null,"errorMessage":"Not a supported tokenized format.","messagePattern":"Not a supported tokenized format\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"server/utils/helpers/tiktoken.js","lineNumber":102,"sourceCode":"   */\n  statsFrom(input) {\n    if (typeof input === \"string\") return this.countFromString(input);\n\n    // What is going on here?\n    // https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb Item 6.\n    // The only option is to estimate. From repeated testing using the static values in the code we are always 2 off,\n    // which means as of Nov 1, 2023 the additional factor on ln: 476 changed from 3 to 5.\n    if (Array.isArray(input)) {\n      const perMessageFactorTokens = input.length * 3;\n      const tokensFromContent = input.reduce(\n        (a, b) => a + this.countFromString(b.content),\n        0\n      );\n      const diffCoefficient = 5;\n      return perMessageFactorTokens + tokensFromContent + diffCoefficient;\n    }\n\n    throw new Error(\"Not a supported tokenized format.\");\n  }\n}\n\nmodule.exports = {\n  TokenManager,\n};\n","sourceCodeStart":84,"sourceCodeEnd":109,"githubUrl":"https://github.com/Mintplex-Labs/anything-llm/blob/526360e320da9d1b36074be5ed64fe76e5bbfbbd/server/utils/helpers/tiktoken.js#L84-L109","documentation":"Thrown by TokenManager.statsFrom() when the input is neither a string nor an array. statsFrom counts tokens for a string directly, or estimates chat-message tokens for an array of {content} objects (OpenAI cookbook heuristic); any other JS type (number, object, null, undefined that slipped past the default param) has no tokenizer path. This is the type-narrowing terminal of the method.","triggerScenarios":"Calling tokenManager.statsFrom(123), statsFrom({content:'x'}), statsFrom(null), or statsFrom([{role:'user'}]) where elements lack a .content property is fine for the array branch, but a non-array object hits the throw. Most commonly a caller passes a single message object instead of an array of messages, or a number token id.","commonSituations":"A prompt-building helper that sometimes passes a single message object and sometimes an array; an LLM provider adapter that hands the raw message without wrapping; a refactor that changed the input shape but missed one call site.","solutions":["Ensure the argument is a string or an array of {content: string} objects.","Normalize at the call site: wrap a single message in an array, stringify non-strings.","Add a type guard (isString || isArray) before calling statsFrom and log the offending type.","Write a unit test covering string, array, and rejected-input cases."],"exampleFix":"// before\nconst n = tokenManager.statsFrom(message); // message is a single object -> throws\n\n// after\nconst n = tokenManager.statsFrom(\n  Array.isArray(message) ? message : String(message)\n);","handlingStrategy":"type-guard","validationCode":"function countTokens(tm, input) {\n  if (typeof input === 'string') return tm.statsFrom(input);\n  if (Array.isArray(input) && input.every(m => m && typeof m.content === 'string'))\n    return tm.statsFrom(input);\n  throw new TypeError('input must be a string or array of {content:string}');\n}","typeGuard":"function isTokenizable(v): v is string | Array<{content:string}> {\n  if (typeof v === 'string') return true;\n  return Array.isArray(v) && v.every(m => !!m && typeof m.content === 'string');\n}","tryCatchPattern":"try {\n  const n = tokenManager.statsFrom(input);\n} catch (e) {\n  if (e.message === 'Not a supported tokenized format.')\n    return fallbackEstimate(input);\n  throw e;\n}","preventionTips":["Normalize input before calling statsFrom: wrap single messages, stringify non-strings.","Add a type guard at every call site.","Unit-test string, array, and rejected-input paths."],"tags":["tokenizer","validation","llm","type-error"],"backgroundTag":null,"analyzedSha":"526360e320da9d1b36074be5ed64fe76e5bbfbbd","analyzedAt":"2026-08-13T01:45:47.170Z","schemaVersion":2},"datasetVersion":"2026-08-13T04:17:16.726Z"}