{"record":{"id":"3ae3b48b1f4b2eb6","repo":"mastra-ai/mastra","slug":"could-not-load-tiktoken-encoding-please-install-i-3ae3b4","errorCode":null,"errorMessage":"Could not load tiktoken encoding. Please install it with `npm install js-tiktoken`.","messagePattern":"Could not load tiktoken encoding\\. Please install it with `npm install js-tiktoken`\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"packages/rag/src/document/transformers/token.ts","lineNumber":62,"sourceCode":"    disallowedSpecial = 'all',\n    options = {},\n  }: {\n    encodingName?: TiktokenEncoding;\n    modelName?: TiktokenModel;\n    tokenizer?: Tiktoken;\n    allowedSpecial?: Set<string> | 'all';\n    disallowedSpecial?: Set<string> | 'all';\n    options: TokenChunkOptions;\n  }) {\n    super(options);\n\n    if (existingTokenizer) {\n      this.tokenizer = existingTokenizer;\n    } else {\n      try {\n        this.tokenizer = modelName ? encodingForModel(modelName) : getEncoding(encodingName);\n      } catch {\n        throw new Error('Could not load tiktoken encoding. ' + 'Please install it with `npm install js-tiktoken`.');\n      }\n    }\n\n    this.allowedArray = allowedSpecial === 'all' ? 'all' : Array.from(allowedSpecial);\n    this.disallowedArray = disallowedSpecial === 'all' ? 'all' : Array.from(disallowedSpecial);\n  }\n\n  splitText({ text }: { text: string }): string[] {\n    const encode = (text: string): number[] => {\n      const processedText = this.stripWhitespace ? text.trim() : text;\n      return Array.from(this.tokenizer.encode(processedText, this.allowedArray, this.disallowedArray));\n    };\n\n    const decode = (tokens: number[]): string => {\n      const text = this.tokenizer.decode(tokens);\n      return this.stripWhitespace ? text.trim() : text;\n    };\n","sourceCodeStart":44,"sourceCodeEnd":80,"githubUrl":"https://github.com/mastra-ai/mastra/blob/75dd419e613fe9c39f846ffc500716141b74fda6/packages/rag/src/document/transformers/token.ts#L44-L80","documentation":"TokenTransformer's constructor builds a tiktoken tokenizer via encodingForModel or getEncoding when no existing tokenizer is supplied. If tiktoken cannot resolve the model/encoding (typically because js-tiktoken is not installed), the failure is rethrown as this installation error. It guards the core dependency of token-based chunking.","triggerScenarios":"new TokenTransformer({...}) without passing existingTokenizer when js-tiktoken is absent, or when modelName/encodingName is not recognized by the installed tiktoken data.","commonSituations":"js-tiktoken missing from dependencies after a fresh install or CI image; typo in model name; js-tiktoken version too old for a newer model; passing a fake encoding name like 'cl100k' instead of 'cl100k_base'.","solutions":["Run `npm install js-tiktoken`.","Pass a known-good encodingName such as 'cl100k_base' instead of a modelName if the model isn't recognized.","Update js-tiktoken to the latest version.","Reuse an already-constructed Tiktoken instance via the existingTokenizer option to avoid re-initialization."],"exampleFix":"// before\nconst t = new TokenTransformer({});\n// after\nnpm install js-tiktoken\nconst t = new TokenTransformer({ encodingName: 'cl100k_base' });","handlingStrategy":"try-catch","validationCode":"let tiktokenReady = false;\ntry { require.resolve('js-tiktoken'); tiktokenReady = true; } catch {}\nif (!tiktokenReady) throw new Error('TokenTransformer requires js-tiktoken: npm install js-tiktoken');","typeGuard":null,"tryCatchPattern":"try {\n  const t = new TokenTransformer({ encodingName: 'cl100k_base' });\n} catch (e) {\n  if ((e as Error).message.includes('tiktoken encoding')) {\n    console.error('js-tiktoken missing or unknown encoding; run npm install js-tiktoken');\n  }\n  throw e;\n}","preventionTips":["Ensure js-tiktoken is a runtime dependency of the package that imports TokenTransformer.","Use standard encoding names ('cl100k_base', 'o200k_base').","Reuse an existing Tiktoken instance via existingTokenizer to isolate init failures.","Test tokenizer construction at app startup (fail fast) rather than at first chunk request."],"tags":["rag","tiktoken","missing-dependency","tokenizer"],"backgroundTag":"missing-npm-dependency","analyzedSha":"75dd419e613fe9c39f846ffc500716141b74fda6","analyzedAt":"2026-08-30T00:15:31.844Z","schemaVersion":2},"datasetVersion":"2026-08-30T03:17:51.788Z"}