{"record":{"id":"d3d4086c7751803e","repo":"mastra-ai/mastra","slug":"could-not-load-tiktoken-encoding-please-install-i","errorCode":null,"errorMessage":"Could not load tiktoken encoding. Please install it with `npm install js-tiktoken`.","messagePattern":"Could not load tiktoken encoding\\. Please install it with `npm install js-tiktoken`\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"packages/rag/src/document/transformers/semantic-markdown.ts","lineNumber":42,"sourceCode":"    modelName,\n    tokenizer: existingTokenizer,\n    allowedSpecial = new Set(),\n    disallowedSpecial = 'all',\n    ...baseOptions\n  }: SemanticMarkdownChunkOptions & { tokenizer?: Tiktoken } = {}) {\n    super(baseOptions);\n\n    this.joinThreshold = joinThreshold;\n    this.allowedArray = allowedSpecial === 'all' ? 'all' : Array.from(allowedSpecial);\n    this.disallowedArray = disallowedSpecial === 'all' ? 'all' : Array.from(disallowedSpecial);\n\n    if (existingTokenizer) {\n      this.tokenizer = existingTokenizer;\n    } else {\n      try {\n        this.tokenizer = modelName ? encodingForModel(modelName) : getEncoding(encodingName);\n      } catch {\n        throw new Error('Could not load tiktoken encoding. Please install it with `npm install js-tiktoken`.');\n      }\n    }\n  }\n\n  private countTokens(text: string): number {\n    const processedText = this.stripWhitespace ? text.trim() : text;\n    return this.tokenizer.encode(processedText, this.allowedArray, this.disallowedArray).length;\n  }\n\n  private splitMarkdownByHeaders(markdown: string): MarkdownNode[] {\n    const sections: MarkdownNode[] = [];\n    const lines = markdown.split('\\n');\n    let currentContent = '';\n    let currentTitle = '';\n    let currentDepth = 0;\n    let inCodeBlock = false;\n\n    // Bounded quantifiers avoid the polynomial backtracking that","sourceCodeStart":24,"sourceCodeEnd":60,"githubUrl":"https://github.com/mastra-ai/mastra/blob/75dd419e613fe9c39f846ffc500716141b74fda6/packages/rag/src/document/transformers/semantic-markdown.ts#L24-L60","documentation":"The semantic markdown splitter constructor needs a tiktoken tokenizer and obtains it via encodingForModel(modelName) or getEncoding(encodingName) from js-tiktoken. If that call throws (package missing or encoding name unknown), it rethrows this error advising installation of js-tiktoken.","triggerScenarios":"Constructing the semantic-markdown splitter without js-tiktoken installed, or passing an unknown encodingName/modelName that js-tiktoken cannot resolve.","commonSituations":"js-tiktoken present at dev time but missing in the deploy bundle (tree-shaken/optional peer dep), or a typo'd encoding name like 'p50k_base_' after a tiktoken version change.","solutions":["Install the dependency: npm install js-tiktoken (or pnpm add js-tiktoken in the workspace).","Verify the encodingName/modelName values are valid for your js-tiktoken version (e.g. 'cl100k_base').","Ensure js-tiktoken is a regular (not optional/dev) dependency so bundlers include it in production builds."],"exampleFix":"// before\n// js-tiktoken not in package.json\nconst splitter = new SemanticMarkdownSplitter({ embeddingDimension: 1536 });\n// after\n// terminal: pnpm add js-tiktoken\nconst splitter = new SemanticMarkdownSplitter({ embeddingDimension: 1536 });","handlingStrategy":"fallback","validationCode":"let tokenizerOk = true;\ntry {\n  require('js-tiktoken').getEncoding('cl100k_base');\n} catch {\n  tokenizerOk = false;\n}\nif (!tokenizerOk) throw new Error('Install js-tiktoken before using SemanticMarkdownSplitter');","typeGuard":null,"tryCatchPattern":"try {\n  return new SemanticMarkdownSplitter(opts);\n} catch (e) {\n  if (e instanceof Error && e.message.includes('Could not load tiktoken encoding')) {\n    return new CharacterTextSplitter({ separator: '\n\n' }); // non-token fallback\n  }\n  throw e;\n}","preventionTips":["List js-tiktoken as a regular dependency, not optional/dev.","Smoke-test splitter construction in CI so bundling regressions surface early.","Pin valid encoding names ('cl100k_base', etc.) in config instead of free-form strings."],"tags":["dependency","tokenizer","rag"],"backgroundTag":"missing-dependency","analyzedSha":"75dd419e613fe9c39f846ffc500716141b74fda6","analyzedAt":"2026-08-30T00:15:31.844Z","schemaVersion":2},"datasetVersion":"2026-08-30T03:17:51.788Z"}