Could not load tiktoken encoding. Please install it with `np

Error message

Could not load tiktoken encoding. Please install it with `npm install js-tiktoken`.

What it means

The semantic markdown splitter constructor needs a tiktoken tokenizer and obtains it via encodingForModel(modelName) or getEncoding(encodingName) from js-tiktoken. If that call throws (package missing or encoding name unknown), it rethrows this error advising installation of js-tiktoken.

Source

Thrown at packages/rag/src/document/transformers/semantic-markdown.ts:42

    modelName,
    tokenizer: existingTokenizer,
    allowedSpecial = new Set(),
    disallowedSpecial = 'all',
    ...baseOptions
  }: SemanticMarkdownChunkOptions & { tokenizer?: Tiktoken } = {}) {
    super(baseOptions);

    this.joinThreshold = joinThreshold;
    this.allowedArray = allowedSpecial === 'all' ? 'all' : Array.from(allowedSpecial);
    this.disallowedArray = disallowedSpecial === 'all' ? 'all' : Array.from(disallowedSpecial);

    if (existingTokenizer) {
      this.tokenizer = existingTokenizer;
    } else {
      try {
        this.tokenizer = modelName ? encodingForModel(modelName) : getEncoding(encodingName);
      } catch {
        throw new Error('Could not load tiktoken encoding. Please install it with `npm install js-tiktoken`.');
      }
    }
  }

  private countTokens(text: string): number {
    const processedText = this.stripWhitespace ? text.trim() : text;
    return this.tokenizer.encode(processedText, this.allowedArray, this.disallowedArray).length;
  }

  private splitMarkdownByHeaders(markdown: string): MarkdownNode[] {
    const sections: MarkdownNode[] = [];
    const lines = markdown.split('\n');
    let currentContent = '';
    let currentTitle = '';
    let currentDepth = 0;
    let inCodeBlock = false;

    // Bounded quantifiers avoid the polynomial backtracking that

View on GitHub (pinned to 75dd419e61)

Solutions

Install the dependency: npm install js-tiktoken (or pnpm add js-tiktoken in the workspace).
Verify the encodingName/modelName values are valid for your js-tiktoken version (e.g. 'cl100k_base').
Ensure js-tiktoken is a regular (not optional/dev) dependency so bundlers include it in production builds.

Example fix

// before
// js-tiktoken not in package.json
const splitter = new SemanticMarkdownSplitter({ embeddingDimension: 1536 });
// after
// terminal: pnpm add js-tiktoken
const splitter = new SemanticMarkdownSplitter({ embeddingDimension: 1536 });

Defensive patterns

Strategy: fallback

Validate before calling

let tokenizerOk = true;
try {
  require('js-tiktoken').getEncoding('cl100k_base');
} catch {
  tokenizerOk = false;
}
if (!tokenizerOk) throw new Error('Install js-tiktoken before using SemanticMarkdownSplitter');

Try / catch

try {
  return new SemanticMarkdownSplitter(opts);
} catch (e) {
  if (e instanceof Error && e.message.includes('Could not load tiktoken encoding')) {
    return new CharacterTextSplitter({ separator: '

' }); // non-token fallback
  }
  throw e;
}

Prevention

List js-tiktoken as a regular dependency, not optional/dev.
Smoke-test splitter construction in CI so bundling regressions surface early.
Pin valid encoding names ('cl100k_base', etc.) in config instead of free-form strings.

When it happens

Trigger: Constructing the semantic-markdown splitter without js-tiktoken installed, or passing an unknown encodingName/modelName that js-tiktoken cannot resolve.

Common situations: js-tiktoken present at dev time but missing in the deploy bundle (tree-shaken/optional peer dep), or a typo'd encoding name like 'p50k_base_' after a tiktoken version change.

Understand the failure class

Background: "not installed", "pip install", "required for": how missing-dependency errors surface across open-source libraries — this error's family across 34 libraries.

Related errors

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/d3d4086c7751803e. Report an issue: GitHub.