Could not load tiktoken encoding. Please install it with `np

Error message

Could not load tiktoken encoding. Please install it with `npm install js-tiktoken`.

What it means

TokenTransformer's constructor builds a tiktoken tokenizer via encodingForModel or getEncoding when no existing tokenizer is supplied. If tiktoken cannot resolve the model/encoding (typically because js-tiktoken is not installed), the failure is rethrown as this installation error. It guards the core dependency of token-based chunking.

Source

Thrown at packages/rag/src/document/transformers/token.ts:62

    disallowedSpecial = 'all',
    options = {},
  }: {
    encodingName?: TiktokenEncoding;
    modelName?: TiktokenModel;
    tokenizer?: Tiktoken;
    allowedSpecial?: Set<string> | 'all';
    disallowedSpecial?: Set<string> | 'all';
    options: TokenChunkOptions;
  }) {
    super(options);

    if (existingTokenizer) {
      this.tokenizer = existingTokenizer;
    } else {
      try {
        this.tokenizer = modelName ? encodingForModel(modelName) : getEncoding(encodingName);
      } catch {
        throw new Error('Could not load tiktoken encoding. ' + 'Please install it with `npm install js-tiktoken`.');
      }
    }

    this.allowedArray = allowedSpecial === 'all' ? 'all' : Array.from(allowedSpecial);
    this.disallowedArray = disallowedSpecial === 'all' ? 'all' : Array.from(disallowedSpecial);
  }

  splitText({ text }: { text: string }): string[] {
    const encode = (text: string): number[] => {
      const processedText = this.stripWhitespace ? text.trim() : text;
      return Array.from(this.tokenizer.encode(processedText, this.allowedArray, this.disallowedArray));
    };

    const decode = (tokens: number[]): string => {
      const text = this.tokenizer.decode(tokens);
      return this.stripWhitespace ? text.trim() : text;
    };

View on GitHub (pinned to 75dd419e61)

Solutions

Run `npm install js-tiktoken`.
Pass a known-good encodingName such as 'cl100k_base' instead of a modelName if the model isn't recognized.
Update js-tiktoken to the latest version.
Reuse an already-constructed Tiktoken instance via the existingTokenizer option to avoid re-initialization.

Example fix

// before
const t = new TokenTransformer({});
// after
npm install js-tiktoken
const t = new TokenTransformer({ encodingName: 'cl100k_base' });

Defensive patterns

Strategy: try-catch

Validate before calling

let tiktokenReady = false;
try { require.resolve('js-tiktoken'); tiktokenReady = true; } catch {}
if (!tiktokenReady) throw new Error('TokenTransformer requires js-tiktoken: npm install js-tiktoken');

Try / catch

try {
  const t = new TokenTransformer({ encodingName: 'cl100k_base' });
} catch (e) {
  if ((e as Error).message.includes('tiktoken encoding')) {
    console.error('js-tiktoken missing or unknown encoding; run npm install js-tiktoken');
  }
  throw e;
}

Prevention

Ensure js-tiktoken is a runtime dependency of the package that imports TokenTransformer.
Use standard encoding names ('cl100k_base', 'o200k_base').
Reuse an existing Tiktoken instance via existingTokenizer to isolate init failures.
Test tokenizer construction at app startup (fail fast) rather than at first chunk request.

When it happens

Trigger: new TokenTransformer({...}) without passing existingTokenizer when js-tiktoken is absent, or when modelName/encodingName is not recognized by the installed tiktoken data.

Common situations: js-tiktoken missing from dependencies after a fresh install or CI image; typo in model name; js-tiktoken version too old for a newer model; passing a fake encoding name like 'cl100k' instead of 'cl100k_base'.

Related errors

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/3ae3b48b1f4b2eb6. Report an issue: GitHub.