mastra-ai/mastra · error · Error

Tokens map file not found at ${tokensMapPath}

Error message

Tokens map file not found at ${tokensMapPath}

What it means

The loader reads special_tokens_map.json to know which tokens are special (CLS, SEP, padding, etc.) when constructing the tokenizers Tokenizer. If <modelDir>/special_tokens_map.json is absent, it throws with the resolved path. This is the last of the four required tokenizer sidecar files.

Source

Thrown at packages/fastembed/src/fastembed.ts:169

    throw new Error(`Tokenizer file not found at ${tokenizerPath}`);
  }

  const configPath = path.join(modelDir.toString(), 'config.json');
  if (!fs.existsSync(configPath)) {
    throw new Error(`Config file not found at ${configPath}`);
  }
  const config = JSON.parse(fs.readFileSync(configPath, 'utf-8'));

  const tokenizerFilePath = path.join(modelDir.toString(), 'tokenizer_config.json');
  if (!fs.existsSync(tokenizerFilePath)) {
    throw new Error(`Tokenizer config file not found at ${tokenizerFilePath}`);
  }
  const tokenizerConfig = JSON.parse(fs.readFileSync(tokenizerFilePath, 'utf-8'));
  maxLength = Math.min(maxLength, tokenizerConfig['model_max_length']);

  const tokensMapPath = path.join(modelDir.toString(), 'special_tokens_map.json');
  if (!fs.existsSync(tokensMapPath)) {
    throw new Error(`Tokens map file not found at ${tokensMapPath}`);
  }
  const tokensMap = JSON.parse(fs.readFileSync(tokensMapPath, 'utf-8'));

  const tokenizer = Tokenizer.fromFile(tokenizerPath);

  tokenizer.setTruncation(maxLength);
  tokenizer.setPadding({
    maxLength,
    padId: config['pad_token_id'],
    padToken: tokenizerConfig['pad_token'],
  });

  for (const token of Object.values(tokensMap)) {
    if (typeof token === 'string') {
      tokenizer.addSpecialTokens([token]);
    } else if (isAddedTokenMap(token)) {
      const addedToken = new AddedToken(token['content'], true, {
        singleWord: token['single_word'],

View on GitHub (pinned to 75dd419e61)

Solutions

1. Copy special_tokens_map.json from the source Hugging Face model repo into modelAbsoluteDirPath.
2. Create a minimal special_tokens_map.json (e.g. {"[CLS]": "[CLS]", "[SEP]": "[SEP]", "padding": "[PAD]", "unk": "[UNK]"}) matching your tokenizer.
3. Re-download the complete model directory to restore all required files.

Example fix

// before
$ cp tokenizer.json config.json tokenizer_config.json ./deploy-model/
// after
$ cp tokenizer.json config.json tokenizer_config.json special_tokens_map.json ./deploy-model/

import fs from 'node:fs';
const REQUIRED = ['tokenizer.json', 'config.json', 'tokenizer_config.json', 'special_tokens_map.json'];
const missing = REQUIRED.filter((f) => !fs.existsSync(path.join(dir, f)));
if (missing.length) throw new Error(`Model dir ${dir} missing: ${missing.join(', ')}`);

try {
  const embedder = await FlagEmbedding.init(opts);
} catch (err) {
  if (err instanceof Error && err.message.includes('Tokens map file not found')) {
    // copy special_tokens_map.json from source repo, then retry
  } else throw err;
}

When it happens

Trigger: FlagEmbedding.init where the model dir contains tokenizer.json, config.json, and tokenizer_config.json but lacks special_tokens_map.json — e.g. a custom export that omits it or a pruned copy of the model folder.

Common situations: Building a minimal custom model directory and assuming only tokenizer.json is needed; scripts that copy files individually and miss this one; interrupted downloads.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Tokenizer config file not found at ${tokenizerFilePath}
• Tokenizer file not found at ${tokenizerPath}
• Config file not found at ${configPath}
• Model file not found at ${modelPath}
• Snapshot file not found: ${snapshotPath} Run with { updateSn

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/6b2e8e155150fc3f. Report an issue: GitHub.