mastra-ai/mastra · error · Error

Tokenizer config file not found at ${tokenizerFilePath}

Error message

Tokenizer config file not found at ${tokenizerFilePath}

What it means

loadTokenizerFromDir reads tokenizer_config.json to determine model_max_length, which it uses to clamp truncation length. If <modelDir>/tokenizer_config.json is missing, the loader throws with the full path. Without this file the library cannot know the model's maximum sequence length.

Source

Thrown at packages/fastembed/src/fastembed.ts:162

    'normalized' in token
  );
}

function loadTokenizerFromDir(modelDir: PathLike, maxLength: number): Tokenizer {
  const tokenizerPath = path.join(modelDir.toString(), 'tokenizer.json');
  if (!fs.existsSync(tokenizerPath)) {
    throw new Error(`Tokenizer file not found at ${tokenizerPath}`);
  }

  const configPath = path.join(modelDir.toString(), 'config.json');
  if (!fs.existsSync(configPath)) {
    throw new Error(`Config file not found at ${configPath}`);
  }
  const config = JSON.parse(fs.readFileSync(configPath, 'utf-8'));

  const tokenizerFilePath = path.join(modelDir.toString(), 'tokenizer_config.json');
  if (!fs.existsSync(tokenizerFilePath)) {
    throw new Error(`Tokenizer config file not found at ${tokenizerFilePath}`);
  }
  const tokenizerConfig = JSON.parse(fs.readFileSync(tokenizerFilePath, 'utf-8'));
  maxLength = Math.min(maxLength, tokenizerConfig['model_max_length']);

  const tokensMapPath = path.join(modelDir.toString(), 'special_tokens_map.json');
  if (!fs.existsSync(tokensMapPath)) {
    throw new Error(`Tokens map file not found at ${tokensMapPath}`);
  }
  const tokensMap = JSON.parse(fs.readFileSync(tokensMapPath, 'utf-8'));

  const tokenizer = Tokenizer.fromFile(tokenizerPath);

  tokenizer.setTruncation(maxLength);
  tokenizer.setPadding({
    maxLength,
    padId: config['pad_token_id'],
    padToken: tokenizerConfig['pad_token'],
  });

View on GitHub (pinned to 75dd419e61)

Solutions

1. Copy tokenizer_config.json from the source Hugging Face repo into the model directory.
2. Alternatively create a minimal tokenizer_config.json containing {"model_max_length": <N>} matching your model's context length.
3. Re-download the model directory from the hub if the local copy is partial.

Example fix

// before
my-model/{tokenizer.json,config.json}  // missing tokenizer_config.json
// after
echo '{"model_max_length": 512}' > my-model/tokenizer_config.json

import fs from 'node:fs';
const p = path.join(dir, 'tokenizer_config.json');
if (!fs.existsSync(p)) {
  fs.writeFileSync(p, JSON.stringify({ model_max_length: 512 })); // minimal fallback
}

try {
  const embedder = await FlagEmbedding.init(opts);
} catch (err) {
  if (err instanceof Error && err.message.includes('Tokenizer config file not found')) {
    // synthesize tokenizer_config.json with model_max_length, then retry
  } else throw err;
}

When it happens

Trigger: FlagEmbedding.init on a model directory containing tokenizer.json and config.json but no tokenizer_config.json — common with custom model dirs assembled by hand or checkpoints exported by tooling that emits tokenizer.json but not the legacy *_config.json.

Common situations: Exporting a custom fine-tuned model via tooling that skips tokenizer_config.json; selectively copying files; older custom checkpoints that predate the file; a corrupted partial download.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Tokens map file not found at ${tokensMapPath}
• Tokenizer file not found at ${tokenizerPath}
• Config file not found at ${configPath}
• Model file not found at ${modelPath}
• Snapshot file not found: ${snapshotPath} Run with { updateSn

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/40bba19a68d5a9de. Report an issue: GitHub.