mastra-ai/mastra · error · Error

Tokenizer file not found at ${tokenizerPath}

Error message

Tokenizer file not found at ${tokenizerPath}

What it means

FlagEmbedding loads a local ONNX embedding model directory that must contain tokenizer.json (used to build the tokenizers Tokenizer). loadTokenizerFromDir checks fs.existsSync on <modelDir>/tokenizer.json and throws if absent. This means the model directory is incomplete, corrupt, or points to the wrong path.

Source

Thrown at packages/fastembed/src/fastembed.ts:151

  normalized: boolean;
}

function isAddedTokenMap(token: unknown): token is AddedTokenMap {
  return (
    typeof token === 'object' &&
    token !== null &&
    'content' in token &&
    'single_word' in token &&
    'rstrip' in token &&
    'lstrip' in token &&
    'normalized' in token
  );
}

function loadTokenizerFromDir(modelDir: PathLike, maxLength: number): Tokenizer {
  const tokenizerPath = path.join(modelDir.toString(), 'tokenizer.json');
  if (!fs.existsSync(tokenizerPath)) {
    throw new Error(`Tokenizer file not found at ${tokenizerPath}`);
  }

  const configPath = path.join(modelDir.toString(), 'config.json');
  if (!fs.existsSync(configPath)) {
    throw new Error(`Config file not found at ${configPath}`);
  }
  const config = JSON.parse(fs.readFileSync(configPath, 'utf-8'));

  const tokenizerFilePath = path.join(modelDir.toString(), 'tokenizer_config.json');
  if (!fs.existsSync(tokenizerFilePath)) {
    throw new Error(`Tokenizer config file not found at ${tokenizerFilePath}`);
  }
  const tokenizerConfig = JSON.parse(fs.readFileSync(tokenizerFilePath, 'utf-8'));
  maxLength = Math.min(maxLength, tokenizerConfig['model_max_length']);

  const tokensMapPath = path.join(modelDir.toString(), 'special_tokens_map.json');
  if (!fs.existsSync(tokensMapPath)) {
    throw new Error(`Tokens map file not found at ${tokensMapPath}`);

View on GitHub (pinned to 75dd419e61)

Solutions

1. Point the model dir at a folder that actually contains tokenizer.json (the HF model repo root, not its onnx/ subfolder).
2. Delete the corrupted/partial download cache and re-run init with showDownloadProgress so the full model re-downloads.
3. If using EmbeddingModel.CUSTOM, copy tokenizer.json, config.json, tokenizer_config.json, special_tokens_map.json and the .onnx file into modelAbsoluteDirPath.
4. Check the volume/mount actually contains the files: ls <modelDir>/tokenizer.json.

Example fix

// before
const embedder = await FlagEmbedding.init({ model: EmbeddingModel.CUSTOM, modelAbsoluteDirPath: './my-model/onnx' });
// after
const embedder = await FlagEmbedding.init({ model: EmbeddingModel.CUSTOM, modelAbsoluteDirPath: './my-model' }); // dir includes tokenizer.json + model.onnx

import fs from 'node:fs';
export function assertModelDirComplete(dir: string) {
  for (const f of ['tokenizer.json', 'config.json', 'tokenizer_config.json', 'special_tokens_map.json']) {
    if (!fs.existsSync(path.join(dir, f))) throw new Error(`${dir} is missing ${f}`);
  }
}

try {
  const embedder = await FlagEmbedding.init({ model, modelAbsoluteDirPath });
} catch (err) {
  if (err instanceof Error && err.message.includes('Tokenizer file not found')) {
    // re-download or fix modelAbsoluteDirPath before retrying
  } else throw err;
}

When it happens

Trigger: FlagEmbedding.init resolves modelDir (downloaded cache dir or a CUSTOM modelAbsoluteDirPath) and that directory lacks tokenizer.json — e.g. a custom model dir exported without the tokenizer file, a truncated download, or pointing at the ONNX model file's directory instead of the model root.

Common situations: Passing modelAbsoluteDirPath that contains only model.onnx (copied from HF 'onnx' subfolder rather than the repo root); a partially deleted or failed model download cache; using a custom fine-tuned model exported without tokenizer files; running in a container where the volume with the model was not mounted.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Config file not found at ${configPath}
• Model file not found at ${modelPath}
• Tokenizer config file not found at ${tokenizerFilePath}
• Tokens map file not found at ${tokensMapPath}
• Snapshot file not found: ${snapshotPath} Run with { updateSn

AI-assisted analysis of mastra-ai/mastra@75dd419e61 (2026-08-30). Data as JSON: /api/errors/b0864c1c51cc465a. Report an issue: GitHub.