{"record":{"id":"b0864c1c51cc465a","repo":"mastra-ai/mastra","slug":"tokenizer-file-not-found-at-tokenizerpath","errorCode":null,"errorMessage":"Tokenizer file not found at ${tokenizerPath}","messagePattern":"Tokenizer file not found at (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/fastembed/src/fastembed.ts","lineNumber":151,"sourceCode":"  normalized: boolean;\n}\n\nfunction isAddedTokenMap(token: unknown): token is AddedTokenMap {\n  return (\n    typeof token === 'object' &&\n    token !== null &&\n    'content' in token &&\n    'single_word' in token &&\n    'rstrip' in token &&\n    'lstrip' in token &&\n    'normalized' in token\n  );\n}\n\nfunction loadTokenizerFromDir(modelDir: PathLike, maxLength: number): Tokenizer {\n  const tokenizerPath = path.join(modelDir.toString(), 'tokenizer.json');\n  if (!fs.existsSync(tokenizerPath)) {\n    throw new Error(`Tokenizer file not found at ${tokenizerPath}`);\n  }\n\n  const configPath = path.join(modelDir.toString(), 'config.json');\n  if (!fs.existsSync(configPath)) {\n    throw new Error(`Config file not found at ${configPath}`);\n  }\n  const config = JSON.parse(fs.readFileSync(configPath, 'utf-8'));\n\n  const tokenizerFilePath = path.join(modelDir.toString(), 'tokenizer_config.json');\n  if (!fs.existsSync(tokenizerFilePath)) {\n    throw new Error(`Tokenizer config file not found at ${tokenizerFilePath}`);\n  }\n  const tokenizerConfig = JSON.parse(fs.readFileSync(tokenizerFilePath, 'utf-8'));\n  maxLength = Math.min(maxLength, tokenizerConfig['model_max_length']);\n\n  const tokensMapPath = path.join(modelDir.toString(), 'special_tokens_map.json');\n  if (!fs.existsSync(tokensMapPath)) {\n    throw new Error(`Tokens map file not found at ${tokensMapPath}`);","sourceCodeStart":133,"sourceCodeEnd":169,"githubUrl":"https://github.com/mastra-ai/mastra/blob/75dd419e613fe9c39f846ffc500716141b74fda6/packages/fastembed/src/fastembed.ts#L133-L169","documentation":"FlagEmbedding loads a local ONNX embedding model directory that must contain tokenizer.json (used to build the tokenizers Tokenizer). loadTokenizerFromDir checks fs.existsSync on <modelDir>/tokenizer.json and throws if absent. This means the model directory is incomplete, corrupt, or points to the wrong path.","triggerScenarios":"FlagEmbedding.init resolves modelDir (downloaded cache dir or a CUSTOM modelAbsoluteDirPath) and that directory lacks tokenizer.json — e.g. a custom model dir exported without the tokenizer file, a truncated download, or pointing at the ONNX model file's directory instead of the model root.","commonSituations":"Passing modelAbsoluteDirPath that contains only model.onnx (copied from HF 'onnx' subfolder rather than the repo root); a partially deleted or failed model download cache; using a custom fine-tuned model exported without tokenizer files; running in a container where the volume with the model was not mounted.","solutions":["Point the model dir at a folder that actually contains tokenizer.json (the HF model repo root, not its onnx/ subfolder).","Delete the corrupted/partial download cache and re-run init with showDownloadProgress so the full model re-downloads.","If using EmbeddingModel.CUSTOM, copy tokenizer.json, config.json, tokenizer_config.json, special_tokens_map.json and the .onnx file into modelAbsoluteDirPath.","Check the volume/mount actually contains the files: ls <modelDir>/tokenizer.json."],"exampleFix":"// before\nconst embedder = await FlagEmbedding.init({ model: EmbeddingModel.CUSTOM, modelAbsoluteDirPath: './my-model/onnx' });\n// after\nconst embedder = await FlagEmbedding.init({ model: EmbeddingModel.CUSTOM, modelAbsoluteDirPath: './my-model' }); // dir includes tokenizer.json + model.onnx","handlingStrategy":"validation","validationCode":"import fs from 'node:fs';\nexport function assertModelDirComplete(dir: string) {\n  for (const f of ['tokenizer.json', 'config.json', 'tokenizer_config.json', 'special_tokens_map.json']) {\n    if (!fs.existsSync(path.join(dir, f))) throw new Error(`${dir} is missing ${f}`);\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  const embedder = await FlagEmbedding.init({ model, modelAbsoluteDirPath });\n} catch (err) {\n  if (err instanceof Error && err.message.includes('Tokenizer file not found')) {\n    // re-download or fix modelAbsoluteDirPath before retrying\n  } else throw err;\n}","preventionTips":["Point at the Hugging Face repo root, never the onnx/ subfolder.","Probe the directory with fs.existsSync for all four tokenizer files before init.","Re-download instead of trusting partially transferred model dirs."],"tags":["filesystem","missing-file","model-loading","embeddings"],"backgroundTag":"file-not-found","analyzedSha":"75dd419e613fe9c39f846ffc500716141b74fda6","analyzedAt":"2026-08-30T00:15:31.844Z","schemaVersion":2},"datasetVersion":"2026-08-30T03:17:51.788Z"}