{"record":{"id":"40bba19a68d5a9de","repo":"mastra-ai/mastra","slug":"tokenizer-config-file-not-found-at-tokenizerfile","errorCode":null,"errorMessage":"Tokenizer config file not found at ${tokenizerFilePath}","messagePattern":"Tokenizer config file not found at (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/fastembed/src/fastembed.ts","lineNumber":162,"sourceCode":"    'normalized' in token\n  );\n}\n\nfunction loadTokenizerFromDir(modelDir: PathLike, maxLength: number): Tokenizer {\n  const tokenizerPath = path.join(modelDir.toString(), 'tokenizer.json');\n  if (!fs.existsSync(tokenizerPath)) {\n    throw new Error(`Tokenizer file not found at ${tokenizerPath}`);\n  }\n\n  const configPath = path.join(modelDir.toString(), 'config.json');\n  if (!fs.existsSync(configPath)) {\n    throw new Error(`Config file not found at ${configPath}`);\n  }\n  const config = JSON.parse(fs.readFileSync(configPath, 'utf-8'));\n\n  const tokenizerFilePath = path.join(modelDir.toString(), 'tokenizer_config.json');\n  if (!fs.existsSync(tokenizerFilePath)) {\n    throw new Error(`Tokenizer config file not found at ${tokenizerFilePath}`);\n  }\n  const tokenizerConfig = JSON.parse(fs.readFileSync(tokenizerFilePath, 'utf-8'));\n  maxLength = Math.min(maxLength, tokenizerConfig['model_max_length']);\n\n  const tokensMapPath = path.join(modelDir.toString(), 'special_tokens_map.json');\n  if (!fs.existsSync(tokensMapPath)) {\n    throw new Error(`Tokens map file not found at ${tokensMapPath}`);\n  }\n  const tokensMap = JSON.parse(fs.readFileSync(tokensMapPath, 'utf-8'));\n\n  const tokenizer = Tokenizer.fromFile(tokenizerPath);\n\n  tokenizer.setTruncation(maxLength);\n  tokenizer.setPadding({\n    maxLength,\n    padId: config['pad_token_id'],\n    padToken: tokenizerConfig['pad_token'],\n  });","sourceCodeStart":144,"sourceCodeEnd":180,"githubUrl":"https://github.com/mastra-ai/mastra/blob/75dd419e613fe9c39f846ffc500716141b74fda6/packages/fastembed/src/fastembed.ts#L144-L180","documentation":"loadTokenizerFromDir reads tokenizer_config.json to determine model_max_length, which it uses to clamp truncation length. If <modelDir>/tokenizer_config.json is missing, the loader throws with the full path. Without this file the library cannot know the model's maximum sequence length.","triggerScenarios":"FlagEmbedding.init on a model directory containing tokenizer.json and config.json but no tokenizer_config.json — common with custom model dirs assembled by hand or checkpoints exported by tooling that emits tokenizer.json but not the legacy *_config.json.","commonSituations":"Exporting a custom fine-tuned model via tooling that skips tokenizer_config.json; selectively copying files; older custom checkpoints that predate the file; a corrupted partial download.","solutions":["Copy tokenizer_config.json from the source Hugging Face repo into the model directory.","Alternatively create a minimal tokenizer_config.json containing {\"model_max_length\": <N>} matching your model's context length.","Re-download the model directory from the hub if the local copy is partial."],"exampleFix":"// before\nmy-model/{tokenizer.json,config.json}  // missing tokenizer_config.json\n// after\necho '{\"model_max_length\": 512}' > my-model/tokenizer_config.json","handlingStrategy":"validation","validationCode":"import fs from 'node:fs';\nconst p = path.join(dir, 'tokenizer_config.json');\nif (!fs.existsSync(p)) {\n  fs.writeFileSync(p, JSON.stringify({ model_max_length: 512 })); // minimal fallback\n}","typeGuard":null,"tryCatchPattern":"try {\n  const embedder = await FlagEmbedding.init(opts);\n} catch (err) {\n  if (err instanceof Error && err.message.includes('Tokenizer config file not found')) {\n    // synthesize tokenizer_config.json with model_max_length, then retry\n  } else throw err;\n}","preventionTips":["Keep tokenizer_config.json with any exported custom checkpoint.","Know your model's true max sequence length so a handcrafted fallback is correct.","Verify all tokenizer sidecar files exist in CI before shipping model artifacts."],"tags":["filesystem","missing-file","tokenizer","embeddings"],"backgroundTag":"file-not-found","analyzedSha":"75dd419e613fe9c39f846ffc500716141b74fda6","analyzedAt":"2026-08-30T00:15:31.844Z","schemaVersion":2},"datasetVersion":"2026-08-30T03:17:51.788Z"}