{"record":{"id":"6b2e8e155150fc3f","repo":"mastra-ai/mastra","slug":"tokens-map-file-not-found-at-tokensmappath","errorCode":null,"errorMessage":"Tokens map file not found at ${tokensMapPath}","messagePattern":"Tokens map file not found at (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/fastembed/src/fastembed.ts","lineNumber":169,"sourceCode":"    throw new Error(`Tokenizer file not found at ${tokenizerPath}`);\n  }\n\n  const configPath = path.join(modelDir.toString(), 'config.json');\n  if (!fs.existsSync(configPath)) {\n    throw new Error(`Config file not found at ${configPath}`);\n  }\n  const config = JSON.parse(fs.readFileSync(configPath, 'utf-8'));\n\n  const tokenizerFilePath = path.join(modelDir.toString(), 'tokenizer_config.json');\n  if (!fs.existsSync(tokenizerFilePath)) {\n    throw new Error(`Tokenizer config file not found at ${tokenizerFilePath}`);\n  }\n  const tokenizerConfig = JSON.parse(fs.readFileSync(tokenizerFilePath, 'utf-8'));\n  maxLength = Math.min(maxLength, tokenizerConfig['model_max_length']);\n\n  const tokensMapPath = path.join(modelDir.toString(), 'special_tokens_map.json');\n  if (!fs.existsSync(tokensMapPath)) {\n    throw new Error(`Tokens map file not found at ${tokensMapPath}`);\n  }\n  const tokensMap = JSON.parse(fs.readFileSync(tokensMapPath, 'utf-8'));\n\n  const tokenizer = Tokenizer.fromFile(tokenizerPath);\n\n  tokenizer.setTruncation(maxLength);\n  tokenizer.setPadding({\n    maxLength,\n    padId: config['pad_token_id'],\n    padToken: tokenizerConfig['pad_token'],\n  });\n\n  for (const token of Object.values(tokensMap)) {\n    if (typeof token === 'string') {\n      tokenizer.addSpecialTokens([token]);\n    } else if (isAddedTokenMap(token)) {\n      const addedToken = new AddedToken(token['content'], true, {\n        singleWord: token['single_word'],","sourceCodeStart":151,"sourceCodeEnd":187,"githubUrl":"https://github.com/mastra-ai/mastra/blob/75dd419e613fe9c39f846ffc500716141b74fda6/packages/fastembed/src/fastembed.ts#L151-L187","documentation":"The loader reads special_tokens_map.json to know which tokens are special (CLS, SEP, padding, etc.) when constructing the tokenizers Tokenizer. If <modelDir>/special_tokens_map.json is absent, it throws with the resolved path. This is the last of the four required tokenizer sidecar files.","triggerScenarios":"FlagEmbedding.init where the model dir contains tokenizer.json, config.json, and tokenizer_config.json but lacks special_tokens_map.json — e.g. a custom export that omits it or a pruned copy of the model folder.","commonSituations":"Building a minimal custom model directory and assuming only tokenizer.json is needed; scripts that copy files individually and miss this one; interrupted downloads.","solutions":["Copy special_tokens_map.json from the source Hugging Face model repo into modelAbsoluteDirPath.","Create a minimal special_tokens_map.json (e.g. {\"[CLS]\": \"[CLS]\", \"[SEP]\": \"[SEP]\", \"padding\": \"[PAD]\", \"unk\": \"[UNK]\"}) matching your tokenizer.","Re-download the complete model directory to restore all required files."],"exampleFix":"// before\n$ cp tokenizer.json config.json tokenizer_config.json ./deploy-model/\n// after\n$ cp tokenizer.json config.json tokenizer_config.json special_tokens_map.json ./deploy-model/","handlingStrategy":"validation","validationCode":"import fs from 'node:fs';\nconst REQUIRED = ['tokenizer.json', 'config.json', 'tokenizer_config.json', 'special_tokens_map.json'];\nconst missing = REQUIRED.filter((f) => !fs.existsSync(path.join(dir, f)));\nif (missing.length) throw new Error(`Model dir ${dir} missing: ${missing.join(', ')}`);","typeGuard":null,"tryCatchPattern":"try {\n  const embedder = await FlagEmbedding.init(opts);\n} catch (err) {\n  if (err instanceof Error && err.message.includes('Tokens map file not found')) {\n    // copy special_tokens_map.json from source repo, then retry\n  } else throw err;\n}","preventionTips":["Treat all four tokenizer JSON files as a single atomic artifact when copying/downloading.","Use the hub download path (built-in models) rather than manual file copying where possible.","Add an artifact integrity checklist to model packaging scripts."],"tags":["filesystem","missing-file","tokenizer","embeddings"],"backgroundTag":"file-not-found","analyzedSha":"75dd419e613fe9c39f846ffc500716141b74fda6","analyzedAt":"2026-08-30T00:15:31.844Z","schemaVersion":2},"datasetVersion":"2026-08-30T03:17:51.788Z"}