{"record":{"id":"195c12c023ba1f52","repo":"krisk/Fuse","slug":"fuse-tokenize-function-must-return-string-rec","errorCode":null,"errorMessage":"[Fuse] tokenize function must return string[]; received ${Array.isArray(result) ? 'array containing non-strings' : typeof result}.","messagePattern":"\\[Fuse\\] tokenize function must return string\\[\\]; received (.+?)\\.","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"src/search/token/analyzer.ts","lineNumber":45,"sourceCode":"        `first match per text will be returned. Add the 'g' flag.`\n    )\n  }\n}\n\nfunction resolveTokenize(\n  tokenize: RegExp | TokenizeFunction | undefined\n): TokenizeFunction {\n  if (typeof tokenize === 'function') {\n    let validated = false\n    return (text: string): string[] => {\n      const result = tokenize(text)\n      if (process.env.NODE_ENV === 'development' && !validated) {\n        validated = true\n        if (\n          !Array.isArray(result) ||\n          result.some((t) => typeof t !== 'string')\n        ) {\n          throw new Error(\n            `[Fuse] tokenize function must return string[]; received ${\n              Array.isArray(result)\n                ? 'array containing non-strings'\n                : typeof result\n            }.`\n          )\n        }\n      }\n      return result\n    }\n  }\n  if (tokenize instanceof RegExp) {\n    if (!tokenize.global) warnNonGlobal(tokenize)\n    return (text: string): string[] => text.match(tokenize) || []\n  }\n  return (text: string): string[] => text.match(DEFAULT_TOKEN) || []\n}\n","sourceCodeStart":27,"sourceCodeEnd":63,"githubUrl":"https://github.com/krisk/Fuse/blob/edf2fb608eca0461508d1d71317e6e58309ffada/src/search/token/analyzer.ts#L27-L63","documentation":"Development-mode validation in resolveTokenize: a custom tokenize function returned something other than an array of strings (either a non-array or an array containing non-string tokens). The analyzer expects string[] so downstream indexing/normalization works; the error names what was received instead.","triggerScenarios":"Passing a custom tokenize option that returns undefined, a string, a single token (not wrapped in an array), or an array containing numbers/objects — but only when NODE_ENV === 'development' and the result fails Array.isArray/result.every(typeof string) checks, via tokenizeFn.","commonSituations":"Writing a tokenizer that returns a string instead of splitting into an array; forgetting a .map(String) after splitting on numeric fields; refactoring a tokenizer that previously returned tokens but now returns a Promise or iterator.","solutions":["Make the tokenize function always return string[]: return input.split(/\\s+/).filter(Boolean).","Coerce token elements to strings before returning (tokens.map(String)).","Run in development mode during testing so this validation fires early instead of failing silently in production.","Add your own return-type check or TypeScript annotation (tokenize: (s: string) => string[]) so build tooling catches it."],"exampleFix":"// before\nconst tokenize = (s) => s.split(' ') // may include '' or non-strings\n// after\nconst tokenize = (s) => s.split(/\\s+/).filter(Boolean).map(String) // guarantees string[]","handlingStrategy":"type-guard","validationCode":"const out = myTokenize(input);\nif (!Array.isArray(out) || out.some((t) => typeof t !== 'string')) throw new TypeError('tokenize must return string[]');","typeGuard":"function isStringArray(v: unknown): v is string[] {\n  return Array.isArray(v) && v.every((t) => typeof t === 'string');\n}","tryCatchPattern":"try {\n  const idx = new Fuse(docs, { tokenize: myTokenize });\n} catch (e) {\n  if (String(e.message).includes('tokenize function must return string[]')) {\n    // wrap/fix the tokenizer: (s) => toTokenArray(s)\n  } else throw e;\n}","preventionTips":["Annotate tokenizers as (s: string) => string[] in TypeScript","Run the app in NODE_ENV=development during tests to trigger validation","Always .map(String) and filter empties before returning tokens","Never return a Promise or iterator from tokenize"],"tags":["search","tokenization","dev-mode"],"backgroundTag":"invalid-tokenizer-return-type","analyzedSha":"edf2fb608eca0461508d1d71317e6e58309ffada","analyzedAt":"2026-09-02T02:46:54.623Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}