{"record":{"id":"ff282837792ecada","repo":"weaviate/weaviate","slug":"asciifoldignore-requires-asciifold-to-be-enabled","errorCode":null,"errorMessage":"asciiFoldIgnore requires asciiFold to be enabled","messagePattern":"asciiFoldIgnore requires asciiFold to be enabled","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"adapters/handlers/rest/handlers_tokenize.go","lineNumber":251,"sourceCode":"\t\treturn schemaops.NewSchemaObjectsPropertiesTokenizeUnprocessableEntity().WithPayload(\n\t\t\terrPayloadFromSingleErr(principal, fmt.Errorf(\"unknown stopword preset %q; must be a built-in preset ('en', 'none') or defined in invertedIndexConfig.stopwordPresets\", prop.TextAnalyzer.StopwordPreset)))\n\t}\n\n\tprepared := tokenizer.NewPreparedAnalyzer(prop.TextAnalyzer)\n\tresult := tokenizer.Analyze(*params.Body.Text, prop.Tokenization, className, prepared, detector)\n\n\treturn schemaops.NewSchemaObjectsPropertiesTokenizeOK().WithPayload(&models.TokenizeResponse{\n\t\tIndexed: result.Indexed,\n\t\tQuery:   result.Query,\n\t})\n}\n\nfunc validateAnalyzerConfig(cfg *models.TextAnalyzerConfig) error {\n\tif cfg == nil {\n\t\treturn nil\n\t}\n\tif !cfg.ASCIIFold && len(cfg.ASCIIFoldIgnore) > 0 {\n\t\treturn fmt.Errorf(\"asciiFoldIgnore requires asciiFold to be enabled\")\n\t}\n\tfor _, entry := range cfg.ASCIIFoldIgnore {\n\t\tif utf8.RuneCountInString(norm.NFC.String(entry)) != 1 {\n\t\t\treturn fmt.Errorf(\"each asciiFoldIgnore entry must be a single character, got %q\", entry)\n\t\t}\n\t}\n\treturn nil\n}\n","sourceCodeStart":233,"sourceCodeEnd":260,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/handlers/rest/handlers_tokenize.go#L233-L260","documentation":"validateAnalyzerConfig rejects a TextAnalyzerConfig where asciiFoldIgnore entries are supplied while asciiFold is disabled (false). The ignore list only has meaning as an exception list for ASCII folding, so it is invalid on its own and returns a 4xx error to the caller of genericTokenize.","triggerScenarios":"POST tokenize (generic) with body.textAnalyzer containing a non-empty asciiFoldIgnore array but asciiFold absent or false.","commonSituations":"Developer adds ignore characters to preserve accented letters but forgets to also enable asciiFold; older clients that predate the asciiFold flag sending only the ignore list.","solutions":["Set asciiFold: true in the same textAnalyzer config","Remove the asciiFoldIgnore array if folding is not wanted","Validate the analyzer config client-side before calling the endpoint"],"exampleFix":"// before\n{\"textAnalyzer\":{\"asciiFoldIgnore\":[\"é\"]}}\n// after\n{\"textAnalyzer\":{\"asciiFold\":true,\"asciiFoldIgnore\":[\"é\"]}}","handlingStrategy":"validation","validationCode":"if (cfg?.asciiFoldIgnore?.length && !cfg.asciiFold) {\n  throw new Error('asciiFoldIgnore requires asciiFold to be enabled')\n}","typeGuard":"function hasValidFoldConfig(cfg) {\n  return !cfg?.asciiFoldIgnore?.length || cfg.asciiFold === true\n}","tryCatchPattern":"try {\n  return await api.tokenize(req)\n} catch (e) {\n  if (e.status === 422 && /asciiFoldIgnore requires asciiFold/.test(e.message)) {\n    return api.tokenize({...req, textAnalyzer: {...req.textAnalyzer, asciiFold: true}})\n  }\n  throw e\n}","preventionTips":["Always set asciiFold:true alongside asciiFoldIgnore","Add a client-side schema validator for textAnalyzer configs","Document the dependency between the two flags in shared config types"],"tags":["rest","tokenization","ascii-fold","validation"],"backgroundTag":"schema-validation-failed","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}