{"record":{"id":"3cc24a1dc3c5c87b","repo":"weaviate/weaviate","slug":"property-s-unsupported-tokenization-s","errorCode":null,"errorMessage":"property '%s': unsupported tokenization '%s'","messagePattern":"property '(.+?)': unsupported tokenization '(.+?)'","errorType":"validation","errorClass":null,"httpStatus":422,"severity":"error","filePath":"usecases/schema/class.go","lineNumber":1376,"sourceCode":"\t}\n\n\tfor _, entry := range prop.TextAnalyzer.ASCIIFoldIgnore {\n\t\tif utf8.RuneCountInString(norm.NFC.String(entry)) != 1 {\n\t\t\treturn fmt.Errorf(\"property '%s': each asciiFoldIgnore entry must be a single character, got %q\",\n\t\t\t\tprop.Name, entry)\n\t\t}\n\t}\n\n\t// explicitly check for support for tokenizers:\n\tif prop.Tokenization != \"\" {\n\t\tswitch prop.Tokenization {\n\t\tcase models.PropertyTokenizationLowercase,\n\t\t\tmodels.PropertyTokenizationTrigram,\n\t\t\tmodels.PropertyTokenizationWord,\n\t\t\tmodels.PropertyTokenizationWhitespace,\n\t\t\tmodels.PropertyTokenizationField: // supported tokenizers, do nothing\n\t\tdefault:\n\t\t\treturn fmt.Errorf(\"property '%s': unsupported tokenization '%s'\", prop.Name, prop.Tokenization)\n\t\t}\n\t}\n\n\tif prop.TextAnalyzer.StopwordPreset != \"\" {\n\t\tif prop.Tokenization != models.PropertyTokenizationWord {\n\t\t\treturn fmt.Errorf(\"property '%s': stopwordPreset is only supported with tokenization %q, got %q\",\n\t\t\t\tprop.Name, models.PropertyTokenizationWord, prop.Tokenization)\n\t\t}\n\t\t_, builtIn := stopwords.Presets[prop.TextAnalyzer.StopwordPreset]\n\t\t_, userDefined := userPresets[prop.TextAnalyzer.StopwordPreset]\n\t\tif !builtIn && !userDefined {\n\t\t\treturn fmt.Errorf(\"property '%s': unknown stopword preset %q; must be a built-in preset ('en', 'none') or defined in invertedIndexConfig.stopwordPresets\",\n\t\t\t\tprop.Name, prop.TextAnalyzer.StopwordPreset)\n\t\t}\n\t}\n\n\treturn nil\n}","sourceCodeStart":1358,"sourceCodeEnd":1394,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/usecases/schema/class.go#L1358-L1394","documentation":"Each property tokenization must be one of the supported tokenizers (lowercase, trigram, word, whitespace, field, and related variants accepted in the switch). This error is thrown in validatePropertyProcessing when the property's tokenization field is set to a value outside the supported set, because Weaviate cannot determine how to split and normalize the property value for the inverted index.","triggerScenarios":"Creating or updating a class with a property whose tokenization is misspelled (e.g. 'whitespac', 'Word', 'GSEN') or uses a tokenizer removed from the installed Weaviate version. Any schema PUT/POST carrying that property fails validation.","commonSituations":"Typo in tokenization name; copying a config from another vector DB; using a tokenizer introduced in a newer Weaviate release than the one running; wrong casing since values are compared exactly.","solutions":["Set tokenization to one of the supported values: 'lowercase', 'trigram', 'word', 'whitespace', 'field' (check docs for your version)","Fix the casing/typo in the tokenization value (comparison is exact string match)","Check your Weaviate version's supported tokenizers (e.g. 'field' and 'trigram' exist only in newer versions) and upgrade or downgrade the config accordingly"],"exampleFix":"// before\n{\"name\": \"text\", \"dataType\": [\"text\"], \"tokenization\": \"whitespac\"}\n// after\n{\"name\": \"text\", \"dataType\": [\"text\"], \"tokenization\": \"whitespace\"}","handlingStrategy":"validation","validationCode":"var validTokenizations = map[string]bool{\"lowercase\": true, \"trigram\": true, \"word\": true, \"whitespace\": true, \"field\": true}\n\nfunc validTokenization(p *models.Property) bool {\n  return p.Tokenization == \"\" || validTokenizations[p.Tokenization]\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep a constant list of supported tokenizers per Weaviate version and validate configs against it in CI","Never free-hand tokenization strings; use generated model constants (models.PropertyTokenization*)"],"tags":["schema","validation","tokenization"],"backgroundTag":"schema-validation-failed","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}