{"record":{"id":"f2e5ce45903c96ad","repo":"weaviate/weaviate","slug":"tokenization-s-is-not-allowed-for-data-type-s","errorCode":null,"errorMessage":"tokenization '%s' is not allowed for data type '%s'","messagePattern":"tokenization '(.+?)' is not allowed for data type '(.+?)'","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"usecases/schema/class.go","lineNumber":1274,"sourceCode":"\t\t\t\treturn nil\n\t\t\tcase models.PropertyTokenizationKagomeKr:\n\t\t\t\tif !entcfg.Enabled(os.Getenv(\"ENABLE_TOKENIZER_KAGOME_KR\")) {\n\t\t\t\t\treturn fmt.Errorf(\"the Korean tokenizer is not enabled; set 'ENABLE_TOKENIZER_KAGOME_KR' to 'true' to enable\")\n\t\t\t\t}\n\t\t\t\treturn nil\n\t\t\tcase models.PropertyTokenizationKagomeJa:\n\t\t\t\tif !entcfg.Enabled(os.Getenv(\"ENABLE_TOKENIZER_KAGOME_JA\")) {\n\t\t\t\t\treturn fmt.Errorf(\"the Japanese tokenizer is not enabled; set 'ENABLE_TOKENIZER_KAGOME_JA' to 'true' to enable\")\n\t\t\t\t}\n\t\t\t\treturn nil\n\t\t\t}\n\t\tdefault:\n\t\t\tif tokenization == \"\" {\n\t\t\t\treturn nil\n\t\t\t}\n\t\t\treturn fmt.Errorf(\"tokenization is not allowed for data type '%s'\", primitiveDataType)\n\t\t}\n\t\treturn fmt.Errorf(\"tokenization '%s' is not allowed for data type '%s'\", tokenization, primitiveDataType)\n\t}\n\n\tif tokenization == \"\" {\n\t\treturn nil\n\t}\n\n\tif propertyDataType.IsNested() {\n\t\treturn fmt.Errorf(\"tokenization is not allowed for object/object[] data types\")\n\t}\n\treturn fmt.Errorf(\"tokenization is not allowed for reference data type\")\n}\n\nfunc (h *Handler) validatePropertyIndexing(prop *models.Property) error {\n\tif prop.IndexInverted != nil {\n\t\tif prop.IndexFilterable != nil || prop.IndexSearchable != nil || prop.IndexRangeFilters != nil {\n\t\t\treturn fmt.Errorf(\"`indexInverted` is deprecated and can not be set together with `indexFilterable`, \" + \"`indexSearchable` or `indexRangeFilters`\")\n\t\t}\n\t}","sourceCodeStart":1256,"sourceCodeEnd":1292,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/usecases/schema/class.go#L1256-L1292","documentation":"Weaviate only supports a `tokenization` setting on properties whose data type has an inverted index that tokenizes text (text/text[] and certain keyword-like types). If you set tokenization on any other primitive data type (e.g. int, number, date, bool), the schema validator in validatePropertyTokenization rejects the class/property update with this message. It exists because tokenization has no meaning for non-textual values and silently ignoring it would hide config mistakes.","triggerScenarios":"POST /v1/schema (create class) or POST /v1/schema/{className} (add property) or PATCH property update where a property has a primitive dataType such as 'int', 'number', 'date', 'boolean' AND a non-empty `tokenization` field (e.g. tokenization: \"word\").","commonSituations":"Copying a property definition from a text property (which legally uses tokenization like 'word', 'field', 'truncate') onto a numeric/date property; client codegen that always serializes an empty-string-defaulted tokenization field; bulk-migrating schemas between collections and blanket-adding tokenization.","solutions":["Remove the `tokenization` field (or set it to \"\") from the offending property definition.","If you intended inverted-index behavior on text, change the property dataType to `text` or `text[]` and keep the tokenization value.","If you only wanted the property excluded from filtering/indexing, use `indexFilterable: false` / `indexSearchable: false` instead of tokenization."],"exampleFix":"// before\n{\"name\":\"age\",\"dataType\":[\"int\"],\"tokenization\":\"word\"}\n// after\n{\"name\":\"age\",\"dataType\":[\"int\"]}","handlingStrategy":"validation","validationCode":"const TOKENIZABLE = new Set(['text','text[]','value','value[]']);\nfunction validateTokenization(prop) {\n  if (prop.tokenization && !TOKENIZABLE.has(prop.dataType[0])) {\n    throw new Error(`property '${prop.name}': tokenization is not supported for dataType '${prop.dataType[0]}'`);\n  }\n}","typeGuard":"function supportsTokenization(prop) {\n  return ['text','text[]'].includes(prop.dataType[0]);\n}","tryCatchPattern":null,"preventionTips":["Only set tokenization on text/text[] (and keyword-style) properties.","Strip empty-string default fields in client serializers (omitempty).","Keep one schema template per dataType category instead of a universal property template."],"tags":["schema","validation","tokenization","weaviate"],"backgroundTag":"invalid-property-tokenization","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}