weaviate/weaviate · error

tokenization is not allowed for data type '%s'

Error message

tokenization is not allowed for data type '%s'

What it means

Tokenization settings only apply to text-like properties (string/text). When a property has a non-text primitive dataType (int, number, boolean, date, etc.) and a non-empty tokenization value, validatePropertyTokenization rejects the schema with this error.

Source

Thrown at usecases/schema/class.go:1272

					return fmt.Errorf("the Chinese tokenizer is not enabled; set 'ENABLE_TOKENIZER_GSE_CH' to 'true' to enable")
				}
				return nil
			case models.PropertyTokenizationKagomeKr:
				if !entcfg.Enabled(os.Getenv("ENABLE_TOKENIZER_KAGOME_KR")) {
					return fmt.Errorf("the Korean tokenizer is not enabled; set 'ENABLE_TOKENIZER_KAGOME_KR' to 'true' to enable")
				}
				return nil
			case models.PropertyTokenizationKagomeJa:
				if !entcfg.Enabled(os.Getenv("ENABLE_TOKENIZER_KAGOME_JA")) {
					return fmt.Errorf("the Japanese tokenizer is not enabled; set 'ENABLE_TOKENIZER_KAGOME_JA' to 'true' to enable")
				}
				return nil
			}
		default:
			if tokenization == "" {
				return nil
			}
			return fmt.Errorf("tokenization is not allowed for data type '%s'", primitiveDataType)
		}
		return fmt.Errorf("tokenization '%s' is not allowed for data type '%s'", tokenization, primitiveDataType)
	}

	if tokenization == "" {
		return nil
	}

	if propertyDataType.IsNested() {
		return fmt.Errorf("tokenization is not allowed for object/object[] data types")
	}
	return fmt.Errorf("tokenization is not allowed for reference data type")
}

func (h *Handler) validatePropertyIndexing(prop *models.Property) error {
	if prop.IndexInverted != nil {
		if prop.IndexFilterable != nil || prop.IndexSearchable != nil || prop.IndexRangeFilters != nil {
			return fmt.Errorf("`indexInverted` is deprecated and can not be set together with `indexFilterable`, " + "`indexSearchable` or `indexRangeFilters`")

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Remove the tokenization field from the non-text property
2. If tokenization is truly needed, change the property dataType to 'text' or 'string'
3. Normalize schema templates to strip tokenization for non-text dataTypes before submitting

Example fix

// before
{"name": "age", "dataType": ["int"], "tokenization": "word"} // fails
// after
{"name": "age", "dataType": ["int"]}

textual := map[string]bool{"text": true, "string": true}
if prop.Tokenization != "" && !textual[prop.DataType[0]] {
	return fmt.Errorf("property %s: tokenization not allowed for dataType %s", prop.Name, prop.DataType[0])
}

func tokenizationAllowed(p *models.Property) bool {
	switch p.DataType[0] {
	case "text", "string":
		return true
	default:
		return p.Tokenization == ""
	}
}

err := client.Schema().ClassCreator().WithClass(class).Do(ctx)
if err != nil && strings.Contains(err.Error(), "tokenization is not allowed for data type") {
	// strip tokenization from non-text properties and retry
}

When it happens

Trigger: Creating/updating a class where a property with dataType such as 'int', 'number', 'boolean', 'date', 'uuid', or a reference type carries a tokenization field (even 'word' or 'whitespace').

Common situations: Copy-pasting a full property JSON including tokenization from a text property onto a numeric/date property; code generators that always emit tokenization; bulk schema templates with tokenization on every property.

Related errors

• invalid tokenization %q
• change-tokenization requires exactly one property
• property %q set as "deleteOn" not found among collection pro
• empty property or empty class name: %w
• '%s' is not a valid property name. Name should not be longer

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/40a43057a892b3ef. Report an issue: GitHub.