weaviate/weaviate · error

tokenization '%s' is not allowed for data type '%s'

Error message

tokenization '%s' is not allowed for data type '%s'

What it means

Weaviate only supports a `tokenization` setting on properties whose data type has an inverted index that tokenizes text (text/text[] and certain keyword-like types). If you set tokenization on any other primitive data type (e.g. int, number, date, bool), the schema validator in validatePropertyTokenization rejects the class/property update with this message. It exists because tokenization has no meaning for non-textual values and silently ignoring it would hide config mistakes.

Source

Thrown at usecases/schema/class.go:1274

				return nil
			case models.PropertyTokenizationKagomeKr:
				if !entcfg.Enabled(os.Getenv("ENABLE_TOKENIZER_KAGOME_KR")) {
					return fmt.Errorf("the Korean tokenizer is not enabled; set 'ENABLE_TOKENIZER_KAGOME_KR' to 'true' to enable")
				}
				return nil
			case models.PropertyTokenizationKagomeJa:
				if !entcfg.Enabled(os.Getenv("ENABLE_TOKENIZER_KAGOME_JA")) {
					return fmt.Errorf("the Japanese tokenizer is not enabled; set 'ENABLE_TOKENIZER_KAGOME_JA' to 'true' to enable")
				}
				return nil
			}
		default:
			if tokenization == "" {
				return nil
			}
			return fmt.Errorf("tokenization is not allowed for data type '%s'", primitiveDataType)
		}
		return fmt.Errorf("tokenization '%s' is not allowed for data type '%s'", tokenization, primitiveDataType)
	}

	if tokenization == "" {
		return nil
	}

	if propertyDataType.IsNested() {
		return fmt.Errorf("tokenization is not allowed for object/object[] data types")
	}
	return fmt.Errorf("tokenization is not allowed for reference data type")
}

func (h *Handler) validatePropertyIndexing(prop *models.Property) error {
	if prop.IndexInverted != nil {
		if prop.IndexFilterable != nil || prop.IndexSearchable != nil || prop.IndexRangeFilters != nil {
			return fmt.Errorf("`indexInverted` is deprecated and can not be set together with `indexFilterable`, " + "`indexSearchable` or `indexRangeFilters`")
		}
	}

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Remove the `tokenization` field (or set it to "") from the offending property definition.
2. If you intended inverted-index behavior on text, change the property dataType to `text` or `text[]` and keep the tokenization value.
3. If you only wanted the property excluded from filtering/indexing, use `indexFilterable: false` / `indexSearchable: false` instead of tokenization.

Example fix

// before
{"name":"age","dataType":["int"],"tokenization":"word"}
// after
{"name":"age","dataType":["int"]}

const TOKENIZABLE = new Set(['text','text[]','value','value[]']);
function validateTokenization(prop) {
  if (prop.tokenization && !TOKENIZABLE.has(prop.dataType[0])) {
    throw new Error(`property '${prop.name}': tokenization is not supported for dataType '${prop.dataType[0]}'`);
  }
}

function supportsTokenization(prop) {
  return ['text','text[]'].includes(prop.dataType[0]);
}

When it happens

Trigger: POST /v1/schema (create class) or POST /v1/schema/{className} (add property) or PATCH property update where a property has a primitive dataType such as 'int', 'number', 'date', 'boolean' AND a non-empty `tokenization` field (e.g. tokenization: "word").

Common situations: Copying a property definition from a text property (which legally uses tokenization like 'word', 'field', 'truncate') onto a numeric/date property; client codegen that always serializes an empty-string-defaulted tokenization field; bulk-migrating schemas between collections and blanket-adding tokenization.

Related errors

• invalid tokenization %q
• unsupported tokenization strategy: %s
• invalid stopwords: %w
• failed to create stopword detector: %w
• cannot handle tokenization '%v' of property '%s'

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/f2e5ce45903c96ad. Report an issue: GitHub.