weaviate/weaviate · error

property '%s': unsupported tokenization '%s'

Error message

property '%s': unsupported tokenization '%s'

What it means

Each property tokenization must be one of the supported tokenizers (lowercase, trigram, word, whitespace, field, and related variants accepted in the switch). This error is thrown in validatePropertyProcessing when the property's tokenization field is set to a value outside the supported set, because Weaviate cannot determine how to split and normalize the property value for the inverted index.

Source

Thrown at usecases/schema/class.go:1376

	}

	for _, entry := range prop.TextAnalyzer.ASCIIFoldIgnore {
		if utf8.RuneCountInString(norm.NFC.String(entry)) != 1 {
			return fmt.Errorf("property '%s': each asciiFoldIgnore entry must be a single character, got %q",
				prop.Name, entry)
		}
	}

	// explicitly check for support for tokenizers:
	if prop.Tokenization != "" {
		switch prop.Tokenization {
		case models.PropertyTokenizationLowercase,
			models.PropertyTokenizationTrigram,
			models.PropertyTokenizationWord,
			models.PropertyTokenizationWhitespace,
			models.PropertyTokenizationField: // supported tokenizers, do nothing
		default:
			return fmt.Errorf("property '%s': unsupported tokenization '%s'", prop.Name, prop.Tokenization)
		}
	}

	if prop.TextAnalyzer.StopwordPreset != "" {
		if prop.Tokenization != models.PropertyTokenizationWord {
			return fmt.Errorf("property '%s': stopwordPreset is only supported with tokenization %q, got %q",
				prop.Name, models.PropertyTokenizationWord, prop.Tokenization)
		}
		_, builtIn := stopwords.Presets[prop.TextAnalyzer.StopwordPreset]
		_, userDefined := userPresets[prop.TextAnalyzer.StopwordPreset]
		if !builtIn && !userDefined {
			return fmt.Errorf("property '%s': unknown stopword preset %q; must be a built-in preset ('en', 'none') or defined in invertedIndexConfig.stopwordPresets",
				prop.Name, prop.TextAnalyzer.StopwordPreset)
		}
	}

	return nil
}

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Set tokenization to one of the supported values: 'lowercase', 'trigram', 'word', 'whitespace', 'field' (check docs for your version)
2. Fix the casing/typo in the tokenization value (comparison is exact string match)
3. Check your Weaviate version's supported tokenizers (e.g. 'field' and 'trigram' exist only in newer versions) and upgrade or downgrade the config accordingly

Example fix

// before
{"name": "text", "dataType": ["text"], "tokenization": "whitespac"}
// after
{"name": "text", "dataType": ["text"], "tokenization": "whitespace"}

var validTokenizations = map[string]bool{"lowercase": true, "trigram": true, "word": true, "whitespace": true, "field": true}

func validTokenization(p *models.Property) bool {
  return p.Tokenization == "" || validTokenizations[p.Tokenization]
}

When it happens

Trigger: Creating or updating a class with a property whose tokenization is misspelled (e.g. 'whitespac', 'Word', 'GSEN') or uses a tokenizer removed from the installed Weaviate version. Any schema PUT/POST carrying that property fails validation.

Common situations: Typo in tokenization name; copying a config from another vector DB; using a tokenizer introduced in a newer Weaviate release than the one running; wrong casing since values are compared exactly.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• invalid tokenization %q
• tokenization is not allowed for data type '%s'
• tokenization '%s' is not allowed for data type '%s'
• tokenization is not allowed for object/object[] data types
• tokenization is not allowed for reference data type

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/3cc24a1dc3c5c87b. Report an issue: GitHub.