weaviate/weaviate · error

text exceeds maximum allowed length of 10,000 characters

Error message

text exceeds maximum allowed length of 10,000 characters

What it means

Length guard on the /v1/tokenize endpoint: the input text surpassed the hard cap of 10,000 characters, protecting the analyzer from oversized payloads. The request is rejected with 422 before any tokenization runs.

Source

Thrown at adapters/handlers/rest/handlers_tokenize.go:59

			return genericTokenize(principal, params)
		})

	api.SchemaSchemaObjectsPropertiesTokenizeHandler = schemaops.SchemaObjectsPropertiesTokenizeHandlerFunc(
		func(params schemaops.SchemaObjectsPropertiesTokenizeParams, principal *models.Principal) middleware.Responder {
			return propertyTokenize(params, principal, schemaManager, namespacesEnabled, logger)
		})
}

func genericTokenize(principal *models.Principal, params tokenizeops.TokenizeParams) middleware.Responder {
	if !slices.Contains(tokenizer.Tokenizations, *params.Body.Tokenization) {
		return tokenizeops.NewTokenizeUnprocessableEntity().WithPayload(
			errPayloadFromSingleErr(principal, fmt.Errorf("unsupported tokenization strategy: %s", *params.Body.Tokenization)))
	}

	// allow a max length of 10k characters to prevent abuse of this endpoint; the tokenizer can handle more, but it may cause performance issues
	if len(*params.Body.Text) > 10000 {
		return tokenizeops.NewTokenizeUnprocessableEntity().WithPayload(
			errPayloadFromSingleErr(principal, errors.New("text exceeds maximum allowed length of 10,000 characters")))
	}

	if err := validateAnalyzerConfig(params.Body.AnalyzerConfig); err != nil {
		return tokenizeops.NewTokenizeUnprocessableEntity().WithPayload(errPayloadFromSingleErr(principal, err))
	}

	// `stopwords` and `stopwordPresets` are mutually exclusive on this
	// endpoint. `stopwords` is for the simple "apply one base preset
	// optionally tweaked with additions/removals" case. `stopwordPresets`
	// is for the "define named presets and select one via analyzerConfig"
	// case. Allowing both on the same request creates subtle resolution
	// corner cases (e.g. stopwords.preset="en" vs stopwordPresets.en=[...]);
	// forcing callers to pick one keeps the mental model simple.
	if params.Body.Stopwords != nil && len(params.Body.StopwordPresets) > 0 {
		return tokenizeops.NewTokenizeUnprocessableEntity().WithPayload(
			errPayloadFromSingleErr(principal, errors.New("stopwords and stopwordPresets are mutually exclusive; pass only one")))
	}

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Shorten the text to at most 10,000 characters
2. Split large documents into chunks and tokenize each separately

When it happens

Trigger: Thrown at adapters/handlers/rest/handlers_tokenize.go:59 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/dbd20544e05dbb81. Report an issue: GitHub.