{"record":{"id":"dbd20544e05dbb81","repo":"weaviate/weaviate","slug":"text-exceeds-maximum-allowed-length-of-10-000-char","errorCode":null,"errorMessage":"text exceeds maximum allowed length of 10,000 characters","messagePattern":"text exceeds maximum allowed length of 10,000 characters","errorType":"http","errorClass":null,"httpStatus":422,"severity":"error","filePath":"adapters/handlers/rest/handlers_tokenize.go","lineNumber":59,"sourceCode":"\t\t\treturn genericTokenize(principal, params)\n\t\t})\n\n\tapi.SchemaSchemaObjectsPropertiesTokenizeHandler = schemaops.SchemaObjectsPropertiesTokenizeHandlerFunc(\n\t\tfunc(params schemaops.SchemaObjectsPropertiesTokenizeParams, principal *models.Principal) middleware.Responder {\n\t\t\treturn propertyTokenize(params, principal, schemaManager, namespacesEnabled, logger)\n\t\t})\n}\n\nfunc genericTokenize(principal *models.Principal, params tokenizeops.TokenizeParams) middleware.Responder {\n\tif !slices.Contains(tokenizer.Tokenizations, *params.Body.Tokenization) {\n\t\treturn tokenizeops.NewTokenizeUnprocessableEntity().WithPayload(\n\t\t\terrPayloadFromSingleErr(principal, fmt.Errorf(\"unsupported tokenization strategy: %s\", *params.Body.Tokenization)))\n\t}\n\n\t// allow a max length of 10k characters to prevent abuse of this endpoint; the tokenizer can handle more, but it may cause performance issues\n\tif len(*params.Body.Text) > 10000 {\n\t\treturn tokenizeops.NewTokenizeUnprocessableEntity().WithPayload(\n\t\t\terrPayloadFromSingleErr(principal, errors.New(\"text exceeds maximum allowed length of 10,000 characters\")))\n\t}\n\n\tif err := validateAnalyzerConfig(params.Body.AnalyzerConfig); err != nil {\n\t\treturn tokenizeops.NewTokenizeUnprocessableEntity().WithPayload(errPayloadFromSingleErr(principal, err))\n\t}\n\n\t// `stopwords` and `stopwordPresets` are mutually exclusive on this\n\t// endpoint. `stopwords` is for the simple \"apply one base preset\n\t// optionally tweaked with additions/removals\" case. `stopwordPresets`\n\t// is for the \"define named presets and select one via analyzerConfig\"\n\t// case. Allowing both on the same request creates subtle resolution\n\t// corner cases (e.g. stopwords.preset=\"en\" vs stopwordPresets.en=[...]);\n\t// forcing callers to pick one keeps the mental model simple.\n\tif params.Body.Stopwords != nil && len(params.Body.StopwordPresets) > 0 {\n\t\treturn tokenizeops.NewTokenizeUnprocessableEntity().WithPayload(\n\t\t\terrPayloadFromSingleErr(principal, errors.New(\"stopwords and stopwordPresets are mutually exclusive; pass only one\")))\n\t}\n","sourceCodeStart":41,"sourceCodeEnd":77,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/handlers/rest/handlers_tokenize.go#L41-L77","documentation":"Length guard on the /v1/tokenize endpoint: the input text surpassed the hard cap of 10,000 characters, protecting the analyzer from oversized payloads. The request is rejected with 422 before any tokenization runs.","triggerScenarios":"Thrown at adapters/handlers/rest/handlers_tokenize.go:59 when the library encounters an invalid state.","commonSituations":"See trigger scenarios.","solutions":["Shorten the text to at most 10,000 characters","Split large documents into chunks and tokenize each separately"],"exampleFix":null,"handlingStrategy":"validation","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}