{"record":{"id":"e75441c5e563ffb4","repo":"weaviate/weaviate","slug":"could-not-marshal-targetvectorsoffsets-max-lengt","errorCode":null,"errorMessage":"could not marshal 'targetVectorsOffsets' max length exceeded (%d/%d)","messagePattern":"could not marshal 'targetVectorsOffsets' max length exceeded \\((.+?)/(.+?)\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"entities/storobj/storage_object.go","lineNumber":1188,"sourceCode":"\t\t\toffsetsMap[name] = uint32(targetVectorsSegmentLength)\n\t\t\ttargetVectorsSegmentLength += 2 + 4*len(vec) // 2 for vec length + vec bytes\n\n\t\t\tif targetVectorsSegmentLength > maxTargetVectorsSegmentLength {\n\t\t\t\treturn pm,\n\t\t\t\t\tfmt.Errorf(\"could not marshal '%s' max length exceeded (%d/%d)\",\n\t\t\t\t\t\t\"targetVectorsSegmentLength\", targetVectorsSegmentLength, maxTargetVectorsSegmentLength)\n\t\t\t}\n\n\t\t\tpm.targetVectors = append(pm.targetVectors, vec)\n\t\t}\n\n\t\tif len(offsetsMap) > 0 {\n\t\t\tpm.targetVectorsOffsets, err = msgpack.Marshal(offsetsMap)\n\t\t\tif err != nil {\n\t\t\t\treturn pm, fmt.Errorf(\"could not marshal target vectors offsets: %w\", err)\n\t\t\t}\n\t\t\tif len(pm.targetVectorsOffsets) > maxTargetVectorsOffsetsLength {\n\t\t\t\treturn pm, fmt.Errorf(\"could not marshal '%s' max length exceeded (%d/%d)\", \"targetVectorsOffsets\", len(pm.targetVectorsOffsets), maxTargetVectorsOffsetsLength)\n\t\t\t}\n\t\t}\n\t}\n\tpm.targetVectorsSegmentLength = uint32(targetVectorsSegmentLength)\n\n\tvar multiVectorsSegmentLength int\n\tif (includeAllTargetVectors || includeSpecificTargetVectors) && len(ko.MultiVectors) > 0 {\n\t\toffsetsMap := map[string]uint32{}\n\t\tpm.multiVectors = make([][][]float32, 0, len(ko.MultiVectors))\n\t\tfor name, vecs := range ko.MultiVectors {\n\t\t\t// Skip if we're filtering and this vector wasn't requested\n\t\t\tif includeSpecificTargetVectors {\n\t\t\t\tif _, ok := requestedVectors[name]; !ok {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t}\n\t\t\toffsetsMap[name] = uint32(multiVectorsSegmentLength)\n\t\t\t// 4 bytes for number of vectors","sourceCodeStart":1170,"sourceCodeEnd":1206,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/entities/storobj/storage_object.go#L1170-L1206","documentation":"prepareMarshal rejects the msgpack-encoded target-vectors offsets map when it exceeds maxTargetVectorsOffsetsLength (math.MaxUint32 ≈ 4 GiB). The offsets blob is stored with a uint32 length prefix in the binary format. With offsets being a name->uint32 map, reaching 4 GiB implies millions of named vectors, so this practically indicates a data-construction bug rather than legitimate use.","triggerScenarios":"Marshalling a multi-vector object where msgpack-encoded offsetsMap exceeds ~4 GiB — i.e. an object with an absurd number of named target vectors (buggy ingestion creating unbounded named vectors) during batch import or replication.","commonSituations":"Ingestion bugs generating a named vector per token/line without a cap; runaway loops appending named vectors to one object; corrupted objects in replication queues.","solutions":["Find and cap the code path that creates named vectors — legitimate schemas have bounded named-vector counts.","Validate schema definitions so only declared named vectors are written.","Inspect the offending object and drop excess named vectors before re-ingesting.","Add a pre-marshal assertion: len(namedVectors) within a sane bound (e.g. < 1000)."],"exampleFix":"// before\nfor i, v := range allChunks {\n    obj.WithVector(fmt.Sprintf(\"vec_%d\", i), v) // unbounded named vectors\n}\n\n// after\nconst maxNamedVectors = 1000\nif len(allChunks) > maxNamedVectors {\n    return errors.New(\"too many named vectors; chunk into separate objects\")\n}","handlingStrategy":"validation","validationCode":"const maxNamedVectors = 1000 // sane bound, far below the 4GiB offsets limit\nif len(obj.NamedVectors) > maxNamedVectors {\n    return fmt.Errorf(\"object has %d named vectors, max %d\", len(obj.NamedVectors), maxNamedVectors)\n}","typeGuard":null,"tryCatchPattern":"if err := marshal(o); err != nil {\n    if strings.Contains(err.Error(), \"could not marshal 'targetVectorsOffsets'\") {\n        logger.Errorf(\"object %s has runaway named-vector count: %v\", o.ID, err)\n    }\n    return err\n}","preventionTips":["Cap named-vector creation loops (e.g. one vector per chunk) with an upper bound.","Reject schemas with unbounded dynamic named-vector names at ingestion time.","Alert on objects whose offsets map size grows abnormally during imports.","If triggered by a loop bug, split chunks across multiple objects instead of one."],"tags":["go","marshalling","named-vectors","limit-exceeded"],"backgroundTag":"payload-size-limit-exceeded","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}