could not marshal 'targetVectorsOffsets' max length exceeded

Error message

could not marshal 'targetVectorsOffsets' max length exceeded (%d/%d)

What it means

prepareMarshal rejects the msgpack-encoded target-vectors offsets map when it exceeds maxTargetVectorsOffsetsLength (math.MaxUint32 ≈ 4 GiB). The offsets blob is stored with a uint32 length prefix in the binary format. With offsets being a name->uint32 map, reaching 4 GiB implies millions of named vectors, so this practically indicates a data-construction bug rather than legitimate use.

Source

Thrown at entities/storobj/storage_object.go:1188

			offsetsMap[name] = uint32(targetVectorsSegmentLength)
			targetVectorsSegmentLength += 2 + 4*len(vec) // 2 for vec length + vec bytes

			if targetVectorsSegmentLength > maxTargetVectorsSegmentLength {
				return pm,
					fmt.Errorf("could not marshal '%s' max length exceeded (%d/%d)",
						"targetVectorsSegmentLength", targetVectorsSegmentLength, maxTargetVectorsSegmentLength)
			}

			pm.targetVectors = append(pm.targetVectors, vec)
		}

		if len(offsetsMap) > 0 {
			pm.targetVectorsOffsets, err = msgpack.Marshal(offsetsMap)
			if err != nil {
				return pm, fmt.Errorf("could not marshal target vectors offsets: %w", err)
			}
			if len(pm.targetVectorsOffsets) > maxTargetVectorsOffsetsLength {
				return pm, fmt.Errorf("could not marshal '%s' max length exceeded (%d/%d)", "targetVectorsOffsets", len(pm.targetVectorsOffsets), maxTargetVectorsOffsetsLength)
			}
		}
	}
	pm.targetVectorsSegmentLength = uint32(targetVectorsSegmentLength)

	var multiVectorsSegmentLength int
	if (includeAllTargetVectors || includeSpecificTargetVectors) && len(ko.MultiVectors) > 0 {
		offsetsMap := map[string]uint32{}
		pm.multiVectors = make([][][]float32, 0, len(ko.MultiVectors))
		for name, vecs := range ko.MultiVectors {
			// Skip if we're filtering and this vector wasn't requested
			if includeSpecificTargetVectors {
				if _, ok := requestedVectors[name]; !ok {
					continue
				}
			}
			offsetsMap[name] = uint32(multiVectorsSegmentLength)
			// 4 bytes for number of vectors

View on GitHub (pinned to 75aa4b6d11)

Solutions

Find and cap the code path that creates named vectors — legitimate schemas have bounded named-vector counts.
Validate schema definitions so only declared named vectors are written.
Inspect the offending object and drop excess named vectors before re-ingesting.
Add a pre-marshal assertion: len(namedVectors) within a sane bound (e.g. < 1000).

Example fix

// before
for i, v := range allChunks {
    obj.WithVector(fmt.Sprintf("vec_%d", i), v) // unbounded named vectors
}

// after
const maxNamedVectors = 1000
if len(allChunks) > maxNamedVectors {
    return errors.New("too many named vectors; chunk into separate objects")
}

Defensive patterns

Strategy: validation

Validate before calling

const maxNamedVectors = 1000 // sane bound, far below the 4GiB offsets limit
if len(obj.NamedVectors) > maxNamedVectors {
    return fmt.Errorf("object has %d named vectors, max %d", len(obj.NamedVectors), maxNamedVectors)
}

Try / catch

if err := marshal(o); err != nil {
    if strings.Contains(err.Error(), "could not marshal 'targetVectorsOffsets'") {
        logger.Errorf("object %s has runaway named-vector count: %v", o.ID, err)
    }
    return err
}

Prevention

Cap named-vector creation loops (e.g. one vector per chunk) with an upper bound.
Reject schemas with unbounded dynamic named-vector names at ingestion time.
Alert on objects whose offsets map size grows abnormally during imports.
If triggered by a loop bug, split chunks across multiple objects instead of one.

When it happens

Trigger: Marshalling a multi-vector object where msgpack-encoded offsetsMap exceeds ~4 GiB — i.e. an object with an absurd number of named target vectors (buggy ingestion creating unbounded named vectors) during batch import or replication.

Common situations: Ingestion bugs generating a named vector per token/line without a cap; runaway loops appending named vectors to one object; corrupted objects in replication queues.

Related errors

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/e75441c5e563ffb4. Report an issue: GitHub.