weaviate/weaviate · error

count %d too large for remaining %d bytes

Error message

count %d too large for remaining %d bytes

What it means

The declared map count exceeds what the remaining bytes could possibly hold: each entry requires at least 2 bytes (1 for key, 1 for value), so count > remaining/2 is impossible for well-formed data. This guard exists to prevent a huge count from reaching make([]uint64, count) and panicking on allocation.

Source

Thrown at adapters/repos/db/lsmkv/gobenc/gobenc.go:239

	}
	pos++

	count, n, err := readGobUint(data, pos)
	if err != nil {
		return nil, nil, fmt.Errorf("read map count: %w", err)
	}
	pos += n

	// the count varint is bounded only against len(data), so it can run past
	// msgEnd; without this, uint64(msgEnd-pos) underflows and the guard below
	// admits a huge count that make([]uint64, count) panics on.
	if pos > msgEnd {
		return nil, nil, fmt.Errorf("map count truncated: read to offset %d past message end %d", pos, msgEnd)
	}

	// Each entry is at least 2 bytes (1 byte key + 1 byte value).
	if count > uint64(msgEnd-pos)/2 {
		return nil, nil, fmt.Errorf("count %d too large for remaining %d bytes", count, msgEnd-pos)
	}

	ids := make([]uint64, count)
	lens := make([]uint32, count)

	for i := range count {
		key, n, err := readGobUint(data, pos)
		if err != nil {
			return nil, nil, fmt.Errorf("read key %d: %w", i, err)
		}
		pos += n

		val, n, err := readGobUint(data, pos)
		if err != nil {
			return nil, nil, fmt.Errorf("read value %d: %w", i, err)
		}
		pos += n

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Reject the record as malformed; do not attempt re-decoding the same bytes.
2. Restore the record from a replica/backup or replay it from the WAL.
3. Compare the count field against the message length manually to locate the corruption.
4. Keep this guard in place — removing it reintroduces an allocation-panic DoS on untrusted input.

remaining := uint64(msgEnd - pos)
if count > remaining/2 { /* malformed: reject before allocating */ }

if _, _, err := gobenc.DecodePairs(data); err != nil {
    log.Warnf("rejecting malformed map record: %v", err)
    return errSkipRecord
}

When it happens

Trigger: Calling DecodePairs/Decode on data where the count varint decodes to a value larger than (msgEnd-pos)/2 — typical of corrupted or adversarial/fuzz input.

Common situations: Flipped bits in the count field, data written by a different encoder version, or deliberate malformed payloads surfaced by fuzz testing (see TestDecodePairsMalformedCountNoPanic).

Related errors

• map count truncated: read to offset %d past message end %d
• unexpected error decoding map values: %w
• read map count: %w
• read key %d: %w
• read value %d: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/9822027d2bfe3507. Report an issue: GitHub.