weaviate/weaviate · error

read key

Error message

read key

What it means

ParseReplaceNode reads the primary key after decoding its 4-byte length: it allocates keyLength bytes and calls io.ReadFull to fill them. This error means the underlying read returned fewer bytes than keyLength — almost always io.ErrUnexpectedEOF — so the key bytes are missing. Since the key length itself was decoded successfully, this usually indicates a value/length mismatch: a corrupted, truncated, or misparsed record whose key region extends past the end of the data.

Source

Thrown at adapters/repos/db/lsmkv/segment_serialization.go:153

	out.tombstone = tmpBuf[0] != 0
	valueLength := binary.LittleEndian.Uint64(tmpBuf[1:9])
	out.value = make([]byte, valueLength)
	if n, err := io.ReadFull(r, out.value); err != nil {
		return out, errors.Wrap(err, "read value")
	} else {
		out.offset += n
	}

	if n, err := io.ReadFull(r, tmpBuf[0:4]); err != nil {
		return out, errors.Wrap(err, "read key length encoding")
	} else {
		out.offset += n
	}

	keyLength := binary.LittleEndian.Uint32(tmpBuf[0:4])
	out.primaryKey = make([]byte, keyLength)
	if n, err := io.ReadFull(r, out.primaryKey); err != nil {
		return out, errors.Wrap(err, "read key")
	} else {
		out.offset += n
	}

	out.secondaryIndexCount = secondaryIndexCount
	if secondaryIndexCount > 0 {
		out.secondaryKeys = make([][]byte, secondaryIndexCount)
	}

	for j := 0; j < int(secondaryIndexCount); j++ {
		if n, err := io.ReadFull(r, tmpBuf[0:4]); err != nil {
			return out, errors.Wrap(err, "read secondary key length encoding")
		} else {
			out.offset += n
		}
		secKeyLen := binary.LittleEndian.Uint32(tmpBuf[0:4])
		if secKeyLen == 0 {
			continue

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Identify the affected segment file from the shard logs, stop Weaviate, and let segment recovery/validation drop or repair the truncated segment.
2. Restore the shard from a verified backup taken while the instance was stopped or via the backup module.
3. Check disk space, dmesg/filesystem errors; repair the volume before restarting.
4. If unrecoverable, delete the shard/collection and re-ingest.
5. Report to weaviate if reproducible on healthy storage — a writer bug may be producing malformed records.

Example fix

// before: trusting a length parsed from possibly-corrupt bytes
keyLength := binary.LittleEndian.Uint32(tmpBuf[0:4])
out.primaryKey = make([]byte, keyLength)
// after: validate the declared length against the remaining record budget
remaining := recEnd - uint64(out.offset)
if uint64(keyLength) > remaining {
    return out, fmt.Errorf("key length %d exceeds remaining record bytes %d", keyLength, remaining)
}
out.primaryKey = make([]byte, keyLength)

// Validate segment integrity before opening it for reads (while stopped):
size := fileSize(segPath)
// each record's declared keyLength must fit within [recordStart, size]
if recStart + 9 + valueLen + 4 + uint64(keyLength) > size {
    return fmt.Errorf("segment %s record at %d overruns file", segPath, recStart)
}

func isTruncatedSegmentError(err error) bool {
    return errors.Is(err, io.ErrUnexpectedEOF)
}

if err := lsmkv.ParseReplaceNodeInto(bufReader, secIdxCount, node); err != nil {
    if isTruncatedSegmentError(err) {
        log.Warnf("segment truncated mid-key, quarantining file: %v", err)
        return quarantineSegment(segPath)
    }
    return err
}

When it happens

Trigger: Calling ParseReplaceNode (via bucket cursors, compaction, or segment recovery) on a segment whose file ends inside the key region: file truncated after the key-length field, or an earlier desynchronized field caused an absurd keyLength (up to 4 GiB) to be allocated and read from a short reader.

Common situations: Torn writes from a crash or power loss; disk-full during a flush; incomplete backup restore; a truncated segment file copied out of band; byte-shifted records after corruption upstream in the same record.

Related errors

• read key length encoding
• read secondary key
• read secondary key length encoding
• failed adding to prop '%s' value bucket
• failed adding to prop '%s' length bucket

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/af44a17a452c4eee. Report an issue: GitHub.