weaviate/weaviate · error

write key length encoding for node

Error message

write key length encoding for node

What it means

At the end of KeyIndexAndWriteToRedux the primary key is serialized: a 4-byte little-endian key length header first. This error wraps failure writing those 4 bytes. It means the destination writer failed after all values were written, so the node was left partially serialized — the segment on disk is incomplete and must not be trusted. Root cause is a storage-level failure of the writer.

Source

Thrown at adapters/repos/db/lsmkv/segment_serialization.go:441

		valueLen := uint64(len(value.value))
		binary.LittleEndian.PutUint64(buf[1:9], valueLen)
		if _, err := w.Write(buf[0:9]); err != nil {
			return segmentindex.KeyRedux{}, errors.Wrapf(err, "write len of value %d", i)
		}
		written += 9

		n, err := w.Write(value.value)
		if err != nil {
			return segmentindex.KeyRedux{}, errors.Wrapf(err, "write value %d", i)
		}
		written += n
	}

	keyLength := uint32(len(s.primaryKey))
	binary.LittleEndian.PutUint32(buf[0:4], keyLength)
	if _, err := w.Write(buf[0:4]); err != nil {
		return segmentindex.KeyRedux{}, errors.Wrapf(err, "write key length encoding for node")
	}
	written += 4

	n, err := w.Write(s.primaryKey)
	if err != nil {
		return segmentindex.KeyRedux{}, errors.Wrapf(err, "write node")
	}
	written += n

	return segmentindex.KeyRedux{
		ValueEnd: s.offset + written,
		Key:      s.primaryKey,
	}, nil
}

func (s segmentCollectionNode) KeyIndexAndWriteTo(w io.Writer) (segmentindex.Key, error) {
	out := segmentindex.Key{}
	written := 0

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Fix the storage issue (space/IO/permissions) and retry the flush or compaction
2. Remove the partially written segment file so startup does not parse a torn node
3. Re-run compaction from the source segments once storage is healthy
4. Alert on disk usage; keep headroom for compaction (can temporarily need ~2x segment space)
5. Inspect the wrapped root error for the exact errno

// check remaining space covers the tail of the node being written
if freeBytes(dataDir) < int64(4+len(node.primaryKey))+minHeadroom {
    return fmt.Errorf("insufficient space to finish node serialization")
}

redux, err := node.KeyIndexAndWriteToRedux(w, buf)
if err != nil {
    if strings.Contains(err.Error(), "write key length encoding") {
        // node left partially written: remove segment output, fix storage, retry
    }
    return err
}

When it happens

Trigger: During collection flush/compaction, the writer fails while emitting the key-length header after values were written: ENOSPC, EIO, or closed file descriptor. Also possible when writing to an in-memory buffer that is exhausted (rare, mainly tests/benchmarks).

Common situations: Disk hitting capacity during the tail end of a compaction; disk error late in a large flush; process resource exhaustion causing fd errors.

Related errors

• write values len for node
• write len of value %d
• write value %d
• write node
• write empty header: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/b000e2c4c9bdf7e0. Report an issue: GitHub.