weaviate/weaviate · critical

read RQ mean

Error message

read RQ mean

What it means

This error is wrapped around a binary.Read failure while deserializing the centering mean vector of a centered-RQ (residual quantization) payload from an HNSW snapshot file. Each mean entry is a little-endian uint32 of float bits; if the underlying reader returns an error (almost always io.EOF/unexpected EOF), the mean vector cannot be reconstructed and snapshot deserialization aborts. It indicates the snapshot file ended (or the reader failed) mid-way through the mean section.

Source

Thrown at adapters/repos/db/vector/hnsw/compact/snapshot_reader.go:301

	}
	if err := applyRQCenteredFlags(res.CompressionRQData(), flags); err != nil {
		return err
	}
	var meanLen uint32
	if err := binary.Read(reader, binary.LittleEndian, &meanLen); err != nil {
		return errors.Wrap(err, "read RQ mean length")
	}
	// The mean always has exactly InputDim entries; validating before the
	// allocation stops a damaged snapshot from requesting an arbitrarily
	// large slice.
	if inputDim := res.CompressionRQData().InputDim; meanLen != inputDim {
		return errors.Errorf("centered RQ mean length %d does not match input dimension %d", meanLen, inputDim)
	}
	mean := make([]float32, meanLen)
	for i := range mean {
		var bits uint32
		if err := binary.Read(reader, binary.LittleEndian, &bits); err != nil {
			return errors.Wrap(err, "read RQ mean")
		}
		mean[i] = math.Float32frombits(bits)
	}
	res.CompressionRQData().Mean = mean
	return nil
}

// readPQData reads Product Quantization data from the reader.
func (r *SnapshotReader) readPQData(reader io.Reader, res *ent.DeserializationResult) error {
	var dims, ks, m uint16
	var encoderType, dist uint8
	var useBitsEncoding uint8

	if err := binary.Read(reader, binary.LittleEndian, &dims); err != nil {
		return errors.Wrap(err, "read PQ dimensions")
	}
	if err := binary.Read(reader, binary.LittleEndian, &ks); err != nil {
		return errors.Wrap(err, "read PQ Ks")

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Re-create the snapshot (re-export/re-upload the shard) and restore from the fresh, complete file; verify its byte size matches the source.
2. Checksum (e.g. sha256) the snapshot file against the source and re-transfer if it differs.
3. Confirm the snapshot was produced by a compatible Weaviate version and that the whole file (including the trailing mean section) was serialized, not just the header.
4. If truncation is recurring, check disk space and write-flush behavior on the node producing snapshots; a full disk mid-write yields truncated files.

Example fix

// before: reading a locally truncated/partial snapshot stream
f, _ := os.Open("shard-snapshot.bin")
r := compact.NewSnapshotReader(f)
res, err := r.Do(context.Background(), shardName, className) // fails: read RQ mean: unexpected EOF

// after: validate completeness (size/checksum) before deserializing
want, _ := sourceObject.Size() // e.g. from remote storage API
got, _ := os.Stat("shard-snapshot.bin")
if got.Size() != want {
    return errors.New("snapshot incomplete, re-download before restoring")
}
f, _ := os.Open("shard-snapshot.bin")
r := compact.NewSnapshotReader(f)
res, err := r.Do(context.Background(), shardName, className)

info, err := os.Stat(snapshotPath)
if err != nil { return err }
// A centered-RQ snapshot must be at least large enough for the header + meanLen*4 bytes;
// reject obviously short files before attempting deserialization.
if info.Size() < minViableSnapshotBytes {
    return fmt.Errorf("snapshot %s is %d bytes; expected >= %d — re-export before restore", snapshotPath, info.Size(), minViableSnapshotBytes)
}
if sum := sha256File(snapshotPath); sum != expectedChecksum {
    return fmt.Errorf("snapshot checksum mismatch: got %s, want %s", sum, expectedChecksum)
}

func snapshotLooksComplete(info os.FileInfo, expectedSize int64) bool {
    return info != nil && !info.IsDir() && info.Size() >= expectedSize
}

res, err := snapshotReader.Do(ctx, shard, class)
if err != nil {
    var eof *fs.PathError
    if errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) || strings.Contains(err.Error(), "read RQ mean") {
        return fmt.Errorf("snapshot truncated or corrupt (centered-RQ mean unreadable): %w — re-export the snapshot", err)
    }
    return err
}

When it happens

Trigger: Deserializing an HNSW snapshot whose compression type is SnapshotCompressionTypeRQCentered via SnapshotReader.readRQCenteredData (called from readCompressionData), when the reader is exhausted before meanLen float32 values can be read — i.e. truncated file, partially written/failed snapshot upload, or reading beyond the metadata section.

Common situations: Restoring a shard from a replica snapshot that was cut short by a crash or network interruption; copying snapshot files with rsync/scp before they finished flushing; mixing snapshot files from different Weaviate versions where the centered-RQ layout changed; a corrupted disk sector in the snapshot object on the backup target.

Related errors

• read PQ dimensions
• read PQ Ks
• read PQ M
• read PQ encoder type
• read level

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/8ab83079bb3b2223. Report an issue: GitHub.