weaviate/weaviate · error

read RQ outputDim

Error message

read RQ outputDim

What it means

Produced in SnapshotReader.readRQData (snapshot_reader.go:469) when binary.Read fails decoding the RQ 'outputDim' uint32 — the third field of the RQ header. inputDim and bits were read, so the stream ended (EOF/unexpected EOF) or errored partway through the 12-byte RQ header. This signals a truncated or corrupt snapshot, or a byte-layout/version mismatch between writer and reader.

Source

Thrown at adapters/repos/db/vector/hnsw/compact/snapshot_reader.go:469

		Dimensions: dims,
		A:          math.Float32frombits(aBits),
		B:          math.Float32frombits(bBits),
	})
	return nil
}

// readRQData reads Rotational Quantization data from the reader.
func (r *SnapshotReader) readRQData(reader io.Reader, res *ent.DeserializationResult) error {
	var inputDim, bits, outputDim, rounds uint32

	if err := binary.Read(reader, binary.LittleEndian, &inputDim); err != nil {
		return errors.Wrap(err, "read RQ inputDim")
	}
	if err := binary.Read(reader, binary.LittleEndian, &bits); err != nil {
		return errors.Wrap(err, "read RQ bits")
	}
	if err := binary.Read(reader, binary.LittleEndian, &outputDim); err != nil {
		return errors.Wrap(err, "read RQ outputDim")
	}
	if err := binary.Read(reader, binary.LittleEndian, &rounds); err != nil {
		return errors.Wrap(err, "read RQ rounds")
	}

	// Read swaps
	swaps := make([][]compression.Swap, rounds)
	for i := uint32(0); i < rounds; i++ {
		swaps[i] = make([]compression.Swap, outputDim/2)
		for j := uint32(0); j < outputDim/2; j++ {
			if err := binary.Read(reader, binary.LittleEndian, &swaps[i][j].I); err != nil {
				return errors.Wrap(err, "read RQ swap I")
			}
			if err := binary.Read(reader, binary.LittleEndian, &swaps[i][j].J); err != nil {
				return errors.Wrap(err, "read RQ swap J")
			}
		}
	}

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Re-acquire the snapshot from the source and validate its size/checksum; truncated data inside the header cannot be reconstructed.
2. Match Weaviate versions between snapshot writer and reader; regenerate the snapshot after upgrading if formats differ.
3. Rule out disk-full conditions on the node that wrote the snapshot (ENOSPC produces short writes that surface as short reads here).
4. If I/O error rather than EOF, check storage health (dmesg, smartctl, mount logs).
5. Rebuild the shard from a replica or re-ingest data; consider disabling RQ temporarily to keep the collection available.

Example fix

// before: restore proceeds even when the copy was partial
err := reader.Deserialize(snapshotFile)

// after: compare against source-reported size before use
if snapshotFileStat.Size() != manifest.Size {
    return fmt.Errorf("snapshot incomplete: have %d bytes, manifest says %d",
        snapshotFileStat.Size(), manifest.Size)
}
err := reader.Deserialize(snapshotFile)

func checkSnapshotSizeAgainstManifest(path string, manifestSize int64) error {
    fi, err := os.Stat(path)
    if err != nil {
        return err
    }
    if fi.Size() != manifestSize {
        return fmt.Errorf("snapshot size mismatch: file=%d manifest=%d", fi.Size(), manifestSize)
    }
    return nil
}

func isTruncatedStream(err error) bool {
    return errors.Is(err, io.ErrUnexpectedEOF) || errors.Is(err, io.EOF)
}

if err := reader.Deserialize(snapshot); err != nil {
    if isTruncatedStream(err) && strings.Contains(err.Error(), "read RQ outputDim") {
        logger.Warnf("snapshot ended inside RQ header; failing over to replica: %v", err)
        return failoverToReplica()
    }
    return err
}

When it happens

Trigger: Snapshot deserialization (via readCompressionData or readRQCenteredData) where only 4–8 bytes of the RQ header are present in the reader; typical of a file truncated mid-header or a reader whose position is offset from the true RQ section start.

Common situations: Interrupted snapshot flush (process killed, disk full); partial rsync/scp of shard directories; restoring backups across Weaviate versions with changed compression serialization; corrupt object-store downloads used for snapshot restore.

Related errors

• read RQ inputDim
• read RQ bits
• read RQ rounds
• read RQ swap I
• read center value

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/c2170ffdb395903e. Report an issue: GitHub.