{"record":{"id":"c2170ffdb395903e","repo":"weaviate/weaviate","slug":"read-rq-outputdim","errorCode":null,"errorMessage":"read RQ outputDim","messagePattern":"read RQ outputDim","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"adapters/repos/db/vector/hnsw/compact/snapshot_reader.go","lineNumber":469,"sourceCode":"\t\tDimensions: dims,\n\t\tA:          math.Float32frombits(aBits),\n\t\tB:          math.Float32frombits(bBits),\n\t})\n\treturn nil\n}\n\n// readRQData reads Rotational Quantization data from the reader.\nfunc (r *SnapshotReader) readRQData(reader io.Reader, res *ent.DeserializationResult) error {\n\tvar inputDim, bits, outputDim, rounds uint32\n\n\tif err := binary.Read(reader, binary.LittleEndian, &inputDim); err != nil {\n\t\treturn errors.Wrap(err, \"read RQ inputDim\")\n\t}\n\tif err := binary.Read(reader, binary.LittleEndian, &bits); err != nil {\n\t\treturn errors.Wrap(err, \"read RQ bits\")\n\t}\n\tif err := binary.Read(reader, binary.LittleEndian, &outputDim); err != nil {\n\t\treturn errors.Wrap(err, \"read RQ outputDim\")\n\t}\n\tif err := binary.Read(reader, binary.LittleEndian, &rounds); err != nil {\n\t\treturn errors.Wrap(err, \"read RQ rounds\")\n\t}\n\n\t// Read swaps\n\tswaps := make([][]compression.Swap, rounds)\n\tfor i := uint32(0); i < rounds; i++ {\n\t\tswaps[i] = make([]compression.Swap, outputDim/2)\n\t\tfor j := uint32(0); j < outputDim/2; j++ {\n\t\t\tif err := binary.Read(reader, binary.LittleEndian, &swaps[i][j].I); err != nil {\n\t\t\t\treturn errors.Wrap(err, \"read RQ swap I\")\n\t\t\t}\n\t\t\tif err := binary.Read(reader, binary.LittleEndian, &swaps[i][j].J); err != nil {\n\t\t\t\treturn errors.Wrap(err, \"read RQ swap J\")\n\t\t\t}\n\t\t}\n\t}","sourceCodeStart":451,"sourceCodeEnd":487,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/vector/hnsw/compact/snapshot_reader.go#L451-L487","documentation":"Produced in SnapshotReader.readRQData (snapshot_reader.go:469) when binary.Read fails decoding the RQ 'outputDim' uint32 — the third field of the RQ header. inputDim and bits were read, so the stream ended (EOF/unexpected EOF) or errored partway through the 12-byte RQ header. This signals a truncated or corrupt snapshot, or a byte-layout/version mismatch between writer and reader.","triggerScenarios":"Snapshot deserialization (via readCompressionData or readRQCenteredData) where only 4–8 bytes of the RQ header are present in the reader; typical of a file truncated mid-header or a reader whose position is offset from the true RQ section start.","commonSituations":"Interrupted snapshot flush (process killed, disk full); partial rsync/scp of shard directories; restoring backups across Weaviate versions with changed compression serialization; corrupt object-store downloads used for snapshot restore.","solutions":["Re-acquire the snapshot from the source and validate its size/checksum; truncated data inside the header cannot be reconstructed.","Match Weaviate versions between snapshot writer and reader; regenerate the snapshot after upgrading if formats differ.","Rule out disk-full conditions on the node that wrote the snapshot (ENOSPC produces short writes that surface as short reads here).","If I/O error rather than EOF, check storage health (dmesg, smartctl, mount logs).","Rebuild the shard from a replica or re-ingest data; consider disabling RQ temporarily to keep the collection available."],"exampleFix":"// before: restore proceeds even when the copy was partial\nerr := reader.Deserialize(snapshotFile)\n\n// after: compare against source-reported size before use\nif snapshotFileStat.Size() != manifest.Size {\n    return fmt.Errorf(\"snapshot incomplete: have %d bytes, manifest says %d\",\n        snapshotFileStat.Size(), manifest.Size)\n}\nerr := reader.Deserialize(snapshotFile)","handlingStrategy":"validation","validationCode":"func checkSnapshotSizeAgainstManifest(path string, manifestSize int64) error {\n    fi, err := os.Stat(path)\n    if err != nil {\n        return err\n    }\n    if fi.Size() != manifestSize {\n        return fmt.Errorf(\"snapshot size mismatch: file=%d manifest=%d\", fi.Size(), manifestSize)\n    }\n    return nil\n}","typeGuard":"func isTruncatedStream(err error) bool {\n    return errors.Is(err, io.ErrUnexpectedEOF) || errors.Is(err, io.EOF)\n}","tryCatchPattern":"if err := reader.Deserialize(snapshot); err != nil {\n    if isTruncatedStream(err) && strings.Contains(err.Error(), \"read RQ outputDim\") {\n        logger.Warnf(\"snapshot ended inside RQ header; failing over to replica: %v\", err)\n        return failoverToReplica()\n    }\n    return err\n}","preventionTips":["Compare restored snapshot byte counts against the source manifest before deserializing.","Check for ENOSPC on the writing node; full disks are a leading cause of half-written snapshots.","Do not share snapshot directories over unreliable network mounts without integrity checks.","Keep shard replicas so any single corrupt snapshot can be rebuilt from a peer.","Re-export snapshots after Weaviate upgrades instead of carrying old-format files forward."],"tags":["go","deserialization","snapshot","eof","hnsw","rq"],"backgroundTag":"unexpected-eof-while-reading","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}