{"record":{"id":"44b071fa9dd7ce05","repo":"weaviate/weaviate","slug":"checksum-validation-of-file-q-failed-expected-d","errorCode":null,"errorMessage":"checksum validation of file %q failed, expected %d, got %d","messagePattern":"checksum validation of file %q failed, expected (.+?), got (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"cluster/replication/copier/copier.go","lineNumber":397,"sourceCode":"\n\t\t\terr = f.Sync()\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"fsyncing file %q for writing: %w\", tmpPath, err)\n\t\t\t}\n\n\t\t\terr = f.Close()\n\t\t\tf = nil // prevent deferred close\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"closing file: %w\", err)\n\t\t\t}\n\n\t\t\t_, checksum, err = integrity.CRC32(tmpPath)\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"calculating checksum for file %q: %w\", tmpPath, err)\n\t\t\t}\n\n\t\t\tif checksum != meta.Crc32 {\n\t\t\t\treturn fmt.Errorf(\"checksum validation of file %q failed, expected %d, got %d\", tmpPath, meta.Crc32, checksum)\n\t\t\t}\n\n\t\t\terr = os.Rename(tmpPath, localFilePath)\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"renaming temporary file %q to final path %q: %w\", tmpPath, localFilePath, err)\n\t\t\t}\n\n\t\t\treturn nil\n\t\t}(); err != nil {\n\t\t\trerr := os.Remove(tmpPath)\n\t\t\tif rerr != nil {\n\t\t\t\tc.logger.Warnf(\"failed to remove temporary file %q after error\", tmpPath)\n\t\t\t}\n\t\t\treturn err\n\t\t}\n\t}\n}\n","sourceCodeStart":379,"sourceCodeEnd":415,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/cluster/replication/copier/copier.go#L379-L415","documentation":"After a snapshot file is downloaded to a temp path, its CRC32 is compared with the checksum recorded by the source node in the file metadata. A mismatch means the transferred bytes differ from what the source advertised — corrupted transfer or stale/incorrect metadata. The copy aborts, the temp file is deleted, and the target refuses to promote a file it cannot trust.","triggerScenarios":"GetReplicaSnapshotFile stream delivered bytes whose CRC32 != meta.Crc32: truncated/garbled stream, source file mutated mid-snapshot without metadata refresh, or bit-rot on either node.","commonSituations":"Unstable network between cluster nodes corrupting gRPC streams; source node under concurrent writes to an inconsistent snapshot; mismatched Weaviate versions where snapshot metadata is stale.","solutions":["Retry the shard replication; a transient network corruption will usually not repeat.","Verify network stability between nodes (packet loss, MTU issues).","Ensure the source node is healthy and the snapshot was taken with the shard quiesced appropriately.","Check both nodes run compatible Weaviate versions; upgrade if snapshot/checksum handling changed."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := copier.CopyReplicaFiles(ctx, class, shard, files, opID); err != nil {\n    if strings.Contains(err.Error(), \"checksum validation of file\") {\n        // corrupted transfer: retry with backoff; escalate if persistent\n        return retryWithBackoff(3, func() error {\n            return copier.CopyReplicaFiles(ctx, class, shard, files, opID)\n        })\n    }\n    return err\n}","preventionTips":["Use reliable, low-loss networking between cluster nodes.","Avoid taking snapshots while the source shard is under heavy concurrent writes.","Keep all nodes on the same Weaviate version so snapshot metadata is consistent.","Enable data-path monitoring/alerting for CRC and I/O errors."],"tags":["replication","integrity","network","corruption"],"backgroundTag":"checksum-mismatch","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}