weaviate/weaviate · critical

checksum validation of file %q failed, expected %d, got %d

Error message

checksum validation of file %q failed, expected %d, got %d

What it means

After a snapshot file is downloaded to a temp path, its CRC32 is compared with the checksum recorded by the source node in the file metadata. A mismatch means the transferred bytes differ from what the source advertised — corrupted transfer or stale/incorrect metadata. The copy aborts, the temp file is deleted, and the target refuses to promote a file it cannot trust.

Source

Thrown at cluster/replication/copier/copier.go:397

			err = f.Sync()
			if err != nil {
				return fmt.Errorf("fsyncing file %q for writing: %w", tmpPath, err)
			}

			err = f.Close()
			f = nil // prevent deferred close
			if err != nil {
				return fmt.Errorf("closing file: %w", err)
			}

			_, checksum, err = integrity.CRC32(tmpPath)
			if err != nil {
				return fmt.Errorf("calculating checksum for file %q: %w", tmpPath, err)
			}

			if checksum != meta.Crc32 {
				return fmt.Errorf("checksum validation of file %q failed, expected %d, got %d", tmpPath, meta.Crc32, checksum)
			}

			err = os.Rename(tmpPath, localFilePath)
			if err != nil {
				return fmt.Errorf("renaming temporary file %q to final path %q: %w", tmpPath, localFilePath, err)
			}

			return nil
		}(); err != nil {
			rerr := os.Remove(tmpPath)
			if rerr != nil {
				c.logger.Warnf("failed to remove temporary file %q after error", tmpPath)
			}
			return err
		}
	}
}

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Retry the shard replication; a transient network corruption will usually not repeat.
2. Verify network stability between nodes (packet loss, MTU issues).
3. Ensure the source node is healthy and the snapshot was taken with the shard quiesced appropriately.
4. Check both nodes run compatible Weaviate versions; upgrade if snapshot/checksum handling changed.

if err := copier.CopyReplicaFiles(ctx, class, shard, files, opID); err != nil {
    if strings.Contains(err.Error(), "checksum validation of file") {
        // corrupted transfer: retry with backoff; escalate if persistent
        return retryWithBackoff(3, func() error {
            return copier.CopyReplicaFiles(ctx, class, shard, files, opID)
        })
    }
    return err
}

When it happens

Trigger: GetReplicaSnapshotFile stream delivered bytes whose CRC32 != meta.Crc32: truncated/garbled stream, source file mutated mid-snapshot without metadata refresh, or bit-rot on either node.

Common situations: Unstable network between cluster nodes corrupting gRPC streams; source node under concurrent writes to an inconsistent snapshot; mismatched Weaviate versions where snapshot metadata is stale.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• connect: %w
• connect: %w
• read compare digests record: %w
• read digest: %w
• get gRPC connection to %s: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/44b071fa9dd7ce05. Report an issue: GitHub.