{"record":{"id":"75c4639f39818dc2","repo":"hashicorp/nomad","slug":"failed-to-read-or-write-snapshot-data-v","errorCode":null,"errorMessage":"failed to read or write snapshot data: %v","messagePattern":"failed to read or write snapshot data: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"helper/snapshot/archive.go","lineNumber":217,"sourceCode":"\t\t\t// Previously we used json.Decode to decode the archive stream. There are\n\t\t\t// edgecases in which it doesn't read all the bytes from the stream, even\n\t\t\t// though the json object is still being parsed properly. Since we\n\t\t\t// simultaneously feeded everything to metaHash, our hash ended up being\n\t\t\t// different than what we calculated when creating the snapshot. Which in\n\t\t\t// turn made the snapshot verification fail. By explicitly reading the\n\t\t\t// whole thing first we ensure that we calculate the correct hash\n\t\t\t// independent of how json.Decode works internally.\n\t\t\tbuf, err := io.ReadAll(io.TeeReader(archive, metaHash))\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to read snapshot metadata: %v\", err)\n\t\t\t}\n\t\t\tif err := json.Unmarshal(buf, &metadata); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to decode snapshot metadata: %v\", err)\n\t\t\t}\n\n\t\tcase \"state.bin\":\n\t\t\tif _, err := io.Copy(io.MultiWriter(snap, snapHash), archive); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to read or write snapshot data: %v\", err)\n\t\t\t}\n\n\t\tcase \"SHA256SUMS\":\n\t\t\tif _, err := io.Copy(&shaBuffer, archive); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to read snapshot hashes: %v\", err)\n\t\t\t}\n\n\t\tdefault:\n\t\t\treturn fmt.Errorf(\"unexpected file %q in snapshot\", hdr.Name)\n\t\t}\n\t}\n\n\t// Verify all the hashes.\n\tif err := hl.DecodeAndVerify(&shaBuffer); err != nil {\n\t\treturn fmt.Errorf(\"failed checking integrity of snapshot: %v\", err)\n\t}\n\n\treturn nil","sourceCodeStart":199,"sourceCodeEnd":235,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/snapshot/archive.go#L199-L235","documentation":"This error wraps an io.Copy failure while streaming the state.bin entry (the Raft state payload) from the snapshot tar into both the destination writer and the SHA-256 hash. Either reading from the archive stream failed or writing to the caller-supplied snap io.Writer failed. It covers the bulk snapshot-data transfer step of read().","triggerScenarios":"read() (via Restore/CopySnapshot) processes the 'state.bin' entry and io.Copy(io.MultiWriter(snap, snapHash), archive) errors: the source stream dies mid-state, or the destination writer fails (e.g. restoring into a raft log store / file sink that errors — disk full, closed pipe, backend unavailable).","commonSituations":"Disk fills up while restoring a large snapshot into the raft data directory; connection reset during a long snapshot download; restore target file handle closed early; writing to a pipe whose consumer exited.","solutions":["Free disk space on the restore target and retry the restore","Retry the operation if the wrapped error indicates a transient network/stream failure","Ensure the destination io.Writer passed to Restore remains open and healthy for the entire restore","Check the wrapped %v error to identify whether the read side (source snapshot) or write side (destination) failed"],"exampleFix":"// before\nerr := snapshot.Restore(logOut, snap)\n// after\nif err := disk.CheckFreeSpace(dataDir, snapshotSize+margin); err != nil {\n    return fmt.Errorf(\"insufficient space to restore snapshot: %w\", err)\n}\nif err := snapshot.Restore(logOut, snap); err != nil {\n    return fmt.Errorf(\"restore failed; ensure destination writable and stream stable: %w\", err)\n}","handlingStrategy":"try-catch","validationCode":"// ensure destination has room and is writable before starting the restore\nfunc ensureRestoreTarget(path string, needBytes int64) error {\n    var st syscall.Statfs_t\n    if err := syscall.Statfs(path, &st); err != nil {\n        return err\n    }\n    avail := int64(st.Bavail) * st.Bsize\n    if avail < needBytes {\n        return fmt.Errorf(\"only %d bytes free, need %d\", avail, needBytes)\n    }\n    return nil\n}","typeGuard":"func isStateCopyErr(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"failed to read or write snapshot data\")\n}","tryCatchPattern":"if err := snapshot.Restore(logOut, snap); err != nil {\n    if strings.Contains(err.Error(), \"failed to read or write snapshot data\") {\n        log.Error(\"state.bin transfer failed; check destination disk space and stream stability\", \"err\", err)\n        return retryRestoreWithFreshStream()\n    }\n    return err\n}","preventionTips":["Monitor free disk space on the raft data directory before large restores","Keep the destination writer open until Restore returns; never close it early","Use stable, retriable transports (resumable downloads) for large snapshots","Compare snapshot size from meta.json against available space before restoring"],"tags":["go","io","snapshot","disk","stream"],"backgroundTag":"stream-copy-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}