hashicorp/nomad · error

failed to read or write snapshot data: %v

Error message

failed to read or write snapshot data: %v

What it means

This error wraps an io.Copy failure while streaming the state.bin entry (the Raft state payload) from the snapshot tar into both the destination writer and the SHA-256 hash. Either reading from the archive stream failed or writing to the caller-supplied snap io.Writer failed. It covers the bulk snapshot-data transfer step of read().

Source

Thrown at helper/snapshot/archive.go:217

			// Previously we used json.Decode to decode the archive stream. There are
			// edgecases in which it doesn't read all the bytes from the stream, even
			// though the json object is still being parsed properly. Since we
			// simultaneously feeded everything to metaHash, our hash ended up being
			// different than what we calculated when creating the snapshot. Which in
			// turn made the snapshot verification fail. By explicitly reading the
			// whole thing first we ensure that we calculate the correct hash
			// independent of how json.Decode works internally.
			buf, err := io.ReadAll(io.TeeReader(archive, metaHash))
			if err != nil {
				return fmt.Errorf("failed to read snapshot metadata: %v", err)
			}
			if err := json.Unmarshal(buf, &metadata); err != nil {
				return fmt.Errorf("failed to decode snapshot metadata: %v", err)
			}

		case "state.bin":
			if _, err := io.Copy(io.MultiWriter(snap, snapHash), archive); err != nil {
				return fmt.Errorf("failed to read or write snapshot data: %v", err)
			}

		case "SHA256SUMS":
			if _, err := io.Copy(&shaBuffer, archive); err != nil {
				return fmt.Errorf("failed to read snapshot hashes: %v", err)
			}

		default:
			return fmt.Errorf("unexpected file %q in snapshot", hdr.Name)
		}
	}

	// Verify all the hashes.
	if err := hl.DecodeAndVerify(&shaBuffer); err != nil {
		return fmt.Errorf("failed checking integrity of snapshot: %v", err)
	}

	return nil

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Free disk space on the restore target and retry the restore
2. Retry the operation if the wrapped error indicates a transient network/stream failure
3. Ensure the destination io.Writer passed to Restore remains open and healthy for the entire restore
4. Check the wrapped %v error to identify whether the read side (source snapshot) or write side (destination) failed

Example fix

// before
err := snapshot.Restore(logOut, snap)
// after
if err := disk.CheckFreeSpace(dataDir, snapshotSize+margin); err != nil {
    return fmt.Errorf("insufficient space to restore snapshot: %w", err)
}
if err := snapshot.Restore(logOut, snap); err != nil {
    return fmt.Errorf("restore failed; ensure destination writable and stream stable: %w", err)
}

// ensure destination has room and is writable before starting the restore
func ensureRestoreTarget(path string, needBytes int64) error {
    var st syscall.Statfs_t
    if err := syscall.Statfs(path, &st); err != nil {
        return err
    }
    avail := int64(st.Bavail) * st.Bsize
    if avail < needBytes {
        return fmt.Errorf("only %d bytes free, need %d", avail, needBytes)
    }
    return nil
}

func isStateCopyErr(err error) bool {
    return err != nil && strings.Contains(err.Error(), "failed to read or write snapshot data")
}

if err := snapshot.Restore(logOut, snap); err != nil {
    if strings.Contains(err.Error(), "failed to read or write snapshot data") {
        log.Error("state.bin transfer failed; check destination disk space and stream stability", "err", err)
        return retryRestoreWithFreshStream()
    }
    return err
}

When it happens

Trigger: read() (via Restore/CopySnapshot) processes the 'state.bin' entry and io.Copy(io.MultiWriter(snap, snapHash), archive) errors: the source stream dies mid-state, or the destination writer fails (e.g. restoring into a raft log store / file sink that errors — disk full, closed pipe, backend unavailable).

Common situations: Disk fills up while restoring a large snapshot into the raft data directory; connection reset during a long snapshot download; restore target file handle closed early; writing to a pipe whose consumer exited.

Related errors

• failed to finalize snapshot: %v
• Failed to copy snapshot to temporary file: %v
• failed to read snapshot metadata: %v
• failed to read snapshot hashes: %v
• failed to open snapshot: %v:

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/75c4639f39818dc2. Report an issue: GitHub.