hashicorp/nomad · error

failed to read snapshot metadata: %v

Error message

failed to read snapshot metadata: %v

What it means

This error wraps an io.ReadAll failure while reading the meta.json entry (Raft snapshot metadata) from the tar archive, with bytes teed into the SHA-256 hash. It indicates the underlying snapshot stream failed mid-read of the metadata file (I/O error on the source reader), not a JSON problem. The wrapped %v contains the underlying read error.

Source

Thrown at helper/snapshot/archive.go:209

			break
		}
		if err != nil {
			return fmt.Errorf("failed reading snapshot: %v", err)
		}

		switch hdr.Name {
		case "meta.json":
			// Previously we used json.Decode to decode the archive stream. There are
			// edgecases in which it doesn't read all the bytes from the stream, even
			// though the json object is still being parsed properly. Since we
			// simultaneously feeded everything to metaHash, our hash ended up being
			// different than what we calculated when creating the snapshot. Which in
			// turn made the snapshot verification fail. By explicitly reading the
			// whole thing first we ensure that we calculate the correct hash
			// independent of how json.Decode works internally.
			buf, err := io.ReadAll(io.TeeReader(archive, metaHash))
			if err != nil {
				return fmt.Errorf("failed to read snapshot metadata: %v", err)
			}
			if err := json.Unmarshal(buf, &metadata); err != nil {
				return fmt.Errorf("failed to decode snapshot metadata: %v", err)
			}

		case "state.bin":
			if _, err := io.Copy(io.MultiWriter(snap, snapHash), archive); err != nil {
				return fmt.Errorf("failed to read or write snapshot data: %v", err)
			}

		case "SHA256SUMS":
			if _, err := io.Copy(&shaBuffer, archive); err != nil {
				return fmt.Errorf("failed to read snapshot hashes: %v", err)
			}

		default:
			return fmt.Errorf("unexpected file %q in snapshot", hdr.Name)
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the snapshot download/restore operation; transient stream failures are the usual cause
2. Verify stability of the storage/network path hosting the snapshot file
3. Re-take the snapshot if the source copy is consistently unreadable
4. Check the wrapped error for the concrete I/O failure (connection reset, permission, device error) and address it directly

Example fix

// before
err := snapshot.Restore(logOut, snap)
// after
var lastErr error
for i := 0; i < 3; i++ {
    if err := snapshot.Restore(logOut, openSnapshot()); err != nil {
        lastErr = err
        time.Sleep(time.Second)
        continue
    }
    lastErr = nil
    break
}
return lastErr

// pre-check: source is reachable and stable before starting
f, err := os.Open(snapshotPath)
if err != nil {
    return fmt.Errorf("snapshot source unavailable: %w", err)
}
if _, err := f.Stat(); err != nil {
    return fmt.Errorf("snapshot source unreadable: %w", err)
}

func isMetadataReadErr(err error) bool {
    return err != nil && strings.Contains(err.Error(), "failed to read snapshot metadata")
}

var err error
for attempt := 0; attempt < 3; attempt++ {
    err = snapshot.Restore(logOut, reopenSnapshot())
    if err == nil || !strings.Contains(err.Error(), "failed to read snapshot metadata") {
        break
    }
    time.Sleep(2 * time.Second) // transient stream failure: back off and retry
}
return err

When it happens

Trigger: read() hits the 'meta.json' entry and io.ReadAll(io.TeeReader(archive, metaHash)) errors: the input stream (file, network connection, HTTP body) fails or is reset partway through reading the metadata section of the tar.

Common situations: Network connection reset while downloading a snapshot from a remote agent; snapshot file on flaky storage; TLS/SSH tunnel dropping mid-transfer; reading a snapshot mounted over a failing network share.

Related errors

• Failed to copy snapshot to temporary file: %v
• failed to finalize snapshot: %v
• failed to read or write snapshot data: %v
• failed to read snapshot hashes: %v
• failed to open snapshot: %v:

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/f451006d8e1d6cff. Report an issue: GitHub.