hashicorp/nomad · error

failed reading snapshot: %v

Error message

failed reading snapshot: %v

What it means

This error is returned from read() when archive.Next() on the tar reader fails with an error other than io.EOF, i.e. the snapshot stream is not a readable tar archive. It means the bytes being read as a Consul/Raft snapshot are corrupted, truncated, or not a snapshot at all. The wrapped %v contains the tar package's underlying parse error.

Source

Thrown at helper/snapshot/archive.go:194

	// Create a hash list that we will use to compare with the SHA256SUMS
	// file in the archive.
	hl := newHashList()

	// Populate the hashes for all the files we expect to see. The check at
	// the end will make sure these are all present in the SHA256SUMS file
	// and that the hashes match.
	metaHash := hl.Add("meta.json")
	snapHash := hl.Add("state.bin")

	// Look through the archive for the pieces we care about.
	var shaBuffer bytes.Buffer
	for {
		hdr, err := archive.Next()
		if err == io.EOF {
			break
		}
		if err != nil {
			return fmt.Errorf("failed reading snapshot: %v", err)
		}

		switch hdr.Name {
		case "meta.json":
			// Previously we used json.Decode to decode the archive stream. There are
			// edgecases in which it doesn't read all the bytes from the stream, even
			// though the json object is still being parsed properly. Since we
			// simultaneously feeded everything to metaHash, our hash ended up being
			// different than what we calculated when creating the snapshot. Which in
			// turn made the snapshot verification fail. By explicitly reading the
			// whole thing first we ensure that we calculate the correct hash
			// independent of how json.Decode works internally.
			buf, err := io.ReadAll(io.TeeReader(archive, metaHash))
			if err != nil {
				return fmt.Errorf("failed to read snapshot metadata: %v", err)
			}
			if err := json.Unmarshal(buf, &metadata); err != nil {
				return fmt.Errorf("failed to decode snapshot metadata: %v", err)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the file/stream is an unmodified Consul snapshot: run 'consul snapshot inspect <file>' to validate it
2. Check you are not double-decompressing or feeding a gzip stream where raw tar is expected
3. Re-download or re-take the snapshot; the source copy is likely truncated or corrupt
4. Check proxies/LBs that may have replaced the body with an error page — compare content-length and content-type
5. Confirm the snapshot was taken with a compatible Consul version

Example fix

// before
snap, err := agent.Snapshot().Open()
restored, err := snapshot.Restore(logOut, snap)
// after
snap, err := agent.Snapshot().Open()
if err != nil { return err }
// sanity-check the stream is a tar before restoring
var probe [512]byte
if _, err := io.ReadFull(snap, probe[:]); err != nil || !bytes.Equal(probe[257:262], []byte("ustar")) {
    return fmt.Errorf("stream is not a valid snapshot tar archive")
}

func looksLikeTar(r io.Reader) (bool, error) {
    hdr := make([]byte, 512)
    if _, err := io.ReadFull(r, hdr); err != nil {
        return false, err
    }
    // ustar magic at offset 257
    return string(hdr[257:262]) == "ustar", nil
}

func isInvalidSnapshotErr(err error) bool {
    return err != nil && strings.Contains(err.Error(), "failed reading snapshot")
}

if err := snapshot.Restore(logOut, snap); err != nil {
    if strings.Contains(err.Error(), "failed reading snapshot") {
        return fmt.Errorf("snapshot source is corrupt or not a tar archive; re-take snapshot: %w", err)
    }
    return err
}

When it happens

Trigger: Calling read() (via Snapshot.Open/Restore/CopySnapshot/API snapshot download) on a stream where tar.Reader.Next() fails: the snapshot file was truncated, the tar header block is corrupt, the response is an HTML/JSON error page instead of a tarball, or bytes were mangled in transit (proxy/TLS termination issues).

Common situations: Downloading a snapshot from a load balancer that returns a 502/503 error body; snapshot file truncated by an incomplete copy; restoring from a snapshot file that was gzip-compressed but not decompressed first; corrupted backup on disk.

Related errors

• failed to finalize snapshot: %v
• failed to decode snapshot metadata: %v
• failed to read snapshot hashes: %v
• failed checking integrity of snapshot: %v
• failed to snapshot %s: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/cba0d88805097c10. Report an issue: GitHub.