hashicorp/nomad · critical

failed checking integrity of snapshot: %v

Error message

failed checking integrity of snapshot: %v

What it means

The SHA256 checksums recorded in the snapshot's SHA256SUMS file did not match the data actually read from meta.json and state.bin. The library throws this as the integrity check after unpacking the tar, to refuse restoring or copying a snapshot whose contents were altered or corrupted. Root causes inside DecodeAndVerify include hash mismatch, missing hash entries, or missing files.

Source

Thrown at helper/snapshot/archive.go:232

		case "state.bin":
			if _, err := io.Copy(io.MultiWriter(snap, snapHash), archive); err != nil {
				return fmt.Errorf("failed to read or write snapshot data: %v", err)
			}

		case "SHA256SUMS":
			if _, err := io.Copy(&shaBuffer, archive); err != nil {
				return fmt.Errorf("failed to read snapshot hashes: %v", err)
			}

		default:
			return fmt.Errorf("unexpected file %q in snapshot", hdr.Name)
		}
	}

	// Verify all the hashes.
	if err := hl.DecodeAndVerify(&shaBuffer); err != nil {
		return fmt.Errorf("failed checking integrity of snapshot: %v", err)
	}

	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Treat the snapshot as corrupt: re-take it with nomad operator snapshot save from a healthy server or restore from a different verified backup.
2. Run sha256sum on meta.json/state.bin extracted from the archive and compare to SHA256SUMS to confirm which file is corrupted and rule out transfer issues.
3. Verify the complete snapshot with nomad operator snapshot inspect (or snapshot.Verify) on the source node before shipping it to the target cluster.
4. Never hand-edit snapshot contents; if metadata surgery is truly required, re-create the archive and its SHA256SUMS consistently (unsupported, do this only with vendor guidance).

Example fix

// before: blindly restoring a snapshot that was edited by hand
// (meta.json changed so its SHA256 no longer matches SHA256SUMS)
snapshot.Restore(logger, editedSnapshot, r)

// after: verify integrity first and abort on mismatch
if _, err := snapshot.Verify(editedSnapshot); err != nil {
    return fmt.Errorf("snapshot failed integrity check, use an unmodified copy: %w", err)
}
editedSnapshot.Seek(0, 0)
snapshot.Restore(logger, editedSnapshot, r)

// Detect integrity failure before attempting a restore
func integrityCheck(in io.Reader) error {
    _, err := snapshot.Verify(in)
    return err // nil means the SHA256SUMS all matched
}

meta, err := snapshot.CopySnapshot(in, dest)
if err != nil {
    if strings.Contains(err.Error(), "failed checking integrity of snapshot") {
        // checksum mismatch: snapshot contents altered/corrupt
        log.Error("snapshot failed SHA256 integrity check; re-take snapshot, do not restore")
        return err // never fall back to restoring corrupt data
    }
    return err
}

When it happens

Trigger: snapshot.Verify, CopySnapshot, or Restore on a snapshot where state.bin or meta.json bytes differ from when the snapshot was written: bit rot on disk, partial upload/download followed by padding, editing of archive contents, gzip corruption after the sums were read, or a hash-list entry for a file not present in the tar.

Common situations: Snapshots stored on failing disks or unreliable object storage; snapshots modified by backup agents; truncated-then-padded transfers; manually edited meta.json (e.g. to force-restore older data) which breaks the recorded hash.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• failed reading snapshot: %v
• failed to decode snapshot metadata: %v
• failed to read snapshot hashes: %v
• invalid digest format
• unsupported checksum format

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/049bacf63c30666f. Report an issue: GitHub.