hashicorp/nomad · error

failed to sync snapshot: %v

Error message

failed to sync snapshot: %v

What it means

writeSnapshot calls archive.Sync() to flush the compressed snapshot file to stable storage before handing it to the caller. If the OS cannot sync the file (fsync failure), snapshot creation aborts with this error, since an unsynced snapshot cannot be trusted for later re-reading.

Source

Thrown at helper/snapshot/snapshot.go:135

	out := io.MultiWriter(hash, archive)

	// Wrap the file writer in a gzip compressor.
	compressor := gzip.NewWriter(out)

	// Write the archive.
	if err := write(compressor, metadata, snap); err != nil {
		return nil, fmt.Errorf("failed to write snapshot file: %v", err)
	}

	// Finish the compressed stream.
	if err := compressor.Close(); err != nil {
		return nil, fmt.Errorf("failed to compress snapshot file: %v", err)
	}

	// Sync the compressed file and rewind it so it's ready to be streamed
	// out by the caller.
	if err := archive.Sync(); err != nil {
		return nil, fmt.Errorf("failed to sync snapshot: %v", err)
	}
	if _, err := archive.Seek(0, 0); err != nil {
		return nil, fmt.Errorf("failed to rewind snapshot: %v", err)
	}

	checksum := "sha-256=" + base64.StdEncoding.EncodeToString(hash.Sum(nil))

	keep = true
	return &Snapshot{archive, metadata.Index, checksum}, nil
}

// Index returns the index of the snapshot. This is safe to call on a nil
// snapshot, it will just return 0.
func (s *Snapshot) Index() uint64 {
	if s == nil {
		return 0
	}
	return s.index

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check dmesg/system logs for storage device errors (EIO) around the failure time.
2. Ensure the temp directory is on a local, reliable filesystem rather than NFS or flaky network storage (set TMPDIR accordingly).
3. Verify disk health (SMART) and free space on the volume holding the temp files.
4. Retry the snapshot after storage is healthy; the temp file from the failed attempt is cleaned up in Close().

Example fix

// before: TMPDIR on NFS → fsync errors
// after
export TMPDIR=/var/lib/myapp/tmp   # local disk
mkdir -p $TMPDIR

// require a local filesystem for temp files
if isNetworkMount(os.TempDir()) {
	return fmt.Errorf("TMPDIR %q is a network mount; use local disk", os.TempDir())
}

snap, err := snapshot.New(...)
if err != nil && strings.Contains(err.Error(), "failed to sync snapshot") {
	logger.Error("fsync of snapshot temp failed; check storage health", "err", err)
	return err
}

When it happens

Trigger: New or NewFromFSM where archive.Sync() returns an error: disk I/O errors (EIO), filesystem errors, disk full manifests at sync time, or underlying device problems on the temp volume.

Common situations: Failing or over-subscribed disks (especially in VMs/cloud instances); NFS or unusual filesystems for TMPDIR that don't support fsync well; kernel-level storage errors under heavy load.

Related errors

• failed to create snapshot file: %v
• failed to write snapshot file: %v
• failed to rewind snapshot: %v
• failed to sync temp snapshot: %v
• unable to read rooted allocation directory

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0003522484f90ba2. Report an issue: GitHub.