hashicorp/nomad · error

failed to sync temp snapshot: %v

Error message

failed to sync temp snapshot: %v

What it means

Once the snapshot has been fully written to the temp file, Restore() calls snap.Sync() to flush it to disk before rewinding and handing the file to Raft. This error wraps a failure of that fsync. It is rare and points to low-level I/O problems persisting the temp file's data.

Source

Thrown at helper/snapshot/snapshot.go:292

		}
		if err := os.Remove(snap.Name()); err != nil {
			logger.Error("Failed to clean up temp snapshot", "error", err)
		}
	}()

	// Read the archive.
	var metadata raft.SnapshotMeta
	if err := read(decomp, &metadata, snap); err != nil {
		return fmt.Errorf("failed to read snapshot file: %v", err)
	}

	if err := concludeGzipRead(decomp); err != nil {
		return err
	}

	// Sync and rewind the file so it's ready to be read again.
	if err := snap.Sync(); err != nil {
		return fmt.Errorf("failed to sync temp snapshot: %v", err)
	}
	if _, err := snap.Seek(0, 0); err != nil {
		return fmt.Errorf("failed to rewind temp snapshot: %v", err)
	}

	// Feed the snapshot into Raft.
	if err := r.Restore(&metadata, snap, 0); err != nil {
		return fmt.Errorf("Raft error when restoring snapshot: %v", err)
	}

	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped %v cause and check dmesg/system logs for disk I/O errors.
2. Verify the filesystem holding the temp directory has free space (df) and is not read-only.
3. Check/replace failing storage hardware or migrate the node.
4. Retry the restore after the storage issue is resolved.

if err := syscall.Access(os.TempDir(), os.O_RDWR); err != nil {
    return fmt.Errorf("temp storage not writable: %w", err)
}

if err := snapStore.Restore(data); err != nil && strings.Contains(err.Error(), "failed to sync temp snapshot") {
    // check dmesg / disk health, then retry after storage recovers
}

When it happens

Trigger: snap.Sync() fails during Restore — typically EIO from the disk, ENOSPC (filesystem full while flushing buffered data), or the underlying file descriptor becoming invalid (closed or I/O error on the device).

Common situations: Failing or full disks on the node; storage backend errors in containers/VMs; hardware or SAN issues surfacing as EIO during fsync.

Related errors

• failed to create snapshot file: %v
• failed to sync snapshot: %v
• unable to read rooted allocation directory
• can't seek to offset %d: %w
• Couldn't copy %q to %q: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d894120122c746a7. Report an issue: GitHub.