hashicorp/nomad · critical

failed to restore state

Error message

failed to restore state

What it means

NewClient returns this when c.restoreState() fails while reloading allocations from the boltdb state database. Nomad logs an explicit guidance block about corrupt state right before returning this (now a bare, non-wrapped) error because corrupt state usually stems from a Nomad bug.

Source

Thrown at client/client.go:650

	if clientIdentity != "" {
		c.setNodeIdentityToken(clientIdentity)
	}

	// Register and then start heartbeating to the servers.
	c.shutdownGroup.Go(c.registerAndHeartbeat)

	// Restore the state
	if err := c.restoreState(); err != nil {
		logger.Error("failed to restore state", "error", err)
		logger.Error("Nomad is unable to start due to corrupt state. "+
			"The safest way to proceed is to manually stop running task processes "+
			"and remove Nomad's state and alloc directories before "+
			"restarting. Lost allocations will be rescheduled.",
			"state_dir", cfg.StateDir, "alloc_dir", cfg.AllocDir)
		logger.Error("Corrupt state is often caused by a bug. Please " +
			"report as much information as possible to " +
			"https://github.com/hashicorp/nomad/issues")
		return nil, fmt.Errorf("failed to restore state")
	}

	// Begin periodic snapshotting of state.
	c.shutdownGroup.Go(c.periodicSnapshot)

	// Begin syncing allocations to the server
	c.shutdownGroup.Go(c.allocSync)

	// Ensure our base labels are generated and stored before we start the
	// client and begin emitting stats.
	c.setupStatsLabels()

	// Start the client! Don't use the shutdownGroup as run handles
	// shutdowns manually to prevent updates from being applied during
	// shutdown.
	go c.run()

	// Start collecting stats

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Back up the state_dir and alloc_dir, stop Nomad, then remove Nomad's state and alloc directories and restart — lost allocations are rescheduled.
2. Check disk health (dmesg for I/O errors, SMART) since corruption often indicates hardware failure.
3. Report details to https://github.com/hashicorp/nomad/issues as the log instructs — corrupt state usually indicates a Nomad bug.
4. Try opening the state DB with a boltdb tool to confirm corruption before deleting.

Example fix

# before: client refuses to start with "failed to restore state"
systemctl stop nomad
mv /var/lib/nomad/state /var/lib/nomad/state.bak
mv /var/lib/nomad/alloc /var/lib/nomad/alloc.bak
# after
systemctl start nomad  # fresh state; allocs rescheduled

client, err := client.NewClient(cfg, logger)
if err != nil && err.Error() == "failed to restore state" {
    // Nomad already logged guidance; automated remediation path:
    // 1) back up state_dir/alloc_dir  2) move them aside  3) restart agent
    logger.Error("state restore failed; backing up state and restarting clean", "err", err)
}

When it happens

Trigger: restoreState returns an error: the state DB snapshot cannot be read, an alloc's restore hook fails unrecoverably, or the state database file is corrupt/truncated after a crash.

Common situations: Client machine crashed or was OOM-killed mid-snapshot leaving a corrupt boltdb; disk corruption on state_dir; upgrading across a version where state schema restore breaks; out-of-disk during restore.

Related errors

• node setup failed: %v
• failed to restore from snapshot: %v
• error parsing: root should be an object
• cannot specify Accessor ID
• network already configured but not found in state

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d5e5f4570f151b03. Report an issue: GitHub.