{"record":{"id":"d5e5f4570f151b03","repo":"hashicorp/nomad","slug":"failed-to-restore-state","errorCode":null,"errorMessage":"failed to restore state","messagePattern":"failed to restore state","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"client/client.go","lineNumber":650,"sourceCode":"\tif clientIdentity != \"\" {\n\t\tc.setNodeIdentityToken(clientIdentity)\n\t}\n\n\t// Register and then start heartbeating to the servers.\n\tc.shutdownGroup.Go(c.registerAndHeartbeat)\n\n\t// Restore the state\n\tif err := c.restoreState(); err != nil {\n\t\tlogger.Error(\"failed to restore state\", \"error\", err)\n\t\tlogger.Error(\"Nomad is unable to start due to corrupt state. \"+\n\t\t\t\"The safest way to proceed is to manually stop running task processes \"+\n\t\t\t\"and remove Nomad's state and alloc directories before \"+\n\t\t\t\"restarting. Lost allocations will be rescheduled.\",\n\t\t\t\"state_dir\", cfg.StateDir, \"alloc_dir\", cfg.AllocDir)\n\t\tlogger.Error(\"Corrupt state is often caused by a bug. Please \" +\n\t\t\t\"report as much information as possible to \" +\n\t\t\t\"https://github.com/hashicorp/nomad/issues\")\n\t\treturn nil, fmt.Errorf(\"failed to restore state\")\n\t}\n\n\t// Begin periodic snapshotting of state.\n\tc.shutdownGroup.Go(c.periodicSnapshot)\n\n\t// Begin syncing allocations to the server\n\tc.shutdownGroup.Go(c.allocSync)\n\n\t// Ensure our base labels are generated and stored before we start the\n\t// client and begin emitting stats.\n\tc.setupStatsLabels()\n\n\t// Start the client! Don't use the shutdownGroup as run handles\n\t// shutdowns manually to prevent updates from being applied during\n\t// shutdown.\n\tgo c.run()\n\n\t// Start collecting stats","sourceCodeStart":632,"sourceCodeEnd":668,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/client.go#L632-L668","documentation":"NewClient returns this when c.restoreState() fails while reloading allocations from the boltdb state database. Nomad logs an explicit guidance block about corrupt state right before returning this (now a bare, non-wrapped) error because corrupt state usually stems from a Nomad bug.","triggerScenarios":"restoreState returns an error: the state DB snapshot cannot be read, an alloc's restore hook fails unrecoverably, or the state database file is corrupt/truncated after a crash.","commonSituations":"Client machine crashed or was OOM-killed mid-snapshot leaving a corrupt boltdb; disk corruption on state_dir; upgrading across a version where state schema restore breaks; out-of-disk during restore.","solutions":["Back up the state_dir and alloc_dir, stop Nomad, then remove Nomad's state and alloc directories and restart — lost allocations are rescheduled.","Check disk health (dmesg for I/O errors, SMART) since corruption often indicates hardware failure.","Report details to https://github.com/hashicorp/nomad/issues as the log instructs — corrupt state usually indicates a Nomad bug.","Try opening the state DB with a boltdb tool to confirm corruption before deleting."],"exampleFix":"# before: client refuses to start with \"failed to restore state\"\nsystemctl stop nomad\nmv /var/lib/nomad/state /var/lib/nomad/state.bak\nmv /var/lib/nomad/alloc /var/lib/nomad/alloc.bak\n# after\nsystemctl start nomad  # fresh state; allocs rescheduled","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"client, err := client.NewClient(cfg, logger)\nif err != nil && err.Error() == \"failed to restore state\" {\n    // Nomad already logged guidance; automated remediation path:\n    // 1) back up state_dir/alloc_dir  2) move them aside  3) restart agent\n    logger.Error(\"state restore failed; backing up state and restarting clean\", \"err\", err)\n}\n","preventionTips":["Back up state_dir before Nomad version upgrades","Use UPS-backed disks and monitor SMART/I-O errors on state volumes","Enable graceful shutdown (systemd timeout) so snapshots complete before kill","Alert on unclean agent exits — repeated crashes corrupt boltdb","Report recurring corruption to Nomad with logs and state DB backups"],"tags":["client","state-store","restore"],"backgroundTag":"corrupt-state-db","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}