{"record":{"id":"ac6d79f76fbc2142","repo":"hashicorp/nomad","slug":"failed-to-upgrade-state-database-v","errorCode":null,"errorMessage":"failed to upgrade state database: %v","messagePattern":"failed to upgrade state database: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"client/client.go","lineNumber":721,"sourceCode":"\t\t\tc.StateDir = p\n\t\t})\n\t}\n\tc.logger.Info(\"using state directory\", \"state_dir\", conf.StateDir)\n\n\t// Open the state database\n\tdb, err := conf.StateDBFactory(c.logger, conf.StateDir)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to open state database: %v\", err)\n\t}\n\n\t// Upgrade the state database\n\tif err := db.Upgrade(); err != nil {\n\t\t// Upgrade only returns an error on critical persistence\n\t\t// failures in which an operator should intervene before the\n\t\t// node is accessible. Upgrade drops and logs corrupt state it\n\t\t// encounters, so failing to start the agent should be extremely\n\t\t// rare.\n\t\treturn fmt.Errorf(\"failed to upgrade state database: %v\", err)\n\t}\n\n\tc.stateDB = db\n\n\t// Ensure host_volumes_dir config is not empty.\n\tif conf.HostVolumesDir == \"\" {\n\t\tconf = c.UpdateConfig(func(c *config.Config) {\n\t\t\tc.HostVolumesDir = filepath.Join(conf.StateDir, \"host_volumes\")\n\t\t})\n\t}\n\n\t// Ensure the alloc mounts dir exists if we are configured with a custom path.\n\tif conf.AllocMountsDir != \"\" {\n\t\tif err := os.MkdirAll(conf.AllocMountsDir, 0o711); err != nil {\n\t\t\treturn fmt.Errorf(\"failed creating alloc mounts dir: %w\", err)\n\t\t}\n\t}\n","sourceCodeStart":703,"sourceCodeEnd":739,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/client.go#L703-L739","documentation":"After opening, the client calls db.Upgrade() to migrate persisted state. Per the source comment, Upgrade drops and logs corrupt state it encounters, so it returns an error only on critical persistence failures where an operator must intervene. Init then fails with this wrapper and the agent refuses to start.","triggerScenarios":"client init: db.Upgrade() returns error — critical persistence failure during state migration (e.g. DB unusable at a structural level, not merely corrupt individual entries).","commonSituations":"Downgrading Nomad to an older version whose state schema cannot be handled, severe disk corruption, or a state DB stored on unreliable storage (network filesystems).","solutions":["Check agent logs for the upgrade-time errors preceding this failure to see which state was rejected.","Verify you are not downgrading Nomad across incompatible state schema versions.","Back up the state directory and remove/rename the state DB so the client starts fresh and re-syncs allocations from servers."],"exampleFix":"// before: agent fails to start after downgrade\n// after: start fresh\nsudo systemctl stop nomad\nsudo mv /var/lib/nomad/client/state.db /var/lib/nomad/client/state.db.bak\nsudo systemctl start nomad","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := clientInit(); err != nil {\n    if strings.Contains(err.Error(), \"failed to upgrade state database\") {\n        // operator intervention: stop agent, back up state dir, remove state.db,\n        // restart so client re-syncs from servers\n    }\n}","preventionTips":["Never downgrade Nomad across incompatible state schema versions","Back up the client state directory before version changes","Keep state storage on local disks; avoid network filesystems"],"tags":["client-init","state-database","schema-migration"],"backgroundTag":"state-db-upgrade-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}