{"record":{"id":"a6eb20e4d29d3342","repo":"hashicorp/nomad","slug":"failed-to-open-state-database-v","errorCode":null,"errorMessage":"failed to open state database: %v","messagePattern":"failed to open state database: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"client/client.go","lineNumber":711,"sourceCode":"\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed creating temporary directory for the StateDir: %v\", err)\n\t\t}\n\n\t\tp, err = filepath.EvalSymlinks(p)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to find temporary directory for the StateDir: %v\", err)\n\t\t}\n\n\t\tconf = c.UpdateConfig(func(c *config.Config) {\n\t\t\tc.StateDir = p\n\t\t})\n\t}\n\tc.logger.Info(\"using state directory\", \"state_dir\", conf.StateDir)\n\n\t// Open the state database\n\tdb, err := conf.StateDBFactory(c.logger, conf.StateDir)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to open state database: %v\", err)\n\t}\n\n\t// Upgrade the state database\n\tif err := db.Upgrade(); err != nil {\n\t\t// Upgrade only returns an error on critical persistence\n\t\t// failures in which an operator should intervene before the\n\t\t// node is accessible. Upgrade drops and logs corrupt state it\n\t\t// encounters, so failing to start the agent should be extremely\n\t\t// rare.\n\t\treturn fmt.Errorf(\"failed to upgrade state database: %v\", err)\n\t}\n\n\tc.stateDB = db\n\n\t// Ensure host_volumes_dir config is not empty.\n\tif conf.HostVolumesDir == \"\" {\n\t\tconf = c.UpdateConfig(func(c *config.Config) {\n\t\t\tc.HostVolumesDir = filepath.Join(conf.StateDir, \"host_volumes\")","sourceCodeStart":693,"sourceCodeEnd":729,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/client.go#L693-L729","documentation":"After determining the state directory, the client opens the persistent state database via conf.StateDBFactory (BoltDB-backed). If the factory returns an error, init fails with this message. The agent cannot start because all client state (allocs, task handles metadata) lives in this DB.","triggerScenarios":"client init: StateDBFactory(logger, conf.StateDir) returns error, typically when the bolt file (state.db) is corrupt, locked, or the state dir is unwritable/uncreatable.","commonSituations":"Corrupt bolt.db after host crash or disk-full, another Nomad process holding the file lock, state_dir on a read-only or full filesystem, restored state files with wrong permissions.","solutions":["Read the wrapped %v error to identify the cause (lock contention vs corruption vs permission).","Ensure no other Nomad client process is using the same data_dir/state_dir and that the directory is writable.","If the DB file is corrupt and unrecoverable, back it up and remove/rename it, accepting loss of local client state (the client will re-sync from servers)."],"exampleFix":"// before\nsudo nomad agent -client   # fails: another agent holds state.db lock\n// after\nsudo systemctl stop nomad; pgrep nomad  # ensure single process\nsudo chown -R nomad:nomad /var/lib/nomad/client\nsudo systemctl start nomad","handlingStrategy":"validation","validationCode":"dbPath := filepath.Join(stateDir, \"state.db\")\nif f, err := os.OpenFile(filepath.Dir(dbPath), os.O_WRONLY, 0o700); err != nil {\n    return fmt.Errorf(\"state dir not writable: %v\", err)\n} else { f.Close() }\nif fi, err := os.Stat(dbPath); err == nil && fi.Size() == 0 {\n    return fmt.Errorf(\"state.db is 0 bytes — likely corrupt\")\n}","typeGuard":null,"tryCatchPattern":"if err := clientInit(); err != nil {\n    if strings.Contains(err.Error(), \"failed to open state database\") {\n        // check for lock holder: lsof state.db; restore from backup or start fresh\n    }\n}","preventionTips":["Run exactly one Nomad client per data_dir","Keep the state directory on reliable local storage (not NFS)","Monitor disk space and ensure correct file ownership for the nomad user","Back up state.db before Nomad upgrades/downgrades"],"tags":["client-init","boltdb","state-database","filesystem"],"backgroundTag":"database-open-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}