{"record":{"id":"b76a62be1a4156e2","repo":"hashicorp/nomad","slug":"timed-out-while-opening-database-is-another-nomad","errorCode":null,"errorMessage":"timed out while opening database, is another Nomad process accessing data_dir %s?","messagePattern":"timed out while opening database, is another Nomad process accessing data_dir (.+?)\\?","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/state/db_bolt.go","lineNumber":202,"sourceCode":"// NewBoltStateDB creates or opens an existing boltdb state file or returns an\n// error.\nfunc NewBoltStateDB(logger hclog.Logger, stateDir string) (StateDB, error) {\n\tfn := filepath.Join(stateDir, \"state.db\")\n\n\t// Check to see if the DB already exists\n\tfi, err := os.Stat(fn)\n\tif err != nil && !os.IsNotExist(err) {\n\t\treturn nil, err\n\t}\n\tfirstRun := fi == nil\n\n\t// Timeout to force failure when accessing a data dir that is already in use\n\ttimeout := &bbolt.Options{Timeout: 5 * time.Second}\n\n\t// Create or open the boltdb state database\n\tdb, err := boltdd.Open(fn, 0600, timeout)\n\tif err == bbolt.ErrTimeout {\n\t\treturn nil, fmt.Errorf(\"timed out while opening database, is another Nomad process accessing data_dir %s?\", stateDir)\n\t} else if err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to create state database: %v\", err)\n\t}\n\n\tsdb := &BoltStateDB{\n\t\tstateDir: stateDir,\n\t\tdb:       db,\n\t\tlogger:   logger,\n\t}\n\n\t// If db did not already exist, initialize metadata fields\n\tif firstRun {\n\t\tif err := sdb.init(); err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t}\n\n\treturn sdb, nil","sourceCodeStart":184,"sourceCodeEnd":220,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/state/db_bolt.go#L184-L220","documentation":"NewBoltStateDB opens the client's bbolt-backed state database with a 5-second lock timeout. bbolt allows only one writer/process at a time; when Open fails specifically with bbolt.ErrTimeout (the file lock could not be acquired in time), Nomad converts it into this explanatory error naming the data_dir.","triggerScenarios":"Starting a Nomad client whose data_dir/state database is already locked — typically a second Nomad agent process pointing at the same data_dir while the first still runs, or a dead process left an unclean lock in an environment where the lock lingers.","commonSituations":"Duplicate Nomad client services (systemd + manual run) on the same host; container/orchestrator restarts overlapping old instances; running nomad client commands against a live agent's data_dir in tests or debugging.","solutions":["Find and stop the other Nomad process using the data_dir (pgrep nomad; check systemd/launchd units)","Verify only one Nomad client is configured to use this data_dir","Wait/retry if a previous process is still shutting down and holds the lock","Point the new agent at a different data_dir if concurrent instances are intentional"],"exampleFix":"# before\nnomad agent -client -data-dir /var/nomad  # second instance, same dir\n# after\nsystemctl stop nomad   # stop the existing holder first\nnomad agent -client -data-dir /var/nomad","handlingStrategy":"retry","validationCode":"if _, err := os.Open(fname); err == nil {\n    if isLockedByOtherProcess(fname) {\n        return fmt.Errorf(\"data_dir %s state db is locked by another process; stop it first\", stateDir)\n    }\n}","typeGuard":null,"tryCatchPattern":"sdb, err := NewBoltStateDB(stateDir, logger)\nif err != nil && strings.Contains(err.Error(), \"timed out while opening database\") {\n    logger.Error(\"state db lock held; is another Nomad client running on this data_dir?\", \"dir\", stateDir)\n    // backoff and retry after ensuring the other process exited\n    time.Sleep(5 * time.Second)\n    return retry()\n}","preventionTips":["Run exactly one Nomad client per data_dir; enforce via systemd singleton/pidfile","Avoid overlapping restarts in containers; use graceful shutdown with stop timeouts","Never point ad-hoc nomad commands at a live agent's data_dir","Monitor for duplicate nomad agent processes on hosts"],"tags":["nomad","boltdb","file-lock","client-startup"],"backgroundTag":"database-locked","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}