{"record":{"id":"ba62294925fee1fc","repo":"hashicorp/nomad","slug":"error-backing-up-state-db-v","errorCode":null,"errorMessage":"error backing up state db: %v","messagePattern":"error backing up state db: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/state/db_bolt.go","lineNumber":1230,"sourceCode":"}\n\n// Upgrade bolt state db from 0.8 schema to 0.9 schema. Noop if already using\n// 0.9 schema. Creates a backup before upgrading.\nfunc (s *BoltStateDB) Upgrade() error {\n\t// Check to see if the underlying DB needs upgrading.\n\tupgrade09, upgrade13, err := NeedsUpgrade(s.db.BoltDB())\n\tif err != nil {\n\t\treturn err\n\t}\n\tif !upgrade09 && !upgrade13 {\n\t\t// No upgrade needed!\n\t\treturn nil\n\t}\n\n\t// Upgraded needed. Backup the boltdb first.\n\tbackupFileName := filepath.Join(s.stateDir, \"state.db.backup\")\n\tif err := backupDB(s.db.BoltDB(), backupFileName); err != nil {\n\t\treturn fmt.Errorf(\"error backing up state db: %v\", err)\n\t}\n\n\t// Perform the upgrade\n\tif err := s.db.Update(func(tx *boltdd.Tx) error {\n\n\t\tif upgrade09 {\n\t\t\tif err := UpgradeAllocs(s.logger, tx); err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t}\n\t\tif upgrade13 {\n\t\t\tif err := UpgradeDynamicPluginRegistry(s.logger, tx); err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t}\n\n\t\t// Add standard metadata\n\t\tif err := addMeta(tx.BoltTx()); err != nil {","sourceCodeStart":1212,"sourceCodeEnd":1248,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/state/db_bolt.go#L1212-L1248","documentation":"Returned during state DB upgrade when the pre-upgrade backup of the BoltDB to state.db.backup fails. Upgrading aborts before any mutation so the live database is left untouched; this is a fail-safe step before destructive schema changes.","triggerScenarios":"backupDB(s.db.BoltDB(), state.db.backup) fails — most commonly the destination file already exists (backup refuses to overwrite), or copy/read errors on the source DB (permissions, disk full).","commonSituations":"Leftover state.db.backup from a previously failed/aborted upgrade, read-only state directory, or insufficient disk space during a Nomad version upgrade on a client node.","solutions":["Check disk space (df) and directory permissions on state_dir.","If a stale state.db.backup exists and you have another backup, remove or rename it, then restart the agent to retry the upgrade.","Verify the source state.db is readable by the Nomad process user.","If the backup keeps failing, copy state.db manually, delete the manual copy's lock files guidance aside, and restore it if the upgrade later corrupts."],"exampleFix":"// before\nupgrade fails: error backing up state db: backup file already exists\n// after\nsystemctl stop nomad\nmv /var/lib/nomad/state.db.backup /var/lib/nomad/state.db.backup.old\nsystemctl start nomad # upgrade retries and creates a fresh backup","handlingStrategy":"validation","validationCode":"backupPath := filepath.Join(stateDir, \"state.db.backup\")\nif fi, err := os.Stat(backupPath); err == nil && !fi.IsDir() {\n    return fmt.Errorf(\"stale backup %s exists; archive it before upgrading\", backupPath)\n}\nif err := diskFreeAtLeast(stateDir, dbSize*2); err != nil { return err }","typeGuard":"func backupPreconditionsOK(stateDir string, dbSize int64) bool {\n    _, err := os.Stat(filepath.Join(stateDir, \"state.db.backup\"))\n    return os.IsNotExist(err)\n}","tryCatchPattern":"if err := upgradeState(); err != nil {\n    if strings.Contains(err.Error(), \"error backing up state db\") {\n        // remove stale backup and retry once\n        os.Rename(backupPath, backupPath+\".old\")\n        err = upgradeState()\n    }\n}","preventionTips":["Clean up stale state.db.backup files before upgrades","Ensure 2x db size free disk space on state_dir","Grant the nomad user write access to state_dir","Stop the agent before touching state files"],"tags":["bolt","backup","upgrade","disk"],"backgroundTag":"state-db-backup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}