{"record":{"id":"0cea554cb3ba1616","repo":"hyperledger/fabric","slug":"as-another-peer-node-command-is-executing-wait-fo-0cea55","errorCode":null,"errorMessage":"as another peer node command is executing, wait for that command to complete its execution or terminate it before retrying","messagePattern":"as another peer node command is executing, wait for that command to complete its execution or terminate it before retrying","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"core/ledger/kvledger/upgrade_dbs.go","lineNumber":26,"sourceCode":"\nimport (\n\t\"github.com/hyperledger/fabric/common/ledger/blkstorage\"\n\t\"github.com/hyperledger/fabric/common/ledger/util/leveldbhelper\"\n\t\"github.com/hyperledger/fabric/core/ledger\"\n\t\"github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb\"\n\t\"github.com/pkg/errors\"\n)\n\n// UpgradeDBs upgrades existing ledger databases to the latest formats.\n// It checks the format of idStore and does not drop any databases\n// if the format is already the latest version. Otherwise, it drops\n// ledger databases and upgrades the idStore format.\nfunc UpgradeDBs(config *ledger.Config) error {\n\trootFSPath := config.RootFSPath\n\tfileLockPath := fileLockPath(rootFSPath)\n\tfileLock := leveldbhelper.NewFileLock(fileLockPath)\n\tif err := fileLock.Lock(); err != nil {\n\t\treturn errors.Wrap(err, \"as another peer node command is executing,\"+\n\t\t\t\" wait for that command to complete its execution or terminate it before retrying\")\n\t}\n\tdefer fileLock.Unlock()\n\n\tlogger.Infof(\"Ledger data folder from config = [%s]\", rootFSPath)\n\n\tdbPath := LedgerProviderPath(rootFSPath)\n\tdb := leveldbhelper.CreateDB(&leveldbhelper.Conf{DBPath: dbPath})\n\tdb.Open()\n\tdefer db.Close()\n\tidStore := &idStore{db, dbPath}\n\n\t// Check upfront whether we should upgrade the data format before dropping databases.\n\t// If someone mistakenly executes the upgrade command in a peer that has some channels that\n\t// are bootstrapped from a snapshot, the peer will not be able to start as the data for those channels\n\t// cannot be recovered\n\tisEligible, err := idStore.checkUpgradeEligibility()\n\tif err != nil {","sourceCodeStart":8,"sourceCodeEnd":44,"githubUrl":"https://github.com/hyperledger/fabric/blob/2736b63f8fd5932511d56fe68b7039d15977f7f6/core/ledger/kvledger/upgrade_dbs.go#L8-L44","documentation":"UpgradeDBs serializes the ledger data-format upgrade with other peer commands using a file lock (fileLock.Lock()) on a lockfile under the peer's root FS path. If the lock cannot be acquired — because another peer process (peer node start, upgrade, etc.) already holds it, or a previous run crashed leaving a stale lock — the underlying error is wrapped with this message and returned.","triggerScenarios":"Calling kvledger.UpgradeDBs (or 'peer node upgrade') while another peer command holds the ledger file lock; a previous peer process crashed without releasing the lock; two upgrade/repair commands run concurrently; the lock file resides on a filesystem where locking fails (e.g. NFS without proper lock support).","commonSituations":"Running 'peer node upgrade' while the peer is still starting or running; stale lock file left after an unclean shutdown (kill -9, OOM); duplicated automation scripts starting two peer commands at once; shared data directories on NFS mounts.","solutions":["Find and stop the other peer process holding the lock (check ps for peer node processes) and retry the command once it exits.","If no peer process is running, remove the stale lock file in the peer's ledger data directory (fileLockPath under config.RootFSPath) and re-run.","Ensure only one peer command runs at a time; serialize operations in deployment scripts.","Move the ledger data directory off NFS to a local filesystem that supports flock properly."],"exampleFix":"// before (concurrent, fails)\ngo peer node upgrade &\npeer node upgrade // lock already held\n\n// after (serialized)\nwait $UPGRADE_PID // let first command finish and release the lock\npeer node upgrade","handlingStrategy":"retry","validationCode":"// check no other peer process is running and lock is free before invoking\nif err := exec.Command(\"pgrep\", \"-x\", \"peer\").Run(); err == nil {\n    return fmt.Errorf(\"another peer process appears to be running\")\n}","typeGuard":null,"tryCatchPattern":"for attempt := 0; attempt < 3; attempt++ {\n    err := kvledger.UpgradeDBs(config)\n    if err == nil || !strings.Contains(err.Error(), \"another peer node command is executing\") {\n        return err\n    }\n    time.Sleep(30 * time.Second) // wait for holder to release lock\n}","preventionTips":["Never run peer node commands concurrently against the same data directory","Check for stale lock files after unclean shutdowns before retrying","Serialize upgrade/repair steps in deployment automation","Avoid NFS for ledger data directories; use local filesystems with flock support"],"tags":["go","file-lock","hyperledger-fabric","concurrency"],"backgroundTag":"file-lock-contention","analyzedSha":"2736b63f8fd5932511d56fe68b7039d15977f7f6","analyzedAt":"2026-09-04T08:52:36.465Z","contentChangedAt":"2026-09-04T08:52:36.465Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}