{"record":{"id":"04085126cd380cf4","repo":"hyperledger/fabric","slug":"as-another-peer-node-command-is-executing-wait-fo-040851","errorCode":null,"errorMessage":"as another peer node command is executing, wait for that command to complete its execution or terminate it before retrying","messagePattern":"as another peer node command is executing, wait for that command to complete its execution or terminate it before retrying","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"core/ledger/kvledger/rebuild_dbs.go","lineNumber":24,"sourceCode":"\npackage kvledger\n\nimport (\n\t\"github.com/hyperledger/fabric/common/ledger/blkstorage\"\n\t\"github.com/hyperledger/fabric/common/ledger/util/leveldbhelper\"\n\t\"github.com/hyperledger/fabric/core/ledger\"\n\t\"github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/statecouchdb\"\n\t\"github.com/pkg/errors\"\n)\n\n// RebuildDBs drops existing ledger databases.\n// Dropped database will be rebuilt upon server restart\nfunc RebuildDBs(config *ledger.Config) error {\n\trootFSPath := config.RootFSPath\n\tfileLockPath := fileLockPath(rootFSPath)\n\tfileLock := leveldbhelper.NewFileLock(fileLockPath)\n\tif err := fileLock.Lock(); err != nil {\n\t\treturn errors.Wrap(err, \"as another peer node command is executing,\"+\n\t\t\t\" wait for that command to complete its execution or terminate it before retrying\")\n\t}\n\tdefer fileLock.Unlock()\n\n\tblockstorePath := BlockStorePath(rootFSPath)\n\tledgerIDs, err := blkstorage.GetLedgersBootstrappedFromSnapshot(blockstorePath)\n\tif err != nil {\n\t\treturn errors.WithMessage(err, \"error while checking if any ledger has been bootstrapped from snapshot\")\n\t}\n\tif len(ledgerIDs) > 0 {\n\t\treturn errors.Errorf(\"cannot rebuild databases because the peer contains channel(s) %s that were bootstrapped from snapshot\", ledgerIDs)\n\t}\n\n\tif config.StateDBConfig.StateDatabase == ledger.CouchDB {\n\t\tif err := statecouchdb.DropApplicationDBs(config.StateDBConfig.CouchDB); err != nil {\n\t\t\treturn err\n\t\t}\n\t}","sourceCodeStart":6,"sourceCodeEnd":42,"githubUrl":"https://github.com/hyperledger/fabric/blob/2736b63f8fd5932511d56fe68b7039d15977f7f6/core/ledger/kvledger/rebuild_dbs.go#L6-L42","documentation":"RebuildDBs (used by 'peer node rebuild-dbs') first takes the exclusive ledger file lock before dropping and rebuilding state/index databases. If the lock is already held by another concurrent ledger command, the operation fails with this message. The file lock serializes all mutating peer node admin commands (pause/resume/reset/rollback/rebuild).","triggerScenarios":"Running 'peer node rebuild-dbs' concurrently with any other peer node command (start, reset, rollback, pause/resume) that holds the fileLock file.","commonSituations":"Rebuild attempted while the peer process is still running; orchestration scripts launching rebuild and rollback together; a stale lock file left by a previously killed command.","solutions":["Stop the running peer / wait for the in-flight command to complete, then run rebuild-dbs again.","Kill any lingering 'peer node' process that holds the lock.","If a prior command crashed, verify no process holds <peer fs path>/fileLock (lsof), remove the stale lock file, and retry.","Automate: always run admin commands serially against one peer data directory."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// ensure peer is not running before rebuild-dbs\nfunc peerIsRunning(pidFile string) bool {\n\tdata, err := os.ReadFile(pidFile)\n\tif err != nil {\n\t\treturn false\n\t}\n\tpid, err := strconv.Atoi(strings.TrimSpace(string(data)))\n\tif err != nil {\n\t\treturn false\n\t}\n\treturn unix.Kill(pid, 0) == nil\n}","typeGuard":null,"tryCatchPattern":"if err := ledger.RebuildDBs(config); err != nil {\n\tif strings.Contains(err.Error(), \"another peer node command is executing\") {\n\t\t// back off and retry once the peer is stopped\n\t\ttime.Sleep(retryInterval)\n\t\treturn retryRebuild()\n\t}\n\treturn err\n}","preventionTips":["Stop the peer container/process before rebuild-dbs","Run admin commands from a single orchestration step, never concurrently","Remove stale fileLock files only after confirming no live process holds them","Document a strict ops runbook ordering for ledger maintenance commands"],"tags":["hyperledger-fabric","file-lock","concurrency"],"backgroundTag":"file-lock-contention","analyzedSha":"2736b63f8fd5932511d56fe68b7039d15977f7f6","analyzedAt":"2026-09-04T08:52:36.465Z","contentChangedAt":"2026-09-04T08:52:36.465Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}