{"record":{"id":"77fb2b38fd431ead","repo":"phacility/phabricator","slug":"failed-to-acquire-lock-for-resource-s-while-t","errorCode":null,"errorMessage":"Failed to acquire lock for resource (\"%s\") while trying to acquire lease (\"%s\").","messagePattern":"Failed to acquire lock for resource \\(\"(.+?)\"\\) while trying to acquire lease \\(\"(.+?)\"\\)\\.","errorType":"exception","errorClass":"DrydockResourceLockException","httpStatus":null,"severity":"warning","filePath":"src/applications/drydock/storage/DrydockLease.php","lineNumber":257,"sourceCode":"            'You can not immediately activate leases on resources which '.\n            'need time to start up.'));\n      }\n    }\n\n    // Before we associate the lease with the resource, we lock the resource\n    // and reload it to make sure it is still pending or active. If we don't\n    // do this, the resource may have just been reclaimed. (Once we acquire\n    // the resource that stops it from being released, so we're nearly safe.)\n\n    $resource_phid = $resource->getPHID();\n    $hash = PhabricatorHash::digestForIndex($resource_phid);\n    $lock_key = 'drydock.resource:'.$hash;\n    $lock = PhabricatorGlobalLock::newLock($lock_key);\n\n    try {\n      $lock->lock(15);\n    } catch (Exception $ex) {\n      throw new DrydockResourceLockException(\n        pht(\n          'Failed to acquire lock for resource (\"%s\") while trying to '.\n          'acquire lease (\"%s\").',\n          $resource->getPHID(),\n          $this->getPHID()));\n    }\n\n    $resource->reload();\n\n    if (($resource->getStatus() !== DrydockResourceStatus::STATUS_ACTIVE) &&\n        ($resource->getStatus() !== DrydockResourceStatus::STATUS_PENDING)) {\n      throw new DrydockAcquiredBrokenResourceException(\n        pht(\n          'Trying to acquire lease (\"%s\") on a resource (\"%s\") in the '.\n          'wrong status (\"%s\").',\n          $this->getPHID(),\n          $resource->getPHID(),\n          $resource->getStatus()));","sourceCodeStart":239,"sourceCodeEnd":275,"githubUrl":"https://github.com/phacility/phabricator/blob/5720a38cfe95b00ca4be5016dd0d2f3195f4fa04/src/applications/drydock/storage/DrydockLease.php#L239-L275","documentation":"Before binding a lease to a resource, Drydock takes a global lock on the resource (`drydock.resource:<hash>`) with a 15-second budget to make the status check + attach atomic. If the lock cannot be grabbed in time (another allocator, updater, or release workflow is holding it), the code throws DrydockResourceLockException wrapping this message. It is inherently transient contention, not corruption: the daemon layer catches this exception type and yields the task to retry later (see DrydockLeaseUpdateWorker, which collects it into `$yields` and throws PhabricatorWorkerYieldException(15)).","triggerScenarios":"Many leases being allocated concurrently against the same resource (build storm landing on one host); a release-resource command holding the lock while leases try to acquire; long-running blueprint customizations under lock; lock holder crashed leaving the global lock held until its lease expires.","commonSituations":"Harbormaster fleets running dozens of builds against a single shared host blueprint; contention spikes after daemons restart and re-process queued lease updates; an operator releasing resources while builds are starting.","solutions":["If seen occasionally in daemon logs: no action — the worker yields and retries automatically within ~15 seconds","If persistent: check for orphaned global locks (`./bin/daemon lantern`/lock inspection or the phabricator_lock table) left by killed daemons and clear them","Spread load: more blueprints/resources so allocators do not converge on one resource lock","Reduce work done while holding the resource lock in custom blueprint code"],"exampleFix":"// before (custom allocator)\ntry {\n  $lease->acquireOnResource($resource);\n} catch (Exception $ex) {\n  throw $ex; // lock contention surfaces as fatal\n}\n\n// after\ntry {\n  $lease->acquireOnResource($resource);\n} catch (DrydockResourceLockException $ex) {\n  // transient contention: wait and retry, as DrydockLeaseUpdateWorker does\n  sleep(15);\n  $lease = id(new DrydockLeaseQuery())\n    ->setViewer($viewer)\n    ->withPHIDs(array($lease->getPHID()))\n    ->executeOne();\n  $lease->acquireOnResource($resource);\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Mirror DrydockLeaseUpdateWorker: treat as transient contention.\ntry {\n  $lease->acquireOnResource($resource);\n} catch (DrydockResourceLockException $ex) {\n  // yield / wait ~15s, reload lease + resource, retry on next pass\n  throw new PhabricatorWorkerYieldException(15);\n}","preventionTips":["Never treat DrydockResourceLockException as fatal — the daemon layer yields and retries by design","Scale out resources/blueprints so allocators stop contending on one resource lock","Keep custom code's resource-lock critical sections short; avoid long provisioning work under the lock","Clean up global locks left by killed daemons when this error persists"],"tags":["drydock","phabricator","lease","lock-contention","transient","concurrency"],"backgroundTag":"lock-acquisition-timeout","analyzedSha":"5720a38cfe95b00ca4be5016dd0d2f3195f4fa04","analyzedAt":"2026-08-21T05:07:25.672Z","schemaVersion":2},"datasetVersion":"2026-08-21T11:28:35.574Z"}