phacility/phabricator · warning · DrydockResourceLockException

Failed to acquire lock for resource ("%s") while trying to a

Error message

Failed to acquire lock for resource ("%s") while trying to acquire lease ("%s").

What it means

Before binding a lease to a resource, Drydock takes a global lock on the resource (`drydock.resource:<hash>`) with a 15-second budget to make the status check + attach atomic. If the lock cannot be grabbed in time (another allocator, updater, or release workflow is holding it), the code throws DrydockResourceLockException wrapping this message. It is inherently transient contention, not corruption: the daemon layer catches this exception type and yields the task to retry later (see DrydockLeaseUpdateWorker, which collects it into `$yields` and throws PhabricatorWorkerYieldException(15)).

Source

Thrown at src/applications/drydock/storage/DrydockLease.php:257

            'You can not immediately activate leases on resources which '.
            'need time to start up.'));
      }
    }

    // Before we associate the lease with the resource, we lock the resource
    // and reload it to make sure it is still pending or active. If we don't
    // do this, the resource may have just been reclaimed. (Once we acquire
    // the resource that stops it from being released, so we're nearly safe.)

    $resource_phid = $resource->getPHID();
    $hash = PhabricatorHash::digestForIndex($resource_phid);
    $lock_key = 'drydock.resource:'.$hash;
    $lock = PhabricatorGlobalLock::newLock($lock_key);

    try {
      $lock->lock(15);
    } catch (Exception $ex) {
      throw new DrydockResourceLockException(
        pht(
          'Failed to acquire lock for resource ("%s") while trying to '.
          'acquire lease ("%s").',
          $resource->getPHID(),
          $this->getPHID()));
    }

    $resource->reload();

    if (($resource->getStatus() !== DrydockResourceStatus::STATUS_ACTIVE) &&
        ($resource->getStatus() !== DrydockResourceStatus::STATUS_PENDING)) {
      throw new DrydockAcquiredBrokenResourceException(
        pht(
          'Trying to acquire lease ("%s") on a resource ("%s") in the '.
          'wrong status ("%s").',
          $this->getPHID(),
          $resource->getPHID(),
          $resource->getStatus()));

View on GitHub (pinned to 5720a38cfe)

Solutions

If seen occasionally in daemon logs: no action — the worker yields and retries automatically within ~15 seconds
If persistent: check for orphaned global locks (`./bin/daemon lantern`/lock inspection or the phabricator_lock table) left by killed daemons and clear them
Spread load: more blueprints/resources so allocators do not converge on one resource lock
Reduce work done while holding the resource lock in custom blueprint code

Example fix

// before (custom allocator)
try {
  $lease->acquireOnResource($resource);
} catch (Exception $ex) {
  throw $ex; // lock contention surfaces as fatal
}

// after
try {
  $lease->acquireOnResource($resource);
} catch (DrydockResourceLockException $ex) {
  // transient contention: wait and retry, as DrydockLeaseUpdateWorker does
  sleep(15);
  $lease = id(new DrydockLeaseQuery())
    ->setViewer($viewer)
    ->withPHIDs(array($lease->getPHID()))
    ->executeOne();
  $lease->acquireOnResource($resource);
}

Defensive patterns

Strategy: retry

Try / catch

// Mirror DrydockLeaseUpdateWorker: treat as transient contention.
try {
  $lease->acquireOnResource($resource);
} catch (DrydockResourceLockException $ex) {
  // yield / wait ~15s, reload lease + resource, retry on next pass
  throw new PhabricatorWorkerYieldException(15);
}

Prevention

Never treat DrydockResourceLockException as fatal — the daemon layer yields and retries by design
Scale out resources/blueprints so allocators stop contending on one resource lock
Keep custom code's resource-lock critical sections short; avoid long provisioning work under the lock
Clean up global locks left by killed daemons when this error persists

When it happens

Trigger: Many leases being allocated concurrently against the same resource (build storm landing on one host); a release-resource command holding the lock while leases try to acquire; long-running blueprint customizations under lock; lock holder crashed leaving the global lock held until its lease expires.

Common situations: Harbormaster fleets running dozens of builds against a single shared host blueprint; contention spikes after daemons restart and re-process queued lease updates; an operator releasing resources while builds are starting.

Related errors

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/77fb2b38fd431ead. Report an issue: GitHub.