phacility/phabricator · warning · DrydockAcquiredBrokenResourceException

Trying to acquire lease ("%s") on a resource ("%s") in the w

Error message

Trying to acquire lease ("%s") on a resource ("%s") in the wrong status ("%s").

What it means

After taking the resource lock, acquireOnResource() reloads the resource and re-checks its status; only ACTIVE and PENDING resources may receive leases. If the reload reveals the resource was released, broken, or destroyed in the window between allocator selection and lock acquisition (a classic TOCTOU race — e.g. it was reclaimed by a release command or failed an update), it throws DrydockAcquiredBrokenResourceException. The daemon layer treats this as bad luck, not error: DrydockLeaseUpdateWorker catches it, skips that resource, and retries allocation on other resources or after a yield.

Source

Thrown at src/applications/drydock/storage/DrydockLease.php:269

    $lock_key = 'drydock.resource:'.$hash;
    $lock = PhabricatorGlobalLock::newLock($lock_key);

    try {
      $lock->lock(15);
    } catch (Exception $ex) {
      throw new DrydockResourceLockException(
        pht(
          'Failed to acquire lock for resource ("%s") while trying to '.
          'acquire lease ("%s").',
          $resource->getPHID(),
          $this->getPHID()));
    }

    $resource->reload();

    if (($resource->getStatus() !== DrydockResourceStatus::STATUS_ACTIVE) &&
        ($resource->getStatus() !== DrydockResourceStatus::STATUS_PENDING)) {
      throw new DrydockAcquiredBrokenResourceException(
        pht(
          'Trying to acquire lease ("%s") on a resource ("%s") in the '.
          'wrong status ("%s").',
          $this->getPHID(),
          $resource->getPHID(),
          $resource->getStatus()));
    }

    $caught = null;
    try {
      $this->openTransaction();

      try {
        DrydockSlotLock::acquireLocks($this->getPHID(), $this->slotLocks);
        $this->slotLocks = array();
      } catch (DrydockSlotLockException $ex) {
        $this->killTransaction();

View on GitHub (pinned to 5720a38cfe)

Solutions

1. If seen in worker logs sporadically: expected behavior — the lease allocation is retried on another resource; verify with the lease's eventual status
2. Avoid destructive `release-resource --all` sweeps while builds are actively allocating; pause the build first
3. Investigate why the resource went BROKEN (blueprint log) if the same resource repeatedly dies at acquisition
4. In custom code, catch DrydockAcquiredBrokenResourceException and move to the next candidate resource, mirroring the worker loop

Example fix

// before
foreach ($resources as $resource) {
  $lease->acquireOnResource($resource);
  break;
}
// Exception: Trying to acquire lease ("PHID-X") on a resource ("PHID-Y") in the wrong status ("released").

// after
foreach ($resources as $resource) {
  try {
    $lease->acquireOnResource($resource);
    break;
  } catch (DrydockAcquiredBrokenResourceException $ex) {
    continue; // resource died mid-flight; try the next candidate
  }
}

// Mirror the worker's candidate loop:
foreach ($ranked_resources as $resource) {
  try {
    $lease->acquireOnResource($resource);
    $allocated = $resource;
    break;
  } catch (DrydockAcquiredBrokenResourceException $ex) {
    continue; // resource died mid-flight; try next candidate
  }
}
if (!$allocated) {
  throw new PhabricatorWorkerYieldException(15); // retry allocation later
}

When it happens

Trigger: A release-resource command (or auto-reaper) destroying the resource between ranking and acquisition; the resource's update worker marking it BROKEN while a lease is mid-acquisition; two allocators racing where one releases/destroys what the other is acquiring.

Common situations: Operators running `drydock release-resource --all` while builds are allocating; resources failing health checks and being broken during allocation; heavy churn windows after daemon restarts.

Related errors

• Trying to acquire an active lease on a pending resource. You
• Failed to acquire lock for resource ("%s") while trying to a
• Trying to activate a lease on a pending resource.
• Trying to activate lease ("%s") on a resource ("%s") in the
• Working copy lease is missing required attribute "%s". Attr

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/d843be093edaf2f0. Report an issue: GitHub.