phacility/phabricator · error · DrydockAcquiredBrokenResourceException

Trying to activate lease ("%s") on a resource ("%s") in the

Error message

Trying to activate lease ("%s") on a resource ("%s") in the wrong status ("%s").

What it means

activateLease() requires the target resource to be STATUS_ACTIVE. If it is still STATUS_PENDING the worker yields 15 seconds and retries; any other status (broken, released, destroyed) throws DrydockAcquiredBrokenResourceException, which marks the acquired lease broken. This is the guard against activating leases on resources that died between acquisition and activation.

Source

Thrown at src/applications/drydock/worker/DrydockLeaseUpdateWorker.php:990

  /**
   * @task activate
   */
  private function activateLease(DrydockLease $lease) {
    $resource = $lease->getResource();
    if (!$resource) {
      throw new Exception(
        pht('Trying to activate lease with no resource.'));
    }

    $resource_status = $resource->getStatus();

    if ($resource_status == DrydockResourceStatus::STATUS_PENDING) {
      throw new PhabricatorWorkerYieldException(15);
    }

    if ($resource_status != DrydockResourceStatus::STATUS_ACTIVE) {
      throw new DrydockAcquiredBrokenResourceException(
        pht(
          'Trying to activate lease ("%s") on a resource ("%s") in '.
          'the wrong status ("%s").',
          $lease->getPHID(),
          $resource->getPHID(),
          $resource_status));
    }

    // NOTE: We can race resource destruction here. Between the time we
    // performed the read above and now, the resource might have closed, so
    // we may activate leases on dead resources. At least for now, this seems
    // fine: a resource dying right before we activate a lease on it should not
    // be distinguishable from a resource dying right after we activate a lease
    // on it. We end up with an active lease on a dead resource either way, and
    // can not prevent resources dying from lightning strikes.

    $blueprint = $resource->getBlueprint();
    $blueprint->activateLease($resource, $lease);

View on GitHub (pinned to 5720a38cfe)

Solutions

If the status is pending, just wait: the worker yields and retries automatically.
If broken/destroyed, open the resource's event log to find why activation failed and fix that (command error, credentials, disk).
Re-run the lease acquisition against a healthy resource once the root cause is fixed.

Defensive patterns

Strategy: retry

Validate before calling

$status = $resource->getStatus();
if ($status === DrydockResourceStatus::STATUS_PENDING) {
  throw new PhabricatorWorkerYieldException(15); // still activating
}
if ($status !== DrydockResourceStatus::STATUS_ACTIVE) {
  // broken/released/destroyed: fail before attempting activation
  throw new Exception(pht('Resource not active: %s', $status));
}

Try / catch

try {
  $this->activateLease($lease);
} catch (DrydockAcquiredBrokenResourceException $ex) {
  // resource died under the lease: log and re-acquire elsewhere
  phlog($ex->getMessage());
  $this->executeAllocator($lease);
}

Prevention

Yield while the resource is pending instead of treating it as a hard failure.
Keep resource activation healthy (fix provisioning errors early) so leases never wait on broken resources.

When it happens

Trigger: The resource broke while the lease was waiting for it to finish activating (activation command failed); the resource was destroyed by reclamation after the lease was acquired; the resource was released by an admin while leases were pending.

Common situations: Working-copy resource whose git/hg clone fails during activation; host resource whose provisioning errored; lease activation racing resource destruction (the code comment explicitly acknowledges this race is tolerated upstream).

Related errors

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/66880ced257d2250. Report an issue: GitHub.