What causes "Unable to acquire lease "%s" on any resource." in phacility/phabricator?

Iterate getAggregateExceptions() (or read the lease event log) to find the dominant underlying error instead of guessing from the outer message.

phacility/phabricator · error · PhutilAggregateException

Unable to acquire lease "%s" on any resource.

Error message

Unable to acquire lease "%s" on any resource.

What it means

After the allocator finds acquirable resources, it tries to acquire the lease on each one and collects every exception. If nothing was acquired and nothing yielded, it throws PhutilAggregateException wrapping all the per-resource failures. The outer message is generic; the real causes are the nested exceptions and the corresponding resource/blueprint logs.

Source

Thrown at src/applications/drydock/worker/DrydockLeaseUpdateWorker.php:564

        $yields[] = $ex;
      } catch (PhabricatorWorkerYieldException $ex) {
        // We can be told to yield, particularly by the supplemental allocator
        // trying to give us a supplemental resource.
        $yields[] = $ex;
      } catch (Exception $ex) {
        $exceptions[] = $ex;
      }
    }

    if ($allocated) {
      return $allocated;
    }

    if ($yields) {
      throw new PhabricatorWorkerYieldException(15);
    }

    throw new PhutilAggregateException(
      pht(
        'Unable to acquire lease "%s" on any resource.',
        $lease->getPHID()),
      $exceptions);
  }


  /**
   * Get all the concrete @{class:DrydockBlueprint}s which can possibly
   * build a resource to satisfy a lease.
   *
   * @param DrydockLease Requested lease.
   * @return list<DrydockBlueprint> List of qualifying blueprints.
   * @task allocator
   */
  private function loadBlueprintsForAllocatingLease(
    DrydockLease $lease) {
    $viewer = $this->getViewer();

View on GitHub (pinned to 5720a38cfe)

Solutions

Iterate getAggregateExceptions() (or read the lease event log) to find the dominant underlying error instead of guessing from the outer message.
Fix the most common nested cause first (credentials, network, blueprint bug).
If all resources are at capacity, allocate more resources (raise blueprint limits) or wait for leases to release.
Re-run the lease update; with the root cause fixed this error clears on the next attempt.

Example fix

try {
  $lease->waitUntilActive();
} catch (PhutilAggregateException $ex) {
  foreach ($ex->getAggregateExceptions() as $inner) {
    phlog($inner->getMessage()); // real cause lives here
  }
  throw $ex;
}

Defensive patterns

Strategy: retry

Try / catch

try {
  $lease->waitUntilActive();
} catch (PhutilAggregateException $ex) {
  foreach ($ex->getAggregateExceptions() as $inner) {
    phlog($inner); // inspect real causes before deciding
  }
  throw new PhabricatorWorkerYieldException(15); // transient: retry
}

Prevention

Always inspect the aggregate's nested exceptions rather than the outer message.
Keep at least one healthy, under-capacity resource per type so acquisition has a target.
Monitor resource status; broken resources shrink the acquirable pool until this error surfaces.

When it happens

Trigger: Every candidate resource's blueprint acquireLease() threw (SSH/command failures, full capacity); all matching resources are in a non-allocatable status so the acquire attempts fail immediately; several concurrent leases win the race for the last free slots and the losers see only failures.

Common situations: Host blueprints whose remote SSH target is down; working-copy leases when every existing clone is broken or at its lease limit; transient races right after a new resource starts activating; daemon/cluster connectivity problems causing every acquire command to fail.

Related errors

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/af9aa7981b71d493. Report an issue: GitHub.