phacility/phabricator · error · PhabricatorWorkerPermanentFailureException

Permanent failure while activating lease ("%s"): %s

Error message

Permanent failure while activating lease ("%s"): %s

What it means

This is the terminal report of lease activation failure: the blueprint's activation threw, the worker already marked the lease BROKEN and logged a DrydockLeaseActivationFailureLogType event, and then throws PhabricatorWorkerPermanentFailureException so the task will not be retried. The '%s' suffix carries the original exception message. The lease is dead; the message exists to surface why.

Source

Thrown at src/applications/drydock/worker/DrydockLeaseUpdateWorker.php:1096

    $lease->logEvent(
      DrydockLeaseActivationFailureLogType::LOGCONST,
      array(
        'class' => get_class($ex),
        'message' => $ex->getMessage(),
      ));

    $lease->awakenTasks();

    $this->queueTask(
      __CLASS__,
      array(
        'leasePHID' => $lease->getPHID(),
      ),
      array(
        'objectPHID' => $lease->getPHID(),
      ));

    throw new PhabricatorWorkerPermanentFailureException(
      pht(
        'Permanent failure while activating lease ("%s"): %s',
        $lease->getPHID(),
        $ex->getMessage()));
  }


/* -(  Destroying Leases  )-------------------------------------------------- */


  /**
   * @task destroy
   */
  private function destroyLease(DrydockLease $lease) {
    $resource = $lease->getResource();

    if ($resource) {
      $blueprint = $resource->getBlueprint();

View on GitHub (pinned to 5720a38cfe)

Solutions

1. Open the lease's event log in Drydock: the DrydockLeaseActivationFailureLogType event records the original failure details.
2. Fix the underlying activation error (credentials, network, command) identified in the message and log.
3. Acquire a new lease; the broken one will not recover and the task will not retry.
4. If activation can transiently fail, make the blueprint throw PhabricatorWorkerYieldException instead so the worker retries rather than breaking the lease.

try {
  $lease->waitUntilActive();
} catch (PhabricatorWorkerPermanentFailureException $ex) {
  // lease is broken; inspect its event log, then acquire a new lease
  phlog($ex->getMessage());
  $lease = DrydockLease::newFromBlueprint($blueprint)->queueForActivation();
}

When it happens

Trigger: Blueprint activateLease() throwing (command execution failure, missing credential, environment problem); activation racing a resource that broke; any exception escaping activateLease during the activation worker run.

Common situations: Working-copy activation failing because the VCS clone/checkout command errored; host blueprint activation failing on SSH or provisioning errors; expired or missing Almanac credentials during lease activation.

Related errors

• Lease "%s" never activated.
• Lease "%s" could not be loaded.
• Trying to acquire an active lease on a pending resource. You
• Trying to activate a lease which has the wrong status: statu
• Trying to activate a lease on a pending resource.

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/18dd524cb7a7e203. Report an issue: GitHub.