What causes "Unexpected failure while destroying lease ("%s")." in phacility/phabricator?

Read the proxied original exception (PhutilProxyException::getPreviousException()) and fix that root cause.

phacility/phabricator · error · PhutilProxyException

Unexpected failure while destroying lease ("%s").

Error message

Unexpected failure while destroying lease ("%s").

What it means

breakLease() is the worker's last-resort handler: when processing a lease throws, it marks the lease BROKEN. But if the lease is already broken, released, or destroyed, marking it broken again makes no sense, so the original exception is wrapped in a PhutilProxyException with this message. Seeing it means the failure happened while cleaning up an already-finalized lease, and the nested exception is the real story.

Source

Thrown at src/applications/drydock/worker/DrydockLeaseUpdateWorker.php:1067

      $blueprint->didReleaseLease($resource, $lease);
    }

    $this->destroyLease($lease);
  }


/* -(  Breaking Leases  )---------------------------------------------------- */


  /**
   * @task break
   */
  protected function breakLease(DrydockLease $lease, Exception $ex) {
    switch ($lease->getStatus()) {
      case DrydockLeaseStatus::STATUS_BROKEN:
      case DrydockLeaseStatus::STATUS_RELEASED:
      case DrydockLeaseStatus::STATUS_DESTROYED:
        throw new PhutilProxyException(
          pht(
            'Unexpected failure while destroying lease ("%s").',
            $lease->getPHID()),
          $ex);
    }

    $lease
      ->setStatus(DrydockLeaseStatus::STATUS_BROKEN)
      ->save();

    $lease->logEvent(
      DrydockLeaseActivationFailureLogType::LOGCONST,
      array(
        'class' => get_class($ex),
        'message' => $ex->getMessage(),
      ));

    $lease->awakenTasks();

View on GitHub (pinned to 5720a38cfe)

Solutions

Read the proxied original exception (PhutilProxyException::getPreviousException()) and fix that root cause.
Make blueprint destroy/cleanup logic idempotent so a second failure on an already-final lease does not throw.
Manually release/destroy the stuck lease via Drydock CLI/console once the underlying issue is fixed.

Defensive patterns

Strategy: try-catch

Validate before calling

if (in_array($lease->getStatus(), array(
  DrydockLeaseStatus::STATUS_BROKEN,
  DrydockLeaseStatus::STATUS_RELEASED,
  DrydockLeaseStatus::STATUS_DESTROYED,
))) {
  return; // nothing to break; skip cleanup re-entry
}

Try / catch

try {
  $this->destroyLease($lease);
} catch (PhutilProxyException $ex) {
  $root = $ex->getPreviousException(); // real failure
  phlog($root ? $root->getMessage() : $ex->getMessage());
}

Prevention

Make lease destruction idempotent: tolerate missing remote state instead of throwing.
Always unwrap PhutilProxyException and fix the nested cause; the outer message only says cleanup hit a finalized lease.

When it happens

Trigger: A lease destruction task fails (e.g. blueprint destroyLease throws) while the lease is already STATUS_BROKEN/RELEASED/DESTROYED; two workers breaking the same lease concurrently, where the second failure hits an already-broken lease.

Common situations: Destroy command on the remote host failing (SSH down, permissions) for a lease that was already broken by an earlier failure; retries of the lease update task after the lease was released.

Related errors

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/bfc70f3441c2a0a9. Report an issue: GitHub.