What causes "Lease has already been destroyed!" in phacility/phabricator?

Re-run the lease command to get a new lease on a live resource.

phacility/phabricator · error · Exception

Lease has already been destroyed!

Error message

Lease has already been destroyed!

What it means

Same wait loop in `bin/drydock lease`, but the lease status is STATUS_DESTROYED: the underlying resource (or the lease itself) was destroyed before activation completed. Destruction can be triggered by an operator, by the resource being broken and cleaned up, or by daemons reclaiming resources, and it is terminal — the CLI stops waiting immediately.

Source

Thrown at src/applications/drydock/management/DrydockManagementLeaseWorkflow.php:250

        }

        echo tsprintf(
          "(Lease #%d) <%s> %B\n",
          $lease->getID(),
          $type,
          $data);
      }

      $status = $lease->getStatus();

      switch ($status) {
        case DrydockLeaseStatus::STATUS_ACTIVE:
          $is_active = true;
          break;
        case DrydockLeaseStatus::STATUS_RELEASED:
          throw new Exception(pht('Lease has already been released!'));
        case DrydockLeaseStatus::STATUS_DESTROYED:
          throw new Exception(pht('Lease has already been destroyed!'));
        case DrydockLeaseStatus::STATUS_BROKEN:
          throw new Exception(pht('Lease has been broken!'));
        case DrydockLeaseStatus::STATUS_PENDING:
        case DrydockLeaseStatus::STATUS_ACQUIRED:
          break;
        default:
          throw new Exception(
            pht(
              'Lease has unknown status "%s".',
              $status));
      }

      if ($is_active) {
        break;
      } else {
        sleep(1);
      }
    }

View on GitHub (pinned to 5720a38cfe)

Solutions

Re-run the lease command to get a new lease on a live resource.
Check the Drydock resource console for why the resource was destroyed (operator action vs. daemon) and address the root cause.
Avoid destroying resources while leases against them are still activating; check active leases first.

Defensive patterns

Strategy: try-catch

Validate before calling

if (!$lease->isActive() &&
    $lease->getStatus() === DrydockLeaseStatus::STATUS_DESTROYED) {
  throw new RuntimeException(
    "Lease {$lease->getID()} was destroyed; re-acquire.");
}

Try / catch

try {
  $lease->waitUntilActive();
} catch (Exception $ex) {
  if (preg_match('/already been destroyed/', $ex->getMessage())) {
    // terminal: verify the backing resource still exists, then re-acquire
    $lease = acquireNewLease($attributes);
  } else {
    throw $ex;
  }
}

Prevention

Do not destroy Drydock resources from automation or the console while leases against them are still activating.
Treat destroy-during-wait as an operational signal: find who issued the destroy in the Drydock log.

When it happens

Trigger: Polling a lease whose resource was destroyed concurrently: an operator destroys the host resource from the Drydock console, or a cleaner daemon destroys pending resources while the lease is still being allocated.

Common situations: Manual cleanup of a stuck allocation while the lease command is still waiting; aggressive reclamation settings destroying slow-pending resources; test runs on blueprints that get torn down mid-flight.

Related errors

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/3e328e1426233606. Report an issue: GitHub.