What causes "Lease has already been released!" in phacility/phabricator?

Re-run `bin/drydock lease` to acquire a fresh lease once the interfering release source is gone.

phacility/phabricator · error · Exception

Lease has already been released!

Error message

Lease has already been released!

What it means

Thrown by the wait loop in `bin/drydock lease` while polling a lease until it becomes ACTIVE. If the lease's status is STATUS_RELEASED, the command aborts: the lease was released (by another actor, by expiry of its until time, or by a concurrent drydock process) before it could ever activate, so waiting is pointless.

Source

Thrown at src/applications/drydock/management/DrydockManagementLeaseWorkflow.php:248

          $type = pht('Unknown ("%s")', $type_key);
          $data = null;
        }

        echo tsprintf(
          "(Lease #%d) <%s> %B\n",
          $lease->getID(),
          $type,
          $data);
      }

      $status = $lease->getStatus();

      switch ($status) {
        case DrydockLeaseStatus::STATUS_ACTIVE:
          $is_active = true;
          break;
        case DrydockLeaseStatus::STATUS_RELEASED:
          throw new Exception(pht('Lease has already been released!'));
        case DrydockLeaseStatus::STATUS_DESTROYED:
          throw new Exception(pht('Lease has already been destroyed!'));
        case DrydockLeaseStatus::STATUS_BROKEN:
          throw new Exception(pht('Lease has been broken!'));
        case DrydockLeaseStatus::STATUS_PENDING:
        case DrydockLeaseStatus::STATUS_ACQUIRED:
          break;
        default:
          throw new Exception(
            pht(
              'Lease has unknown status "%s".',
              $status));
      }

      if ($is_active) {
        break;
      } else {
        sleep(1);

View on GitHub (pinned to 5720a38cfe)

Solutions

Re-run `bin/drydock lease` to acquire a fresh lease once the interfering release source is gone.
Find who released it: check the Drydock log for the lease (Drydock console) for a 'release' actor, and check whether --until was set too aggressively.
If concurrent automation is releasing leases, serialize the acquire/use/release cycle with a lock or unique lease attributes.

Example fix

# before
bin/drydock lease --type host --until '+30 seconds'  # lease expires mid-allocation

# after
bin/drydock lease --type host --until '+1 hour'

Defensive patterns

Strategy: try-catch

Validate before calling

$status = $lease->getStatus();
if ($status !== DrydockLeaseStatus::STATUS_ACTIVE &&
    $status !== DrydockLeaseStatus::STATUS_PENDING &&
    $status !== DrydockLeaseStatus::STATUS_ACQUIRED) {
  throw new RuntimeException(
    "Lease {$lease->getID()} is in terminal status {$status}; re-acquire.");
}

Try / catch

try {
  // acquire + wait for activation (e.g. bin/drydock lease or $lease->waitUntilActive())
} catch (Exception $ex) {
  if (preg_match('/already been released/', $ex->getMessage())) {
    // terminal: log, then re-acquire a fresh lease (single retry)
    $lease = acquireNewLease($attributes);
  } else {
    throw $ex;
  }
}

Prevention

Give --until enough headroom to cover slow allocations.
Ensure only one owner releases a given lease; serialize acquire/use/release in concurrent automation.
Check the lease status in the Drydock console before re-running commands against old lease IDs.

When it happens

Trigger: `bin/drydock lease --type host` creates a lease, the CLI polls getStatus(), and some other process (web UI release action, a cleanup daemon, another CLI run, lease expiration) releases it while it is still PENDING/ACQUIRED.

Common situations: Two operators/scripts racing on the same lease; a very short --until causing the reaper to release the lease during allocation; the allocation being slow enough that cleanup daemons reap pending leases.

Related errors

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/62e739e03891f4da. Report an issue: GitHub.