{"record":{"id":"bfc70f3441c2a0a9","repo":"phacility/phabricator","slug":"unexpected-failure-while-destroying-lease-s","errorCode":null,"errorMessage":"Unexpected failure while destroying lease (\"%s\").","messagePattern":"Unexpected failure while destroying lease \\(\"(.+?)\"\\)\\.","errorType":"exception","errorClass":"PhutilProxyException","httpStatus":null,"severity":"error","filePath":"src/applications/drydock/worker/DrydockLeaseUpdateWorker.php","lineNumber":1067,"sourceCode":"      $blueprint->didReleaseLease($resource, $lease);\n    }\n\n    $this->destroyLease($lease);\n  }\n\n\n/* -(  Breaking Leases  )---------------------------------------------------- */\n\n\n  /**\n   * @task break\n   */\n  protected function breakLease(DrydockLease $lease, Exception $ex) {\n    switch ($lease->getStatus()) {\n      case DrydockLeaseStatus::STATUS_BROKEN:\n      case DrydockLeaseStatus::STATUS_RELEASED:\n      case DrydockLeaseStatus::STATUS_DESTROYED:\n        throw new PhutilProxyException(\n          pht(\n            'Unexpected failure while destroying lease (\"%s\").',\n            $lease->getPHID()),\n          $ex);\n    }\n\n    $lease\n      ->setStatus(DrydockLeaseStatus::STATUS_BROKEN)\n      ->save();\n\n    $lease->logEvent(\n      DrydockLeaseActivationFailureLogType::LOGCONST,\n      array(\n        'class' => get_class($ex),\n        'message' => $ex->getMessage(),\n      ));\n\n    $lease->awakenTasks();","sourceCodeStart":1049,"sourceCodeEnd":1085,"githubUrl":"https://github.com/phacility/phabricator/blob/5720a38cfe95b00ca4be5016dd0d2f3195f4fa04/src/applications/drydock/worker/DrydockLeaseUpdateWorker.php#L1049-L1085","documentation":"breakLease() is the worker's last-resort handler: when processing a lease throws, it marks the lease BROKEN. But if the lease is already broken, released, or destroyed, marking it broken again makes no sense, so the original exception is wrapped in a PhutilProxyException with this message. Seeing it means the failure happened while cleaning up an already-finalized lease, and the nested exception is the real story.","triggerScenarios":"A lease destruction task fails (e.g. blueprint destroyLease throws) while the lease is already STATUS_BROKEN/RELEASED/DESTROYED; two workers breaking the same lease concurrently, where the second failure hits an already-broken lease.","commonSituations":"Destroy command on the remote host failing (SSH down, permissions) for a lease that was already broken by an earlier failure; retries of the lease update task after the lease was released.","solutions":["Read the proxied original exception (PhutilProxyException::getPreviousException()) and fix that root cause.","Make blueprint destroy/cleanup logic idempotent so a second failure on an already-final lease does not throw.","Manually release/destroy the stuck lease via Drydock CLI/console once the underlying issue is fixed."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if (in_array($lease->getStatus(), array(\n  DrydockLeaseStatus::STATUS_BROKEN,\n  DrydockLeaseStatus::STATUS_RELEASED,\n  DrydockLeaseStatus::STATUS_DESTROYED,\n))) {\n  return; // nothing to break; skip cleanup re-entry\n}","typeGuard":null,"tryCatchPattern":"try {\n  $this->destroyLease($lease);\n} catch (PhutilProxyException $ex) {\n  $root = $ex->getPreviousException(); // real failure\n  phlog($root ? $root->getMessage() : $ex->getMessage());\n}","preventionTips":["Make lease destruction idempotent: tolerate missing remote state instead of throwing.","Always unwrap PhutilProxyException and fix the nested cause; the outer message only says cleanup hit a finalized lease."],"tags":["drydock","php","lease","cleanup","idempotency","proxy-exception"],"backgroundTag":"cleanup-state-conflict","analyzedSha":"5720a38cfe95b00ca4be5016dd0d2f3195f4fa04","analyzedAt":"2026-08-21T05:07:25.672Z","schemaVersion":2},"datasetVersion":"2026-08-21T11:28:35.574Z"}