{"record":{"id":"47b92da37b79fae2","repo":"phacility/phabricator","slug":"unexpected-failure-while-destroying-resource-s","errorCode":null,"errorMessage":"Unexpected failure while destroying resource (\"%s\").","messagePattern":"Unexpected failure while destroying resource \\(\"(.+?)\"\\)\\.","errorType":"exception","errorClass":"PhutilProxyException","httpStatus":null,"severity":"error","filePath":"src/applications/drydock/worker/DrydockResourceUpdateWorker.php","lineNumber":269,"sourceCode":"\n    $this->destroyResource($resource);\n  }\n\n\n/* -(  Breaking Resources  )------------------------------------------------- */\n\n\n  /**\n   * @task break\n   */\n  private function breakResource(DrydockResource $resource, Exception $ex) {\n    switch ($resource->getStatus()) {\n      case DrydockResourceStatus::STATUS_BROKEN:\n      case DrydockResourceStatus::STATUS_RELEASED:\n      case DrydockResourceStatus::STATUS_DESTROYED:\n        // If the resource was already broken, just throw a normal exception.\n        // This will retry the task eventually.\n        throw new PhutilProxyException(\n          pht(\n            'Unexpected failure while destroying resource (\"%s\").',\n            $resource->getPHID()),\n          $ex);\n    }\n\n    $resource\n      ->setStatus(DrydockResourceStatus::STATUS_BROKEN)\n      ->save();\n\n    $resource->scheduleUpdate();\n\n    $resource->logEvent(\n      DrydockResourceActivationFailureLogType::LOGCONST,\n      array(\n        'class' => get_class($ex),\n        'message' => $ex->getMessage(),\n      ));","sourceCodeStart":251,"sourceCodeEnd":287,"githubUrl":"https://github.com/phacility/phabricator/blob/5720a38cfe95b00ca4be5016dd0d2f3195f4fa04/src/applications/drydock/worker/DrydockResourceUpdateWorker.php#L251-L287","documentation":"breakResource() is the resource update worker's failure handler: when processing a resource throws, it marks the resource BROKEN. If the resource is already broken, released, or destroyed, there is nothing left to break, so the original exception is wrapped in a PhutilProxyException with this message. As the comment notes, this plain exception makes the task retry later; the nested exception carries the real cause.","triggerScenarios":"A destroy/update task fails (blueprint destroyResource throws, remote host unreachable) while the resource is already STATUS_BROKEN/RELEASED/DESTROYED; two workers failing the same resource concurrently, the second one hitting an already-broken row.","commonSituations":"Destruction of a broken resource failing because the host is gone; SSH or permission errors during cleanup; retried resource update tasks after the resource was already broken.","solutions":["Inspect the proxied original exception to find the actual cleanup failure.","Make the blueprint's destroy logic tolerant of partially-destroyed resources (idempotent cleanup).","If the resource is a zombie (broken but undeletable), finish the destroy manually via Drydock CLI after fixing the underlying error."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if (in_array($resource->getStatus(), array(\n  DrydockResourceStatus::STATUS_BROKEN,\n  DrydockResourceStatus::STATUS_RELEASED,\n  DrydockResourceStatus::STATUS_DESTROYED,\n))) {\n  return; // already finalized; skip\n}","typeGuard":null,"tryCatchPattern":"try {\n  $this->destroyResource($resource);\n} catch (PhutilProxyException $ex) {\n  $root = $ex->getPrevious_exception ? $ex->getPreviousException() : $ex;\n  phlog($root->getMessage()); // the real cleanup failure\n}","preventionTips":["Make blueprint destroyResource() idempotent so cleanup can be retried on partially-destroyed resources.","Unwrap PhutilProxyException to find the real error; the outer text only flags a state conflict."],"tags":["drydock","php","resource-lifecycle","cleanup","idempotency","proxy-exception"],"backgroundTag":"cleanup-state-conflict","analyzedSha":"5720a38cfe95b00ca4be5016dd0d2f3195f4fa04","analyzedAt":"2026-08-21T05:07:25.672Z","schemaVersion":2},"datasetVersion":"2026-08-21T11:28:35.574Z"}