{"record":{"id":"eac6c644240875a1","repo":"phacility/phabricator","slug":"permanent-failure-while-activating-resource-s","errorCode":null,"errorMessage":"Permanent failure while activating resource (\"%s\"): %s","messagePattern":"Permanent failure while activating resource \\(\"(.+?)\"\\): (.+?)","errorType":"exception","errorClass":"PhabricatorWorkerPermanentFailureException","httpStatus":null,"severity":"error","filePath":"src/applications/drydock/worker/DrydockResourceUpdateWorker.php","lineNumber":289,"sourceCode":"            'Unexpected failure while destroying resource (\"%s\").',\n            $resource->getPHID()),\n          $ex);\n    }\n\n    $resource\n      ->setStatus(DrydockResourceStatus::STATUS_BROKEN)\n      ->save();\n\n    $resource->scheduleUpdate();\n\n    $resource->logEvent(\n      DrydockResourceActivationFailureLogType::LOGCONST,\n      array(\n        'class' => get_class($ex),\n        'message' => $ex->getMessage(),\n      ));\n\n    throw new PhabricatorWorkerPermanentFailureException(\n      pht(\n        'Permanent failure while activating resource (\"%s\"): %s',\n        $resource->getPHID(),\n        $ex->getMessage()));\n  }\n\n\n/* -(  Destroying Resources  )----------------------------------------------- */\n\n\n  /**\n   * @task destroy\n   */\n  private function destroyResource(DrydockResource $resource) {\n    $blueprint = $resource->getBlueprint();\n    $blueprint->destroyResource($resource);\n\n    DrydockSlotLock::releaseLocks($resource->getPHID());","sourceCodeStart":271,"sourceCodeEnd":307,"githubUrl":"https://github.com/phacility/phabricator/blob/5720a38cfe95b00ca4be5016dd0d2f3195f4fa04/src/applications/drydock/worker/DrydockResourceUpdateWorker.php#L271-L307","documentation":"Thrown by the Drydock resource update worker when activating a resource raises an exception. Before throwing, breakResource() marks the resource BROKEN, schedules a status update, and records a DrydockResourceActivationFailureLogType event carrying the original exception class and message. Wrapping it in PhabricatorWorkerPermanentFailureException tells the task queue to never retry this task.","triggerScenarios":"A DrydockResourceUpdateWorker task activates a resource and the blueprint's activation logic (e.g. activateResource() / underlying host, CLI, or API calls) throws any Exception; breakResource() then converts it into this permanent failure. Placeholders are the resource PHID and the original exception message.","commonSituations":"Blueprint implementation bugs: unreachable remote host, missing or invalid credentials, misconfigured blueprint fields, or an external system rejecting the activation request. Also happens after blueprint code changes leave stored pending resources unactivatable.","solutions":["Open the resource in the Drydock UI and read its log: the DrydockResourceActivationFailureLogType event names the original exception class and message.","Fix the root cause in the blueprint's activation implementation (host reachability, credentials, blueprint configuration values).","Release or destroy the broken resource so new allocations get a clean resource, then retry the operation that requested it.","If failures can be transient, make the blueprint retry or fail with a non-exception signal before the worker permanently breaks the resource."],"exampleFix":"// before: any exception during activation permanently breaks the resource\npublic function activateResource(DrydockResource $resource) {\n  $this->getSSHCLI()->execx('start-service'); // throws on transient SSH failure\n}\n\n// after: absorb transient failures before the worker marks the resource broken\npublic function activateResource(DrydockResource $resource) {\n  $retries = 3;\n  while (true) {\n    try {\n      $this->getSSHCLI()->execx('start-service');\n      return;\n    } catch (PhutilExecException $ex) {\n      if (--$retries <= 0) {\n        throw $ex;\n      }\n      sleep(1);\n    }\n  }\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// inside blueprint activation: absorb or contextualize transient failures\ntry {\n  $this->activateOnRemoteHost($resource);\n} catch (PhutilExecException $ex) {\n  // retry, degrade, or rethrow with context — anything uncaught\n  // reaches breakResource() and permanently breaks the resource\n  throw new Exception(pht('Host %s unreachable: %s', $host, $ex->getMessage()), $ex->getCode());\n}","preventionTips":["Test blueprint activation against a real backend before enabling allocation.","Log rich context (host, credentials name, command) in activation code so the DrydockResourceActivationFailureLogType event is diagnosable.","Monitor drydock resource status; break BROKEN resources quickly so new allocations are not starved.","Keep blueprint configuration validated in the UI so activation never starts with bad values."],"tags":["drydock","worker","resource-activation","permanent-failure","php"],"backgroundTag":"drydock-resource-activation-failed","analyzedSha":"5720a38cfe95b00ca4be5016dd0d2f3195f4fa04","analyzedAt":"2026-08-21T05:07:25.672Z","schemaVersion":2},"datasetVersion":"2026-08-21T11:28:35.574Z"}