What causes "Permanent failure while activating resource ("%s"): %s" in phacility/phabricator?

Open the resource in the Drydock UI and read its log: the DrydockResourceActivationFailureLogType event names the original exception class and message.

phacility/phabricator · error · PhabricatorWorkerPermanentFailureException

Permanent failure while activating resource ("%s"): %s

Error message

Permanent failure while activating resource ("%s"): %s

What it means

Thrown by the Drydock resource update worker when activating a resource raises an exception. Before throwing, breakResource() marks the resource BROKEN, schedules a status update, and records a DrydockResourceActivationFailureLogType event carrying the original exception class and message. Wrapping it in PhabricatorWorkerPermanentFailureException tells the task queue to never retry this task.

Source

Thrown at src/applications/drydock/worker/DrydockResourceUpdateWorker.php:289

            'Unexpected failure while destroying resource ("%s").',
            $resource->getPHID()),
          $ex);
    }

    $resource
      ->setStatus(DrydockResourceStatus::STATUS_BROKEN)
      ->save();

    $resource->scheduleUpdate();

    $resource->logEvent(
      DrydockResourceActivationFailureLogType::LOGCONST,
      array(
        'class' => get_class($ex),
        'message' => $ex->getMessage(),
      ));

    throw new PhabricatorWorkerPermanentFailureException(
      pht(
        'Permanent failure while activating resource ("%s"): %s',
        $resource->getPHID(),
        $ex->getMessage()));
  }


/* -(  Destroying Resources  )----------------------------------------------- */


  /**
   * @task destroy
   */
  private function destroyResource(DrydockResource $resource) {
    $blueprint = $resource->getBlueprint();
    $blueprint->destroyResource($resource);

    DrydockSlotLock::releaseLocks($resource->getPHID());

View on GitHub (pinned to 5720a38cfe)

Solutions

Open the resource in the Drydock UI and read its log: the DrydockResourceActivationFailureLogType event names the original exception class and message.
Fix the root cause in the blueprint's activation implementation (host reachability, credentials, blueprint configuration values).
Release or destroy the broken resource so new allocations get a clean resource, then retry the operation that requested it.
If failures can be transient, make the blueprint retry or fail with a non-exception signal before the worker permanently breaks the resource.

Example fix

// before: any exception during activation permanently breaks the resource
public function activateResource(DrydockResource $resource) {
  $this->getSSHCLI()->execx('start-service'); // throws on transient SSH failure
}

// after: absorb transient failures before the worker marks the resource broken
public function activateResource(DrydockResource $resource) {
  $retries = 3;
  while (true) {
    try {
      $this->getSSHCLI()->execx('start-service');
      return;
    } catch (PhutilExecException $ex) {
      if (--$retries <= 0) {
        throw $ex;
      }
      sleep(1);
    }
  }
}

Defensive patterns

Strategy: try-catch

Try / catch

// inside blueprint activation: absorb or contextualize transient failures
try {
  $this->activateOnRemoteHost($resource);
} catch (PhutilExecException $ex) {
  // retry, degrade, or rethrow with context — anything uncaught
  // reaches breakResource() and permanently breaks the resource
  throw new Exception(pht('Host %s unreachable: %s', $host, $ex->getMessage()), $ex->getCode());
}

Prevention

Test blueprint activation against a real backend before enabling allocation.
Log rich context (host, credentials name, command) in activation code so the DrydockResourceActivationFailureLogType event is diagnosable.
Monitor drydock resource status; break BROKEN resources quickly so new allocations are not starved.
Keep blueprint configuration validated in the UI so activation never starts with bad values.

When it happens

Trigger: A DrydockResourceUpdateWorker task activates a resource and the blueprint's activation logic (e.g. activateResource() / underlying host, CLI, or API calls) throws any Exception; breakResource() then converts it into this permanent failure. Placeholders are the resource PHID and the original exception message.

Common situations: Blueprint implementation bugs: unreachable remote host, missing or invalid credentials, misconfigured blueprint fields, or an external system rejecting the activation request. Also happens after blueprint code changes leave stored pending resources unactivatable.

Related errors

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/eac6c644240875a1. Report an issue: GitHub.