phacility/phabricator · error · PhabricatorWorkerPermanentFailureException

Found unexpected task status

Error message

Found unexpected task status ("%s").

What it means

PhabricatorWorkerBulkJobTaskWorker executes one unit of a bulk job and requires the bulk task row to still be in STATUS_WAITING when it starts (checked while holding a per-task global lock). A different status means the task already ran, is running elsewhere, or was completed — so the worker permanently fails instead of re-executing side effects.

Solutions

1. Check the bulk task's status; if it is RUNNING or DONE, the failure is the guard working — ignore the duplicate.
2. If a task is stuck in RUNNING after a crash, resolve the stale row via the worker management workflows before retrying.
3. Do not re-queue bulk task rows by hand.

try {
  // execute the bulk task worker
} catch (PhabricatorWorkerPermanentFailureException $ex) {
  // terminal: check the bulk task status; ignore duplicates of DONE/RUNNING tasks
}

When it happens

Trigger: Duplicate task rows for one bulk task; a task re-queued after completion; a stale RUNNING row from a crashed daemon being executed again.

Common situations: Database restores that duplicate worker rows; aggressive manual use of bin/worker retry; crashed taskmasters leaving stale rows that later get re-executed after lease expiry.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Found unexpected job status
• Worker has invalid actor PHID
• Worker has invalid job ID
• Worker has invalid task ID
• Worker has no job ID.

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/dfd05e5f0d2b2052. Report an issue: GitHub.

Appendix: source

Thrown at src/infrastructure/daemon/workers/bulk/PhabricatorWorkerBulkJobTaskWorker.php:16

<?php

final class PhabricatorWorkerBulkJobTaskWorker
  extends PhabricatorWorkerBulkJobWorker {

  protected function doWork() {
    $lock = $this->acquireTaskLock();

    $task = $this->loadTask();
    $status = $task->getStatus();
    switch ($task->getStatus()) {
      case PhabricatorWorkerBulkTask::STATUS_WAITING:
        // This is what we expect.
        break;
      default:
        throw new PhabricatorWorkerPermanentFailureException(
          pht(
            'Found unexpected task status ("%s").',
            $status));
    }

    $task
      ->setStatus(PhabricatorWorkerBulkTask::STATUS_RUNNING)
      ->save();

    $lock->unlock();

    $job = $this->loadJob();
    $actor = $this->loadActor($job);

    try {
      $job->runTask($actor, $task);
      $status = PhabricatorWorkerBulkTask::STATUS_DONE;
    } catch (Exception $ex) {

View on GitHub (pinned to 5720a38cfe)