What causes "Cannot mark the stage: [%s] finished" in apache/druid?

Verify the stage reached an effectively-finished state before calling finishStage with strict=true

apache/druid · error · IllegalArgumentException

Cannot mark the stage: [%s] finished

Error message

Cannot mark the stage: [%s] finished

What it means

Thrown by ControllerQueryKernel.finishStage(stageId, strict=true) as an IllegalArgumentException when the stage is not in the effectivelyFinishedStages set, i.e. it has not been pre-marked as effectively finished, so strictly finishing it would violate the state machine.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernel.java:612

      if (stageTracker.getPhase() != ControllerStagePhase.NEW) {
        throw new ISE("Cannot start the stage: [%s]", stageId);
      }

      stageTracker.start();
    });
  }

  /**
   * Checks if the stage can be finished, delegates call to {@link ControllerStageTracker#finish()} for internal phase
   * transition and registers the transition in this query kernel
   * <p>
   * If the method is called with strict = true, we confirm if the stage can be marked as finished or else
   * throw illegal argument exception
   */
  public void finishStage(final StageId stageId, final boolean strict)
  {
    if (strict && !effectivelyFinishedStages.contains(stageId)) {
      throw new IAE("Cannot mark the stage: [%s] finished", stageId);
    }
    doWithStageTracker(stageId, stageTracker -> {
      stageTracker.finish();
      effectivelyFinishedStages.remove(stageId);
    });
    stageWorkOrders.remove(stageId);
  }

  /**
   * Delegates call to {@link ControllerStageTracker#getWorkerInputs()}
   */
  public WorkerInputs getWorkerInputsForStage(final StageId stageId)
  {
    return getStageTrackerOrThrow(stageId).getWorkerInputs();
  }

  /**
   * Delegates call to {@link ControllerStageTracker#addPartialKeyInformationForWorker(int, PartialKeyStatisticsInformation)}.

View on GitHub (pinned to 9b90983fd2)

Solutions

Verify the stage reached an effectively-finished state before calling finishStage with strict=true
Pass strict=false if best-effort finishing is acceptable and the stage may legitimately not be effectively finished
Inspect the state-machine step that registers effectively-finished stages (StagePhase equivalents) and confirm it ran for this stage
If this follows a controller restart, check that kernel transitions were fully persisted before replaying terminal-stage finishing

Example fix

// before
kernel.finishStage(stageId, true);

// after
if (kernel.isStageEffectivelyFinished(stageId)) {
  kernel.finishStage(stageId, true);
} else {
  kernel.finishStage(stageId, false);
}

Defensive patterns

Strategy: validation

Validate before calling

if (!queryKernel.isStageEffectivelyFinished(stageId)) {
  queryKernel.finishStage(stageId, false);
} else {
  queryKernel.finishStage(stageId, true);
}

Try / catch

try {
  queryKernel.finishStage(stageId, true);
} catch (IllegalArgumentException e) {
  LOG.warn(e, "Stage %s not effectively finished; finishing non-strictly", stageId);
  queryKernel.finishStage(stageId, false);
}

Prevention

Register stages as effectively finished before strict finishing
Use strict=false unless the caller is certain of the stage's completion state
Keep markSuccessfulTerminalStagesAsFinished ordering consistent with kernel state
Persist effectively-finished transitions so restart replay stays in sync

When it happens

Trigger: Calling finishStage(stageId, true) for a stage whose completion was never registered via markEffectivelyFinishedToKernel (the effectivelyFinishedStages set does not contain the stage). Typically from markSuccessfulTerminalStagesAsFinished when the controller believes a terminal stage succeeded but the kernel never recorded it.

Common situations: Controller/state-machine logic marking terminal stages finished out of order; stages skipped due to earlier failures so they never become effectively finished; desync between the controller's ChatHandler/state machine and the kernel after a restart.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/53397ed08567f38a. Report an issue: GitHub.