apache/druid · error · IllegalArgumentException

Supervisor[ ] must be in a RUNNING state to perform a reset…

Error message

Supervisor[%s] must be in a RUNNING state to perform a reset and backfill

What it means

SupervisorManager only allows reset-and-backfill while the supervisor is in the RUNNING state; any other lifecycle state (SUSPENDED, STOPPING, UNABLE_TO_CONNECT, etc.) causes this IllegalArgumentException naming the supervisor id. The guard prevents resetting offsets for a supervisor not actively managing tasks. No state changes occur when it fires.

Solutions

Resume a suspended supervisor with POST /druid/indexer/v1/supervisor/<id>/resume and wait for state RUNNING (check GET /supervisor/<id>/status) before retrying
If the supervisor is unhealthy, fix the underlying cause (broker connectivity, credentials, task failures) so it reaches RUNNING
If the supervisor was stopped/terminated intentionally, re-create and start it, or use a batch ingestion task for the backfill instead

Example fix

// before
POST /druid/indexer/v1/supervisor/kafka-wiki/reset  // supervisor suspended
// after
POST /druid/indexer/v1/supervisor/kafka-wiki/resume
curl .../supervisor/kafka-wiki/status | jq '.[0].state'  # wait for RUNNING
POST /druid/indexer/v1/supervisor/kafka-wiki/reset

Defensive patterns

Strategy: validation

Validate before calling

const st = await (await fetch(`${overlord}/druid/indexer/v1/supervisor/${id}/status`)).json();
if (st[0]?.state !== 'RUNNING') throw new Error(`supervisor ${id} not RUNNING (state=${st[0]?.state})`);

Type guard

function isRunning(status) { return status?.[0]?.state === 'RUNNING'; }

Try / catch

try { await resetBackfill(id); } catch (e) { if (/must be in a RUNNING state/.test(e.message)) { await resumeAndWaitForRunning(id); await resetBackfill(id); } else throw e; }

Prevention

Poll supervisor status until RUNNING before issuing reset/backfill
Automate resume-after-maintenance so supervisors aren't left suspended
Alert on supervisors stuck in UNABLE_TO_CONNECT or other non-RUNNING states

When it happens

Trigger: resetToLatestAndBackfill called when streamSupervisor.getState() != BasicState.RUNNING, e.g. the supervisor is suspended via POST /supervisor/<id>/suspend, is still starting up, or is in a failed/unhealthy state.

Common situations: Operator suspended the supervisor for maintenance and forgot it before requesting backfill; supervisor stuck in UNABLE_TO_CONNECT due to Kafka/Kinesis connectivity so it never reaches RUNNING; request raced with supervisor startup right after submit.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/2cc2ef8772300f82. Report an issue: GitHub.

Appendix: source

Thrown at indexing-service/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorManager.java:522

  private void validateResetAndBackfill(
      String id,
      SeekableStreamSupervisor streamSupervisor,
      SeekableStreamSupervisorSpec streamSpec
  )
  {
    if (streamSupervisor.getIoConfig().isUseEarliestSequenceNumber()) {
      throw new IAE("Reset with skipped offsets is not supported when useEarliestOffset is true.");
    }

    if (!specHasConcurrentLocks(streamSpec)) {
      throw new IAE(
          "Backfill tasks require 'useConcurrentLocks' to be set to true in the supervisor context to allow concurrent writes with the main supervisor tasks"
      );
    }

    if (streamSupervisor.getState() != SupervisorStateManager.BasicState.RUNNING) {
      throw new IAE("Supervisor[%s] must be in a RUNNING state to perform a reset and backfill", id);
    }
  }

  public boolean checkPointDataSourceMetadata(
      String supervisorId,
      int taskGroupId,
      DataSourceMetadata previousDataSourceMetadata
  )
  {
    try {
      Preconditions.checkState(started, "SupervisorManager not started");
      Preconditions.checkNotNull(supervisorId, "supervisorId cannot be null");

      Pair<Supervisor, SupervisorSpec> supervisor = supervisors.get(supervisorId);

      Preconditions.checkNotNull(supervisor, "supervisor could not be found");

      final StreamSupervisor streamSupervisor = requireStreamSupervisor(supervisorId, "checkPoint");

View on GitHub (pinned to 9b90983fd2)