{"record":{"id":"2cc2ef8772300f82","repo":"apache/druid","slug":"supervisor-s-must-be-in-a-running-state-to-perfo","errorCode":null,"errorMessage":"Supervisor[%s] must be in a RUNNING state to perform a reset and backfill","messagePattern":"Supervisor\\[(.+?)\\] must be in a RUNNING state to perform a reset and backfill","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"indexing-service/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorManager.java","lineNumber":522,"sourceCode":"\n  private void validateResetAndBackfill(\n      String id,\n      SeekableStreamSupervisor streamSupervisor,\n      SeekableStreamSupervisorSpec streamSpec\n  )\n  {\n    if (streamSupervisor.getIoConfig().isUseEarliestSequenceNumber()) {\n      throw new IAE(\"Reset with skipped offsets is not supported when useEarliestOffset is true.\");\n    }\n\n    if (!specHasConcurrentLocks(streamSpec)) {\n      throw new IAE(\n          \"Backfill tasks require 'useConcurrentLocks' to be set to true in the supervisor context to allow concurrent writes with the main supervisor tasks\"\n      );\n    }\n\n    if (streamSupervisor.getState() != SupervisorStateManager.BasicState.RUNNING) {\n      throw new IAE(\"Supervisor[%s] must be in a RUNNING state to perform a reset and backfill\", id);\n    }\n  }\n\n  public boolean checkPointDataSourceMetadata(\n      String supervisorId,\n      int taskGroupId,\n      DataSourceMetadata previousDataSourceMetadata\n  )\n  {\n    try {\n      Preconditions.checkState(started, \"SupervisorManager not started\");\n      Preconditions.checkNotNull(supervisorId, \"supervisorId cannot be null\");\n\n      Pair<Supervisor, SupervisorSpec> supervisor = supervisors.get(supervisorId);\n\n      Preconditions.checkNotNull(supervisor, \"supervisor could not be found\");\n\n      final StreamSupervisor streamSupervisor = requireStreamSupervisor(supervisorId, \"checkPoint\");","sourceCodeStart":504,"sourceCodeEnd":540,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/indexing-service/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorManager.java#L504-L540","documentation":"SupervisorManager only allows reset-and-backfill while the supervisor is in the RUNNING state; any other lifecycle state (SUSPENDED, STOPPING, UNABLE_TO_CONNECT, etc.) causes this IllegalArgumentException naming the supervisor id. The guard prevents resetting offsets for a supervisor not actively managing tasks. No state changes occur when it fires.","triggerScenarios":"resetToLatestAndBackfill called when streamSupervisor.getState() != BasicState.RUNNING, e.g. the supervisor is suspended via POST /supervisor/<id>/suspend, is still starting up, or is in a failed/unhealthy state.","commonSituations":"Operator suspended the supervisor for maintenance and forgot it before requesting backfill; supervisor stuck in UNABLE_TO_CONNECT due to Kafka/Kinesis connectivity so it never reaches RUNNING; request raced with supervisor startup right after submit.","solutions":["Resume a suspended supervisor with POST /druid/indexer/v1/supervisor/<id>/resume and wait for state RUNNING (check GET /supervisor/<id>/status) before retrying","If the supervisor is unhealthy, fix the underlying cause (broker connectivity, credentials, task failures) so it reaches RUNNING","If the supervisor was stopped/terminated intentionally, re-create and start it, or use a batch ingestion task for the backfill instead"],"exampleFix":"// before\nPOST /druid/indexer/v1/supervisor/kafka-wiki/reset  // supervisor suspended\n// after\nPOST /druid/indexer/v1/supervisor/kafka-wiki/resume\ncurl .../supervisor/kafka-wiki/status | jq '.[0].state'  # wait for RUNNING\nPOST /druid/indexer/v1/supervisor/kafka-wiki/reset","handlingStrategy":"validation","validationCode":"const st = await (await fetch(`${overlord}/druid/indexer/v1/supervisor/${id}/status`)).json();\nif (st[0]?.state !== 'RUNNING') throw new Error(`supervisor ${id} not RUNNING (state=${st[0]?.state})`);","typeGuard":"function isRunning(status) { return status?.[0]?.state === 'RUNNING'; }","tryCatchPattern":"try { await resetBackfill(id); } catch (e) { if (/must be in a RUNNING state/.test(e.message)) { await resumeAndWaitForRunning(id); await resetBackfill(id); } else throw e; }","preventionTips":["Poll supervisor status until RUNNING before issuing reset/backfill","Automate resume-after-maintenance so supervisors aren't left suspended","Alert on supervisors stuck in UNABLE_TO_CONNECT or other non-RUNNING states"],"tags":["supervisor","state-machine","druid-overlord","kafka"],"backgroundTag":"invalid-state-transition","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}