apache/druid · error · ISE

Expected [%d] tasks to succeed, but we got [%d] succeeded ta

Error message

Expected [%d] tasks to succeed, but we got [%d] succeeded tasks and [%d] failed tasks

What it means

Thrown when all subtasks finished (none running, no pending completion events) but the number of succeeded subtasks does not equal the number of subtask specs that were submitted. Parallel index tasks require every spec's subtask to succeed; any failure or missing success makes the whole phase fail.

Source

Thrown at indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexPhaseRunner.java:167

          final TaskState completeState = taskCompleteEvent.getLastState();
          getSubtaskCompletionCallback(taskCompleteEvent).run();
          switch (completeState) {
            case SUCCESS:
              final TaskStatusPlus completeStatus = taskCompleteEvent.getLastStatus();
              if (completeStatus == null) {
                throw new ISE("Last status of complete task is missing!");
              }
              if (!subTaskSpecIterator.hasNext()) {
                // We have no more subTasks to run
                if (taskMonitor.getNumRunningTasks() == 0 && taskCompleteEvents.isEmpty()) {
                  subTaskScheduleAndMonitorStopped = true;
                  if (subTaskSpecIterator.count == taskMonitor.getNumSucceededTasks()) {
                    // Succeeded
                    state = TaskState.SUCCESS;
                  } else {
                    // Failed
                    final ParallelIndexingPhaseProgress monitorStatus = taskMonitor.getProgress();
                    throw new ISE(
                        "Expected [%d] tasks to succeed, but we got [%d] succeeded tasks and [%d] failed tasks",
                        subTaskSpecIterator.count,
                        monitorStatus.getSucceeded(),
                        monitorStatus.getFailed()
                    );
                  }
                }
              } else if (taskMonitor.getNumRunningTasks() < maxNumConcurrentSubTasks) {
                // We have more subTasks to run
                submitNewTask(taskMonitor, subTaskSpecIterator.next());
              } else {
                // We have more subTasks to run, but don't have enough available task slots
                // do nothing
              }
              break;
            case FAILED:
              // TaskMonitor already tried everything it can do for failed tasks. We failed.
              state = TaskState.FAILED;

View on GitHub (pinned to 9b90983fd2)

Solutions

Inspect the failed subtasks' logs via the overlord UI to find the root-cause failure and fix the input/tuning config.
Increase maxNumConcurrentSubTasks / maxRetry in ParallelIndexTuningConfig so transient failures are retried.
Fix the underlying data problem (bad rows, huge splits) — lower splitHintSpec targetPartitionSize or filter bad input files.
Re-run the ingestion task after fixing; successful partial work is discarded for this task run.

Example fix

// before
// tuningConfig: { "type": "parallel_index", "maxNumSubTasks": 4 }
// after: allow retries for transient subtask failures
// tuningConfig: { "type": "parallel_index", "maxNumSubTasks": 4, "maxRetry": 3 }

Defensive patterns

Strategy: retry

Try / catch

catch (ISE e) {
  if (e.getMessage().startsWith("Expected [")) {
    // inspect failed subtask specs via /subtaskspecs?state=FAILED, fix root cause, resubmit task
  }
}

Prevention

Set maxRetry > 0 in ParallelIndexTuningConfig for transient failures
Pre-validate input files/splits before submitting
Monitor subtask logs for OOM/oversize segment errors
Size splits with splitHintSpec so no subtask is overloaded

When it happens

Trigger: One or more subtasks of a ParallelIndexTask fail (e.g. bad input splits, oversized segments, worker losses) so numSucceededTasks < subTaskSpecIterator.count while the phase winds down.

Common situations: Subtasks hitting out-of-memory or row-too-large errors; workers killed during rebalancing; replace-mode tasks whose intervals were trimmed so some subtasks had no data; retryable subtask failures exhausting maxRetry.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/4822a15f8bf980e8. Report an issue: GitHub.