apache/druid · error · IllegalStateException

Could not find remote outputs of stage [%d] partition [%d] f

Error message

Could not find remote outputs of stage [%d] partition [%d] for worker [%d] at the path [%s]

What it means

DurableStorageInputChannelFactory.openChannel throws this ISE when the remote deep-storage partition output path for a given stage/partition/worker does not exist (storageConnector.pathExists returned false). It means the consuming worker could not locate the outputs a producer worker was supposed to have written to durable storage. This usually indicates the producer task failed, its outputs were cleaned up (cleanup intervals / retention), or the deep-storage path configuration is inconsistent between tasks.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/shuffle/input/DurableStorageInputChannelFactory.java:127

  public ReadableFrameChannel openChannel(StageId stageId, int workerNumber, int partitionNumber) throws IOException
  {

    try {
      final String remotePartitionPath = findSuccessfulPartitionOutput(
          controllerTaskId,
          workerNumber,
          stageId.getStageNumber(),
          partitionNumber
      );
      LOG.debug(
          "Reading output of stage [%d], partition [%d] for worker [%d] from the file at path [%s]",
          stageId.getStageNumber(),
          partitionNumber,
          workerNumber,
          remotePartitionPath
      );
      if (!storageConnector.pathExists(remotePartitionPath)) {
        throw new ISE(
            "Could not find remote outputs of stage [%d] partition [%d] for worker [%d] at the path [%s]",
            stageId.getStageNumber(),
            partitionNumber,
            workerNumber,
            remotePartitionPath
        );
      }
      final InputStream inputStream = storageConnector.read(remotePartitionPath);

      return ReadableInputStreamFrameChannel.open(
          inputStream,
          remotePartitionPath,
          remoteInputStreamPool,
          false,
          wireTransferableContext
      );
    }
    catch (Exception e) {

View on GitHub (pinned to 9b90983fd2)

Solutions

Verify the query's intermediate outputs still exist in deep storage at the reported path (check with the storage console/CLI).
Check logs of the producing worker task for failure or cleanup; retry the query if outputs were lost by a transient failure.
Ensure all Druid services use the same durable storage configuration (storageConnector type, bucket, base key prefix) and that no lifecycle/expiry rule deletes the prefix during query execution.
If deep storage is eventually consistent (e.g. S3), ensure read-after-write consistency settings and retry logic are in place.

Example fix

// before: blindly opening the channel
Channel channel = durableStorageInputChannelFactory.openChannel(stageId, partitionNumber, workerNumber);

// after: check path and controller state, retry transiently
if (!storageConnector.pathExists(remotePartitionPath)) {
  LOG.warn("remote output %s missing, retrying...");
  // retry with backoff, then fail with context about the worker task
  throw new QueryInterruptedException(new ResourceLimitException(...));
}

Defensive patterns

Strategy: retry

Validate before calling

// Java (client-side pseudo-check before parsing results)
if (!storageConnector.pathExists(remotePartitionPath)) {
  throw new IllegalStateException("Deep storage output missing before query start: " + remotePartitionPath);
}

Try / catch

// catch ISE and retry the query with backoff, capping attempts
try {
  runMsqQuery(query);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("Could not find remote outputs")) { retryWithBackoff(query, 3); }
  else { throw e; }
}

Prevention

Keep deep-storage lifecycle/expiry rules from touching Druid's durable-storage prefix during queries.
Align durable storage location/prefix config across broker, overlord, and task runtimes.
Monitor worker task failure rates; fix OOM/kill issues before they orphan stage outputs.

When it happens

Trigger: Calling openChannel (e.g. via getResultYielder) when DurableStorage is enabled and remotePartitionPath for the (stageId, partitionNumber, workerNumber) triple is absent in the configured deep storage; a retry reads a path whose producer output was already deleted; a misconfigured storageConnector pointing at the wrong bucket/prefix so the path check fails.

Common situations: Task retries after a controller/worker failure where outputs were cleaned; deep storage retention/cleanup job (e.g. S3 lifecycle rule) deleting intermediate outputs mid-query; mismatched druid.storage durableStorage location config between broker/overlord and tasks; transient deep-storage consistency lag (S3 read-after-write issues).

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/a971be590052935c. Report an issue: GitHub.