apache/druid · error · IOException

Encountered error while reading the output of stage [%d], pa

Error message

Encountered error while reading the output of stage [%d], partition [%d] for worker [%d]

What it means

This IOE wraps any exception raised while opening/reading a remote partition output channel from durable storage. It is the generic failure point for downloading or streaming a stage's output frames when DurableStorage is enabled; the cause carries the underlying error (I/O, network, deserialization, missing file).

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/shuffle/input/DurableStorageInputChannelFactory.java:146

            "Could not find remote outputs of stage [%d] partition [%d] for worker [%d] at the path [%s]",
            stageId.getStageNumber(),
            partitionNumber,
            workerNumber,
            remotePartitionPath
        );
      }
      final InputStream inputStream = storageConnector.read(remotePartitionPath);

      return ReadableInputStreamFrameChannel.open(
          inputStream,
          remotePartitionPath,
          remoteInputStreamPool,
          false,
          wireTransferableContext
      );
    }
    catch (Exception e) {
      throw new IOE(
          e,
          "Encountered error while reading the output of stage [%d], partition [%d] for worker [%d]",
          stageId.getStageNumber(),
          partitionNumber,
          workerNumber
      );
    }
  }

  /**
   * Given an input worker number, stage number and the partition number, this method figures out the exact location
   * where the outputs would be present in the durable storage and returns the complete path or throws an exception
   * if no such file exists in the durable storage
   * More information at {@link DurableStorageOutputChannelFactory#createSuccessFile(String)}
   */
  public String findSuccessfulPartitionOutput(
      final String controllerTaskId,
      final int workerNo,

View on GitHub (pinned to 9b90983fd2)

Solutions

Inspect the cause exception in the stack trace to find the true failure (auth, timeout, missing object, corruption).
Verify deep-storage credentials/connectivity from the worker host (e.g. aws s3 cp of the same path).
Check the producing task's logs for a crash that left a truncated output file; rerun the query.
Increase deep-storage client timeout/retry settings if errors are transient network issues.

Example fix

null

Defensive patterns

Strategy: retry

Validate before calling

// verify deep storage connectivity before issuing queries
storageConnector.pathExists("/health-check-probe-key");

Try / catch

try {
  readRemoteOutput();
} catch (IOException e) {
  LOG.warn("remote output read failed", e.getCause());
  if (isTransient(e.getCause())) { retryWithBackoff(); } else { throw e; }
}

Prevention

Rotate cloud storage credentials before expiry and test read/write from worker hosts.
Configure generous timeouts/retries on the deep-storage client.
Alert on object-store throttling (429/503) during heavy MSQ usage.

When it happens

Trigger: openChannel calls FrameFileChannelFactory/input stream creation over the storageConnector and any exception (IOException, client errors from S3/Azure/GCS SDKs, corrupt remote file, missing path) is caught and rethrown as IOE with stage/partition/worker context.

Common situations: Network or credential problems talking to deep storage; partially written or truncated partition output from a failed producer; deep-storage client timeouts; reading a file deleted between the pathExists check and read.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/cbf47330c5365591. Report an issue: GitHub.