apache/druid · error · IOException

Encountered error while reading the output of stage

Error message

Encountered error while reading the output of stage [%d], partition [%d] for worker [%d]

What it means

This IOE wraps any exception raised while opening/reading a remote partition output channel from durable storage. It is the generic failure point for downloading or streaming a stage's output frames when DurableStorage is enabled; the cause carries the underlying error (I/O, network, deserialization, missing file).

Solutions

1. Inspect the cause exception in the stack trace to find the true failure (auth, timeout, missing object, corruption).
2. Verify deep-storage credentials/connectivity from the worker host (e.g. aws s3 cp of the same path).
3. Check the producing task's logs for a crash that left a truncated output file; rerun the query.
4. Increase deep-storage client timeout/retry settings if errors are transient network issues.

Example fix

null

// verify deep storage connectivity before issuing queries
storageConnector.pathExists("/health-check-probe-key");

try {
  readRemoteOutput();
} catch (IOException e) {
  LOG.warn("remote output read failed", e.getCause());
  if (isTransient(e.getCause())) { retryWithBackoff(); } else { throw e; }
}

When it happens

Trigger: openChannel calls FrameFileChannelFactory/input stream creation over the storageConnector and any exception (IOException, client errors from S3/Azure/GCS SDKs, corrupt remote file, missing path) is caught and rethrown as IOE with stage/partition/worker context.

Common situations: Network or credential problems talking to deep storage; partially written or truncated partition output from a failed producer; deep-storage client timeouts; reading a file deleted between the pathExists check and read.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Unable to read file
• Could not find remote outputs of stage
• File does not exist
• No file present at the location
• Unable to read the task id from the file

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/cbf47330c5365591. Report an issue: GitHub.

Appendix: source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/shuffle/input/DurableStorageInputChannelFactory.java:146

            "Could not find remote outputs of stage [%d] partition [%d] for worker [%d] at the path [%s]",
            stageId.getStageNumber(),
            partitionNumber,
            workerNumber,
            remotePartitionPath
        );
      }
      final InputStream inputStream = storageConnector.read(remotePartitionPath);

      return ReadableInputStreamFrameChannel.open(
          inputStream,
          remotePartitionPath,
          remoteInputStreamPool,
          false,
          wireTransferableContext
      );
    }
    catch (Exception e) {
      throw new IOE(
          e,
          "Encountered error while reading the output of stage [%d], partition [%d] for worker [%d]",
          stageId.getStageNumber(),
          partitionNumber,
          workerNumber
      );
    }
  }

  /**
   * Given an input worker number, stage number and the partition number, this method figures out the exact location
   * where the outputs would be present in the durable storage and returns the complete path or throws an exception
   * if no such file exists in the durable storage
   * More information at {@link DurableStorageOutputChannelFactory#createSuccessFile(String)}
   */
  public String findSuccessfulPartitionOutput(
      final String controllerTaskId,
      final int workerNo,

View on GitHub (pinned to 9b90983fd2)