{"record":{"id":"cbf47330c5365591","repo":"apache/druid","slug":"encountered-error-while-reading-the-output-of-stag","errorCode":null,"errorMessage":"Encountered error while reading the output of stage [%d], partition [%d] for worker [%d]","messagePattern":"Encountered error while reading the output of stage \\[(.+?)\\], partition \\[(.+?)\\] for worker \\[(.+?)\\]","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/shuffle/input/DurableStorageInputChannelFactory.java","lineNumber":146,"sourceCode":"            \"Could not find remote outputs of stage [%d] partition [%d] for worker [%d] at the path [%s]\",\n            stageId.getStageNumber(),\n            partitionNumber,\n            workerNumber,\n            remotePartitionPath\n        );\n      }\n      final InputStream inputStream = storageConnector.read(remotePartitionPath);\n\n      return ReadableInputStreamFrameChannel.open(\n          inputStream,\n          remotePartitionPath,\n          remoteInputStreamPool,\n          false,\n          wireTransferableContext\n      );\n    }\n    catch (Exception e) {\n      throw new IOE(\n          e,\n          \"Encountered error while reading the output of stage [%d], partition [%d] for worker [%d]\",\n          stageId.getStageNumber(),\n          partitionNumber,\n          workerNumber\n      );\n    }\n  }\n\n  /**\n   * Given an input worker number, stage number and the partition number, this method figures out the exact location\n   * where the outputs would be present in the durable storage and returns the complete path or throws an exception\n   * if no such file exists in the durable storage\n   * More information at {@link DurableStorageOutputChannelFactory#createSuccessFile(String)}\n   */\n  public String findSuccessfulPartitionOutput(\n      final String controllerTaskId,\n      final int workerNo,","sourceCodeStart":128,"sourceCodeEnd":164,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/shuffle/input/DurableStorageInputChannelFactory.java#L128-L164","documentation":"This IOE wraps any exception raised while opening/reading a remote partition output channel from durable storage. It is the generic failure point for downloading or streaming a stage's output frames when DurableStorage is enabled; the cause carries the underlying error (I/O, network, deserialization, missing file).","triggerScenarios":"openChannel calls FrameFileChannelFactory/input stream creation over the storageConnector and any exception (IOException, client errors from S3/Azure/GCS SDKs, corrupt remote file, missing path) is caught and rethrown as IOE with stage/partition/worker context.","commonSituations":"Network or credential problems talking to deep storage; partially written or truncated partition output from a failed producer; deep-storage client timeouts; reading a file deleted between the pathExists check and read.","solutions":["Inspect the cause exception in the stack trace to find the true failure (auth, timeout, missing object, corruption).","Verify deep-storage credentials/connectivity from the worker host (e.g. aws s3 cp of the same path).","Check the producing task's logs for a crash that left a truncated output file; rerun the query.","Increase deep-storage client timeout/retry settings if errors are transient network issues."],"exampleFix":"null","handlingStrategy":"retry","validationCode":"// verify deep storage connectivity before issuing queries\nstorageConnector.pathExists(\"/health-check-probe-key\");","typeGuard":null,"tryCatchPattern":"try {\n  readRemoteOutput();\n} catch (IOException e) {\n  LOG.warn(\"remote output read failed\", e.getCause());\n  if (isTransient(e.getCause())) { retryWithBackoff(); } else { throw e; }\n}","preventionTips":["Rotate cloud storage credentials before expiry and test read/write from worker hosts.","Configure generous timeouts/retries on the deep-storage client.","Alert on object-store throttling (429/503) during heavy MSQ usage."],"tags":["deep-storage","msq","io-error","wrapped-exception"],"backgroundTag":"file-read-failed","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}