{"record":{"id":"a971be590052935c","repo":"apache/druid","slug":"could-not-find-remote-outputs-of-stage-d-partit","errorCode":null,"errorMessage":"Could not find remote outputs of stage [%d] partition [%d] for worker [%d] at the path [%s]","messagePattern":"Could not find remote outputs of stage \\[(.+?)\\] partition \\[(.+?)\\] for worker \\[(.+?)\\] at the path \\[(.+?)\\]","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/shuffle/input/DurableStorageInputChannelFactory.java","lineNumber":127,"sourceCode":"  public ReadableFrameChannel openChannel(StageId stageId, int workerNumber, int partitionNumber) throws IOException\n  {\n\n    try {\n      final String remotePartitionPath = findSuccessfulPartitionOutput(\n          controllerTaskId,\n          workerNumber,\n          stageId.getStageNumber(),\n          partitionNumber\n      );\n      LOG.debug(\n          \"Reading output of stage [%d], partition [%d] for worker [%d] from the file at path [%s]\",\n          stageId.getStageNumber(),\n          partitionNumber,\n          workerNumber,\n          remotePartitionPath\n      );\n      if (!storageConnector.pathExists(remotePartitionPath)) {\n        throw new ISE(\n            \"Could not find remote outputs of stage [%d] partition [%d] for worker [%d] at the path [%s]\",\n            stageId.getStageNumber(),\n            partitionNumber,\n            workerNumber,\n            remotePartitionPath\n        );\n      }\n      final InputStream inputStream = storageConnector.read(remotePartitionPath);\n\n      return ReadableInputStreamFrameChannel.open(\n          inputStream,\n          remotePartitionPath,\n          remoteInputStreamPool,\n          false,\n          wireTransferableContext\n      );\n    }\n    catch (Exception e) {","sourceCodeStart":109,"sourceCodeEnd":145,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/shuffle/input/DurableStorageInputChannelFactory.java#L109-L145","documentation":"DurableStorageInputChannelFactory.openChannel throws this ISE when the remote deep-storage partition output path for a given stage/partition/worker does not exist (storageConnector.pathExists returned false). It means the consuming worker could not locate the outputs a producer worker was supposed to have written to durable storage. This usually indicates the producer task failed, its outputs were cleaned up (cleanup intervals / retention), or the deep-storage path configuration is inconsistent between tasks.","triggerScenarios":"Calling openChannel (e.g. via getResultYielder) when DurableStorage is enabled and remotePartitionPath for the (stageId, partitionNumber, workerNumber) triple is absent in the configured deep storage; a retry reads a path whose producer output was already deleted; a misconfigured storageConnector pointing at the wrong bucket/prefix so the path check fails.","commonSituations":"Task retries after a controller/worker failure where outputs were cleaned; deep storage retention/cleanup job (e.g. S3 lifecycle rule) deleting intermediate outputs mid-query; mismatched druid.storage durableStorage location config between broker/overlord and tasks; transient deep-storage consistency lag (S3 read-after-write issues).","solutions":["Verify the query's intermediate outputs still exist in deep storage at the reported path (check with the storage console/CLI).","Check logs of the producing worker task for failure or cleanup; retry the query if outputs were lost by a transient failure.","Ensure all Druid services use the same durable storage configuration (storageConnector type, bucket, base key prefix) and that no lifecycle/expiry rule deletes the prefix during query execution.","If deep storage is eventually consistent (e.g. S3), ensure read-after-write consistency settings and retry logic are in place."],"exampleFix":"// before: blindly opening the channel\nChannel channel = durableStorageInputChannelFactory.openChannel(stageId, partitionNumber, workerNumber);\n\n// after: check path and controller state, retry transiently\nif (!storageConnector.pathExists(remotePartitionPath)) {\n  LOG.warn(\"remote output %s missing, retrying...\");\n  // retry with backoff, then fail with context about the worker task\n  throw new QueryInterruptedException(new ResourceLimitException(...));\n}","handlingStrategy":"retry","validationCode":"// Java (client-side pseudo-check before parsing results)\nif (!storageConnector.pathExists(remotePartitionPath)) {\n  throw new IllegalStateException(\"Deep storage output missing before query start: \" + remotePartitionPath);\n}","typeGuard":null,"tryCatchPattern":"// catch ISE and retry the query with backoff, capping attempts\ntry {\n  runMsqQuery(query);\n} catch (IllegalStateException e) {\n  if (e.getMessage().contains(\"Could not find remote outputs\")) { retryWithBackoff(query, 3); }\n  else { throw e; }\n}","preventionTips":["Keep deep-storage lifecycle/expiry rules from touching Druid's durable-storage prefix during queries.","Align durable storage location/prefix config across broker, overlord, and task runtimes.","Monitor worker task failure rates; fix OOM/kill issues before they orphan stage outputs."],"tags":["deep-storage","msq","distributed-shuffle","missing-output"],"backgroundTag":"resource-not-found","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}