{"record":{"id":"818370dcb2d573b2","repo":"apache/druid","slug":"failed-to-stream-logs-for-task-s-starting-at-off","errorCode":null,"errorMessage":"Failed to stream logs for task[%s] starting at offset[%d]","messagePattern":"Failed to stream logs for task\\[(.+?)\\] starting at offset\\[(.+?)\\]","errorType":"exception","errorClass":"IOException (IOE)","httpStatus":null,"severity":"error","filePath":"extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3TaskLogs.java","lineNumber":112,"sourceCode":"  }\n\n  @Override\n  public Optional<InputStream> streamTaskPayload(String taskid) throws IOException\n  {\n    final String taskKey = getTaskLogKey(taskid, \"task.json\");\n    return streamTaskFileWithRetry(0, taskKey);\n  }\n\n  /**\n   * Using the retry conditions defined in {@link S3Utils#S3RETRY}.\n   */\n  private Optional<InputStream> streamTaskFileWithRetry(final long offset, String taskKey) throws IOException\n  {\n    try {\n      return S3Utils.retryS3Operation(() -> streamTaskFile(offset, taskKey));\n    }\n    catch (Exception e) {\n      throw new IOE(e, \"Failed to stream logs for task[%s] starting at offset[%d]\", taskKey, offset);\n    }\n  }\n\n  private Optional<InputStream> streamTaskFile(final long offset, String taskKey)\n  {\n    try {\n      final HeadObjectResponse objectMetadata = service.get().getObjectMetadata(config.getS3Bucket(), taskKey);\n\n      final long start;\n      final long end = objectMetadata.contentLength() - 1;\n\n      long contentLength = objectMetadata.contentLength();\n      if (offset >= contentLength || offset <= -contentLength) {\n        start = 0;\n      } else if (offset >= 0) {\n        start = offset;\n      } else {\n        start = contentLength + offset;","sourceCodeStart":94,"sourceCodeEnd":130,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3TaskLogs.java#L94-L130","documentation":"S3TaskLogs.streamTaskFileWithRetry wraps S3Utils.retryS3Operation around the actual S3 GET of a task log/report/status/payload. If all retries fail, the underlying exception is rethrown as an IOException with this message, keyed by task ID and byte offset.","triggerScenarios":"Streaming task logs (or reports/status/payload) from S3 when the GetObject persistently fails after retries: object missing, access denied, throttling that exhausted retries, network outage.","commonSituations":"Task logs already deleted by log retention/cleanup before someone opens the console; wrong druid.storage.taskLog bucket config; S3 throttling during large log streaming; region mismatch.","solutions":["Verify the task's log object exists in the configured task-log bucket/prefix.","Check IAM permissions for s3:GetObject on the task-log prefix.","Confirm druid.storage.taskLog(S3) bucket/keyPrefix/region configuration.","Check whether a log-cleanup policy deleted logs for older tasks.","Inspect the wrapped cause for the definitive AWS error code."],"exampleFix":"// before\nOptional<InputStream> s = taskLogs.streamTaskLog(taskId, 0);\n// after\nOptional<InputStream> s = taskLogs.streamTaskLog(taskId, 0);\nif (s.isEmpty()) {\n  log.warn(\"No logs retained for task %s (possibly expired)\", taskId);\n}","handlingStrategy":"try-catch","validationCode":"if (!S3Utils.isObjectInBucket(s3Client, taskLogBucket, taskLogKey)) return Optional.empty();","typeGuard":null,"tryCatchPattern":"try { logs = taskLogs.streamTaskLog(taskId, offset); } catch (IOException e) { log.warn(\"task logs unavailable for %s: %s\", taskId, e.getMessage()); }","preventionTips":["Keep log retention longer than typical debugging windows","Verify task-log bucket/prefix config","Check IAM GetObject on task-log prefix","Watch for S3 throttling with many concurrent log viewers"],"tags":["s3","task-logs","io","retry-exhausted"],"backgroundTag":"http-request-failed","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}