apache/druid · error · IOException (IOE)

Failed to stream logs for task

Error message

Failed to stream logs for task[%s] starting at offset[%d]

What it means

S3TaskLogs.streamTaskFileWithRetry wraps S3Utils.retryS3Operation around the actual S3 GET of a task log/report/status/payload. If all retries fail, the underlying exception is rethrown as an IOException with this message, keyed by task ID and byte offset.

Solutions

1. Verify the task's log object exists in the configured task-log bucket/prefix.
2. Check IAM permissions for s3:GetObject on the task-log prefix.
3. Confirm druid.storage.taskLog(S3) bucket/keyPrefix/region configuration.
4. Check whether a log-cleanup policy deleted logs for older tasks.
5. Inspect the wrapped cause for the definitive AWS error code.

Example fix

// before
Optional<InputStream> s = taskLogs.streamTaskLog(taskId, 0);
// after
Optional<InputStream> s = taskLogs.streamTaskLog(taskId, 0);
if (s.isEmpty()) {
  log.warn("No logs retained for task %s (possibly expired)", taskId);
}

if (!S3Utils.isObjectInBucket(s3Client, taskLogBucket, taskLogKey)) return Optional.empty();

try { logs = taskLogs.streamTaskLog(taskId, offset); } catch (IOException e) { log.warn("task logs unavailable for %s: %s", taskId, e.getMessage()); }

When it happens

Trigger: Streaming task logs (or reports/status/payload) from S3 when the GetObject persistently fails after retries: object missing, access denied, throttling that exhausted retries, network outage.

Common situations: Task logs already deleted by log retention/cleanup before someone opens the console; wrong druid.storage.taskLog bucket config; S3 throttling during large log streaming; region mismatch.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• Cannot list directory
• Could not fetch last modified timestamp from URI
• Failed to get object summaries from S3 bucket
• Failed to read log for task
• Failed to stream logs from

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/818370dcb2d573b2. Report an issue: GitHub.

Appendix: source

Thrown at extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3TaskLogs.java:112

  }

  @Override
  public Optional<InputStream> streamTaskPayload(String taskid) throws IOException
  {
    final String taskKey = getTaskLogKey(taskid, "task.json");
    return streamTaskFileWithRetry(0, taskKey);
  }

  /**
   * Using the retry conditions defined in {@link S3Utils#S3RETRY}.
   */
  private Optional<InputStream> streamTaskFileWithRetry(final long offset, String taskKey) throws IOException
  {
    try {
      return S3Utils.retryS3Operation(() -> streamTaskFile(offset, taskKey));
    }
    catch (Exception e) {
      throw new IOE(e, "Failed to stream logs for task[%s] starting at offset[%d]", taskKey, offset);
    }
  }

  private Optional<InputStream> streamTaskFile(final long offset, String taskKey)
  {
    try {
      final HeadObjectResponse objectMetadata = service.get().getObjectMetadata(config.getS3Bucket(), taskKey);

      final long start;
      final long end = objectMetadata.contentLength() - 1;

      long contentLength = objectMetadata.contentLength();
      if (offset >= contentLength || offset <= -contentLength) {
        start = 0;
      } else if (offset >= 0) {
        start = offset;
      } else {
        start = contentLength + offset;

View on GitHub (pinned to 9b90983fd2)