apache/druid · error · IOException (IOE)

Failed to stream logs for task[%s] starting at offset[%d]

Error message

Failed to stream logs for task[%s] starting at offset[%d]

What it means

S3TaskLogs.streamTaskFileWithRetry wraps S3Utils.retryS3Operation around the actual S3 GET of a task log/report/status/payload. If all retries fail, the underlying exception is rethrown as an IOException with this message, keyed by task ID and byte offset.

Source

Thrown at extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3TaskLogs.java:112

  }

  @Override
  public Optional<InputStream> streamTaskPayload(String taskid) throws IOException
  {
    final String taskKey = getTaskLogKey(taskid, "task.json");
    return streamTaskFileWithRetry(0, taskKey);
  }

  /**
   * Using the retry conditions defined in {@link S3Utils#S3RETRY}.
   */
  private Optional<InputStream> streamTaskFileWithRetry(final long offset, String taskKey) throws IOException
  {
    try {
      return S3Utils.retryS3Operation(() -> streamTaskFile(offset, taskKey));
    }
    catch (Exception e) {
      throw new IOE(e, "Failed to stream logs for task[%s] starting at offset[%d]", taskKey, offset);
    }
  }

  private Optional<InputStream> streamTaskFile(final long offset, String taskKey)
  {
    try {
      final HeadObjectResponse objectMetadata = service.get().getObjectMetadata(config.getS3Bucket(), taskKey);

      final long start;
      final long end = objectMetadata.contentLength() - 1;

      long contentLength = objectMetadata.contentLength();
      if (offset >= contentLength || offset <= -contentLength) {
        start = 0;
      } else if (offset >= 0) {
        start = offset;
      } else {
        start = contentLength + offset;

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Verify the task's log object exists in the configured task-log bucket/prefix.
2. Check IAM permissions for s3:GetObject on the task-log prefix.
3. Confirm druid.storage.taskLog(S3) bucket/keyPrefix/region configuration.
4. Check whether a log-cleanup policy deleted logs for older tasks.
5. Inspect the wrapped cause for the definitive AWS error code.

Example fix

// before
Optional<InputStream> s = taskLogs.streamTaskLog(taskId, 0);
// after
Optional<InputStream> s = taskLogs.streamTaskLog(taskId, 0);
if (s.isEmpty()) {
  log.warn("No logs retained for task %s (possibly expired)", taskId);
}

if (!S3Utils.isObjectInBucket(s3Client, taskLogBucket, taskLogKey)) return Optional.empty();

try { logs = taskLogs.streamTaskLog(taskId, offset); } catch (IOException e) { log.warn("task logs unavailable for %s: %s", taskId, e.getMessage()); }

When it happens

Trigger: Streaming task logs (or reports/status/payload) from S3 when the GetObject persistently fails after retries: object missing, access denied, throttling that exhausted retries, network outage.

Common situations: Task logs already deleted by log retention/cleanup before someone opens the console; wrong druid.storage.taskLog bucket config; S3 throttling during large log streaming; region mismatch.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• Failed to stream logs from: %s
• Failed to get object summaries from S3 bucket[%s], prefix[%s
• Could not fetch last modified timestamp from URI [%s]
• Cannot list directory [%s]
• Failed to read log for task: %s

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/818370dcb2d573b2. Report an issue: GitHub.