apache/druid · error · DruidRuntimeException

Failed to get object summaries from S3 bucket[%s], prefix[%s

Error message

Failed to get object summaries from S3 bucket[%s], prefix[%s]; S3 error: %s

What it means

ObjectSummaryIterator.fetchNextBatch lists S3 objects page-by-page with listObjectsV2 wrapped in S3Utils.retryS3Operation. When the listing still fails after maxRetries with an S3Exception, Druid wraps it in a ReportedException (RE) that includes the bucket, prefix, and the S3 error message (e.g. AccessDenied, NoSuchBucket, throttling). This error surfaces during input-source split listing, so ingestion/queries against the S3 input source fail at the planning/listing stage.

Source

Thrown at extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/ObjectSummaryIterator.java:133

    continuationToken = null;
  }

  private void fetchNextBatch()
  {
    try {
      ListObjectsV2Request request = ListObjectsV2Request.builder()
          .bucket(currentBucket)
          .prefix(currentPrefix)
          .maxKeys(maxListingLength)
          .continuationToken(continuationToken)
          .build();

      result = S3Utils.retryS3Operation(() -> s3Client.listObjectsV2(request), maxRetries);
      continuationToken = result.nextContinuationToken();
      objectSummaryIterator = result.contents().iterator();
    }
    catch (S3Exception e) {
      throw new RE(
          e,
          "Failed to get object summaries from S3 bucket[%s], prefix[%s]; S3 error: %s",
          currentBucket,
          currentPrefix,
          e.getMessage()
      );
    }
    catch (Exception e) {
      throw new RE(
          e,
          "Failed to get object summaries from S3 bucket[%s], prefix[%s]",
          currentBucket,
          currentPrefix
      );
    }
  }

  /**

View on GitHub (pinned to 9b90983fd2)

Solutions

Grant the task's IAM credentials s3:ListBucket permission (with the prefix condition) for the target bucket.
Verify the bucket name, region, and endpoint configuration in the S3 input source / druid.s3.* properties.
If the error is SlowDown/throttling, reduce concurrent listing workers, narrow prefixes, or increase retry limits (druid.s3.maxRetries / client retry policy).
Refresh or fix credentials (expired STS tokens, wrong access/secret key) and re-run the task.

Example fix

// before (IAM policy)
{"Effect":"Deny","Action":"s3:ListBucket","Resource":"arn:aws:s3:::my-bucket"}
// after
{"Effect":"Allow","Action":"s3:ListBucket","Resource":"arn:aws:s3:::my-bucket",
 "Condition":{"StringLike":{"s3:prefix":["data/*"]}}}

Defensive patterns

Strategy: retry

Validate before calling

// preflight check before running the job
HeadBucketRequest head = HeadBucketRequest.builder().bucket(bucket).build();
s3Client.headBucket(head); // throws S3Exception 403/404 early if bucket is inaccessible/nonexistent

Try / catch

try {
  iterator.next(); // triggers listing
} catch (ReportedException e) {
  Throwable cause = e.getCause();
  if (cause instanceof S3Exception s3e) {
    int status = s3e.statusCode();
    if (status == 403 || status == 404) {
      throw new IllegalStateException("Check bucket name/IAM ListBucket permission", e);
    }
    if (status == 503) {
      // backoff and retry with fewer concurrent listings
    }
  }
  throw e;
}

Prevention

Grant s3:ListBucket (with prefix condition) to the ingestion task role
Verify bucket name and region before launching tasks
Back off on 503 SlowDown and reduce parallel prefix listings
Monitor for expired STS credentials in long-running tasks
Tune druid.s3 retry settings to tolerate transient S3 errors

When it happens

Trigger: Calling the constructor or advancing the iterator (constructorPostProcessing/advanceObjectSummary -> fetchNextBatch) where s3Client.listObjectsV2 throws S3Exception on every attempt up to maxRetries — e.g. 403 AccessDenied, 404 NoSuchBucket, 503 SlowDown throttling, or network errors exceeding the retry budget.

Common situations: IAM credentials lacking s3:ListBucket on the bucket/prefix; typo in bucket name or region; S3 request-rate throttling on buckets with many prefixes; expired session credentials (STS); VPC endpoint or proxy blocking S3 traffic.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/48945d64d71274a0. Report an issue: GitHub.