apache/druid · error · DruidRuntimeException

Failed to get object summaries from S3 bucket[%s], prefix[%s

Error message

Failed to get object summaries from S3 bucket[%s], prefix[%s]

What it means

ObjectRefresh: thrown by ObjectSummaryIterator when listing object summaries from S3 fails for reasons other than the retried AmazonServiceException (e.g. unexpected S3 responses or generic exceptions). It wraps the cause and reports the bucket and prefix being listed.

Source

Thrown at extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/ObjectSummaryIterator.java:142

          .maxKeys(maxListingLength)
          .continuationToken(continuationToken)
          .build();

      result = S3Utils.retryS3Operation(() -> s3Client.listObjectsV2(request), maxRetries);
      continuationToken = result.nextContinuationToken();
      objectSummaryIterator = result.contents().iterator();
    }
    catch (S3Exception e) {
      throw new RE(
          e,
          "Failed to get object summaries from S3 bucket[%s], prefix[%s]; S3 error: %s",
          currentBucket,
          currentPrefix,
          e.getMessage()
      );
    }
    catch (Exception e) {
      throw new RE(
          e,
          "Failed to get object summaries from S3 bucket[%s], prefix[%s]",
          currentBucket,
          currentPrefix
      );
    }
  }

  /**
   * Advance objectSummaryIterator to the next non-placeholder, updating "currentObjectSummary".
   */
  private void advanceObjectSummary()
  {
    while (objectSummaryIterator.hasNext() || result.isTruncated() || prefixesIterator.hasNext()) {
      while (objectSummaryIterator.hasNext()) {
        final S3Object candidateObject = objectSummaryIterator.next();
        // skips directories and empty objects
        if (!isDirectoryPlaceholder(candidateObject) && candidateObject.size() > 0) {

View on GitHub (pinned to 9b90983fd2)

Solutions

Check task logs for the wrapped cause (e.getMessage of the inner exception) to find the root S3 error.
Verify the bucket exists and is in the region the S3 client is configured for.
Confirm IAM permissions (s3:ListBucket, s3:GetObject) for the credentials used.
Retry the failed task; transient listing failures usually clear.
Check proxy/network connectivity between the Druid node and the S3 endpoint.

Example fix

// before: generic client config
// after: pin the correct region for the bucket
S3ClientConfig cfg = ...; // ensure druid.s3.client config sets region, e.g. properties.set(Core.AWS_REGION, "us-east-1")

Defensive patterns

Strategy: retry

Validate before calling

// check client reachability & perms before listing
HeadBucketRequest hbr = HeadBucketRequest.builder().bucket(bucket).build();
s3Client.headBucket(hbr); // throws if bucket inaccessible/wrong region

Try / catch

try { iterateSummaries(); } catch (RE e) { log.error("Listing failed for %s/%s; retrying", e.getCause()); scheduleRetry(); }

Prevention

Pin the client region to the bucket's region
Grant s3:ListBucket and s3:GetObject up front
Monitor S3 5xx rates and set retries in SDK config

When it happens

Trigger: fetchNextBatch calls s3Client.listObjectsV2 (via S3Utils) for currentBucket/currentPrefix and the call throws a non-retryable/non-AWS-service exception, or an unexpected AWS SDK result occurs during constructorPostProcessing/advanceObjectSummary.

Common situations: S3 outages returning malformed responses, SDK deserialization failures, invalid region configuration causing client errors, network interruption mid-list, permission changes mid-iteration.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/7a86746309a28b1a. Report an issue: GitHub.