{"record":{"id":"7a86746309a28b1a","repo":"apache/druid","slug":"failed-to-get-object-summaries-from-s3-bucket-s-7a8674","errorCode":null,"errorMessage":"Failed to get object summaries from S3 bucket[%s], prefix[%s]","messagePattern":"Failed to get object summaries from S3 bucket\\[(.+?)\\], prefix\\[(.+?)\\]","errorType":"exception","errorClass":"DruidRuntimeException","httpStatus":null,"severity":"error","filePath":"extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/ObjectSummaryIterator.java","lineNumber":142,"sourceCode":"          .maxKeys(maxListingLength)\n          .continuationToken(continuationToken)\n          .build();\n\n      result = S3Utils.retryS3Operation(() -> s3Client.listObjectsV2(request), maxRetries);\n      continuationToken = result.nextContinuationToken();\n      objectSummaryIterator = result.contents().iterator();\n    }\n    catch (S3Exception e) {\n      throw new RE(\n          e,\n          \"Failed to get object summaries from S3 bucket[%s], prefix[%s]; S3 error: %s\",\n          currentBucket,\n          currentPrefix,\n          e.getMessage()\n      );\n    }\n    catch (Exception e) {\n      throw new RE(\n          e,\n          \"Failed to get object summaries from S3 bucket[%s], prefix[%s]\",\n          currentBucket,\n          currentPrefix\n      );\n    }\n  }\n\n  /**\n   * Advance objectSummaryIterator to the next non-placeholder, updating \"currentObjectSummary\".\n   */\n  private void advanceObjectSummary()\n  {\n    while (objectSummaryIterator.hasNext() || result.isTruncated() || prefixesIterator.hasNext()) {\n      while (objectSummaryIterator.hasNext()) {\n        final S3Object candidateObject = objectSummaryIterator.next();\n        // skips directories and empty objects\n        if (!isDirectoryPlaceholder(candidateObject) && candidateObject.size() > 0) {","sourceCodeStart":124,"sourceCodeEnd":160,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/ObjectSummaryIterator.java#L124-L160","documentation":"ObjectRefresh: thrown by ObjectSummaryIterator when listing object summaries from S3 fails for reasons other than the retried AmazonServiceException (e.g. unexpected S3 responses or generic exceptions). It wraps the cause and reports the bucket and prefix being listed.","triggerScenarios":"fetchNextBatch calls s3Client.listObjectsV2 (via S3Utils) for currentBucket/currentPrefix and the call throws a non-retryable/non-AWS-service exception, or an unexpected AWS SDK result occurs during constructorPostProcessing/advanceObjectSummary.","commonSituations":"S3 outages returning malformed responses, SDK deserialization failures, invalid region configuration causing client errors, network interruption mid-list, permission changes mid-iteration.","solutions":["Check task logs for the wrapped cause (e.getMessage of the inner exception) to find the root S3 error.","Verify the bucket exists and is in the region the S3 client is configured for.","Confirm IAM permissions (s3:ListBucket, s3:GetObject) for the credentials used.","Retry the failed task; transient listing failures usually clear.","Check proxy/network connectivity between the Druid node and the S3 endpoint."],"exampleFix":"// before: generic client config\n// after: pin the correct region for the bucket\nS3ClientConfig cfg = ...; // ensure druid.s3.client config sets region, e.g. properties.set(Core.AWS_REGION, \"us-east-1\")","handlingStrategy":"retry","validationCode":"// check client reachability & perms before listing\nHeadBucketRequest hbr = HeadBucketRequest.builder().bucket(bucket).build();\ns3Client.headBucket(hbr); // throws if bucket inaccessible/wrong region","typeGuard":null,"tryCatchPattern":"try { iterateSummaries(); } catch (RE e) { log.error(\"Listing failed for %s/%s; retrying\", e.getCause()); scheduleRetry(); }","preventionTips":["Pin the client region to the bucket's region","Grant s3:ListBucket and s3:GetObject up front","Monitor S3 5xx rates and set retries in SDK config"],"tags":["aws","s3","storage","listing"],"backgroundTag":"api-error-response","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}