apache/druid · error · IllegalStateException

Failed to get an s3 object for bucket[%s], key[%s], and star

Error message

Failed to get an s3 object for bucket[%s], key[%s], and start[%d]

What it means

This IllegalStateException is raised in S3Entity.readFrom when the AWS v2 SDK client's getObject call returns null despite being asked for a specific bucket, key, and byte range. The S3 v2 SDK contract normally returns a non-null ResponseInputStream or throws; a null here indicates an unexpected/unrecoverable client state (e.g. a closed or misbuilt client), so Druid fails fast with an ISE rather than returning a broken stream.

Source

Thrown at extensions-core/s3-extensions/src/main/java/org/apache/druid/data/input/s3/S3Entity.java:76

  }

  @Override
  public URI getUri()
  {
    return object.toUri(S3StorageDruidModule.SCHEME);
  }

  @Override
  protected InputStream readFrom(long offset) throws IOException
  {
    GetObjectRequest.Builder requestBuilder = GetObjectRequest.builder()
        .bucket(object.getBucket())
        .key(object.getPath())
        .range(AwsBytesRange.from(offset).getBytesRange());
    try {
      final ResponseInputStream<GetObjectResponse> s3Object = s3Client.getObject(requestBuilder);
      if (s3Object == null) {
        throw new ISE(
            "Failed to get an s3 object for bucket[%s], key[%s], and start[%d]",
            object.getBucket(),
            object.getPath(),
            offset
        );
      }
      return s3Object;
    }
    catch (S3Exception e) {
      throw new IOException(e);
    }
  }

  @Override
  protected String getPath()
  {
    return object.getPath();
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

Fix the S3Client supplier so it never returns or yields a null client / null response; stub getObject to return a real ResponseInputStream in tests.
If running tests, add a Mockito stub: when(s3Client.getObject(any(GetObjectRequest.class))).thenReturn(mockStream).
Check that the S3 client is not closed/disposed before readFrom is called (task lifecycle issue).
Upgrade/align the software.amazon.awssdk:s3 version to a release where getObject throws S3Exception instead of returning null.

Example fix

// before
when(s3Client.getObject(any(GetObjectRequest.class))).thenReturn(null);
// after
when(s3Client.getObject(any(GetObjectRequest.class)))
    .thenReturn(new ResponseInputStream<>(GetObjectResponse.builder().build(),
        AbortableInputStream.create(new ByteArrayInputStream(data))));

Defensive patterns

Strategy: type-guard

Validate before calling

S3Client client = supplier.get();
if (client == null) {
  throw new IllegalStateException("S3Client supplier returned null client");
}

Type guard

static boolean isUsableResponse(ResponseInputStream<GetObjectResponse> resp) {
  return resp != null && resp.getResponse() != null;
}

Try / catch

try (ResponseInputStream<GetObjectResponse> in = s3Client.getObject(req)) {
  if (in == null) {
    throw new ISE("S3 client returned null for bucket[%s] key[%s]", bucket, key);
  }
  // consume stream
} catch (S3Exception | NoSuchElementException e) {
  // handle SDK failure; null should be impossible with a healthy client
}

Prevention

Never let custom S3ClientSupplier implementations return null; throw on client construction failure
In tests, always stub getObject to return a real ResponseInputStream (use Mockito RETURNS_DEEP_STUBS or explicit stubs)
Keep AWS SDK v2 versions consistent between Druid and any custom extensions
Ensure clients are not closed before entity read

When it happens

Trigger: Calling readFrom (input source split/reader open) where s3Client.getObject(requestBuilder) yields null for the given bucket/key/offset — typically with a null or improperly initialized S3Client instance returned by a mocked or custom S3ClientSupplier.

Common situations: Unit tests with Mockito mocks that return null for getObject instead of a stubbed stream; a custom S3 client supplier/factory returning null on client creation or after close; SDK version mismatches where an error path returns null instead of throwing.

Understand the failure class

Background: "invalid response format", "malformed payload", "missing data field": when an API returns 200 but the response shape is wrong — this error's family across 23 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/ce1ce54d218c3589. Report an issue: GitHub.