What causes "Cannot acquire closed HTTP client: ${clientKey}" in apache/iceberg?

Do not use the S3FileIO/catalog after closing it; create a new instance for subsequent operations.

apache/iceberg · error · IllegalStateException

Cannot acquire closed HTTP client: ${clientKey}

Error message

Cannot acquire closed HTTP client: ${clientKey}

What it means

HttpClientCache ref-counts shared AWS SDK HTTP clients. acquire() increments the reference count and hands out the wrapper, but if the underlying client has already been fully released (closed), it throws this IllegalStateException rather than handing out a reference to a closed resource.

Source

Thrown at aws/src/main/java/org/apache/iceberg/aws/HttpClientCache.java:123

    private final String clientKey;
    private volatile int refCount = 0;
    private boolean closed = false;

    ManagedHttpClient(SdkHttpClient httpClient, String clientKey) {
      this.httpClient = httpClient;
      this.clientKey = clientKey;
      LOG.debug("Created managed HTTP client: key={}", clientKey);
    }

    /**
     * Acquire a reference to the HTTP client, incrementing the reference count.
     *
     * @return the ref-counted wrapper client
     * @throws IllegalStateException if the client has already been closed
     */
    synchronized ManagedHttpClient acquire() {
      if (closed) {
        throw new IllegalStateException("Cannot acquire closed HTTP client: " + clientKey);
      }
      refCount++;
      LOG.debug("Acquired HTTP client: key={}, refCount={}", clientKey, refCount);
      return this;
    }

    /**
     * Release a reference to the HTTP client, decrementing the reference count. If the count
     * reaches zero, the client is closed.
     *
     * @return true if the client was closed, false otherwise
     */
    synchronized boolean release() {
      if (closed) {
        LOG.warn("Attempted to release already closed HTTP client: key={}", clientKey);
        return false;
      }

View on GitHub (pinned to 86d9c8fc54)

Solutions

Do not use the S3FileIO/catalog after closing it; create a new instance for subsequent operations.
Ensure every user of the shared client calls acquire before use and release when done, so refCount prevents premature close.
Fix thread-lifecycle races so close() happens only after all users are finished (e.g. CloseableGroup managed by a single owner).

Example fix

// before
fileIO.close();
Table table = catalog.loadTable("db.t"); // uses closed HTTP client
// after
Table table = catalog.loadTable("db.t");
fileIO.close(); // close only after all IO is done

Defensive patterns

Strategy: try-catch

Validate before calling

if (fileIO instanceof Closeable && alreadyClosed) {
  throw new IllegalStateException("S3FileIO was closed; create a new instance");
}

Type guard

static boolean usable(S3FileIO io) {
  try { io.properties(); return true; } catch (IllegalStateException e) { return false; }
}

Try / catch

try {
  table.io().newInputFile(location);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("Cannot acquire closed HTTP client")) {
    LOG.error("IO was closed before use; recreate it");
  }
  throw e;
}

Prevention

Treat S3FileIO/catalog as single-use: close only at end of lifetime, never mid-use.
Use try-with-resources and keep all reads/writes inside the try block.
Avoid sharing a closed-prone client across threads without a clear ownership owner calling close().

When it happens

Trigger: Calling acquire() on a cache entry after all previous holders released it and it was closed — e.g. using an S3FileIO/Catalog whose HTTP client was shut down via release/CloseableGroup close, then attempting further requests; test helpers client1/client2 and acquireAfterCloseThrows exercise exactly this.

Common situations: Reusing an S3FileIO instance after calling close(); sharing one cached client across threads where one thread closes it while another still needs it; keeping a table/catalog reference alive past its owner's shutdown.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/a8ba74ee8dab840d. Report an issue: GitHub.