apache/iceberg · warning

Timed out waiting for eviction executor to terminate

Error message

Timed out waiting for eviction executor to terminate

What it means

AuthSessionCache closes its scheduled eviction executor on close() and waits up to 10 seconds for it to terminate. If pending eviction tasks still occupy the executor's single thread after 10 seconds, this warning is logged and shutdownNow() interrupts them. It means cache cleanup did not finish gracefully, though resources are forcibly cancelled.

Source

Thrown at core/src/main/java/org/apache/iceberg/rest/auth/AuthSessionCache.java:105

  public <T extends AuthSession> T cachedSession(String key, Function<String, T> loader) {
    return (T) sessionCache().get(key, loader);
  }

  @Override
  public void close() {
    try {
      Cache<String, AuthSession> cache = sessionCache;
      this.sessionCache = null;
      if (cache != null) {
        cache.invalidateAll();
        cache.cleanUp();
      }
    } finally {
      if (executor instanceof ExecutorService) {
        ExecutorService service = (ExecutorService) executor;
        service.shutdown();
        if (!Uninterruptibles.awaitTerminationUninterruptibly(service, 10, TimeUnit.SECONDS)) {
          LOG.warn("Timed out waiting for eviction executor to terminate");
        }
        service.shutdownNow();
      }
    }
  }

  @VisibleForTesting
  Cache<String, AuthSession> sessionCache() {
    if (sessionCache == null) {
      synchronized (this) {
        if (sessionCache == null) {
          this.sessionCache = newSessionCache();
        }
      }
    }

    return sessionCache;
  }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Check the eviction listener (token revoke) for slow/hanging HTTP calls and add aggressive connect/read timeouts on the OAuth client
2. Increase the 10s termination window if legitimate eviction work needs longer, or drain sessions before close
3. Avoid closing the cache concurrently with high session churn; stop traffic first
4. If interruption is safe, treat this as benign — shutdownNow() cancels the tasks and the JVM can proceed

Example fix

// before: eviction listener blocks close() with unbounded HTTP call
session -> oauthClient.revokeToken(session.token())
// after
session -> {
  oauthClient.revokeToken(session.token())
      .timeout(Duration.ofSeconds(2))
      .toCompletableFuture().get(2, TimeUnit.SECONDS);
}

// Before closing, ensure no pending revocations
if (cache.estimatedSize() > 0 && revokeEndpointReachable) {
  executor.shutdown();
  if (!executor.awaitTermination(10, TimeUnit.SECONDS)) executor.shutdownNow();
}

When it happens

Trigger: Calling close() on an AuthSessionCache while the eviction executor is backed up: many cached sessions expiring at once, a blocked eviction task (e.g. a slow token revoke HTTP call in the eviction listener), or the executor thread stuck in a non-interruptible operation.

Common situations: REST catalog clients under heavy load with short token lifetimes; revocation endpoints that are slow or hang (network issues, proxy); JVM shutdown hooks closing the cache while eviction work is mid-flight.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Timed out waiting for coordinator shutdown
• Commit operation did not complete within {} minutes ({} ms)
• Can't retrieve values from an empty struct
• Can't modify an empty struct
• %s doesn't implement cleanupLevel

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/07f1d198fb4bee63. Report an issue: GitHub.