What causes "Timed out waiting for coordinator shutdown" in apache/iceberg?

Investigate what the coordinator was doing — look for long commit durations or hung HTTP/RPC calls in the preceding logs.

apache/iceberg · error · ConnectException

Timed out waiting for coordinator shutdown

Error message

Timed out waiting for coordinator shutdown

What it means

Coordinator.terminate() shuts down its executor and waits up to 1 minute for graceful termination; if the coordinator thread does not finish in that window, a ConnectException is thrown to fail the sink task rather than hang shutdown indefinitely.

Source

Thrown at kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Coordinator.java:437

      return MAPPER.readValue(value, typeRef);
    } catch (IOException e) {
      throw new UncheckedIOException(e);
    }
  }

  long partialCommitFailureCount() {
    return partialCommitFailures.get();
  }

  void terminate() {
    this.terminated = true;

    exec.shutdownNow();

    // wait for coordinator termination, else cause the sink task to fail
    try {
      if (!exec.awaitTermination(1, TimeUnit.MINUTES)) {
        throw new ConnectException("Timed out waiting for coordinator shutdown");
      }
    } catch (InterruptedException e) {
      throw new ConnectException("Interrupted while waiting for coordinator shutdown", e);
    }
  }
}

View on GitHub (pinned to 86d9c8fc54)

Solutions

Investigate what the coordinator was doing — look for long commit durations or hung HTTP/RPC calls in the preceding logs.
Ensure the catalog endpoint is reachable and responsive; fix network/latency issues to the metastore.
Shrink the work per commit (fewer files/data per interval) so commits complete well within the shutdown window.
Restart the sink task/connector; the commit protocol is designed for safe retry after failure.

Defensive patterns

Strategy: retry

Try / catch

try {
  coordinator.stop(true);
} catch (ConnectException e) {
  if (String.valueOf(e.getMessage()).contains("Timed out waiting for coordinator shutdown")) {
    LOG.error("Coordinator shutdown timed out; check catalog latency and commit sizes", e);
    throw e;
  }
  throw e;
}

Prevention

Keep catalog endpoints healthy and low-latency so commits finish quickly
Limit data volume per commit interval
Monitor JVM GC pauses and network stalls on Connect workers
Alert on shutdown timeouts — they usually precede worse task failures

When it happens

Trigger: exec.shutdownNow() was called but the coordinator runnable remained busy past the 1-minute awaitTermination deadline — typically a long commit, a hung network call to the catalog, or a stuck event poll.

Common situations: Slow or unresponsive Hive/REST catalog during task shutdown; very large commits at the moment of a rebalance; JVM-level stalls (GC, network partition) delaying the coordinator loop.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/5b5268ac8b5716ec. Report an issue: GitHub.