{"record":{"id":"5b5268ac8b5716ec","repo":"apache/iceberg","slug":"timed-out-waiting-for-coordinator-shutdown","errorCode":null,"errorMessage":"Timed out waiting for coordinator shutdown","messagePattern":"Timed out waiting for coordinator shutdown","errorType":"exception","errorClass":"ConnectException","httpStatus":null,"severity":"error","filePath":"kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Coordinator.java","lineNumber":437,"sourceCode":"      return MAPPER.readValue(value, typeRef);\n    } catch (IOException e) {\n      throw new UncheckedIOException(e);\n    }\n  }\n\n  long partialCommitFailureCount() {\n    return partialCommitFailures.get();\n  }\n\n  void terminate() {\n    this.terminated = true;\n\n    exec.shutdownNow();\n\n    // wait for coordinator termination, else cause the sink task to fail\n    try {\n      if (!exec.awaitTermination(1, TimeUnit.MINUTES)) {\n        throw new ConnectException(\"Timed out waiting for coordinator shutdown\");\n      }\n    } catch (InterruptedException e) {\n      throw new ConnectException(\"Interrupted while waiting for coordinator shutdown\", e);\n    }\n  }\n}\n","sourceCodeStart":419,"sourceCodeEnd":444,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Coordinator.java#L419-L444","documentation":"Coordinator.terminate() shuts down its executor and waits up to 1 minute for graceful termination; if the coordinator thread does not finish in that window, a ConnectException is thrown to fail the sink task rather than hang shutdown indefinitely.","triggerScenarios":"exec.shutdownNow() was called but the coordinator runnable remained busy past the 1-minute awaitTermination deadline — typically a long commit, a hung network call to the catalog, or a stuck event poll.","commonSituations":"Slow or unresponsive Hive/REST catalog during task shutdown; very large commits at the moment of a rebalance; JVM-level stalls (GC, network partition) delaying the coordinator loop.","solutions":["Investigate what the coordinator was doing — look for long commit durations or hung HTTP/RPC calls in the preceding logs.","Ensure the catalog endpoint is reachable and responsive; fix network/latency issues to the metastore.","Shrink the work per commit (fewer files/data per interval) so commits complete well within the shutdown window.","Restart the sink task/connector; the commit protocol is designed for safe retry after failure."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  coordinator.stop(true);\n} catch (ConnectException e) {\n  if (String.valueOf(e.getMessage()).contains(\"Timed out waiting for coordinator shutdown\")) {\n    LOG.error(\"Coordinator shutdown timed out; check catalog latency and commit sizes\", e);\n    throw e;\n  }\n  throw e;\n}","preventionTips":["Keep catalog endpoints healthy and low-latency so commits finish quickly","Limit data volume per commit interval","Monitor JVM GC pauses and network stalls on Connect workers","Alert on shutdown timeouts — they usually precede worse task failures"],"tags":["kafka-connect","coordinator","timeout","shutdown"],"backgroundTag":"request-timeout","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}