apache/iceberg · critical · NotRunningException

Coordinator unexpectedly terminated on committer ${taskId}

Error message

Coordinator unexpectedly terminated on committer ${taskId}

What it means

NotRunningException thrown by CommitterImpl.processControlEvents when the background coordinator thread has terminated but the committer is still being asked to process control events. The coordinator is the actor that performs commits; once it dies the sink task cannot make progress and must fail so Connect can restart it.

Source

Thrown at kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java:200

    }

    // Reset offsets to last committed to avoid data loss.
    LOG.info("Seeking to last committed offsets for worker {}.", taskId);
    KafkaUtils.seekToLastCommittedOffsets(context);
  }

  @Override
  public void save(Collection<SinkRecord> sinkRecords) {
    if (sinkRecords != null && !sinkRecords.isEmpty()) {
      startWorker();
      worker.save(sinkRecords);
    }
    processControlEvents();
  }

  private void processControlEvents() {
    if (coordinatorThread != null && coordinatorThread.isTerminated()) {
      throw new NotRunningException(
          String.format("Coordinator unexpectedly terminated on committer %s", taskId));
    }
    if (worker != null) {
      worker.process();
    }
  }

  private void startWorker() {
    if (null == this.worker) {
      LOG.info("Starting commit worker {}", taskId);
      SinkWriter sinkWriter = new SinkWriter(catalog, config);
      worker = new Worker(config, clientFactory, sinkWriter, context);
      worker.start();
    }
  }

  private void startCoordinator() {
    if (null == this.coordinatorThread) {

View on GitHub (pinned to 86d9c8fc54)

Solutions

Inspect the coordinator thread's earlier stack trace in worker logs to find the root-cause exception that terminated it.
Fix the underlying commit failure (catalog credentials, network access to metastore/REST catalog, schema compatibility).
Restart the connector task — the sink requires a live coordinator and cannot recover in-place.
Verify coordinator thread timeouts and the exec.awaitTermination settings are adequate for your commit latency.

Defensive patterns

Strategy: try-catch

Try / catch

try {
  committer.save(request);
} catch (NotRunningException e) {
  LOG.error("Coordinator dead; failing sink task for restart", e);
  throw e; // must propagate so Connect restarts the task
}

Prevention

Monitor coordinator thread liveness and earlier root-cause exceptions in logs
Harden commit dependencies: catalog credentials, network, and schema compatibility
Alert on NotRunningException — it always means an earlier coordinator crash

When it happens

Trigger: An unhandled exception killed the coordinator thread (e.g. a commit failed terminally) and the committer's save()/processControlEvents() path subsequently runs; the thread's isTerminated() check fires before any event processing.

Common situations: Coordinator crashed earlier during a commit due to a catalog auth failure, schema mismatch, or timeout; Kafka Connect keeps the task alive until the committer notices the dead coordinator at the next control-event poll.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/2ea304c94875f96b. Report an issue: GitHub.