apache/seatunnel · critical · RuntimeException

Failed to persist readyToCloseStartingTask to IMap, key: %s

Error message

Failed to persist readyToCloseStartingTask to IMap, key: %s

What it means

CheckpointCoordinator.updateReadyToCloseStartingTask persists the readyToCloseStartingTask marker to the IMap with retries; after exhausting retries it logs and throws a RuntimeException, deliberately failing the job so it doesn't get stuck in an unrecoverable state on master failover. Persistence of this marker is a required step of the checkpoint close protocol.

Source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/checkpoint/CheckpointCoordinator.java:613

                                                    ? new HashSet<>((Set<TaskLocation>) exist)
                                                    : new HashSet<>();
                                    merged.addAll(readyToCloseStartingTask);
                                    return merged;
                                });
                        return null;
                    },
                    new RetryUtils.RetryMaterial(
                            Constant.OPERATION_RETRY_TIME,
                            true,
                            ExceptionUtil::isOperationNeedRetryException,
                            Constant.OPERATION_RETRY_SLEEP));
        } catch (Exception e) {
            LOG.error(
                    "Failed to persist readyToCloseStartingTask to IMap after retries, key: {}."
                            + " Failing the job to avoid an unrecoverable stuck state on master failover.",
                    readyToCloseImapKey,
                    e);
            throw new RuntimeException(
                    "Failed to persist readyToCloseStartingTask to IMap, key: "
                            + readyToCloseImapKey,
                    e);
        }
    }

    protected void readyToCloseIdleTask(TaskLocation taskLocation) {
        if (plan.getStartingSubtasks().contains(taskLocation)) {
            throw new UnsupportedOperationException("Unsupported close starting task");
        }

        LOG.info(
                "Received close idle task, task id: {}, pipeline id: {}, job id: {}, detail: {}",
                taskLocation.getTaskID(),
                taskLocation.getPipelineId(),
                taskLocation.getJobId(),
                taskLocation);
        synchronized (readyToCloseIdleTask) {

View on GitHub (pinned to cf67b549a7)

Solutions

1. Check the wrapped cause and Hazelcast logs for timeouts/partition events during the failure window
2. Verify network stability between cluster nodes; fix the partition or connectivity problem before re-running
3. Increase Hazelcast operation timeout / retry budget if workloads legitimately cause long operation pauses
4. Resubmit the job from a compatible checkpoint state once the cluster is healthy — the job was intentionally failed to avoid a stuck state
5. Review IMap backup counts so the key survives single-node failures

Example fix

// before
// no handling: job fails with raw RuntimeException
coordinator.readyToClose(taskLocation);

// after
try {
    coordinator.readyToClose(taskLocation);
} catch (RuntimeException e) {
    LOG.error("readyToClose persist failed; will resubmit job from last checkpoint", e);
    resubmitJobFromCheckpoint(jobId);
}

// ensure cluster is healthy before protocol-critical writes
boolean healthy = hazelcastInstance.getCluster().getMembers().size() == expectedMembers;

try {
    coordinator.readyToClose(taskLocation);
} catch (RuntimeException e) {
    LOG.error("IMap persist failed for {}", readyToCloseImapKey, e.getCause());
    resubmitJobFromLastCheckpoint(jobId);
}

When it happens

Trigger: All IMap put retries fail during readyToClose: sustained Hazelcast connectivity loss, repeated operation timeouts, partition owner unavailable during failover, or serialization errors on the value.

Common situations: Master node losing Hazelcast connection mid-checkpoint; long network partition between cluster members; Hazelcast cluster overloaded so operations time out beyond the retry budget.

Related errors

• Failed to load readyToCloseStartingTask from IMap, key: %s
• Failed to fetch running jobs from IMap during master switch
• Job id %s restore failed, can not get job state
• The checkpoint coordinator(%s) don't exist
• Unknown checkpoint type:

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/3022f7b4031842d3. Report an issue: GitHub.