Node heartbeat timeout, disconnected for resource manager. N

Error message

Node heartbeat timeout, disconnected for resource manager. Node Address:

What it means

AbstractResourceManager.memberRemoved() is invoked by Hazelcast's membership service when a node leaves the cluster. The resource manager logs a warning that the node was disconnected (heartbeat timeout / membership removal) and removes its address from the registerWorker map. This is an informational reaction to cluster topology change, not a thrown exception.

Source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/resourcemanager/AbstractResourceManager.java:168

    }

    private void waitingWorkerRegister() {
        if (ExecutionMode.LOCAL.equals(mode)) {
            // Local mode, should wait worker(master node) register.
            try {
                while (registerWorker.isEmpty() && isRunning) {
                    log.info("waiting current worker register to resource manager...");
                    Thread.sleep(DEFAULT_WORKER_CHECK_INTERVAL);
                }
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }
        }
    }

    @Override
    public void memberRemoved(MembershipServiceEvent event) {
        log.warn(
                "Node heartbeat timeout, disconnected for resource manager. "
                        + "Node Address: "
                        + event.getMember().getAddress());
        registerWorker.remove(event.getMember().getAddress());
    }

    @Override
    public CompletableFuture<List<SlotProfile>> applyResources(
            long jobId, List<ResourceProfile> resourceProfile, Map<String, String> tagFilter)
            throws NoEnoughResourceException {
        waitingWorkerRegister();
        ConcurrentMap<Address, WorkerProfile> matchedWorker = filterWorkerByTag(tagFilter);
        if (matchedWorker.isEmpty()) {
            log.error("No matched worker with tag filter {}.", tagFilter);
            throw new NoEnoughResourceException();
        }
        return new ResourceRequestHandler(
                        jobId, resourceProfile, matchedWorker, this, slotAllocationStrategy)

View on GitHub (pinned to cf67b549a7)

Solutions

Investigate why the node left (worker logs, kubectl events, OS logs for OOM).
Verify network stability between nodes and Hazelcast heartbeat/merge settings.
Ensure workers deregister gracefully on shutdown so resource slots are released cleanly.
Scale the cluster back up or let the job recovery mechanisms reschedule tasks from the lost node.

Defensive patterns

Strategy: fallback

Validate before calling

// before submitting jobs, verify expected cluster size
int liveMembers = hazelcast.getCluster().getMembers().size();
if (liveMembers < expectedWorkers) { reschedule/hold submission; }

Try / catch

// register a MembershipListener to react to memberRemoved proactively
hazelcast.getCluster().addMembershipListener(new MembershipListener() {
    public void memberRemoved(MembershipServiceEvent e) { relocateTasks(e.getMember()); }
});

Prevention

Configure graceful shutdown so workers deregister before exit
Set sensible Hazelcast heartbeat/failure-detector timeouts for your network
Monitor node liveness (k8s probes) and auto-scale replacements
Investigate OOM/network events on evicted nodes

When it happens

Trigger: A worker node is killed, crashes, or becomes unreachable long enough for Hazelcast membership to evict it; the MembershipServiceEvent fires and memberRemoved runs.

Common situations: Worker pod OOM-killed in Kubernetes; network partition causing heartbeat timeout; node gracefully shut down without deregistering; mismatched heartbeat timeouts in cluster config.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/e919812e9a8fd506. Report an issue: GitHub.