{"record":{"id":"e919812e9a8fd506","repo":"apache/seatunnel","slug":"node-heartbeat-timeout-disconnected-for-resource","errorCode":null,"errorMessage":"Node heartbeat timeout, disconnected for resource manager. Node Address: ","messagePattern":"Node heartbeat timeout, disconnected for resource manager\\. Node Address: ","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/resourcemanager/AbstractResourceManager.java","lineNumber":168,"sourceCode":"    }\n\n    private void waitingWorkerRegister() {\n        if (ExecutionMode.LOCAL.equals(mode)) {\n            // Local mode, should wait worker(master node) register.\n            try {\n                while (registerWorker.isEmpty() && isRunning) {\n                    log.info(\"waiting current worker register to resource manager...\");\n                    Thread.sleep(DEFAULT_WORKER_CHECK_INTERVAL);\n                }\n            } catch (InterruptedException e) {\n                throw new RuntimeException(e);\n            }\n        }\n    }\n\n    @Override\n    public void memberRemoved(MembershipServiceEvent event) {\n        log.warn(\n                \"Node heartbeat timeout, disconnected for resource manager. \"\n                        + \"Node Address: \"\n                        + event.getMember().getAddress());\n        registerWorker.remove(event.getMember().getAddress());\n    }\n\n    @Override\n    public CompletableFuture<List<SlotProfile>> applyResources(\n            long jobId, List<ResourceProfile> resourceProfile, Map<String, String> tagFilter)\n            throws NoEnoughResourceException {\n        waitingWorkerRegister();\n        ConcurrentMap<Address, WorkerProfile> matchedWorker = filterWorkerByTag(tagFilter);\n        if (matchedWorker.isEmpty()) {\n            log.error(\"No matched worker with tag filter {}.\", tagFilter);\n            throw new NoEnoughResourceException();\n        }\n        return new ResourceRequestHandler(\n                        jobId, resourceProfile, matchedWorker, this, slotAllocationStrategy)","sourceCodeStart":150,"sourceCodeEnd":186,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/resourcemanager/AbstractResourceManager.java#L150-L186","documentation":"AbstractResourceManager.memberRemoved() is invoked by Hazelcast's membership service when a node leaves the cluster. The resource manager logs a warning that the node was disconnected (heartbeat timeout / membership removal) and removes its address from the registerWorker map. This is an informational reaction to cluster topology change, not a thrown exception.","triggerScenarios":"A worker node is killed, crashes, or becomes unreachable long enough for Hazelcast membership to evict it; the MembershipServiceEvent fires and memberRemoved runs.","commonSituations":"Worker pod OOM-killed in Kubernetes; network partition causing heartbeat timeout; node gracefully shut down without deregistering; mismatched heartbeat timeouts in cluster config.","solutions":["Investigate why the node left (worker logs, kubectl events, OS logs for OOM).","Verify network stability between nodes and Hazelcast heartbeat/merge settings.","Ensure workers deregister gracefully on shutdown so resource slots are released cleanly.","Scale the cluster back up or let the job recovery mechanisms reschedule tasks from the lost node."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// before submitting jobs, verify expected cluster size\nint liveMembers = hazelcast.getCluster().getMembers().size();\nif (liveMembers < expectedWorkers) { reschedule/hold submission; }","typeGuard":null,"tryCatchPattern":"// register a MembershipListener to react to memberRemoved proactively\nhazelcast.getCluster().addMembershipListener(new MembershipListener() {\n    public void memberRemoved(MembershipServiceEvent e) { relocateTasks(e.getMember()); }\n});","preventionTips":["Configure graceful shutdown so workers deregister before exit","Set sensible Hazelcast heartbeat/failure-detector timeouts for your network","Monitor node liveness (k8s probes) and auto-scale replacements","Investigate OOM/network events on evicted nodes"],"tags":["cluster","heartbeat","zeta-engine","hazelcast"],"backgroundTag":"request-timeout","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}