{"record":{"id":"35b23e9cb3624980","repo":"nathanmarz/storm","slug":"remote-address-is-not-reachable-we-will-close-this-client","errorCode":null,"errorMessage":"Remote address is not reachable. We will close this client.","messagePattern":"Remote address is not reachable\\. We will close this client\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"storm-netty/src/jvm/backtype/storm/messaging/netty/Client.java","lineNumber":100,"sourceCode":"\n        // Start the connection attempt.\n        remote_addr = new InetSocketAddress(host, port);\n        bootstrap.connect(remote_addr);\n    }\n\n    /**\n     * We will retry connection with exponential back-off policy\n     */\n    void reconnect() {\n        try {\n            int tried_count = retries.incrementAndGet();\n            if (tried_count <= max_retries) {\n                Thread.sleep(getSleepTimeMs());\n                LOG.info(\"Reconnect ... [{}]\", tried_count);\n                bootstrap.connect(remote_addr);\n                LOG.debug(\"connection started...\");\n            } else {\n                LOG.warn(\"Remote address is not reachable. We will close this client.\");\n                close();\n            }\n        } catch (InterruptedException e) {\n            LOG.warn(\"connection failed\", e);\n        }\n    }\n\n    /**\n     * # of milliseconds to wait per exponential back-off policy\n     */\n    private int getSleepTimeMs()\n    {\n        int backoff = 1 << retries.get();\n        int sleepMs = base_sleep_ms * Math.max(1, random.nextInt(backoff));\n        if ( sleepMs > max_sleep_ms )\n            sleepMs = max_sleep_ms;\n        return sleepMs;\n    }","sourceCodeStart":82,"sourceCodeEnd":118,"githubUrl":"https://github.com/nathanmarz/storm/blob/cdb116e942666973bc4eaa0df098d5bab82739e7/storm-netty/src/jvm/backtype/storm/messaging/netty/Client.java#L82-L118","documentation":"Client.reconnect retries connecting to the remote (nimbus/worker) address up to max_retries with sleep backoff. When tried_count exceeds max_retries, it logs 'Remote address is not reachable. We will close this client.' and calls close() at Client.java:100 — the client gives up permanently and subsequent sends will fail with the being_closed error.","triggerScenarios":"Repeated failed connects to remote_addr for more than max_retries attempts (typical storm.netty.max.retries default 30) during reconnect loops — remote host down, wrong port, firewall, or process crashed.","commonSituations":"Nimbus or a remote worker dead or restarted while this worker keeps trying; misconfigured transport port in the topology config; network partitions/firewall dropping connections; hostname resolution issues in containers.","solutions":["Verify the remote host/port are correct and the target process is running (check nimbus/worker logs).","Increase storm.netty.max.retries and storm.netty.min/max.sleep.ms to tolerate longer outages.","Check network connectivity (telnet/nc to the port) and firewall/security-group rules.","Restart the worker/supervisor after the remote becomes reachable — the closed client does not recover on its own."],"exampleFix":"// before (storm.yaml)\n// storm.netty.max.retries not set\n// after\nstorm.netty.max.retries: 300\nstorm.netty.min.sleep.ms: 1000\nstorm.netty.max.sleep.ms: 30000","handlingStrategy":"retry","validationCode":"// preflight connectivity check\ntry (Socket s = new Socket()) {\n    s.connect(new InetSocketAddress(host, port), 3000); // throws if unreachable\n}","typeGuard":null,"tryCatchPattern":"try {\n    client.send(task, message);\n} catch (RuntimeException e) {\n    if (e.getMessage().contains(\"does not take requests\")) {\n        // remote became unreachable and client closed itself; recreate client\n    } else { throw e; }\n}","preventionTips":["Monitor nimbus/worker liveness so outages are detected before retry exhaustion.","Tune storm.netty.max.retries and sleep settings to survive restarts and network blips.","Verify ports/firewalls/security groups between workers and nimbus.","Watch logs for 'Reconnect ...' spam — an early signal the remote is unreachable."],"tags":["netty","connectivity","reconnect","retry-exhausted"],"backgroundTag":"connection-refused","analyzedSha":"cdb116e942666973bc4eaa0df098d5bab82739e7","analyzedAt":"2026-09-12T14:30:00.714Z","contentChangedAt":"2026-09-12T14:30:00.714Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}