Remote address is not reachable. We will close this client.

Error message

Remote address is not reachable. We will close this client.

What it means

Client.reconnect retries connecting to the remote (nimbus/worker) address up to max_retries with sleep backoff. When tried_count exceeds max_retries, it logs 'Remote address is not reachable. We will close this client.' and calls close() at Client.java:100 — the client gives up permanently and subsequent sends will fail with the being_closed error.

Solutions

Verify the remote host/port are correct and the target process is running (check nimbus/worker logs).
Increase storm.netty.max.retries and storm.netty.min/max.sleep.ms to tolerate longer outages.
Check network connectivity (telnet/nc to the port) and firewall/security-group rules.
Restart the worker/supervisor after the remote becomes reachable — the closed client does not recover on its own.

Example fix

// before (storm.yaml)
// storm.netty.max.retries not set
// after
storm.netty.max.retries: 300
storm.netty.min.sleep.ms: 1000
storm.netty.max.sleep.ms: 30000

Defensive patterns

Strategy: retry

Validate before calling

// preflight connectivity check
try (Socket s = new Socket()) {
    s.connect(new InetSocketAddress(host, port), 3000); // throws if unreachable
}

Try / catch

try {
    client.send(task, message);
} catch (RuntimeException e) {
    if (e.getMessage().contains("does not take requests")) {
        // remote became unreachable and client closed itself; recreate client
    } else { throw e; }
}

Prevention

Monitor nimbus/worker liveness so outages are detected before retry exhaustion.
Tune storm.netty.max.retries and sleep settings to survive restarts and network blips.
Verify ports/firewalls/security groups between workers and nimbus.
Watch logs for 'Reconnect ...' spam — an early signal the remote is unreachable.

When it happens

Trigger: Repeated failed connects to remote_addr for more than max_retries attempts (typical storm.netty.max.retries default 30) during reconnect loops — remote host down, wrong port, firewall, or process crashed.

Common situations: Nimbus or a remote worker dead or restarted while this worker keeps trying; misconfigured transport port in the topology config; network partitions/firewall dropping connections; hostname resolution issues in containers.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

AI-assisted analysis of nathanmarz/storm@cdb116e942 (2026-09-12). Data as JSON: /api/errors/35b23e9cb3624980. Report an issue: GitHub.

Appendix: source

Thrown at storm-netty/src/jvm/backtype/storm/messaging/netty/Client.java:100

        // Start the connection attempt.
        remote_addr = new InetSocketAddress(host, port);
        bootstrap.connect(remote_addr);
    }

    /**
     * We will retry connection with exponential back-off policy
     */
    void reconnect() {
        try {
            int tried_count = retries.incrementAndGet();
            if (tried_count <= max_retries) {
                Thread.sleep(getSleepTimeMs());
                LOG.info("Reconnect ... [{}]", tried_count);
                bootstrap.connect(remote_addr);
                LOG.debug("connection started...");
            } else {
                LOG.warn("Remote address is not reachable. We will close this client.");
                close();
            }
        } catch (InterruptedException e) {
            LOG.warn("connection failed", e);
        }
    }

    /**
     * # of milliseconds to wait per exponential back-off policy
     */
    private int getSleepTimeMs()
    {
        int backoff = 1 << retries.get();
        int sleepMs = base_sleep_ms * Math.max(1, random.nextInt(backoff));
        if ( sleepMs > max_sleep_ms )
            sleepMs = max_sleep_ms;
        return sleepMs;
    }

View on GitHub (pinned to cdb116e942)