nathanmarz/storm · warning

connection failed

Error message

connection failed

What it means

This is a WARN log (not a thrown exception) emitted by the Netty messaging Client when its connection establishment is interrupted by an InterruptedException during reconnect. After logging, the interrupt status is effectively swallowed and the client returns; the connection stays closed and Storm's supervisor will later respawn the worker. It signals the worker thread was interrupted while trying to reach a remote host (typically during topology shutdown or worker restart).

Solutions

1. No action needed if it occurs during topology kill/rebalance or worker shutdown — it is expected there.
2. If seen while the worker should be running, check that the remote supervisor/port is reachable (telnet/nc to the host:port in the log).
3. Verify network and firewall rules between supervisors for the Storm worker ports (default 6700+).
4. Check that workers are not being repeatedly restarted (supervisor logs) — repeated interruptions during reconnect indicate an underlying instability.
5. Ensure storm.messaging.netty timeout/retry settings give reconnect attempts enough time before shutdown interrupts them.

// before starting workers, check the remote host:port is reachable
InetSocketAddress addr = new InetSocketAddress(host, port);
if (addr.isUnresolved() || !isPortOpen(addr)) { throw new IllegalStateException("remote task host unreachable: " + host + ":" + port); }

// Netty Client handles the interrupt internally; workers are supervised.
// Operator-level guard: alert on repeated reconnect/interrupt logs and rely on Storm to respawn the worker.
try { client.reconnect(); } catch (InterruptedException ie) { Thread.currentThread().interrupt(); /* propagate shutdown */ }

When it happens

Trigger: Client.reconnect() attempts to connect to the remote task host and the calling thread receives Thread.interrupt() mid-connection, most often during worker shutdown/topology kill while a reconnect attempt is in flight.

Common situations: Killing or rebalancing a topology while workers still have pending Netty reconnects; supervisor restarting a worker; JVM shutdown hooks interrupting the messaging threads; slow/unreachable remote host keeping reconnect attempts alive until shutdown.

Related errors

• Client is being closed, and does not take requests any more
• Client connection should not receive any messages
• null object forbidded in message batch
• Unsuppoted object type
• Task ID should not exceed

AI-assisted analysis of nathanmarz/storm@cdb116e942 (2026-09-12). Data as JSON: /api/errors/e8a9a58532af1f74. Report an issue: GitHub.

Appendix: source

Thrown at storm-netty/src/jvm/backtype/storm/messaging/netty/Client.java:104

    }

    /**
     * We will retry connection with exponential back-off policy
     */
    void reconnect() {
        try {
            int tried_count = retries.incrementAndGet();
            if (tried_count <= max_retries) {
                Thread.sleep(getSleepTimeMs());
                LOG.info("Reconnect ... [{}]", tried_count);
                bootstrap.connect(remote_addr);
                LOG.debug("connection started...");
            } else {
                LOG.warn("Remote address is not reachable. We will close this client.");
                close();
            }
        } catch (InterruptedException e) {
            LOG.warn("connection failed", e);
        }
    }

    /**
     * # of milliseconds to wait per exponential back-off policy
     */
    private int getSleepTimeMs()
    {
        int backoff = 1 << retries.get();
        int sleepMs = base_sleep_ms * Math.max(1, random.nextInt(backoff));
        if ( sleepMs > max_sleep_ms )
            sleepMs = max_sleep_ms;
        return sleepMs;
    }

    /**
     * Enqueue a task message to be sent to server
     */

View on GitHub (pinned to cdb116e942)