conductor-oss/conductor · critical · RuntimeException

IO error while connecting to <addresses>

Error message

IO error while connecting to <addresses>

What it means

createConnection caught an IOException from factory.newConnection and retrySettings is null, so the failure propagates immediately as a RuntimeException instead of being retried. IOException from newConnection means a transport/handshake failure: connection refused, DNS failure, socket reset, or an authentication failure (the rabbitmq-client surfaces bad credentials as IOException).

Source

Thrown at amqp/src/main/java/com/netflix/conductor/contribs/queue/amqp/AMQPConnection.java:131

                            @Override
                            public void handleBlocked(String reason) throws IOException {
                                LOGGER.error(
                                        "Connection {} is blocked. reason: {}",
                                        connection.getClientProvidedName(),
                                        reason);
                            }
                        });
                return connection;
            } catch (final IOException e) {
                AMQPRetryPattern retry = retrySettings;
                if (retry == null) {
                    final String error =
                            "IO error while connecting to "
                                    + Arrays.stream(addresses)
                                            .map(address -> address.toString())
                                            .collect(Collectors.joining(","));
                    LOGGER.error(error, e);
                    throw new RuntimeException(error, e);
                }
                try {
                    retry.continueOrPropogate(e, retryIndex);
                } catch (Exception ex) {
                    final String error =
                            "Retries completed. IO error while connecting to "
                                    + Arrays.stream(addresses)
                                            .map(address -> address.toString())
                                            .collect(Collectors.joining(","));
                    LOGGER.error(error, e);
                    throw new RuntimeException(error, e);
                }
                retryIndex++;
            } catch (final TimeoutException e) {
                AMQPRetryPattern retry = retrySettings;
                if (retry == null) {
                    final String error =
                            "Timeout while connecting to "

View on GitHub (pinned to cf7c3e4a8a)

Solutions

1. Verify the broker is reachable at each configured address/port: `nc -zv <host> <port>`.
2. Confirm username, password and vhost are correct, since the rabbitmq client reports an auth failure as IOException.
3. Wire an AMQPRetryPattern so transient IO failures are retried instead of failing fast (the no-retry branch only runs when retrySettings is null).
4. Check DNS resolution for every host in the addresses list.

Example fix

// before: AMQPObservableQueue built with null retry -> fails fast on a flaky broker
new AMQPObservableQueue(factory, addresses, useExchange, settings, null, batchSize, pollTimeInMS);

// after: supply a retry pattern so transient IOExceptions are retried
AMQPRetryPattern retry = new AMQPRetryPattern(limit, duration, type);
new AMQPObservableQueue(factory, addresses, useExchange, settings, retry, batchSize, pollTimeInMS);

// Validate each address is resolvable and the port is open before building the queue
for (Address a : addresses) {
    try (java.net.Socket s = new java.net.Socket()) {
        s.connect(new java.net.InetSocketAddress(a.getHost(), a.getPort()), 2000);
    } catch (IOException e) {
        throw new IllegalStateException("broker unreachable at " + a, e);
    }
}

try {
    amqpConnection.getOrCreateChannel(ConnectionType.SUBSCRIBER, queueName);
} catch (RuntimeException e) {
    if (e.getCause() instanceof IOException) {
        // transport-level failure; back off and retry at the caller layer
    }
    throw e;
}

When it happens

Trigger: getOrCreateChannel triggers createConnection; factory.newConnection throws IOException; getInstance was called with a null AMQPRetryPattern (the module is wired without retry configuration), so the catch block takes the no-retry branch and throws.

Common situations: Wrong host or port in the AMQPEventQueueProperties config; broker is down or firewalled; incorrect username/password/vhost (auth failure is reported as IOException); DNS cannot resolve the broker host.

Related errors

• Failed to open connection
• Retries completed. IO error while connecting to <addresses>
• Timeout while connecting to <addresses>
• Retries completed. Timeout while connecting to <addresses>
• Cannot open <connType> channel on <addresses>

AI-assisted analysis of conductor-oss/conductor@cf7c3e4a8a (2026-08-14). Data as JSON: /api/errors/20c30bc2ac3d895e. Report an issue: GitHub.