conductor-oss/conductor · critical · RuntimeException

Retries completed. IO error while connecting to <addresses>

Error message

Retries completed. IO error while connecting to <addresses>

What it means

createConnection caught IOException, retrySettings is non-null, but retry.continueOrPropogate(e, retryIndex) itself threw, meaning the configured retry budget (attempt limit or duration) has been exhausted while the IOException keeps recurring. The original IOException is attached as the cause of the propagated RuntimeException.

Source

Thrown at amqp/src/main/java/com/netflix/conductor/contribs/queue/amqp/AMQPConnection.java:142

                if (retry == null) {
                    final String error =
                            "IO error while connecting to "
                                    + Arrays.stream(addresses)
                                            .map(address -> address.toString())
                                            .collect(Collectors.joining(","));
                    LOGGER.error(error, e);
                    throw new RuntimeException(error, e);
                }
                try {
                    retry.continueOrPropogate(e, retryIndex);
                } catch (Exception ex) {
                    final String error =
                            "Retries completed. IO error while connecting to "
                                    + Arrays.stream(addresses)
                                            .map(address -> address.toString())
                                            .collect(Collectors.joining(","));
                    LOGGER.error(error, e);
                    throw new RuntimeException(error, e);
                }
                retryIndex++;
            } catch (final TimeoutException e) {
                AMQPRetryPattern retry = retrySettings;
                if (retry == null) {
                    final String error =
                            "Timeout while connecting to "
                                    + Arrays.stream(addresses)
                                            .map(address -> address.toString())
                                            .collect(Collectors.joining(","));
                    LOGGER.error(error, e);
                    throw new RuntimeException(error, e);
                }
                try {
                    retry.continueOrPropogate(e, retryIndex);
                } catch (Exception ex) {
                    final String error =
                            "Retries completed. Timeout while connecting to "

View on GitHub (pinned to cf7c3e4a8a)

Solutions

1. Confirm the broker is actually up and reachable for the full retry window (the outage outlasted the retries).
2. Fix the root cause first: if it is auth/vhost/network, retries cannot succeed and must not be lengthened blindly.
3. If the outage is legitimately transient but longer than expected, raise the retry limit and/or duration in AMQPEventQueueProperties.
4. Add a circuit breaker upstream so a long broker outage fails fast instead of burning the whole retry budget per call.

Example fix

# before: retries exhaust during a multi-minute broker restart
conductor.amqp.retry.limit=3
conductor.amqp.retry.duration=1s

# after: size the budget to the expected restart window
conductor.amqp.retry.limit=10
conductor.amqp.retry.duration=5s

// Retries already exhausted inside the library; treat as a hard failure
try {
    amqpConnection.getOrCreateChannel(type, name);
} catch (RuntimeException e) {
    LOGGER.error("AMQP connection retries exhausted: {}", e.getMessage(), e);
    // trip a circuit breaker; fail the workflow operation rather than loop
    throw e;
}

When it happens

Trigger: factory.newConnection keeps throwing IOException across every retry attempt until the AMQPRetryPattern limit/duration is reached, at which point continueOrPropogate raises and the retries-completed message is thrown.

Common situations: Broker down for longer than the configured retry window; a persistent authentication failure (non-transient, so retries never help); a sustained network partition; a misconfigured port that refuses every attempt.

Related errors

• IO error while connecting to <addresses>
• Retries completed. Timeout while connecting to <addresses>
• Retries completed. Cannot open <connType> channel on <addres
• Failed to open connection
• Timeout while connecting to <addresses>

AI-assisted analysis of conductor-oss/conductor@cf7c3e4a8a (2026-08-14). Data as JSON: /api/errors/a444be8602856a99. Report an issue: GitHub.