conductor-oss/conductor · critical · RuntimeException

Failed to open connection

Error message

Failed to open connection

What it means

Thrown by AMQPConnection.createConnection after factory.newConnection(addresses, clientName) returns a Connection whose isOpen() is false (or, rarely, null). It is a post-handshake sanity check: the TCP/AMQP handshake completed but the resulting Connection is not usable because the broker closed it immediately afterwards. The rabbitmq-client almost never returns null, so the realistic trigger is isOpen()==false.

Source

Thrown at amqp/src/main/java/com/netflix/conductor/contribs/queue/amqp/AMQPConnection.java:91

    // Exposed for UT
    public static void setAMQPConnection(AMQPConnection amqpConnection) {
        AMQPConnection.amqpConnection = amqpConnection;
    }

    public Address[] getAddresses() {
        return addresses;
    }

    private Connection createConnection(String connectionPrefix) {
        int retryIndex = 1;
        while (true) {
            try {
                Connection connection =
                        factory.newConnection(
                                addresses, System.getenv("HOSTNAME") + "-" + connectionPrefix);
                if (connection == null || !connection.isOpen()) {
                    throw new RuntimeException("Failed to open connection");
                }
                connection.addShutdownListener(
                        new ShutdownListener() {
                            @Override
                            public void shutdownCompleted(ShutdownSignalException cause) {
                                LOGGER.error(
                                        "Received a shutdown exception for the connection {}. reason {} cause{}",
                                        connection.getClientProvidedName(),
                                        cause.getMessage(),
                                        cause);
                            }
                        });
                connection.addBlockedListener(
                        new BlockedListener() {
                            @Override
                            public void handleUnblocked() throws IOException {
                                LOGGER.info(
                                        "Connection {} is unblocked",

View on GitHub (pinned to cf7c3e4a8a)

Solutions

  1. Check broker resource alarms and limits: run `rabbitmqctl status` and verify vm_memory_high_watermark and disk free space are not triggered.
  2. Verify the configured vhost exists and the user has access: `rabbitmqctl list_permissions -p <vhost>`.
  3. Confirm the broker is not at its max_connections limit for the user/vhost.
  4. Retry the queue operation: getOrCreateChannel recreates a closed connection on the next call, so transient broker pressure can self-heal.
  5. If TLS is enabled, confirm the client truststore and protocol match the broker to avoid a post-handshake close.

Example fix

# before: broker under memory pressure drops new connections
# rabbitmq.conf
vm_memory_high_watermark.relative = 0.4

# after: raise the watermark (or add memory) so new connections survive the handshake
vm_memory_high_watermark.relative = 0.7
Defensive patterns

Strategy: try-catch

Validate before calling

// Pre-flight: confirm the broker is reachable and the factory can open a test connection
ConnectionFactory probe = new ConnectionFactory();
probe.setUri("amqp://" + username + "@" + host + ":" + port + "/" + vhost);
try (Connection c = probe.newConnection()) {
    if (!c.isOpen()) throw new IllegalStateException("broker opened a closed connection");
}

Try / catch

// getOrCreateChannel(ConnectionType, String) declares `throws Exception`
try {
    Channel chn = amqpConnection.getOrCreateChannel(ConnectionType.PUBLISHER, queueName);
    // ... use channel
} catch (RuntimeException e) {
    // 'Failed to open connection' surfaces here; log, back off, and rethrow or circuit-break
    LOGGER.error("AMQP connection unavailable", e);
    throw e;
}

Prevention

When it happens

Trigger: First call to getOrCreateChannel(SUBSCRIBER|PUBLISHER, name) when the cached connection is null or closed, so createConnection runs and newConnection returns a connection that is already closed. Also on re-establishment after the cached publisher/subscriber connection goes down.

Common situations: Broker is under a memory/disk resource alarm and drops the new connection right after the handshake; broker is at the max_connections limit; vhost/user permission causes the broker to tear the connection down post-auth; a TLS mismatch causes a silent close before the channel layer sees it.

Related errors

AI-assisted analysis of conductor-oss/conductor@cf7c3e4a8a (2026-08-14). Data as JSON: /api/errors/c86bcacd5ceb743c. Report an issue: GitHub.