apache/dolphinscheduler · error · RemoteException

connect to : %s fail

Error message

connect to : %s fail

What it means

doSendSync throws this RemoteException when getOrCreateChannel cannot obtain an active Netty channel to the target host — i.e. the TCP connection to the remote server could not be established. The RPC request was never written to the wire. The formatted message includes the Host (ip:port) that failed.

Source

Thrown at dolphinscheduler-extract/dolphinscheduler-extract-base/src/main/java/org/apache/dolphinscheduler/extract/base/client/NettyRemotingClient.java:163

                    throw (RemoteException) ex;
                } else {
                    throw new RemoteException("Call method to " + host + " failed", ex);
                }
            } finally {
                ClientSyncDurationMetrics clientSyncDurationMetrics = ClientSyncDurationMetrics
                        .of(syncRequestDto)
                        .withMilliseconds(System.currentTimeMillis() - start);
                RpcMetrics.recordClientSyncRequestDuration(clientSyncDurationMetrics);
            }
        }
    }

    private IRpcResponse doSendSync(final Transporter transporter,
                                    final Host serverHost,
                                    long timeoutMills) throws RemoteException, InterruptedException {
        final Channel channel = getOrCreateChannel(serverHost);
        if (channel == null) {
            throw new RemoteException(String.format("connect to : %s fail", serverHost));
        }
        final ResponseFuture responseFuture = new ResponseFuture(transporter.getHeader().getOpaque(), timeoutMills);
        channel.writeAndFlush(transporter).addListener(future -> {
            if (future.isSuccess()) {
                responseFuture.setSendOk(true);
                return;
            } else {
                responseFuture.setSendOk(false);
            }
            responseFuture.setCause(future.cause());
            responseFuture.putResponse(null);
            log.error("Send Sync request {} to host {} failed", transporter, serverHost, responseFuture.getCause());
        });
        /*
         * sync wait for result
         */
        final IRpcResponse iRpcResponse = responseFuture.waitResponse();
        if (iRpcResponse != null) {

View on GitHub (pinned to 02eac45a1b)

Solutions

  1. Verify the remote process is running and listening on the configured port (ss -tlnp | grep <port> or telnet)
  2. Confirm the Host ip/port from the registry (ZooKeeper) is correct and current; stale registry entries are common after an unclean shutdown
  3. Check firewall/security-group rules allow TCP between master and worker
  4. If in containers/K8s, verify DNS resolution of the host name from the calling pod
  5. Increase netty client connect timeout if the network is slow, and retry the request

Example fix

// before
// connect fails silently downstream, only generic exception seen
// after
Channel ch = client.getOrCreateChannel(host);
if (ch == null || !ch.isActive()) {
    log.error("Cannot connect to {}, check process/port/firewall", host);
}
Defensive patterns

Strategy: retry

Validate before calling

boolean canConnect(Host host) {
    try (Socket s = new Socket()) {
        s.connect(new InetSocketAddress(host.getIp(), host.getPort()), 2000);
        return true;
    } catch (IOException e) {
        log.warn("Host {} not connectable: {}", host, e.getMessage());
        return false;
    }
}

Type guard

null

Try / catch

try {
    IRpcResponse resp = client.sendSync(host, request, timeout);
} catch (RemoteException e) {
    if (String.valueOf(e.getMessage()).startsWith("connect to")) {
        // pick another host from registry / schedule retry with backoff
    }
    throw e;
}

Prevention

When it happens

Trigger: sendSync -> doSendSync calls getOrCreateChannel(serverHost); the underlying bootstrap.connect fails (connection refused, timeout, DNS failure) and createChannel throws, or returns a channel that never becomes active, so null is returned and this RemoteException is thrown.

Common situations: Worker process is down or still booting; wrong port in worker registration; container hostname not resolvable from master pod (K8s headless service issues); host firewalled; connect timeout too short for slow network.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/672ecfd47c2e72f5. Report an issue: GitHub.