What causes "RemoteException(serverHost.toString(), cause)" in apache/dolphinscheduler?

Inspect responseFuture.getCause() (via the exception cause) for the write failure root cause

apache/dolphinscheduler · error · RemoteException

RemoteException(serverHost.toString(), cause)

Error message

RemoteException(serverHost.toString(), cause)

What it means

RemoteException(host, cause) thrown by doSendSync when no response was received AND the request was never successfully written to the channel (isSendOK() is false). responseFuture.getCause() holds the underlying failure — typically the write failed because the channel closed mid-flight or the async write listener reported an error. Distinguish this from the timeout case (572): here the send itself failed.

Source

Thrown at dolphinscheduler-extract/dolphinscheduler-extract-base/src/main/java/org/apache/dolphinscheduler/extract/base/client/NettyRemotingClient.java:187

                return;
            } else {
                responseFuture.setSendOk(false);
            }
            responseFuture.setCause(future.cause());
            responseFuture.putResponse(null);
            log.error("Send Sync request {} to host {} failed", transporter, serverHost, responseFuture.getCause());
        });
        /*
         * sync wait for result
         */
        final IRpcResponse iRpcResponse = responseFuture.waitResponse();
        if (iRpcResponse != null) {
            return iRpcResponse;
        }
        if (responseFuture.isSendOK()) {
            throw new RemoteTimeoutException(serverHost.toString(), timeoutMills, responseFuture.getCause());
        } else {
            throw new RemoteException(serverHost.toString(), responseFuture.getCause());
        }
    }

    Channel getOrCreateChannel(Host host) {
        Channel channel = channels.get(host);
        if (channel != null && channel.isActive()) {
            return channel;
        }
        try {
            channelsLock.lock();
            channel = channels.get(host);
            if (channel != null && channel.isActive()) {
                return channel;
            }
            channel = createChannel(host);
            channels.put(host, channel);
        } finally {
            channelsLock.unlock();

View on GitHub (pinned to 02eac45a1b)

Solutions

Inspect responseFuture.getCause() (via the exception cause) for the write failure root cause
Clear/rebuild the cached channel: connections can go stale — retry once with a fresh connection
Check whether the remote host was restarting or killing connections (rolling upgrade window)
Ensure Netty client-side idle detection/reconnect is configured so dead channels are evicted before writes
Add retry with backoff around sendSync for transient connection resets

Example fix

// before
IRpcResponse resp = client.sendSync(host, request, timeout); // fails on stale channel
// after
IRpcResponse resp;
try {
    resp = client.sendSync(host, request, timeout);
} catch (RemoteException e) {
    client.closeChannel(host); // drop stale connection
    resp = client.sendSync(host, request, timeout); // retry on fresh channel
}

Defensive patterns

Strategy: retry

Validate before calling

// Evict stale channels before retrying:
Channel ch = client.getOrCreateChannel(host);
if (ch == null || !ch.isActive()) {
    client.closeChannel(host); // force fresh connection on next call
}

Type guard

null

Try / catch

try {
    return client.sendSync(host, request, timeout);
} catch (RemoteException e) {
    if (e.getCause() != null && !isTimeout(e)) { // send failed, not timed out
        client.closeChannel(host);              // drop stale connection
        return client.sendSync(host, request, timeout); // one retry on fresh channel
    }
    throw e;
}

Prevention

Enable Netty idle-state handling so dead cached channels are evicted before writes fail
Avoid long-lived idle connections through NAT/LBs — set TCP keepalive or reconnect periodically
Coordinate retries with rolling-restart windows so connection resets aren't mistaken for real failures
Always inspect getCause(): write failures vs timeouts require different retry policies

When it happens

Trigger: sendSync -> doSendSync: channel.writeAndFlush future fails (connection reset/closed between getOrCreateChannel and the write, TCP RST, half-closed socket), so setSendOk(true) is never called; waitResponse() then returns null with a cause set and isSendOK false.

Common situations: Worker restarted mid-request (connection dropped); idle connection reaped by an intermediary (NAT/LB timeout) but still cached in the client's channels map; TLS/protocol mismatch closing the socket; remote port briefly unavailable during rolling restarts.

Related errors

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/ff0b0f4dbf9f7c6d. Report an issue: GitHub.