apache/dolphinscheduler · error · RemoteTimeoutException

RemoteTimeoutException(serverHost.toString(), timeoutMills,

Error message

RemoteTimeoutException(serverHost.toString(), timeoutMills, cause)

What it means

RemoteTimeoutException thrown by doSendSync when the request was successfully written to the channel (isSendOK() is true) but responseFuture.waitResponse() returned null before a response arrived within timeoutMills. The remote host received the request but did not answer in time — a response timeout, not a connection failure.

Source

Thrown at dolphinscheduler-extract/dolphinscheduler-extract-base/src/main/java/org/apache/dolphinscheduler/extract/base/client/NettyRemotingClient.java:185

            if (future.isSuccess()) {
                responseFuture.setSendOk(true);
                return;
            } else {
                responseFuture.setSendOk(false);
            }
            responseFuture.setCause(future.cause());
            responseFuture.putResponse(null);
            log.error("Send Sync request {} to host {} failed", transporter, serverHost, responseFuture.getCause());
        });
        /*
         * sync wait for result
         */
        final IRpcResponse iRpcResponse = responseFuture.waitResponse();
        if (iRpcResponse != null) {
            return iRpcResponse;
        }
        if (responseFuture.isSendOK()) {
            throw new RemoteTimeoutException(serverHost.toString(), timeoutMills, responseFuture.getCause());
        } else {
            throw new RemoteException(serverHost.toString(), responseFuture.getCause());
        }
    }

    Channel getOrCreateChannel(Host host) {
        Channel channel = channels.get(host);
        if (channel != null && channel.isActive()) {
            return channel;
        }
        try {
            channelsLock.lock();
            channel = channels.get(host);
            if (channel != null && channel.isActive()) {
                return channel;
            }
            channel = createChannel(host);
            channels.put(host, channel);

View on GitHub (pinned to 02eac45a1b)

Solutions

Increase the RPC timeout passed to sendSync (timeoutMills) to a value that fits the remote operation's worst-case latency
Check the remote host's load: thread pools, GC logs, and whether the receiving dispatcher queue is full
Verify the remote process is alive and processing (check its logs for the corresponding opaque requestId)
Retry the request with backoff; if timeouts cluster on one host, drain or restart that host
Reduce payload size or split large requests (e.g. large task dispatch payloads) that take long to serialize/process

Example fix

// before
IRpcResponse resp = client.sendSync(host, request, 3000); // 3s
// after
IRpcResponse resp = client.sendSync(host, request, 30000); // 30s for heavy calls

Defensive patterns

Strategy: retry

Validate before calling

// Before the call, ensure the timeout covers the remote operation's worst case:
long safeTimeout = Math.max(defaultTimeoutMills, expectedRemoteLatencyMs * 3);
if (safeTimeout > maxAllowedTimeout) { alert("RPC timeout budget too small for " + host); }

Type guard

null

Try / catch

try {
    return client.sendSync(host, request, timeoutMills);
} catch (RemoteTimeoutException e) {
    log.warn("RPC to {} timed out after {}ms, requestId={}", e.getServerHost(), e.getTimeoutMills(), opaque);
    // check remote status before retrying to avoid duplicate execution
    throw e;
}

Prevention

Size timeoutMills to the remote handler's worst-case processing time
Monitor remote worker thread-pool saturation and GC pause logs
Only retry if the command is idempotent — the remote may still execute the request late
Add alerts on p99 RPC latency per host to catch slow hosts early

When it happens

Trigger: sendSync -> doSendSync: channel write succeeded, waitResponse() times out because the remote handler is slow, the processing thread pool is saturated, the remote process is blocked (GC pause, deadlocked task), or the remote dropped the request without replying, and the configured timeoutMills is too small.

Common situations: Default timeouts too low for heavy dispatch commands (e.g. dispatching a large workflow); worker overloaded with tasks so the server business thread pool queue is full; remote host swapping or long GC pauses; network latency spikes; remote process killed after accepting the request.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/31820817e0393c99. Report an issue: GitHub.