{"record":{"id":"31820817e0393c99","repo":"apache/dolphinscheduler","slug":"remotetimeoutexception-serverhost-tostring-time","errorCode":null,"errorMessage":"RemoteTimeoutException(serverHost.toString(), timeoutMills, cause)","messagePattern":"RemoteTimeoutException\\(serverHost\\.toString\\(\\), timeoutMills, cause\\)","errorType":"exception","errorClass":"RemoteTimeoutException","httpStatus":null,"severity":"error","filePath":"dolphinscheduler-extract/dolphinscheduler-extract-base/src/main/java/org/apache/dolphinscheduler/extract/base/client/NettyRemotingClient.java","lineNumber":185,"sourceCode":"            if (future.isSuccess()) {\n                responseFuture.setSendOk(true);\n                return;\n            } else {\n                responseFuture.setSendOk(false);\n            }\n            responseFuture.setCause(future.cause());\n            responseFuture.putResponse(null);\n            log.error(\"Send Sync request {} to host {} failed\", transporter, serverHost, responseFuture.getCause());\n        });\n        /*\n         * sync wait for result\n         */\n        final IRpcResponse iRpcResponse = responseFuture.waitResponse();\n        if (iRpcResponse != null) {\n            return iRpcResponse;\n        }\n        if (responseFuture.isSendOK()) {\n            throw new RemoteTimeoutException(serverHost.toString(), timeoutMills, responseFuture.getCause());\n        } else {\n            throw new RemoteException(serverHost.toString(), responseFuture.getCause());\n        }\n    }\n\n    Channel getOrCreateChannel(Host host) {\n        Channel channel = channels.get(host);\n        if (channel != null && channel.isActive()) {\n            return channel;\n        }\n        try {\n            channelsLock.lock();\n            channel = channels.get(host);\n            if (channel != null && channel.isActive()) {\n                return channel;\n            }\n            channel = createChannel(host);\n            channels.put(host, channel);","sourceCodeStart":167,"sourceCodeEnd":203,"githubUrl":"https://github.com/apache/dolphinscheduler/blob/02eac45a1b6676e639fcbfb4be2243de5771b05d/dolphinscheduler-extract/dolphinscheduler-extract-base/src/main/java/org/apache/dolphinscheduler/extract/base/client/NettyRemotingClient.java#L167-L203","documentation":"RemoteTimeoutException thrown by doSendSync when the request was successfully written to the channel (isSendOK() is true) but responseFuture.waitResponse() returned null before a response arrived within timeoutMills. The remote host received the request but did not answer in time — a response timeout, not a connection failure.","triggerScenarios":"sendSync -> doSendSync: channel write succeeded, waitResponse() times out because the remote handler is slow, the processing thread pool is saturated, the remote process is blocked (GC pause, deadlocked task), or the remote dropped the request without replying, and the configured timeoutMills is too small.","commonSituations":"Default timeouts too low for heavy dispatch commands (e.g. dispatching a large workflow); worker overloaded with tasks so the server business thread pool queue is full; remote host swapping or long GC pauses; network latency spikes; remote process killed after accepting the request.","solutions":["Increase the RPC timeout passed to sendSync (timeoutMills) to a value that fits the remote operation's worst-case latency","Check the remote host's load: thread pools, GC logs, and whether the receiving dispatcher queue is full","Verify the remote process is alive and processing (check its logs for the corresponding opaque requestId)","Retry the request with backoff; if timeouts cluster on one host, drain or restart that host","Reduce payload size or split large requests (e.g. large task dispatch payloads) that take long to serialize/process"],"exampleFix":"// before\nIRpcResponse resp = client.sendSync(host, request, 3000); // 3s\n// after\nIRpcResponse resp = client.sendSync(host, request, 30000); // 30s for heavy calls","handlingStrategy":"retry","validationCode":"// Before the call, ensure the timeout covers the remote operation's worst case:\nlong safeTimeout = Math.max(defaultTimeoutMills, expectedRemoteLatencyMs * 3);\nif (safeTimeout > maxAllowedTimeout) { alert(\"RPC timeout budget too small for \" + host); }","typeGuard":"null","tryCatchPattern":"try {\n    return client.sendSync(host, request, timeoutMills);\n} catch (RemoteTimeoutException e) {\n    log.warn(\"RPC to {} timed out after {}ms, requestId={}\", e.getServerHost(), e.getTimeoutMills(), opaque);\n    // check remote status before retrying to avoid duplicate execution\n    throw e;\n}","preventionTips":["Size timeoutMills to the remote handler's worst-case processing time","Monitor remote worker thread-pool saturation and GC pause logs","Only retry if the command is idempotent — the remote may still execute the request late","Add alerts on p99 RPC latency per host to catch slow hosts early"],"tags":["rpc","timeout","network"],"backgroundTag":"request-timeout","analyzedSha":"02eac45a1b6676e639fcbfb4be2243de5771b05d","analyzedAt":"2026-09-06T17:43:00.555Z","contentChangedAt":"2026-09-06T17:43:00.555Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}