apache/hadoop · warning · IOException

connection has been closed

Error message

connection has been closed

What it means

Client.call submits the call to the connection's executor via sendRpcRequest. A RejectedExecutionException means the executor is already shut down — the shared Connection was closed (idle reaper, prior failure, server-side reset) between obtaining it and queuing the request. It is wrapped in IOException('connection has been closed') so callers see a standard IO failure; the async counter is released before rethrow.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java:1527

   * @return the rpc response
   * Throws exceptions if there are network problems or if the remote code
   * threw an exception.
   */
  Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
      ConnectionId remoteId, int serviceClass,
      AtomicBoolean fallbackToSimpleAuth, AlignmentContext alignmentContext)
      throws IOException {
    final Call call = createCall(rpcKind, rpcRequest);
    call.setAlignmentContext(alignmentContext);
    final Connection connection = getConnection(remoteId, call, serviceClass,
        fallbackToSimpleAuth);

    try {
      checkAsyncCall();
      try {
        connection.sendRpcRequest(call);                 // send the rpc request
      } catch (RejectedExecutionException e) {
        throw new IOException("connection has been closed", e);
      } catch (InterruptedException ie) {
        Thread.currentThread().interrupt();
        IOException ioe = new InterruptedIOException(
            "Interrupted waiting to send RPC request to server");
        ioe.initCause(ie);
        throw ioe;
      }
    } catch (Exception e) {
      if (isAsynchronousMode() && isAsyncCallCheckEabled()) {
        releaseAsyncCall();
      }
      throw e;
    }

    if (isAsynchronousMode()) {
      CompletableFuture<Writable> result = call.rpcResponseFuture.handle(
          (rpcResponse, e) -> {
            if (isAsyncCallCheckEabled()) {

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the operation — the next Client.call acquires/creates a fresh connection; for FileSystem-level ops, rely on RetryPolicy.
2. For custom RPC clients, catch IOException around call() and re-invoke once after a short backoff.
3. Reduce idle churn by keeping connections warm or tuning client idle settings rather than disabling cleanup.
4. If persistent, check whether the server is flapping (restarting/OOM) and fix that root cause.

Example fix

// before
Writable resp = client.call(rpcKind, request, remoteId, serviceClass); // IOException: connection has been closed

// after
Writable resp;
try {
  resp = client.call(rpcKind, request, remoteId, serviceClass);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("connection has been closed")) {
    resp = client.call(rpcKind, request, remoteId, serviceClass); // fresh connection is set up
  } else {
    throw e;
  }
}

try {
  call();
} catch (IOException e) {
  if (e.getCause() instanceof java.util.concurrent.RejectedExecutionException
      || (e.getMessage() != null && e.getMessage().contains("connection has been closed"))) {
    call(); // re-getConnection builds a fresh connection; safe for idempotent calls
  } else { throw e; }
}

When it happens

Trigger: Racing the idle-connection reaper: a cached ConnectionId's connection is closed just as a new call grabs it; a prior call killed the connection after an error and a concurrent thread still holds the old reference; server reset (restart) closing sockets while async calls are in flight.

Common situations: Multithreaded clients reusing RPC connections (normal for FileSystem/DFSClient); bursts right after a server restart or failover; long-idle clients whose connection timed out and the next submit loses the race.

Related errors

• Response is null.
• Client IDs not matched: local ID={}, ID in response={}
• RPC response length mismatch
• Exceeded limit of max asynchronous calls: %d, please configu
• Multipart upload incomplete: expected {} parts but got {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f055f0c5ee6c3c19. Report an issue: GitHub.