{"record":{"id":"f055f0c5ee6c3c19","repo":"apache/hadoop","slug":"connection-has-been-closed","errorCode":null,"errorMessage":"connection has been closed","messagePattern":"connection has been closed","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"warning","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java","lineNumber":1527,"sourceCode":"   * @return the rpc response\n   * Throws exceptions if there are network problems or if the remote code\n   * threw an exception.\n   */\n  Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,\n      ConnectionId remoteId, int serviceClass,\n      AtomicBoolean fallbackToSimpleAuth, AlignmentContext alignmentContext)\n      throws IOException {\n    final Call call = createCall(rpcKind, rpcRequest);\n    call.setAlignmentContext(alignmentContext);\n    final Connection connection = getConnection(remoteId, call, serviceClass,\n        fallbackToSimpleAuth);\n\n    try {\n      checkAsyncCall();\n      try {\n        connection.sendRpcRequest(call);                 // send the rpc request\n      } catch (RejectedExecutionException e) {\n        throw new IOException(\"connection has been closed\", e);\n      } catch (InterruptedException ie) {\n        Thread.currentThread().interrupt();\n        IOException ioe = new InterruptedIOException(\n            \"Interrupted waiting to send RPC request to server\");\n        ioe.initCause(ie);\n        throw ioe;\n      }\n    } catch (Exception e) {\n      if (isAsynchronousMode() && isAsyncCallCheckEabled()) {\n        releaseAsyncCall();\n      }\n      throw e;\n    }\n\n    if (isAsynchronousMode()) {\n      CompletableFuture<Writable> result = call.rpcResponseFuture.handle(\n          (rpcResponse, e) -> {\n            if (isAsyncCallCheckEabled()) {","sourceCodeStart":1509,"sourceCodeEnd":1545,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L1509-L1545","documentation":"Client.call submits the call to the connection's executor via sendRpcRequest. A RejectedExecutionException means the executor is already shut down — the shared Connection was closed (idle reaper, prior failure, server-side reset) between obtaining it and queuing the request. It is wrapped in IOException('connection has been closed') so callers see a standard IO failure; the async counter is released before rethrow.","triggerScenarios":"Racing the idle-connection reaper: a cached ConnectionId's connection is closed just as a new call grabs it; a prior call killed the connection after an error and a concurrent thread still holds the old reference; server reset (restart) closing sockets while async calls are in flight.","commonSituations":"Multithreaded clients reusing RPC connections (normal for FileSystem/DFSClient); bursts right after a server restart or failover; long-idle clients whose connection timed out and the next submit loses the race.","solutions":["Retry the operation — the next Client.call acquires/creates a fresh connection; for FileSystem-level ops, rely on RetryPolicy.","For custom RPC clients, catch IOException around call() and re-invoke once after a short backoff.","Reduce idle churn by keeping connections warm or tuning client idle settings rather than disabling cleanup.","If persistent, check whether the server is flapping (restarting/OOM) and fix that root cause."],"exampleFix":"// before\nWritable resp = client.call(rpcKind, request, remoteId, serviceClass); // IOException: connection has been closed\n\n// after\nWritable resp;\ntry {\n  resp = client.call(rpcKind, request, remoteId, serviceClass);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"connection has been closed\")) {\n    resp = client.call(rpcKind, request, remoteId, serviceClass); // fresh connection is set up\n  } else {\n    throw e;\n  }\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  call();\n} catch (IOException e) {\n  if (e.getCause() instanceof java.util.concurrent.RejectedExecutionException\n      || (e.getMessage() != null && e.getMessage().contains(\"connection has been closed\"))) {\n    call(); // re-getConnection builds a fresh connection; safe for idempotent calls\n  } else { throw e; }\n}","preventionTips":["Wrap RPC invocations with a single retry for races against connection teardown.","Let the framework's RetryPolicy do the work for FileSystem-level APIs instead of raw Client calls.","Keep connections modestly warm under sustained load so the reaper does not race you.","Investigate server flapping if the error recurs frequently."],"tags":["hadoop","ipc","rpc-client","connection","race-condition","retry"],"backgroundTag":"connection-closed-retry","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}