{"record":{"id":"03b048690d60e743","repo":"apache/hadoop","slug":"failed-to-get-connection-for-is-already","errorCode":null,"errorMessage":"Failed to get connection for {}, {}: {} is already stopped","messagePattern":"Failed to get connection for (.+?), (.+?): (.+?) is already stopped","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java","lineNumber":1654,"sourceCode":"      throws IOException {\n    final Consumer<Connection> removeMethod = c -> {\n      final boolean removed = connections.remove(remoteId, c);\n      if (removed && connections.isEmpty()) {\n        synchronized (emptyCondition) {\n          emptyCondition.notify();\n        }\n      }\n    };\n\n    Connection connection;\n    /* we could avoid this allocation for each RPC by having a  \n     * connectionsId object and with set() method. We need to manage the\n     * refs for keys in HashMap properly. For now its ok.\n     */\n    while (true) {\n      synchronized (putLock) { // synchronized to avoid put after stop\n        if (!running.get()) {\n          throw new IOException(\"Failed to get connection for \" + remoteId\n              + \", \" + call + \": \" + this + \" is already stopped\");\n        }\n        connection = connections.computeIfAbsent(remoteId,\n            id -> new Connection(id, serviceClass, removeMethod));\n      }\n\n      if (connection.addCall(call)) {\n        break;\n      } else {\n        // This connection is closed, should be removed. But other thread could\n        // have already known this closedConnection, and replace it with a new\n        // connection. So we should call conditional remove to make sure we only\n        // remove this closedConnection.\n        removeMethod.accept(connection);\n      }\n    }\n\n    // If the server happens to be slow, the method below will take longer to","sourceCodeStart":1636,"sourceCodeEnd":1672,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L1636-L1672","documentation":"Client.getConnection loops until it can attach a Call to a Connection for the remote ConnectionId; before touching the connections map under putLock it checks the client-level running flag. If Client.stop() already flipped running to false, any new or still-looping call fails with this IOException naming the remoteId and the stopped client. It signals use-after-shutdown of an RPC Client instance, not a network problem.","triggerScenarios":"One thread calls client.stop() (directly or indirectly via RPC.stopProxy / closing a cached FileSystem) while another thread issues a new RPC through the same Client; a call retrying the getConnection loop after its connection was closed races a concurrent stop.","commonSituations":"Sharing a FileSystem/proxy across threads and closing it in one thread while another still reads; UGI doAs blocks that finish and close proxies while background threads keep using them; test teardown closing clients before worker threads drain; HA failover code closing the old client while requests are in flight.","solutions":["Fix the lifecycle: guarantee no RPCs are issued after close/stop — drain or join worker threads before closing the FileSystem or stopping the proxy/Client.","Create a fresh proxy/FileSystem (or re-resolve via FileSystem.get with the same URI/UGI) for the new call instead of reusing the stopped instance.","If the close is legitimate (failover), catch this IOException as a signal to rebuild the client and retry once on the new instance."],"exampleFix":"// before\nnew Thread(() -> {\n for (Path p : paths) { fs.open(p); } // may run after main thread called fs.close()\n}).start();\nfs.close(); // stops the underlying IPC Client\n\n// after\nThread worker = new Thread(() -> {\n for (Path p : paths) { fs.open(p); }\n});\nworker.start();\nworker.join();   // all RPCs done first\nfs.close();","handlingStrategy":"try-catch","validationCode":"// before issuing calls on a shared client-backed proxy\nif (client != null && client.isAlive()) {\n  // safe to attempt; still race with a concurrent stop()\n  proxy.call(req);\n}","typeGuard":null,"tryCatchPattern":"try {\n  return proxy.call(req);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().endsWith(\"is already stopped\")) {\n    proxy = rebuildProxy(); // client was stopped; get a fresh one\n    return proxy.call(req);\n  }\n  throw e;\n}","preventionTips":["Single-owner lifecycle: only the component that creates a FileSystem/proxy closes it, and only after joining all users.","Close in finally of the owning scope, never from unrelated callbacks.","In HA failover paths, replace rather than reuse stopped client instances."],"tags":["hadoop","ipc","rpc","client-shutdown","lifecycle","concurrency"],"backgroundTag":"use-after-close","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}