apache/hadoop · error · IOException

Failed to get connection for {}, {}: {} is already stopped

Error message

Failed to get connection for {}, {}: {} is already stopped

What it means

Client.getConnection loops until it can attach a Call to a Connection for the remote ConnectionId; before touching the connections map under putLock it checks the client-level running flag. If Client.stop() already flipped running to false, any new or still-looping call fails with this IOException naming the remoteId and the stopped client. It signals use-after-shutdown of an RPC Client instance, not a network problem.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java:1654

      throws IOException {
    final Consumer<Connection> removeMethod = c -> {
      final boolean removed = connections.remove(remoteId, c);
      if (removed && connections.isEmpty()) {
        synchronized (emptyCondition) {
          emptyCondition.notify();
        }
      }
    };

    Connection connection;
    /* we could avoid this allocation for each RPC by having a  
     * connectionsId object and with set() method. We need to manage the
     * refs for keys in HashMap properly. For now its ok.
     */
    while (true) {
      synchronized (putLock) { // synchronized to avoid put after stop
        if (!running.get()) {
          throw new IOException("Failed to get connection for " + remoteId
              + ", " + call + ": " + this + " is already stopped");
        }
        connection = connections.computeIfAbsent(remoteId,
            id -> new Connection(id, serviceClass, removeMethod));
      }

      if (connection.addCall(call)) {
        break;
      } else {
        // This connection is closed, should be removed. But other thread could
        // have already known this closedConnection, and replace it with a new
        // connection. So we should call conditional remove to make sure we only
        // remove this closedConnection.
        removeMethod.accept(connection);
      }
    }

    // If the server happens to be slow, the method below will take longer to

View on GitHub (pinned to 2add963021)

Solutions

  1. Fix the lifecycle: guarantee no RPCs are issued after close/stop — drain or join worker threads before closing the FileSystem or stopping the proxy/Client.
  2. Create a fresh proxy/FileSystem (or re-resolve via FileSystem.get with the same URI/UGI) for the new call instead of reusing the stopped instance.
  3. If the close is legitimate (failover), catch this IOException as a signal to rebuild the client and retry once on the new instance.

Example fix

// before
new Thread(() -> {
 for (Path p : paths) { fs.open(p); } // may run after main thread called fs.close()
}).start();
fs.close(); // stops the underlying IPC Client

// after
Thread worker = new Thread(() -> {
 for (Path p : paths) { fs.open(p); }
});
worker.start();
worker.join();   // all RPCs done first
fs.close();
Defensive patterns

Strategy: try-catch

Validate before calling

// before issuing calls on a shared client-backed proxy
if (client != null && client.isAlive()) {
  // safe to attempt; still race with a concurrent stop()
  proxy.call(req);
}

Try / catch

try {
  return proxy.call(req);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().endsWith("is already stopped")) {
    proxy = rebuildProxy(); // client was stopped; get a fresh one
    return proxy.call(req);
  }
  throw e;
}

Prevention

When it happens

Trigger: One thread calls client.stop() (directly or indirectly via RPC.stopProxy / closing a cached FileSystem) while another thread issues a new RPC through the same Client; a call retrying the getConnection loop after its connection was closed races a concurrent stop.

Common situations: Sharing a FileSystem/proxy across threads and closing it in one thread while another still reads; UGI doAs blocks that finish and close proxies while background threads keep using them; test teardown closing clients before worker threads drain; HA failover code closing the old client while requests are in flight.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/03b048690d60e743. Report an issue: GitHub.