What causes "Interrupted waiting for the proxy" in apache/hadoop?

Treat it as cancellation, not a connection error: catch InterruptedIOException, call Thread.currentThread().interrupt() to restore the flag, and stop retrying.

apache/hadoop · error · InterruptedIOException

Interrupted waiting for the proxy

Error message

Interrupted waiting for the proxy

What it means

Thrown by RPC.waitForProxy's connection retry loop as an InterruptedIOException when the calling thread's interrupt flag is set between connection attempts (either the flag was already set after a failed attempt, or Thread.sleep(1000) between retries was interrupted). In the sleep-interrupt path the original connection IOException is chained as the cause. It means the wait was deliberately aborted, not that the server is unreachable.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java:444

            .getDefaultSocketFactory(conf), rpcTimeout, connectionRetryPolicy);
      } catch(ConnectException se) {  // namenode has not been started
        LOG.info("Server at " + addr + " not available yet, Zzzzz...");
        ioe = se;
      } catch(SocketTimeoutException te) {  // namenode is busy
        LOG.info("Problem connecting to server: " + addr);
        ioe = te;
      } catch(NoRouteToHostException nrthe) { // perhaps a VIP is failing over
        LOG.info("No route to host for server: " + addr);
        ioe = nrthe;
      }
      // check if timed out
      if (Time.now()-timeout >= startTime) {
        throw ioe;
      }

      if (Thread.currentThread().isInterrupted()) {
        // interrupted during some IO; this may not have been caught
        throw new InterruptedIOException("Interrupted waiting for the proxy");
      }

      // wait for retry
      try {
        Thread.sleep(1000);
      } catch (InterruptedException ie) {
        Thread.currentThread().interrupt();
        throw (IOException) new InterruptedIOException(
            "Interrupted waiting for the proxy").initCause(ioe);
      }
    }
  }

  /**
   * Construct a client-side proxy object that implements the named protocol,
   * talking to a server at the named address. 
   * @param <T> Generics Type T.
   * @param protocol input protocol.

View on GitHub (pinned to 2add963021)

Solutions

Treat it as cancellation, not a connection error: catch InterruptedIOException, call Thread.currentThread().interrupt() to restore the flag, and stop retrying.
Identify the interrupter: check for shutdownNow()/cancel(true) on executors owning this thread; if the proxy wait must finish, run it in a thread no shutdown path interrupts.
Inspect getCause() for the underlying connection exception (e.g., ConnectException) if you also need to know what was being retried.
Prefer bounded waits with an explicit deadline over relying on interrupts for control flow.

Example fix

// before
try {
  proxy = (MyProto) RPC.waitForProxy(theClass, addr, conf, 60000);
} catch (IOException e) {
  LOG.warn("proxy failed", e); // interrupt swallowed, retry logic may misfire
}
// after
try {
  proxy = (MyProto) RPC.waitForProxy(theClass, addr, conf, 60000);
} catch (InterruptedIOException e) {
  Thread.currentThread().interrupt(); // restore flag, abandon the wait
  return;
}

Defensive patterns

Strategy: try-catch

Validate before calling

if (Thread.currentThread().isInterrupted()) {
  // do not enter the proxy wait at all; the loop would abort immediately
  throw new IllegalStateException("caller thread already interrupted");
}
proxy = RPC.waitForProxy(theClass, addr, conf, timeout);

Try / catch

Catch InterruptedIOException in a clause before the generic IOException catch: restore Thread.currentThread().interrupt(), abandon the wait (return/propagate), and log e.getCause() for the original connection failure that was being retried.

Prevention

Never swallow InterruptedIOException inside generic IOException handlers.
Run long proxy waits in dedicated threads that no shutdown path interrupts.
Prefer Future.get with explicit timeout for cancellation instead of interrupting worker threads.

When it happens

Trigger: Calling RPC.waitForProxy()/getProxy with retries from a thread that receives Thread.interrupt() while the loop sleeps 1000ms between attempts, or that already carried the interrupt flag when an attempt threw. Typical interrupt sources: ExecutorService.shutdownNow(), Future.cancel(true), JUnit timeout, or application shutdown racing proxy creation.

Common situations: Startup code waiting for a NameNode/ResourceManager to become reachable while an outer framework cancels the task; tests with @Test(timeout=...) that interrupt the waiting thread; executors shut down while a worker was still waiting for the service.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6da428cac46ad42e. Report an issue: GitHub.