apache/hadoop · error · SocketTimeoutException

{} millis timeout while waiting for channel to be ready for

Error message

{} millis timeout while waiting for channel to be ready for {}. ch : {}

What it means

Inside doIO, when a non-blocking operation returns 0 the code parks in SelectorPool.select(channel, ops, timeout). If select reports zero ready operations by the deadline, SocketTimeoutException is thrown with a message built by timeoutExceptionString naming the operation (read/write) and channel. This is the generic IO-readiness timeout of the stream wrappers.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketIOWithTimeout.java:163

        }
      } catch (IOException e) {
        if (!channel.isOpen()) {
          closed = true;
        }
        throw e;
      }

      //now wait for socket to be ready.
      int count = 0;
      try {
        count = SelectorPool.select(channel, ops, timeout);
      } catch (IOException e) { //unexpected IOException.
        closed = true;
        throw e;
      } 

      if (count == 0) {
        throw new SocketTimeoutException(timeoutExceptionString(channel,
                                                                timeout, ops));
      }
      // otherwise the socket should be ready for io.
    }
    
    return 0; // does not reach here.
  }
  
  /**
   * The contract is similar to {@link SocketChannel#connect(SocketAddress)} 
   * with a timeout.
   * 
   * @see SocketChannel#connect(SocketAddress)
   * 
   * @param channel - this should be a {@link SelectableChannel}
   * @param endpoint
   * @throws IOException
   */

View on GitHub (pinned to 2add963021)

Solutions

1. Raise the timeout (setTimeout / dfs.client.socket-timeout and related ipc properties) to cover worst-case latency
2. Check peer health and network throughput — a timeout here usually means a stalled sender, not slow data
3. Catch SocketTimeoutException at the call site and retry/fail over per client policy instead of treating it as fatal corruption

Example fix

// before
SocketInputStream in = new SocketInputStream(channel, 5_000); // 5s too small

// after
SocketInputStream in = new SocketInputStream(channel, 60_000);
in.setTimeout(120_000); // extend for long transfers

long timeout = Math.max(60_000, expectedMaxTransferMillis());
stream.setTimeout(timeout);

catch (SocketTimeoutException e) {
  // readiness timeout: peer stalled. Verify peer, then retry with backoff
  if (--attempts > 0 && peerHealthy()) {
    reopenStreamAndRetry();
  } else {
    throw e;
  }
}

When it happens

Trigger: A read on SocketInputStream or write on SocketOutputStream where the channel never becomes ready within this.timeout (set via constructor or setTimeout) — the peer stalls mid-response, stops draining its receive window, or the network drops packets.

Common situations: Hung or GC-paused datanode/name-node peers; network partitions mid-transfer; client socket timeouts (e.g. dfs.client.socket-timeout) set below real transfer latencies; large block reads over slow links.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Channel is null. Check how the channel or socket is created.
• Channel should be a SelectableChannel
• Buffer has no data left.
• Interrupted while waiting for IO on channel {}. Total timeou
• getXAttrs on path `{}' is not within a mount point

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/95c103cd6c2a3b51. Report an issue: GitHub.