apache/hadoop · error · IOException

Xceiver count {} exceeds the limit of concurrent xceivers: {

Error message

Xceiver count {} exceeds the limit of concurrent xceivers: {}

What it means

IOException from the DataXceiverServer accept loop: a new peer connected and the DataNode's live xceiver (block transfer) thread count already exceeds dfs.datanode.max.transfer.threads (default 4096, legacy name dfs.datanode.maxReceiverThreads). The server throws instead of spawning another DataXceiver, protecting itself from thread exhaustion; the new connection is effectively rejected and the affected client operation must retry another DataNode. The exception is caught inside the run loop, so the DataNode itself keeps running.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java:247

        DFSConfigKeys.DFS_DATANODE_DATA_READ_BANDWIDTHPERSEC_DEFAULT);
    if (bandwidthPerSec > 0) {
      this.readThrottler = new DataTransferThrottler(bandwidthPerSec);
    } else {
      this.readThrottler = null;
    }
  }

  @Override
  public void run() {
    Peer peer = null;
    while (datanode.shouldRun && !datanode.shutdownForUpgrade) {
      try {
        peer = peerServer.accept();

        // Make sure the xceiver count is not exceeded
        int curXceiverCount = datanode.getXceiverCount();
        if (curXceiverCount > maxXceiverCount) {
          throw new IOException("Xceiver count " + curXceiverCount
              + " exceeds the limit of concurrent xceivers: "
              + maxXceiverCount);
        }

        new Daemon(datanode.threadGroup,
            DataXceiver.create(peer, datanode, this))
            .start();
      } catch (SocketTimeoutException ignored) {
        // wake up to see if should continue to run
      } catch (AsynchronousCloseException ace) {
        // another thread closed our listener socket - that's expected during shutdown,
        // but not in other circumstances
        if (datanode.shouldRun && !datanode.shutdownForUpgrade) {
          LOG.warn("{}:DataXceiverServer", datanode.getDisplayName(), ace);
        }
      } catch (IOException ie) {
        IOUtils.closeStream(peer);
        LOG.warn("{}:DataXceiverServer", datanode.getDisplayName(), ie);

View on GitHub (pinned to 2add963021)

Solutions

1. Increase dfs.datanode.max.transfer.threads (e.g. 8192 or 16384) in hdfs-site.xml on DataNodes and restart; the key is re-loadable at runtime in newer versions via 'hdfs dfsadmin -reconfig'
2. Reduce client-side parallelism for bulk jobs (distcp -p bandwidth/-m mappers, Spark executor concurrency) or schedule them off-peak
3. Check for the actual load driver via DataNode JMX (XceiverCount, thread dumps) - a stuck high count may be leaked sockets rather than genuine load
4. Add DataNodes or rebalance so transfer load spreads instead of concentrating on a few hot nodes

Example fix

<!-- before -->
<property>
  <name>dfs.datanode.max.transfer.threads</name>
  <value>4096</value>
</property>

<!-- after -->
<property>
  <name>dfs.datanode.max.transfer.threads</name>
  <value>8192</value>
</property>
<!-- then: hdfs dfsadmin -reconfig datanode <host:ipc_port> -start  (or restart DN) -->

// Client-side throttle: bound concurrent streams per DN below the DN limit
// (DN default dfs.datanode.max.transfer.threads = 4096)
int maxConcurrentPerDn = 3500;
Semaphore permits = new Semaphore(maxConcurrentPerDn);
// acquire(permits) before opening each stream to a DN, release on close

try {
  openBlockStream(datanode);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("exceeds the limit of concurrent xceivers")) {
    // DN is saturated: back off, let HDFS pick another replica, and reduce parallelism
    Thread.sleep(30_000);
    openBlockStreamOnOtherReplica();
  } else { throw e; }
}

When it happens

Trigger: Peak load where concurrent block reads/writes/replication transfers on one DataNode exceed max.transfer.threads: heavy parallel distcp/Spark/Hive scans, mass replication after node failure, balancer or hdfs mover running during a busy period, or clients with very high dfs.client.threads / parallelism hammering few DataNodes.

Common situations: Leftover default 4096 on large machines handling thousands of concurrent readers; a small cluster where replication storm after a DataNode death pushes all transfers onto remaining nodes; backup jobs (distcp) run during business hours.

Related errors

• Capacity ratio{} is not between 0 to 1: {}
• Capacity ratio config is not with correct format: {}
• URI: {} is not in the expected format
• Invalid value configured for dfs.datanode.disk.check.timeout
• Invalid value configured for dfs.datanode.disk.check.min.gap

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6e064a71092aa871. Report an issue: GitHub.