{"record":{"id":"6e064a71092aa871","repo":"apache/hadoop","slug":"xceiver-count-exceeds-the-limit-of-concurrent-x","errorCode":null,"errorMessage":"Xceiver count {} exceeds the limit of concurrent xceivers: {}","messagePattern":"Xceiver count (.+?) exceeds the limit of concurrent xceivers: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java","lineNumber":247,"sourceCode":"        DFSConfigKeys.DFS_DATANODE_DATA_READ_BANDWIDTHPERSEC_DEFAULT);\n    if (bandwidthPerSec > 0) {\n      this.readThrottler = new DataTransferThrottler(bandwidthPerSec);\n    } else {\n      this.readThrottler = null;\n    }\n  }\n\n  @Override\n  public void run() {\n    Peer peer = null;\n    while (datanode.shouldRun && !datanode.shutdownForUpgrade) {\n      try {\n        peer = peerServer.accept();\n\n        // Make sure the xceiver count is not exceeded\n        int curXceiverCount = datanode.getXceiverCount();\n        if (curXceiverCount > maxXceiverCount) {\n          throw new IOException(\"Xceiver count \" + curXceiverCount\n              + \" exceeds the limit of concurrent xceivers: \"\n              + maxXceiverCount);\n        }\n\n        new Daemon(datanode.threadGroup,\n            DataXceiver.create(peer, datanode, this))\n            .start();\n      } catch (SocketTimeoutException ignored) {\n        // wake up to see if should continue to run\n      } catch (AsynchronousCloseException ace) {\n        // another thread closed our listener socket - that's expected during shutdown,\n        // but not in other circumstances\n        if (datanode.shouldRun && !datanode.shutdownForUpgrade) {\n          LOG.warn(\"{}:DataXceiverServer\", datanode.getDisplayName(), ace);\n        }\n      } catch (IOException ie) {\n        IOUtils.closeStream(peer);\n        LOG.warn(\"{}:DataXceiverServer\", datanode.getDisplayName(), ie);","sourceCodeStart":229,"sourceCodeEnd":265,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L229-L265","documentation":"IOException from the DataXceiverServer accept loop: a new peer connected and the DataNode's live xceiver (block transfer) thread count already exceeds dfs.datanode.max.transfer.threads (default 4096, legacy name dfs.datanode.maxReceiverThreads). The server throws instead of spawning another DataXceiver, protecting itself from thread exhaustion; the new connection is effectively rejected and the affected client operation must retry another DataNode. The exception is caught inside the run loop, so the DataNode itself keeps running.","triggerScenarios":"Peak load where concurrent block reads/writes/replication transfers on one DataNode exceed max.transfer.threads: heavy parallel distcp/Spark/Hive scans, mass replication after node failure, balancer or hdfs mover running during a busy period, or clients with very high dfs.client.threads / parallelism hammering few DataNodes.","commonSituations":"Leftover default 4096 on large machines handling thousands of concurrent readers; a small cluster where replication storm after a DataNode death pushes all transfers onto remaining nodes; backup jobs (distcp) run during business hours.","solutions":["Increase dfs.datanode.max.transfer.threads (e.g. 8192 or 16384) in hdfs-site.xml on DataNodes and restart; the key is re-loadable at runtime in newer versions via 'hdfs dfsadmin -reconfig'","Reduce client-side parallelism for bulk jobs (distcp -p bandwidth/-m mappers, Spark executor concurrency) or schedule them off-peak","Check for the actual load driver via DataNode JMX (XceiverCount, thread dumps) - a stuck high count may be leaked sockets rather than genuine load","Add DataNodes or rebalance so transfer load spreads instead of concentrating on a few hot nodes"],"exampleFix":"<!-- before -->\n<property>\n  <name>dfs.datanode.max.transfer.threads</name>\n  <value>4096</value>\n</property>\n\n<!-- after -->\n<property>\n  <name>dfs.datanode.max.transfer.threads</name>\n  <value>8192</value>\n</property>\n<!-- then: hdfs dfsadmin -reconfig datanode <host:ipc_port> -start  (or restart DN) -->","handlingStrategy":"retry","validationCode":"// Client-side throttle: bound concurrent streams per DN below the DN limit\n// (DN default dfs.datanode.max.transfer.threads = 4096)\nint maxConcurrentPerDn = 3500;\nSemaphore permits = new Semaphore(maxConcurrentPerDn);\n// acquire(permits) before opening each stream to a DN, release on close","typeGuard":null,"tryCatchPattern":"try {\n  openBlockStream(datanode);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"exceeds the limit of concurrent xceivers\")) {\n    // DN is saturated: back off, let HDFS pick another replica, and reduce parallelism\n    Thread.sleep(30_000);\n    openBlockStreamOnOtherReplica();\n  } else { throw e; }\n}","preventionTips":["Size dfs.datanode.max.transfer.threads to real concurrency (large nodes: 8192-16384) and apply via reconfig or rolling restart","Schedule distcp/backups off-peak and cap their parallelism (-m mappers, bandwidth limits)","Monitor XceiverCount via DataNode JMX and alert before it approaches the configured limit","Investigate persistently-high xceiver counts with thread dumps - leaked sockets masquerade as load"],"tags":["hdfs","datanode","xceiver","thread-limit","capacity","config"],"backgroundTag":"connection-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}