{"record":{"id":"3d3d3a0edce0cf00","repo":"apache/hadoop","slug":"failed-to-allocate-new-blockreader-at-position","errorCode":null,"errorMessage":"failed to allocate new BlockReader at position {}","messagePattern":"failed to allocate new BlockReader at position (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java","lineNumber":1879,"sourceCode":"          throws IOException, UnsupportedOperationException {\n    if (maxLength == 0) {\n      return EMPTY_BUFFER;\n    } else if (maxLength < 0) {\n      throw new IllegalArgumentException(\"can't read a negative \" +\n          \"number of bytes.\");\n    }\n    if ((blockReader == null) || (blockEnd == -1)) {\n      if (pos >= getFileLength()) {\n        return null;\n      }\n      /*\n       * If we don't have a blockReader, or the one we have has no more bytes\n       * left to read, we call seekToBlockSource to get a new blockReader and\n       * recalculate blockEnd.  Note that we assume we're not at EOF here\n       * (we check this above).\n       */\n      if ((!seekToBlockSource(pos)) || (blockReader == null)) {\n        throw new IOException(\"failed to allocate new BlockReader \" +\n            \"at position \" + pos);\n      }\n    }\n    ByteBuffer buffer = null;\n    if (dfsClient.getConf().getShortCircuitConf().isShortCircuitMmapEnabled()) {\n      buffer = tryReadZeroCopy(maxLength, opts);\n    }\n    if (buffer != null) {\n      return buffer;\n    }\n    buffer = ByteBufferUtil.fallbackRead(this, bufferPool, maxLength);\n    if (buffer != null) {\n      getExtendedReadBuffers().put(buffer, bufferPool);\n    }\n    return buffer;\n  }\n\n  private synchronized ByteBuffer tryReadZeroCopy(int maxLength,","sourceCodeStart":1861,"sourceCodeEnd":1897,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1861-L1897","documentation":"In the enhanced/zero-copy read path, when the stream needs a block reader it calls seekToBlockSource(pos), which iterates the block's replica locations trying to open a BlockReader to a Datanode. If that returns false (every location failed) or the reader is still null, DFSInputStream throws IOException('failed to allocate new BlockReader at position <pos>'). It means the client could not open a data connection to any Datanode holding that block.","triggerScenarios":"Every replica Datanode of the target block is down, unreachable (network/iptables/security groups), or refuses the client; block locations returned by the NameNode are stale after DNs were decommissioned; short-circuit reader creation fails on every DN (missing/permission-broken dfs.domain.socket.path) and TCP fallback also fails.","commonSituations":"DN outage or rolling restart while a long-running reader holds the stream; misconfigured dfs.domain.socket.path (shared-memory domain socket dir not writable by the client user) with dfs.client.read.shortcircuit=true; firewall rules blocking the DN data port (9866); rack/network partition between client and DNs.","solutions":["Check Datanode liveness and connectivity: hdfs dfsadmin -report, and verify you can reach <dn-host>:9866 from the client host.","If short-circuit reads are on, verify dfs.domain.socket.path exists on the DNs and is writable by the client user; as a workaround set dfs.client.read.shortcircuit=false.","Run hdfs fsck <path> -files -blocks -locations to confirm the block has healthy, current locations.","Retry the open/read with backoff - stale locations refresh and dead DNs get excluded, so a fresh open often succeeds."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"List<LocatedBlock> blocks = ((DistributedFileSystem) fs).getClient()\n    .getLocatedBlocks(path, pos, 1).getLocatedBlocks();\nboolean allReplicasDead = !blocks.isEmpty()\n    && Stream.of(blocks.get(0).getLocations()).allMatch(DatanodeInfo::isDecommitted\n        /* or your own reachability check */);\n// treat allReplicasDead as 'do not even try yet - alert instead'","typeGuard":null,"tryCatchPattern":"for (int attempt = 1; attempt <= 3; attempt++) {\n  try {\n    return doRead(fs, path, pos);\n  } catch (IOException e) {\n    if (!String.valueOf(e.getMessage()).contains(\"failed to allocate new BlockReader\")) throw e;\n    Thread.sleep(500L * attempt); // dead DN gets excluded / topology refreshes\n    if (attempt == 3) {\n      try (FSDataInputStream fresh = fs.open(path)) { fresh.seek(pos); return doRead(fresh, pos); }\n    }\n  }\n}","preventionTips":["Keep replication >= 2 (3 default) so one dead DN never strands a block.","Monitor DN liveness and fix firewall/security-group rules for DN data ports (9866) proactively.","Validate dfs.domain.socket.path permissions when short-circuit reads are enabled, or disable short-circuit.","Re-open the stream (fresh open excludes dead DNs) instead of hammering reads on a stuck stream."],"tags":["hdfs","hdfs-client","block-reader","datanode","connectivity","short-circuit"],"backgroundTag":"block-reader-open-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}