apache/hadoop · error · IOException

failed to allocate new BlockReader at position {}

Error message

failed to allocate new BlockReader at position {}

What it means

In the enhanced/zero-copy read path, when the stream needs a block reader it calls seekToBlockSource(pos), which iterates the block's replica locations trying to open a BlockReader to a Datanode. If that returns false (every location failed) or the reader is still null, DFSInputStream throws IOException('failed to allocate new BlockReader at position <pos>'). It means the client could not open a data connection to any Datanode holding that block.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:1879

          throws IOException, UnsupportedOperationException {
    if (maxLength == 0) {
      return EMPTY_BUFFER;
    } else if (maxLength < 0) {
      throw new IllegalArgumentException("can't read a negative " +
          "number of bytes.");
    }
    if ((blockReader == null) || (blockEnd == -1)) {
      if (pos >= getFileLength()) {
        return null;
      }
      /*
       * If we don't have a blockReader, or the one we have has no more bytes
       * left to read, we call seekToBlockSource to get a new blockReader and
       * recalculate blockEnd.  Note that we assume we're not at EOF here
       * (we check this above).
       */
      if ((!seekToBlockSource(pos)) || (blockReader == null)) {
        throw new IOException("failed to allocate new BlockReader " +
            "at position " + pos);
      }
    }
    ByteBuffer buffer = null;
    if (dfsClient.getConf().getShortCircuitConf().isShortCircuitMmapEnabled()) {
      buffer = tryReadZeroCopy(maxLength, opts);
    }
    if (buffer != null) {
      return buffer;
    }
    buffer = ByteBufferUtil.fallbackRead(this, bufferPool, maxLength);
    if (buffer != null) {
      getExtendedReadBuffers().put(buffer, bufferPool);
    }
    return buffer;
  }

  private synchronized ByteBuffer tryReadZeroCopy(int maxLength,

View on GitHub (pinned to 2add963021)

Solutions

1. Check Datanode liveness and connectivity: hdfs dfsadmin -report, and verify you can reach <dn-host>:9866 from the client host.
2. If short-circuit reads are on, verify dfs.domain.socket.path exists on the DNs and is writable by the client user; as a workaround set dfs.client.read.shortcircuit=false.
3. Run hdfs fsck <path> -files -blocks -locations to confirm the block has healthy, current locations.
4. Retry the open/read with backoff - stale locations refresh and dead DNs get excluded, so a fresh open often succeeds.

List<LocatedBlock> blocks = ((DistributedFileSystem) fs).getClient()
    .getLocatedBlocks(path, pos, 1).getLocatedBlocks();
boolean allReplicasDead = !blocks.isEmpty()
    && Stream.of(blocks.get(0).getLocations()).allMatch(DatanodeInfo::isDecommitted
        /* or your own reachability check */);
// treat allReplicasDead as 'do not even try yet - alert instead'

for (int attempt = 1; attempt <= 3; attempt++) {
  try {
    return doRead(fs, path, pos);
  } catch (IOException e) {
    if (!String.valueOf(e.getMessage()).contains("failed to allocate new BlockReader")) throw e;
    Thread.sleep(500L * attempt); // dead DN gets excluded / topology refreshes
    if (attempt == 3) {
      try (FSDataInputStream fresh = fs.open(path)) { fresh.seek(pos); return doRead(fresh, pos); }
    }
  }
}

When it happens

Trigger: Every replica Datanode of the target block is down, unreachable (network/iptables/security groups), or refuses the client; block locations returned by the NameNode are stale after DNs were decommissioned; short-circuit reader creation fails on every DN (missing/permission-broken dfs.domain.socket.path) and TCP fallback also fails.

Common situations: DN outage or rolling restart while a long-running reader holds the stream; misconfigured dfs.domain.socket.path (shared-memory domain socket dir not writable by the client user) with dfs.client.read.shortcircuit=true; firewall rules blocking the DN data port (9866); rack/network partition between client and DNs.

Related errors

• BlockReader failed to seek to {}. Instead, it seeked to {}.
• there is no shared memory segment registered with shmId {}
• Unable to close file because dfsclient was unable to contac
• Unable to close file because the last block {} does not have
• The BlockReader is null. The BlockReader creation failed or

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3d3d3a0edce0cf00. Report an issue: GitHub.