apache/hadoop · error · IOException

The BlockReader is null. The BlockReader creation failed or

Error message

The BlockReader is null. The BlockReader creation failed or the reader hit exception.

What it means

Thrown by the erasure-coded (striped) read path in DFSStripedInputStream. StripeReader.readCells receives a BlockReader that is null because reader creation failed (getBlockReaderWithRetry exhausted) or an earlier exception invalidated the reader for that internal block. Since no reader exists for the cell, the striped read cannot proceed and the IOException propagates to the caller.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java:281

          + currentNode, e);
      //Clear buffer to make next decode success
      strategy.getReadBuffer().clear();
      if (blockReader != null) {
        blockReader.close();
      }
      throw e;
    }
  }

  private Callable<BlockReadStats> readCells(final BlockReader reader,
      final DatanodeInfo datanode, final long currentReaderOffset,
      final long targetReaderOffset, final ByteBufferStrategy[] strategies,
      final ExtendedBlock currentBlock) {
    return () -> {
      // reader can be null if getBlockReaderWithRetry failed or
      // the reader hit exception before
      if (reader == null) {
        throw new IOException("The BlockReader is null. " +
            "The BlockReader creation failed or the reader hit exception.");
      }
      Preconditions.checkState(currentReaderOffset <= targetReaderOffset);
      if (currentReaderOffset < targetReaderOffset) {
        long skipped = reader.skip(targetReaderOffset - currentReaderOffset);
        Preconditions.checkState(
            skipped == targetReaderOffset - currentReaderOffset);
      }

      int ret = 0;
      for (ByteBufferStrategy strategy : strategies) {
        int bytesReead = readToBuffer(reader, datanode, strategy, currentBlock);
        ret += bytesReead;
      }
      return new BlockReadStats(ret, reader.isShortCircuit(),
          reader.getNetworkDistance());
    };
  }

View on GitHub (pinned to 2add963021)

Solutions

Verify datanode health with hdfs dfsadmin -report and check the file with hdfs fsck /path -files -blocks -locations
Retry the read job: the HDFS client retries against alternate replicas, so transient datanode failures usually clear on the next attempt
Check client-to-datanode connectivity (port 9866 by default) and DataNode logs for handshake or xceiver errors
If one datanode is consistently bad, restart it or take it out of rotation until its internal blocks are re-replicated

Example fix

// before: single read attempt, fails if a datanode blips
try (FSDataInputStream in = fs.open(ecPath)) {
  IOUtils.readFully(in, buf, 0, buf.length);
}

// after: retry with backoff so a new stream re-picks healthy datanodes
for (int i = 0; ; i++) {
  try (FSDataInputStream in = fs.open(ecPath)) {
    IOUtils.readFully(in, buf, 0, buf.length);
    break;
  } catch (IOException e) {
    if (i == maxRetries - 1) throw e;
    Thread.sleep(1000L << i);
  }
}

Defensive patterns

Strategy: retry

Validate before calling

// Pre-check that the internal blocks of the EC file have reachable replicas
DistributedFileSystem dfs = (DistributedFileSystem) FileSystem.get(conf);
LocatedBlock last = dfs.getClient().getLocatedBlocks(ecPath.toString(), 0)
    .getLastLocatedBlock();
boolean allReachable = last.getBlockIndices().stream()
    .flatMap(b -> Arrays.stream(b.getBlockLocations()))
    .map(l -> l.getHosts().length > 0)
    .reduce(true, Boolean::logicalAnd);
if (!allReachable) throw new IOException("missing replicas for " + ecPath);

Try / catch

try {
  readStriped(fs.open(ecPath));
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("BlockReader is null")) {
    // transient datanode failure: a fresh stream re-picks replicas
    readStriped(fs.open(ecPath));
  } else {
    throw e;
  }
}

Prevention

Monitor datanode liveness and network health so EC reads always have reachable replicas
Keep EC internal blocks fully replicated; run hdfs fsck periodically on EC datasets
Wrap striped reads in application-level retry with backoff instead of assuming client retries cover everything

When it happens

Trigger: Reading a file stored with an EC policy (e.g. RS-6-3) when the datanode hosting an internal block is unreachable (connection refused/timeout), too busy, or the reader was closed after a prior IOException; a later chunk task then runs with the null reader and throws this.

Common situations: Datanode outage or rolling restart during EC file reads; network partition between client and datanodes; datanodes with exhausted xceiver threads; heavy cluster load causing reader setup timeouts.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a0268e92fc313786. Report an issue: GitHub.