{"record":{"id":"a0268e92fc313786","repo":"apache/hadoop","slug":"the-blockreader-is-null-the-blockreader-creation","errorCode":null,"errorMessage":"The BlockReader is null. The BlockReader creation failed or the reader hit exception.","messagePattern":"The BlockReader is null\\. The BlockReader creation failed or the reader hit exception\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java","lineNumber":281,"sourceCode":"          + currentNode, e);\n      //Clear buffer to make next decode success\n      strategy.getReadBuffer().clear();\n      if (blockReader != null) {\n        blockReader.close();\n      }\n      throw e;\n    }\n  }\n\n  private Callable<BlockReadStats> readCells(final BlockReader reader,\n      final DatanodeInfo datanode, final long currentReaderOffset,\n      final long targetReaderOffset, final ByteBufferStrategy[] strategies,\n      final ExtendedBlock currentBlock) {\n    return () -> {\n      // reader can be null if getBlockReaderWithRetry failed or\n      // the reader hit exception before\n      if (reader == null) {\n        throw new IOException(\"The BlockReader is null. \" +\n            \"The BlockReader creation failed or the reader hit exception.\");\n      }\n      Preconditions.checkState(currentReaderOffset <= targetReaderOffset);\n      if (currentReaderOffset < targetReaderOffset) {\n        long skipped = reader.skip(targetReaderOffset - currentReaderOffset);\n        Preconditions.checkState(\n            skipped == targetReaderOffset - currentReaderOffset);\n      }\n\n      int ret = 0;\n      for (ByteBufferStrategy strategy : strategies) {\n        int bytesReead = readToBuffer(reader, datanode, strategy, currentBlock);\n        ret += bytesReead;\n      }\n      return new BlockReadStats(ret, reader.isShortCircuit(),\n          reader.getNetworkDistance());\n    };\n  }","sourceCodeStart":263,"sourceCodeEnd":299,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java#L263-L299","documentation":"Thrown by the erasure-coded (striped) read path in DFSStripedInputStream. StripeReader.readCells receives a BlockReader that is null because reader creation failed (getBlockReaderWithRetry exhausted) or an earlier exception invalidated the reader for that internal block. Since no reader exists for the cell, the striped read cannot proceed and the IOException propagates to the caller.","triggerScenarios":"Reading a file stored with an EC policy (e.g. RS-6-3) when the datanode hosting an internal block is unreachable (connection refused/timeout), too busy, or the reader was closed after a prior IOException; a later chunk task then runs with the null reader and throws this.","commonSituations":"Datanode outage or rolling restart during EC file reads; network partition between client and datanodes; datanodes with exhausted xceiver threads; heavy cluster load causing reader setup timeouts.","solutions":["Verify datanode health with hdfs dfsadmin -report and check the file with hdfs fsck /path -files -blocks -locations","Retry the read job: the HDFS client retries against alternate replicas, so transient datanode failures usually clear on the next attempt","Check client-to-datanode connectivity (port 9866 by default) and DataNode logs for handshake or xceiver errors","If one datanode is consistently bad, restart it or take it out of rotation until its internal blocks are re-replicated"],"exampleFix":"// before: single read attempt, fails if a datanode blips\ntry (FSDataInputStream in = fs.open(ecPath)) {\n  IOUtils.readFully(in, buf, 0, buf.length);\n}\n\n// after: retry with backoff so a new stream re-picks healthy datanodes\nfor (int i = 0; ; i++) {\n  try (FSDataInputStream in = fs.open(ecPath)) {\n    IOUtils.readFully(in, buf, 0, buf.length);\n    break;\n  } catch (IOException e) {\n    if (i == maxRetries - 1) throw e;\n    Thread.sleep(1000L << i);\n  }\n}","handlingStrategy":"retry","validationCode":"// Pre-check that the internal blocks of the EC file have reachable replicas\nDistributedFileSystem dfs = (DistributedFileSystem) FileSystem.get(conf);\nLocatedBlock last = dfs.getClient().getLocatedBlocks(ecPath.toString(), 0)\n    .getLastLocatedBlock();\nboolean allReachable = last.getBlockIndices().stream()\n    .flatMap(b -> Arrays.stream(b.getBlockLocations()))\n    .map(l -> l.getHosts().length > 0)\n    .reduce(true, Boolean::logicalAnd);\nif (!allReachable) throw new IOException(\"missing replicas for \" + ecPath);","typeGuard":null,"tryCatchPattern":"try {\n  readStriped(fs.open(ecPath));\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"BlockReader is null\")) {\n    // transient datanode failure: a fresh stream re-picks replicas\n    readStriped(fs.open(ecPath));\n  } else {\n    throw e;\n  }\n}","preventionTips":["Monitor datanode liveness and network health so EC reads always have reachable replicas","Keep EC internal blocks fully replicated; run hdfs fsck periodically on EC datasets","Wrap striped reads in application-level retry with backoff instead of assuming client retries cover everything"],"tags":["hdfs","erasure-coding","block-reader","datanode","io"],"backgroundTag":"datanode-connection-failure","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}