What causes "Unexpected EOS from the reader" in apache/hadoop?

Let the built-in failover run (it retries other replicas); only escalate if it persists

apache/hadoop · error · IOException

Unexpected EOS from the reader

Error message

Unexpected EOS from the reader

What it means

The client expected data at the current position (pos < file length per NN metadata) but readBuffer() returned a negative result: the DataNode closed the block reader before delivering the promised bytes - a short/truncated replica on that DN. The surrounding readWithStrategy loop catches this IOException and fails over to other replicas; it escapes only when the read keeps failing across attempts.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:903

          // error on the same block. See HDFS-3067
          if (pos > blockEnd || currentNode == null) {
            currentNode = blockSeekTo(pos);
          }
          int realLen = (int) Math.min(len, (blockEnd - pos + 1L));
          synchronized(infoLock) {
            if (locatedBlocks.isLastBlockComplete()) {
              realLen = (int) Math.min(realLen,
                  locatedBlocks.getFileLength() - pos);
            }
          }
          long beginReadMS = Time.monotonicNow();
          int result = readBuffer(strategy, realLen, corruptedBlocks, exceptionMap);
          long readTimeMS = Time.monotonicNow() - beginReadMS;
          if (result >= 0) {
            pos += result;
          } else {
            // got a EOS from reader though we expect more data on it.
            throw new IOException("Unexpected EOS from the reader");
          }
          updateReadStatistics(readStatistics, result, blockReader);
          dfsClient.updateFileSystemReadStats(blockReader.getNetworkDistance(),
              result, readTimeMS);
          return result;
        } catch (ChecksumException ce) {
          throw ce;
        } catch (IOException e) {
          checkInterrupted(e);
          if (retries == 1) {
            DFSClient.LOG.warn("DFS Read", e);
          }
          blockEnd = -1;
          if (currentNode != null) {
            addToLocalDeadNodes(currentNode);
            dfsClient.addNodeToDeadNodeDetector(this, currentNode);
          }
          if (--retries == 0) {

View on GitHub (pinned to 2add963021)

Solutions

Let the built-in failover run (it retries other replicas); only escalate if it persists
Run hdfs fsck -files -blocks -locations on the file to find the truncated replica
Delete/move the bad replica (hdfs fsck -delete or -move) so the NN re-replicates from a good copy
Check the suspect DataNode's disks and logs for hardware faults

Defensive patterns

Strategy: retry

Type guard

static boolean isUnexpectedEos(IOException e) {
  return e.getMessage() != null && e.getMessage().equals("Unexpected EOS from the reader");
}

Try / catch

for (int i = 0; i < 3; i++) {
  try { return readRange(in, off, len); }
  catch (IOException e) {
    if (i == 2) throw e; // client already failed over across replicas
    Thread.sleep(2000L);
  }
}

Prevention

Trust but verify: the client retries other replicas; only investigate after repeated failures
Schedule hdfs fsck -block checks so truncated replicas are found before readers hit them
Monitor DN disk health to catch failing drives early

When it happens

Trigger: A replica on the chosen DN is physically shorter than the NameNode's recorded block length (truncated file on that DN, partial write, disk fault); DN restarted mid-read after losing the replica.

Common situations: DN disk corruption or manual tampering under dfs.data.dir; blocks partially written during crashes then bad-handled; recurring reads from one bad replica.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6420c9362ef1a331. Report an issue: GitHub.