apache/hadoop · error · IOException

truncated return from reader.read(): excpected {}, got {}

Error message

truncated return from reader.read(): excpected {}, got {}

What it means

After reading from a DN, the client requires the total bytes read to equal the requested length; a short read without an error or EOF marker throws this IOException (note the upstream typo 'excpected'). It means the DN delivered fewer bytes than the block metadata says are there - typically a truncated or corrupt replica on that DN. The outer fetchBlockByteRange handler records the failure and tries the next replica; it surfaces only after replicas are exhausted.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:1247

        tmp = tmp.slice();
        long beginReadMS = Time.monotonicNow();
        int nread = 0;
        int ret;
        while (true) {
          ret = reader.read(tmp);
          if (ret <= 0) {
            break;
          }
          nread += ret;
        }
        long readTimeMS = Time.monotonicNow() - beginReadMS;
        buf.position(buf.position() + nread);

        IOUtilsClient.updateReadStatistics(readStatistics, nread, reader);
        dfsClient.updateFileSystemReadStats(
            reader.getNetworkDistance(), nread, readTimeMS);
        if (nread != len) {
          throw new IOException("truncated return from reader.read(): " +
              "excpected " + len + ", got " + nread);
        }
        DFSClientFaultInjector.get().readFromDatanodeDelay();
        return;
      } catch (ChecksumException e) {
        String msg = "fetchBlockByteRange(). Got a checksum exception for "
            + src + " at " + block.getBlock() + ":" + e.getPos() + " from "
            + datanode.info;
        DFSClient.LOG.warn(msg);
        // we want to remember what we have tried
        corruptedBlocks.addCorruptedBlock(block.getBlock(), datanode.info);
        addToLocalDeadNodes(datanode.info);
        throw new IOException(msg);
      } catch (IOException e) {
        checkInterrupted(e);
        if (e instanceof InvalidEncryptionKeyException && refetchEncryptionKey > 0) {
          DFSClient.LOG.info("Will fetch a new encryption key and retry, "
              + "encryption key was invalid when connecting to " + datanode.addr

View on GitHub (pinned to 2add963021)

Solutions

1. Rely on automatic failover to other replicas first; escalate only if it persists
2. Run hdfs fsck to identify the short replica and remove it (fsck -delete/-move) so a good copy re-replicates
3. Inspect the suspect DN's disks and logs; evict it for maintenance if faulty

static boolean isTruncatedRead(IOException e) {
  return e.getMessage() != null && e.getMessage().contains("truncated return from reader.read()");
}

for (int i = 0; i < 2; i++) {
  try { in.readFully(buf, off, len); return; }
  catch (IOException e) {
    if (i == 1) throw e;
    // next attempt picks a different replica automatically
  }
}

When it happens

Trigger: Replica on the serving DN is physically shorter than its metadata (truncated block file); DN returned a short read due to disk fault or restart mid-read.

Common situations: Disks dying under dfs.datanode.data.dir; blocks damaged by partial writes during crashes; repeated reads pinned to the same bad replica.

Related errors

• Corrupted replica {replicaInfo} with a length of {numBytes}
• Corrupted replica {replicaInfo} with a length of {replicaLen
• Found fewer bytesOnDisk than bytesAcked for replica {rbw}
• Unexpected EOS from the reader
• Version Mismatch (Expected: {}, Received: {} )

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e56f79a533b1c119. Report an issue: GitHub.