{"record":{"id":"e56f79a533b1c119","repo":"apache/hadoop","slug":"truncated-return-from-reader-read-excpected","errorCode":null,"errorMessage":"truncated return from reader.read(): excpected {}, got {}","messagePattern":"truncated return from reader\\.read\\(\\): excpected (.+?), got (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java","lineNumber":1247,"sourceCode":"        tmp = tmp.slice();\n        long beginReadMS = Time.monotonicNow();\n        int nread = 0;\n        int ret;\n        while (true) {\n          ret = reader.read(tmp);\n          if (ret <= 0) {\n            break;\n          }\n          nread += ret;\n        }\n        long readTimeMS = Time.monotonicNow() - beginReadMS;\n        buf.position(buf.position() + nread);\n\n        IOUtilsClient.updateReadStatistics(readStatistics, nread, reader);\n        dfsClient.updateFileSystemReadStats(\n            reader.getNetworkDistance(), nread, readTimeMS);\n        if (nread != len) {\n          throw new IOException(\"truncated return from reader.read(): \" +\n              \"excpected \" + len + \", got \" + nread);\n        }\n        DFSClientFaultInjector.get().readFromDatanodeDelay();\n        return;\n      } catch (ChecksumException e) {\n        String msg = \"fetchBlockByteRange(). Got a checksum exception for \"\n            + src + \" at \" + block.getBlock() + \":\" + e.getPos() + \" from \"\n            + datanode.info;\n        DFSClient.LOG.warn(msg);\n        // we want to remember what we have tried\n        corruptedBlocks.addCorruptedBlock(block.getBlock(), datanode.info);\n        addToLocalDeadNodes(datanode.info);\n        throw new IOException(msg);\n      } catch (IOException e) {\n        checkInterrupted(e);\n        if (e instanceof InvalidEncryptionKeyException && refetchEncryptionKey > 0) {\n          DFSClient.LOG.info(\"Will fetch a new encryption key and retry, \"\n              + \"encryption key was invalid when connecting to \" + datanode.addr","sourceCodeStart":1229,"sourceCodeEnd":1265,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1229-L1265","documentation":"After reading from a DN, the client requires the total bytes read to equal the requested length; a short read without an error or EOF marker throws this IOException (note the upstream typo 'excpected'). It means the DN delivered fewer bytes than the block metadata says are there - typically a truncated or corrupt replica on that DN. The outer fetchBlockByteRange handler records the failure and tries the next replica; it surfaces only after replicas are exhausted.","triggerScenarios":"Replica on the serving DN is physically shorter than its metadata (truncated block file); DN returned a short read due to disk fault or restart mid-read.","commonSituations":"Disks dying under dfs.datanode.data.dir; blocks damaged by partial writes during crashes; repeated reads pinned to the same bad replica.","solutions":["Rely on automatic failover to other replicas first; escalate only if it persists","Run hdfs fsck to identify the short replica and remove it (fsck -delete/-move) so a good copy re-replicates","Inspect the suspect DN's disks and logs; evict it for maintenance if faulty"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":"static boolean isTruncatedRead(IOException e) {\n  return e.getMessage() != null && e.getMessage().contains(\"truncated return from reader.read()\");\n}","tryCatchPattern":"for (int i = 0; i < 2; i++) {\n  try { in.readFully(buf, off, len); return; }\n  catch (IOException e) {\n    if (i == 1) throw e;\n    // next attempt picks a different replica automatically\n  }\n}","preventionTips":["Let the client's replica failover run before escalating","Use hdfs fsck to find and evict short replicas","Replace failing DN disks; monitor for short/block-file vs metadata mismatches"],"tags":["hdfs","truncated-read","replica-corruption","datanode"],"backgroundTag":"truncated-read","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}