{"record":{"id":"6420c9362ef1a331","repo":"apache/hadoop","slug":"unexpected-eos-from-the-reader","errorCode":null,"errorMessage":"Unexpected EOS from the reader","messagePattern":"Unexpected EOS from the reader","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java","lineNumber":903,"sourceCode":"          // error on the same block. See HDFS-3067\n          if (pos > blockEnd || currentNode == null) {\n            currentNode = blockSeekTo(pos);\n          }\n          int realLen = (int) Math.min(len, (blockEnd - pos + 1L));\n          synchronized(infoLock) {\n            if (locatedBlocks.isLastBlockComplete()) {\n              realLen = (int) Math.min(realLen,\n                  locatedBlocks.getFileLength() - pos);\n            }\n          }\n          long beginReadMS = Time.monotonicNow();\n          int result = readBuffer(strategy, realLen, corruptedBlocks, exceptionMap);\n          long readTimeMS = Time.monotonicNow() - beginReadMS;\n          if (result >= 0) {\n            pos += result;\n          } else {\n            // got a EOS from reader though we expect more data on it.\n            throw new IOException(\"Unexpected EOS from the reader\");\n          }\n          updateReadStatistics(readStatistics, result, blockReader);\n          dfsClient.updateFileSystemReadStats(blockReader.getNetworkDistance(),\n              result, readTimeMS);\n          return result;\n        } catch (ChecksumException ce) {\n          throw ce;\n        } catch (IOException e) {\n          checkInterrupted(e);\n          if (retries == 1) {\n            DFSClient.LOG.warn(\"DFS Read\", e);\n          }\n          blockEnd = -1;\n          if (currentNode != null) {\n            addToLocalDeadNodes(currentNode);\n            dfsClient.addNodeToDeadNodeDetector(this, currentNode);\n          }\n          if (--retries == 0) {","sourceCodeStart":885,"sourceCodeEnd":921,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L885-L921","documentation":"The client expected data at the current position (pos < file length per NN metadata) but readBuffer() returned a negative result: the DataNode closed the block reader before delivering the promised bytes - a short/truncated replica on that DN. The surrounding readWithStrategy loop catches this IOException and fails over to other replicas; it escapes only when the read keeps failing across attempts.","triggerScenarios":"A replica on the chosen DN is physically shorter than the NameNode's recorded block length (truncated file on that DN, partial write, disk fault); DN restarted mid-read after losing the replica.","commonSituations":"DN disk corruption or manual tampering under dfs.data.dir; blocks partially written during crashes then bad-handled; recurring reads from one bad replica.","solutions":["Let the built-in failover run (it retries other replicas); only escalate if it persists","Run hdfs fsck -files -blocks -locations on the file to find the truncated replica","Delete/move the bad replica (hdfs fsck -delete or -move) so the NN re-replicates from a good copy","Check the suspect DataNode's disks and logs for hardware faults"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":"static boolean isUnexpectedEos(IOException e) {\n  return e.getMessage() != null && e.getMessage().equals(\"Unexpected EOS from the reader\");\n}","tryCatchPattern":"for (int i = 0; i < 3; i++) {\n  try { return readRange(in, off, len); }\n  catch (IOException e) {\n    if (i == 2) throw e; // client already failed over across replicas\n    Thread.sleep(2000L);\n  }\n}","preventionTips":["Trust but verify: the client retries other replicas; only investigate after repeated failures","Schedule hdfs fsck -block checks so truncated replicas are found before readers hit them","Monitor DN disk health to catch failing drives early"],"tags":["hdfs","replica-corruption","read","datanode"],"backgroundTag":"premature-end-of-stream","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}