{"record":{"id":"58d050ec7284467b","repo":"apache/hadoop","slug":"first-key-entry-size-out-of-range-size","errorCode":null,"errorMessage":"First key entry size out of range: {size}","messagePattern":"First key entry size out of range: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java","lineNumber":2185,"sourceCode":"    private long sum = 0;\n    \n    /**\n     * For reading from file.\n     * \n     * @throws IOException\n     */\n    public TFileIndex(int entryCount, DataInput in, BytesComparator comparator)\n        throws IOException {\n      // entryCount is derived from the file; only use it as a capacity hint,\n      // bounded, so a corrupt value cannot force a huge pre-allocation. The\n      // loops below are limited by the actual bytes available in the stream.\n      int capacityHint = Math.max(0, Math.min(entryCount, INDEX_CAPACITY_HINT_CAP));\n      index = new ArrayList<>(capacityHint);\n      recordNumIndex = new ArrayList<>(capacityHint);\n      int size = Utils.readVInt(in); // size for the first key entry.\n      if (size > 0) {\n        if (size > MAX_INDEX_ENTRY_SIZE) {\n          throw new IOException(\"First key entry size out of range: \" + size);\n        }\n        byte[] buffer = new byte[size];\n        in.readFully(buffer);\n        DataInputStream firstKeyInputStream =\n            new DataInputStream(new ByteArrayInputStream(buffer, 0, size));\n\n        int firstKeyLength = Utils.readVInt(firstKeyInputStream);\n        if (firstKeyLength < 0 || firstKeyLength > MAX_KEY_SIZE) {\n          throw new IOException(\"First key length out of range: \"\n              + firstKeyLength);\n        }\n        firstKey = new ByteArray(new byte[firstKeyLength]);\n        firstKeyInputStream.readFully(firstKey.buffer());\n\n        for (int i = 0; i < entryCount; i++) {\n          size = Utils.readVInt(in);\n          if (size < 0 || size > MAX_INDEX_ENTRY_SIZE) {\n            throw new IOException(\"Index entry size out of range: \" + size);","sourceCodeStart":2167,"sourceCodeEnd":2203,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java#L2167-L2203","documentation":"IOException from the TFileIndex(DataInput) constructor: the VInt size of the first-key entry read from the file's index region is positive but exceeds MAX_INDEX_ENTRY_SIZE (64KB + 16, TFile.java:179). The first-key entry is a length-prefixed blob containing the first key of the file, so an out-of-range size means the index region is not decodable — corruption, truncation, or wrong read boundaries.","triggerScenarios":"new TFile.Reader(...) parsing a file whose index area is damaged: truncated tail (index lives near the end), bit rot, or a wrong fileLength passed to the Reader so the constructor reads the index from the wrong offset. The size check caps allocation before the readFully, so huge bogus sizes fail here rather than OOM.","commonSituations":"Killed writers (index is written at close), HDFS block corruption, reading a file while it is still being written, or code that reuses an FSDataInputStream without seeking to 0 and passes a stale length.","solutions":["fsck the file and restore from a healthy replica or the original source; regenerate if unreadable.","Ensure writers fully close before readers open, and pipelines don't expose partial files (write to a temp name, atomic rename).","Double-check the (stream, fileLength) pair handed to TFile.Reader matches the actual file."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  TFile.Reader r = new TFile.Reader(fsdis, fileLength, conf);\n} catch (IOException e) {\n  // index region undecodable: restore from replica / regenerate from source\n}","preventionTips":["Write to temp paths and atomically rename after close — the index is written last and is the most truncation-sensitive part.","fsck files before large read jobs.","Pass exact length and a fresh seek(0) stream to the Reader."],"tags":["tfile","hadoop-common","index","data-corruption","io"],"backgroundTag":"malformed-file-format","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}