{"record":{"id":"7ccbf7e4f1351eb6","repo":"apache/hadoop","slug":"index-entry-size-out-of-range-size","errorCode":null,"errorMessage":"Index entry size out of range: {size}","messagePattern":"Index entry size out of range: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java","lineNumber":2203,"sourceCode":"          throw new IOException(\"First key entry size out of range: \" + size);\n        }\n        byte[] buffer = new byte[size];\n        in.readFully(buffer);\n        DataInputStream firstKeyInputStream =\n            new DataInputStream(new ByteArrayInputStream(buffer, 0, size));\n\n        int firstKeyLength = Utils.readVInt(firstKeyInputStream);\n        if (firstKeyLength < 0 || firstKeyLength > MAX_KEY_SIZE) {\n          throw new IOException(\"First key length out of range: \"\n              + firstKeyLength);\n        }\n        firstKey = new ByteArray(new byte[firstKeyLength]);\n        firstKeyInputStream.readFully(firstKey.buffer());\n\n        for (int i = 0; i < entryCount; i++) {\n          size = Utils.readVInt(in);\n          if (size < 0 || size > MAX_INDEX_ENTRY_SIZE) {\n            throw new IOException(\"Index entry size out of range: \" + size);\n          }\n          if (buffer.length < size) {\n            buffer = new byte[size];\n          }\n          in.readFully(buffer, 0, size);\n          TFileIndexEntry idx =\n              new TFileIndexEntry(new DataInputStream(new ByteArrayInputStream(\n                  buffer, 0, size)));\n          index.add(idx);\n          sum += idx.entries();\n          recordNumIndex.add(sum);\n        }\n      } else {\n        if (entryCount != 0) {\n          throw new RuntimeException(\"Internal error\");\n        }\n      }\n      this.comparator = comparator;","sourceCodeStart":2185,"sourceCodeEnd":2221,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java#L2185-L2221","documentation":"IOException from the per-block loop of TFileIndex(DataInput): for one of the entryCount index entries, the VInt size decodes to a negative value or exceeds MAX_INDEX_ENTRY_SIZE (64KB + 16). Each entry describes one data block (its first key plus entry count), so an out-of-range size means the block index region is corrupt from that entry onward.","triggerScenarios":"Reading a TFile whose block-index array is damaged past the first entry: partial index writes, mid-file corruption, or a truncated file where the loop tries to read more entries than the bytes present (subsequent readFully would fail, but a bogus size fails here first). Note entryCount is only a loop bound; garbage sizes are caught per-entry.","commonSituations":"Large multi-block files where corruption lands mid-index; killed jobs; files copied with interrupted transfers that nonetheless preserved length fields.","solutions":["Restore or regenerate the file; the index is not repairable in place.","Verify transfers complete (length + checksum) before publishing a TFile.","Keep writes atomic (temp file + rename) so partial indexes are never visible."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  TFile.Reader r = new TFile.Reader(fsdis, fileLength, conf);\n} catch (IOException e) {\n  // block-index entry undecodable: restore/regenerate\n}","preventionTips":["Atomic publish (temp + rename) protects the multi-entry block index from partial writes.","Verify transfer completeness (length + checksum) when copying TFiles between stores.","Batch readers should catch IOException per file and continue, so one corrupt index doesn't kill the job."],"tags":["tfile","hadoop-common","index","data-corruption","io"],"backgroundTag":"malformed-file-format","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}