{"record":{"id":"d3446f0bc2df0b6e","repo":"apache/hadoop","slug":"first-key-length-out-of-range-firstkeylength","errorCode":null,"errorMessage":"First key length out of range: {firstKeyLength}","messagePattern":"First key length out of range: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java","lineNumber":2194,"sourceCode":"      // entryCount is derived from the file; only use it as a capacity hint,\n      // bounded, so a corrupt value cannot force a huge pre-allocation. The\n      // loops below are limited by the actual bytes available in the stream.\n      int capacityHint = Math.max(0, Math.min(entryCount, INDEX_CAPACITY_HINT_CAP));\n      index = new ArrayList<>(capacityHint);\n      recordNumIndex = new ArrayList<>(capacityHint);\n      int size = Utils.readVInt(in); // size for the first key entry.\n      if (size > 0) {\n        if (size > MAX_INDEX_ENTRY_SIZE) {\n          throw new IOException(\"First key entry size out of range: \" + size);\n        }\n        byte[] buffer = new byte[size];\n        in.readFully(buffer);\n        DataInputStream firstKeyInputStream =\n            new DataInputStream(new ByteArrayInputStream(buffer, 0, size));\n\n        int firstKeyLength = Utils.readVInt(firstKeyInputStream);\n        if (firstKeyLength < 0 || firstKeyLength > MAX_KEY_SIZE) {\n          throw new IOException(\"First key length out of range: \"\n              + firstKeyLength);\n        }\n        firstKey = new ByteArray(new byte[firstKeyLength]);\n        firstKeyInputStream.readFully(firstKey.buffer());\n\n        for (int i = 0; i < entryCount; i++) {\n          size = Utils.readVInt(in);\n          if (size < 0 || size > MAX_INDEX_ENTRY_SIZE) {\n            throw new IOException(\"Index entry size out of range: \" + size);\n          }\n          if (buffer.length < size) {\n            buffer = new byte[size];\n          }\n          in.readFully(buffer, 0, size);\n          TFileIndexEntry idx =\n              new TFileIndexEntry(new DataInputStream(new ByteArrayInputStream(\n                  buffer, 0, size)));\n          index.add(idx);","sourceCodeStart":2176,"sourceCodeEnd":2212,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java#L2176-L2212","documentation":"IOException from TFileIndex(DataInput) while parsing inside the first-key entry blob: after size bytes are read and wrapped in a ByteArrayInputStream, the VInt firstKeyLength decoded from that blob is negative or exceeds MAX_KEY_SIZE (64KB). This indirection (size -> blob -> key length) means the blob itself is corrupt even though its outer size passed the earlier check.","triggerScenarios":"A first-key entry whose outer size is within 64KB+16 but whose inner VInt decodes to a negative value or > 65536 — random bytes where the length prefix should be, from truncation, bit rot, or a misaligned index read.","commonSituations":"Same family as other index corruption: interrupted close, bad replicas, reading with a wrong fileLength. Can also appear when a file written by a non-standard TFile implementation stores a different first-key layout.","solutions":["Restore the file from a healthy replica or regenerate it from source data.","Verify integrity at the storage layer (checksums/fsck) before blaming the library.","Ensure only genuine TFiles (same version) are opened by this reader."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  TFile.Reader r = new TFile.Reader(fsdis, fileLength, conf);\n} catch (IOException e) {\n  // first-key entry corrupt: quarantine and rebuild file\n}","preventionTips":["Verify storage-level checksums for archival TFiles.","Keep first keys modest in size; the first-key entry must fit MAX_INDEX_ENTRY_SIZE.","Treat any index parse IOException as file-level corruption, not a code bug."],"tags":["tfile","hadoop-common","index","key-length","data-corruption"],"backgroundTag":"malformed-file-format","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}