{"record":{"id":"c3e8ce1fa7d629c8","repo":"apache/hadoop","slug":"index-entry-key-length-out-of-range-len","errorCode":null,"errorMessage":"Index entry key length out of range: {len}","messagePattern":"Index entry key length out of range: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java","lineNumber":2348,"sourceCode":"        Utils.writeVInt(out, dob.getLength());\n        out.write(dob.getData(), 0, dob.getLength());\n      }\n    }\n  }\n\n  /**\n   * TFile Data Index entry. We should try to make the memory footprint of each\n   * index entry as small as possible.\n   */\n  static final class TFileIndexEntry implements RawComparable {\n    final byte[] key;\n    // count of <key, value> entries in the block.\n    final long kvEntries;\n\n    public TFileIndexEntry(DataInput in) throws IOException {\n      int len = Utils.readVInt(in);\n      if (len < 0 || len > MAX_KEY_SIZE) {\n        throw new IOException(\"Index entry key length out of range: \" + len);\n      }\n      key = new byte[len];\n      in.readFully(key, 0, len);\n      kvEntries = Utils.readVLong(in);\n    }\n\n    // default entry, without any padding\n    public TFileIndexEntry(byte[] newkey, int offset, int len, long entries) {\n      key = new byte[len];\n      System.arraycopy(newkey, offset, key, 0, len);\n      this.kvEntries = entries;\n    }\n\n    @Override\n    public byte[] buffer() {\n      return key;\n    }\n","sourceCodeStart":2330,"sourceCodeEnd":2366,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java#L2330-L2366","documentation":"IOException from TFileIndexEntry(DataInput), which decodes one block-index entry: the VInt key length must satisfy 0 <= len <= MAX_KEY_SIZE (64KB) before the key bytes are read. An out-of-range length means this index entry's bytes are corrupt — allocation is refused before readFully, so bogus lengths fail here instead of exhausting memory.","triggerScenarios":"Parsing any of the per-block index entries during TFile.Reader construction where the length prefix decodes negative or above 65536: corrupted index region, wrong stream position, or a file whose index was truncated mid-entry.","commonSituations":"Block-level index corruption on large files, transfers that truncate the index tail, or nonconforming writer implementations.","solutions":["Restore the file from a healthy replica or regenerate from source; index entries cannot be reconstituted.","fsck / verify checksums at the storage layer before re-reading.","Ensure complete file transfers and atomic publishes so index tails are never partial."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  TFile.Reader r = new TFile.Reader(fsdis, fileLength, conf);\n} catch (IOException e) {\n  // index entry key length undecodable: restore/regenerate file\n}","preventionTips":["Cap block first-key sizes at 64KB when choosing block boundaries/index keys.","fsck archived files periodically; index entry corruption makes the whole file unopenable.","Atomic publish + verified transfers keep index entries intact."],"tags":["tfile","hadoop-common","index","key-length","data-corruption"],"backgroundTag":"malformed-file-format","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}