apache/hadoop · error · IOException

Corrupted VLong encoding

Error message

Corrupted VLong encoding

What it means

IOException from Utils.readVLong(DataInput): the varint's first byte lands in the undefined part of the encoding space. Per the decoder, first bytes in [-128, -126] compute len = firstByte + 129 in {1,2,3}, which no case handles, so the default throws 'Corrupted VLong encoding'. Valid multi-byte forms start at firstByte -125 (len 4) up to -120 (len 8); shorter extended forms don't exist in this format.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/Utils.java:243

      case 1:
        return ((firstByte + 112) << 24) | (in.readUnsignedShort() << 8)
            | in.readUnsignedByte();
      case 0:
        int len = firstByte + 129;
        switch (len) {
          case 4:
            return in.readInt();
          case 5:
            return ((long) in.readInt()) << 8 | in.readUnsignedByte();
          case 6:
            return ((long) in.readInt()) << 16 | in.readUnsignedShort();
          case 7:
            return ((long) in.readInt()) << 24 | (in.readUnsignedShort() << 8)
                | in.readUnsignedByte();
          case 8:
            return in.readLong();
          default:
            throw new IOException("Corrupted VLong encoding");
        }
      default:
        throw new RuntimeException("Internal error");
    }
  }

  /**
   * Write a String as a VInt n, followed by n Bytes as in Text format.
   * 
   * @param out out.
   * @param s s.
   * @throws IOException raised on errors performing I/O.
   */
  public static void writeString(DataOutput out, String s) throws IOException {
    if (s != null) {
      Text text = new Text(s);
      byte[] buffer = text.getBytes();
      int len = text.getLength();

View on GitHub (pinned to 2add963021)

Solutions

1. Restore or regenerate the file — the byte stream is not decodable varint data.
2. If you produce TFiles with custom tooling, write length prefixes with Utils.writeVInt/writeVLong (not WritableUtils) so the encodings match.
3. Catch IOException around full-file parsing in batch jobs and quarantine the file.

Example fix

// before (writing prefixes with the wrong varint encoder)
org.apache.hadoop.io.WritableUtils.writeVInt(out, len);
// after
org.apache.hadoop.io.file.tfile.Utils.writeVInt(out, len);

try {
  TFile.Reader r = new TFile.Reader(fsdis, fileLength, conf);
} catch (IOException e) {
  if ("Corrupted VLong encoding".equals(e.getMessage())) { // bytes are not TFile varints
    quarantine(path);
  } else throw e;
}

When it happens

Trigger: Decoding a VLong/VInt prefix whose first byte is 0x80..0x82 (-128..-126): random bytes where a length was expected, mid-stream desynchronization, or truncated/garbled TFile regions. Because TFile prefixes every record and index element with varints, corruption surfaces here quickly.

Common situations: Bit rot and truncated files, reading a stream at the wrong offset, or interop code that assumed Hadoop Writable varint semantics (which differ: WritableUtils accepts different leading bytes) and wrote TFiles with the wrong encoder.

Related errors

• Number too large to be represented as Integer
• Key length out of range: {klen}
• Cannot find matching key in block.
• First key entry size out of range: {size}
• First key length out of range: {firstKeyLength}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/41efef990452a984. Report an issue: GitHub.