apache/hadoop · error · RuntimeException

Number too large to be represented as Integer

Error message

Number too large to be represented as Integer

What it means

RuntimeException from Utils.readVInt(DataInput): readVLong decodes fine, but the resulting long falls outside [Integer.MIN_VALUE, Integer.MAX_VALUE], so it cannot be returned as int. TFile uses readVInt for every length/count prefix (key lengths, index sizes, entry counts), so this fires when a corrupt stream decodes to a value that needs more than 32 bits where a small length was expected.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/Utils.java:181

        throw new RuntimeException("Internal error");
    }
  }

  /**
   * Decoding the variable-length integer. Synonymous to
   * <code>(int)Utils#readVLong(in)</code>.
   * 
   * @param in
   *          input stream
   * @return the decoded integer
   * @throws IOException raised on errors performing I/O.
   * 
   * @see Utils#readVLong(DataInput)
   */
  public static int readVInt(DataInput in) throws IOException {
    long ret = readVLong(in);
    if ((ret > Integer.MAX_VALUE) || (ret < Integer.MIN_VALUE)) {
      throw new RuntimeException(
          "Number too large to be represented as Integer");
    }
    return (int) ret;
  }

  /**
   * Decoding the variable-length integer. Suppose the value of the first byte
   * is FB, and the following bytes are NB[*].
   * <ul>
   * <li>if (FB &gt;= -32), return (long)FB;
   * <li>if (FB in [-72, -33]), return (FB+52)&lt;&lt;8 + NB[0]&amp;0xff;
   * <li>if (FB in [-104, -73]), return (FB+88)&lt;&lt;16 +
   * (NB[0]&amp;0xff)&lt;&lt;8 + NB[1]&amp;0xff;
   * <li>if (FB in [-120, -105]), return (FB+112)&lt;&lt;24 + (NB[0]&amp;0xff)
   * &lt;&lt;16 + (NB[1]&amp;0xff)&lt;&lt;8 + NB[2]&amp;0xff;
   * <li>if (FB in [-128, -121]), return interpret NB[FB+129] as a signed
   * big-endian integer.
   * </ul>

View on GitHub (pinned to 2add963021)

Solutions

1. Treat as corruption: fsck/restore the file, or regenerate from source.
2. If you call Utils.readVInt yourself on external data, switch to Utils.readVLong when values may legitimately exceed int range.
3. Catch RuntimeException (not IOException) around TFile parsing if you quarantine bad files in a batch pipeline.

Example fix

// before (calling readVInt on data whose values can exceed int)
int n = Utils.readVInt(in);
// after
long n = Utils.readVLong(in);

try {
  TFile.Reader r = new TFile.Reader(fsdis, fileLength, conf);
} catch (RuntimeException e) {
  if (e.getMessage().contains("Integer")) { // readVInt overflow => corrupt stream
    quarantine(path);
  } else throw e;
}

When it happens

Trigger: Any TFile parse path that hits a VInt prefix whose 5-to-9-byte encoding decodes above 2^31-1 or below -2^31: classic signature of random bytes being interpreted as a length (desynchronized block reader, corrupt data), since legitimate TFile lengths are tiny.

Common situations: Corrupt or truncated files where the reader is decoding garbage as varints; opening a non-TFile stream; partial writes followed by reads. Rare with healthy files because real lengths never approach 2^31.

Related errors

• Corrupted VLong encoding
• Key length out of range: {klen}
• Cannot find matching key in block.
• First key entry size out of range: {size}
• First key length out of range: {firstKeyLength}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/95f274518917b0cf. Report an issue: GitHub.