{"record":{"id":"95f274518917b0cf","repo":"apache/hadoop","slug":"number-too-large-to-be-represented-as-integer","errorCode":null,"errorMessage":"Number too large to be represented as Integer","messagePattern":"Number too large to be represented as Integer","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/Utils.java","lineNumber":181,"sourceCode":"        throw new RuntimeException(\"Internal error\");\n    }\n  }\n\n  /**\n   * Decoding the variable-length integer. Synonymous to\n   * <code>(int)Utils#readVLong(in)</code>.\n   * \n   * @param in\n   *          input stream\n   * @return the decoded integer\n   * @throws IOException raised on errors performing I/O.\n   * \n   * @see Utils#readVLong(DataInput)\n   */\n  public static int readVInt(DataInput in) throws IOException {\n    long ret = readVLong(in);\n    if ((ret > Integer.MAX_VALUE) || (ret < Integer.MIN_VALUE)) {\n      throw new RuntimeException(\n          \"Number too large to be represented as Integer\");\n    }\n    return (int) ret;\n  }\n\n  /**\n   * Decoding the variable-length integer. Suppose the value of the first byte\n   * is FB, and the following bytes are NB[*].\n   * <ul>\n   * <li>if (FB &gt;= -32), return (long)FB;\n   * <li>if (FB in [-72, -33]), return (FB+52)&lt;&lt;8 + NB[0]&amp;0xff;\n   * <li>if (FB in [-104, -73]), return (FB+88)&lt;&lt;16 +\n   * (NB[0]&amp;0xff)&lt;&lt;8 + NB[1]&amp;0xff;\n   * <li>if (FB in [-120, -105]), return (FB+112)&lt;&lt;24 + (NB[0]&amp;0xff)\n   * &lt;&lt;16 + (NB[1]&amp;0xff)&lt;&lt;8 + NB[2]&amp;0xff;\n   * <li>if (FB in [-128, -121]), return interpret NB[FB+129] as a signed\n   * big-endian integer.\n   * </ul>","sourceCodeStart":163,"sourceCodeEnd":199,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/Utils.java#L163-L199","documentation":"RuntimeException from Utils.readVInt(DataInput): readVLong decodes fine, but the resulting long falls outside [Integer.MIN_VALUE, Integer.MAX_VALUE], so it cannot be returned as int. TFile uses readVInt for every length/count prefix (key lengths, index sizes, entry counts), so this fires when a corrupt stream decodes to a value that needs more than 32 bits where a small length was expected.","triggerScenarios":"Any TFile parse path that hits a VInt prefix whose 5-to-9-byte encoding decodes above 2^31-1 or below -2^31: classic signature of random bytes being interpreted as a length (desynchronized block reader, corrupt data), since legitimate TFile lengths are tiny.","commonSituations":"Corrupt or truncated files where the reader is decoding garbage as varints; opening a non-TFile stream; partial writes followed by reads. Rare with healthy files because real lengths never approach 2^31.","solutions":["Treat as corruption: fsck/restore the file, or regenerate from source.","If you call Utils.readVInt yourself on external data, switch to Utils.readVLong when values may legitimately exceed int range.","Catch RuntimeException (not IOException) around TFile parsing if you quarantine bad files in a batch pipeline."],"exampleFix":"// before (calling readVInt on data whose values can exceed int)\nint n = Utils.readVInt(in);\n// after\nlong n = Utils.readVLong(in);","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  TFile.Reader r = new TFile.Reader(fsdis, fileLength, conf);\n} catch (RuntimeException e) {\n  if (e.getMessage().contains(\"Integer\")) { // readVInt overflow => corrupt stream\n    quarantine(path);\n  } else throw e;\n}","preventionTips":["In your own Utils callers, prefer readVLong for any externally-sourced varint that may exceed int range.","Quarantine files whose parse throws RuntimeException — bogus lengths mean desynchronized bytes.","Keep TFile length fields tiny by design: they are varints, but readers still validate int bounds."],"tags":["tfile","hadoop-common","utils","varint","overflow","data-corruption"],"backgroundTag":"numeric-overflow","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}