{"record":{"id":"cf820b72b096fbf2","repo":"apache/hadoop","slug":"string-length-length-exceeds-the-limit-of-max","errorCode":null,"errorMessage":"String length ${length} exceeds the limit of ${maxLength}","messagePattern":"String length (.+?) exceeds the limit of (.+?)","errorType":"exception","errorClass":"EOFException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/Utils.java","lineNumber":305,"sourceCode":"   * @param in The input stream.\n   * @param maxLength The largest permitted encoded length in bytes, negative for no limit.\n   * @return The string or null.\n   * @throws EOFException input data length exceeds {@code maxLength}.\n   * @throws IOException IO failure.\n   * @throws NegativeArraySizeException string length was minus two or less.\n   */\n  public static String readString(DataInput in, int maxLength)\n      throws IOException {\n    int length = readVInt(in);\n    if (length == -1) {\n      return null;\n    }\n    if (length < 0) {\n      throw new NegativeArraySizeException(\"Corrupted data: negative string length \"\n          + length);\n    }\n    if (maxLength >= 0 && length > maxLength) {\n      throw new EOFException(\"String length \" + length\n          + \" exceeds the limit of \" + maxLength);\n    }\n    byte[] buffer = new byte[length];\n    in.readFully(buffer);\n    return Text.decode(buffer);\n  }\n\n  /**\n   * A generic Version class. We suggest applications built on top of TFile use\n   * this class to maintain version information in their meta blocks.\n   * \n   * A version number consists of a major version and a minor version. The\n   * suggested usage of major and minor version number is to increment major\n   * version number when the new storage format is not backward compatible, and\n   * increment the minor version otherwise.\n   */\n  public static final class Version implements Comparable<Version> {\n    private final short major;","sourceCodeStart":287,"sourceCodeEnd":323,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/Utils.java#L287-L323","documentation":"Guard inside Utils.readString(DataInput in, int maxLength): after reading the VInt length prefix, if maxLength >= 0 and length > maxLength the method throws EOFException(\"String length ... exceeds the limit of ...\"). The bound exists to stop a corrupt or hostile length prefix (values near 2^31) from triggering a huge byte[] allocation and OOM; it converts a memory-exhaustion failure into a fast, descriptive one.","triggerScenarios":"Calling readString with a maxLength bound where the stream's length prefix exceeds it — genuinely corrupt data with an inflated length field, a mismatch between the limit the writer assumed and the one the reader enforces, or a crafted/fuzzed input file.","commonSituations":"After upgrading Hadoop, callers now pass a maxLength to data written by older code with longer legitimate strings; an arbitrarily small maxLength chosen by the caller; corrupted TFile metadata where the length field is garbage but positive.","solutions":["Compare the length in the message against your maxLength: if the input is known-good, the caller's limit is wrong — raise it to the format's true maximum.","If the reported length is absurdly large, treat the input as corrupt (same remediation as the negative-length sibling) and regenerate the file.","When you control both ends, derive the limit from the format spec and use the same constant for write and read so they cannot diverge."],"exampleFix":"// before: limit smaller than legal values\nString s = Utils.readString(in, 1024);\n\n// after: limit derived from the format's true maximum string size\nString s = Utils.readString(in, MAX_KEY_LENGTH /* e.g. 64 * 1024 */);","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  String s = Utils.readString(in, maxLength);\n} catch (EOFException e) {\n  // limit exceeded (or truncated stream): decide corrupt vs misconfigured limit\n  // from the reported length before any retry\n}","preventionTips":["Derive maxLength from the format spec's true maximum string size, not an arbitrary small number.","Always pass a maxLength for untrusted inputs — it is the OOM guard.","Catch EOFException separately from IOException so the bound that failed is logged."],"tags":["tfile","data-corruption","oom-guard","hadoop-common"],"backgroundTag":"payload-too-large","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}