{"record":{"id":"8beb8c663adb6c6d","repo":"apache/hadoop","slug":"tried-to-deserialize-bytes-of-data-but-maxleng","errorCode":null,"errorMessage":"tried to deserialize {} bytes of data, but maxLength = {}","messagePattern":"tried to deserialize (.+?) bytes of data, but maxLength = (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java","lineNumber":358,"sourceCode":"      return decode(bytes, 0, length);\n    } catch (CharacterCodingException e) { \n      throw new RuntimeException(\"Should not have happened\", e);\n    }\n  }\n\n  @Override\n  public void readFields(DataInput in) throws IOException {\n    int newLength = WritableUtils.readVInt(in);\n    readWithKnownLength(in, newLength);\n  }\n\n  public void readFields(DataInput in, int maxLength) throws IOException {\n    int newLength = WritableUtils.readVInt(in);\n    if (newLength < 0) {\n      throw new IOException(\"tried to deserialize \" + newLength +\n          \" bytes of data!  newLength must be non-negative.\");\n    } else if (newLength >= maxLength) {\n      throw new IOException(\"tried to deserialize \" + newLength +\n          \" bytes of data, but maxLength = \" + maxLength);\n    }\n    readWithKnownLength(in, newLength);\n  }\n\n  /**\n   * Skips over one Text in the input.\n   * @param in input in.\n   * @throws IOException raised on errors performing I/O.\n   */\n  public static void skip(DataInput in) throws IOException {\n    int length = WritableUtils.readVInt(in);\n    WritableUtils.skipFully(in, length);\n  }\n\n  /**\n   * Read a Text object whose length is already known.\n   * This allows creating Text from a stream which uses a different serialization","sourceCodeStart":340,"sourceCodeEnd":376,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java#L340-L376","documentation":"The defensive overload Text.readFields(DataInput, int maxLength) rejects any encoded Text whose byte length is >= maxLength, throwing IOException(\"tried to deserialize N bytes of data, but maxLength = M\"). The non-negative check passes but the size exceeds the bound the caller imposed; the overload exists so decoders on untrusted input can bound memory allocation.","triggerScenarios":"Reading a legitimately large Text (a big string field) through a decoder that imposes a byte cap; a limit raised on the writer side but not the reader side; corrupt or hostile length VInts that pass the non-negative check but exceed any sane size.","commonSituations":"Config-driven maximum field sizes that differ between producer and consumer; security hardening adding maxLength guards to previously unbounded reads; denial-of-service style oversized payloads.","solutions":["If the data is legitimately larger, raise maxLength at the reading call site (and the config that drives it) to a value above the real maximum.","Enforce the same cap at the writer with Text.writeString(out, s, maxLength) / Text.write(out, maxLength) so both ends agree and failures surface at the producer.","If the size is unexpected, treat as corruption: locate the writer of that record and verify what it serialized."],"exampleFix":"// before: writer unbounded, reader capped\nText.writeString(out, s);               // may write 100 KB\nText t = new Text(); t.readFields(in, 1024);  // throws\n\n// after: both ends share one limit\nprivate static final int MAX_FIELD = 65536;\nText.writeString(out, s, MAX_FIELD);\nText t = new Text(); t.readFields(in, MAX_FIELD);","handlingStrategy":"validation","validationCode":"// Enforce the shared cap at the writer so the reader's maxLength never trips\nprivate static final int MAX_FIELD = 65536;\n\n// write side\nText.writeString(out, s, MAX_FIELD);\n\n// read side — same constant\nText t = new Text();\nt.readFields(in, MAX_FIELD);","typeGuard":null,"tryCatchPattern":"try {\n  t.readFields(in, maxLength);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"but maxLength\")) {\n    // oversized or corrupt field: report limit and move on / fail cleanly\n    LOG.error(\"Field exceeds limit {} — check writer cap alignment\", maxLength);\n  } else {\n    throw e;\n  }\n}","preventionTips":["Define one shared limit constant used by both write and read code","Test size limits with realistic (multi-byte, large) payloads","Fail oversized values at ingestion with an actionable message, not deep in serialization"],"tags":["text","size-limit","deserialization","writable","hadoop"],"backgroundTag":"payload-too-large","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}