apache/hadoop · error · IOException

tried to deserialize {} bytes of data, but maxLength = {}

Error message

tried to deserialize {} bytes of data, but maxLength = {}

What it means

The defensive overload Text.readFields(DataInput, int maxLength) rejects any encoded Text whose byte length is >= maxLength, throwing IOException("tried to deserialize N bytes of data, but maxLength = M"). The non-negative check passes but the size exceeds the bound the caller imposed; the overload exists so decoders on untrusted input can bound memory allocation.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java:358

      return decode(bytes, 0, length);
    } catch (CharacterCodingException e) { 
      throw new RuntimeException("Should not have happened", e);
    }
  }

  @Override
  public void readFields(DataInput in) throws IOException {
    int newLength = WritableUtils.readVInt(in);
    readWithKnownLength(in, newLength);
  }

  public void readFields(DataInput in, int maxLength) throws IOException {
    int newLength = WritableUtils.readVInt(in);
    if (newLength < 0) {
      throw new IOException("tried to deserialize " + newLength +
          " bytes of data!  newLength must be non-negative.");
    } else if (newLength >= maxLength) {
      throw new IOException("tried to deserialize " + newLength +
          " bytes of data, but maxLength = " + maxLength);
    }
    readWithKnownLength(in, newLength);
  }

  /**
   * Skips over one Text in the input.
   * @param in input in.
   * @throws IOException raised on errors performing I/O.
   */
  public static void skip(DataInput in) throws IOException {
    int length = WritableUtils.readVInt(in);
    WritableUtils.skipFully(in, length);
  }

  /**
   * Read a Text object whose length is already known.
   * This allows creating Text from a stream which uses a different serialization

View on GitHub (pinned to 2add963021)

Solutions

1. If the data is legitimately larger, raise maxLength at the reading call site (and the config that drives it) to a value above the real maximum.
2. Enforce the same cap at the writer with Text.writeString(out, s, maxLength) / Text.write(out, maxLength) so both ends agree and failures surface at the producer.
3. If the size is unexpected, treat as corruption: locate the writer of that record and verify what it serialized.

Example fix

// before: writer unbounded, reader capped
Text.writeString(out, s);               // may write 100 KB
Text t = new Text(); t.readFields(in, 1024);  // throws

// after: both ends share one limit
private static final int MAX_FIELD = 65536;
Text.writeString(out, s, MAX_FIELD);
Text t = new Text(); t.readFields(in, MAX_FIELD);

// Enforce the shared cap at the writer so the reader's maxLength never trips
private static final int MAX_FIELD = 65536;

// write side
Text.writeString(out, s, MAX_FIELD);

// read side — same constant
Text t = new Text();
t.readFields(in, MAX_FIELD);

try {
  t.readFields(in, maxLength);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("but maxLength")) {
    // oversized or corrupt field: report limit and move on / fail cleanly
    LOG.error("Field exceeds limit {} — check writer cap alignment", maxLength);
  } else {
    throw e;
  }
}

When it happens

Trigger: Reading a legitimately large Text (a big string field) through a decoder that imposes a byte cap; a limit raised on the writer side but not the reader side; corrupt or hostile length VInts that pass the non-negative check but exceed any sane size.

Common situations: Config-driven maximum field sizes that differ between producer and consumer; security hardening adding maxLength guards to previously unbounded reads; denial-of-service style oversized payloads.

Related errors

• tried to deserialize {} bytes of data! newLength must be no
• data was too long to write! Expected less than or equal to
• string was too long to write! Expected less than or equal t
• Exception while get content summary
• f.toString()

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8beb8c663adb6c6d. Report an issue: GitHub.