apache/hadoop · error · IOException

tried to deserialize

Error message

tried to deserialize {} bytes of data, but maxLength = {}

What it means

The defensive overload Text.readFields(DataInput, int maxLength) rejects any encoded Text whose byte length is >= maxLength, throwing IOException("tried to deserialize N bytes of data, but maxLength = M"). The non-negative check passes but the size exceeds the bound the caller imposed; the overload exists so decoders on untrusted input can bound memory allocation.

Solutions

1. If the data is legitimately larger, raise maxLength at the reading call site (and the config that drives it) to a value above the real maximum.
2. Enforce the same cap at the writer with Text.writeString(out, s, maxLength) / Text.write(out, maxLength) so both ends agree and failures surface at the producer.
3. If the size is unexpected, treat as corruption: locate the writer of that record and verify what it serialized.

Example fix

// before: writer unbounded, reader capped
Text.writeString(out, s);               // may write 100 KB
Text t = new Text(); t.readFields(in, 1024);  // throws

// after: both ends share one limit
private static final int MAX_FIELD = 65536;
Text.writeString(out, s, MAX_FIELD);
Text t = new Text(); t.readFields(in, MAX_FIELD);

// Enforce the shared cap at the writer so the reader's maxLength never trips
private static final int MAX_FIELD = 65536;

// write side
Text.writeString(out, s, MAX_FIELD);

// read side — same constant
Text t = new Text();
t.readFields(in, MAX_FIELD);

try {
  t.readFields(in, maxLength);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("but maxLength")) {
    // oversized or corrupt field: report limit and move on / fail cleanly
    LOG.error("Field exceeds limit {} — check writer cap alignment", maxLength);
  } else {
    throw e;
  }
}

When it happens

Trigger: Reading a legitimately large Text (a big string field) through a decoder that imposes a byte cap; a limit raised on the writer side but not the reader side; corrupt or hostile length VInts that pass the non-negative check but exceed any sane size.

Common situations: Config-driven maximum field sizes that differ between producer and consumer; security hardening adding maxLength guards to previously unbounded reads; denial-of-service style oversized payloads.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• data was too long to write! Expected less than or equal to
• tried to deserialize
• string was too long to write! Expected less than or equal…
• adding an additional class would exceed the maximum number…
• bad framework group id

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8beb8c663adb6c6d. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java:358

      return decode(bytes, 0, length);
    } catch (CharacterCodingException e) { 
      throw new RuntimeException("Should not have happened", e);
    }
  }

  @Override
  public void readFields(DataInput in) throws IOException {
    int newLength = WritableUtils.readVInt(in);
    readWithKnownLength(in, newLength);
  }

  public void readFields(DataInput in, int maxLength) throws IOException {
    int newLength = WritableUtils.readVInt(in);
    if (newLength < 0) {
      throw new IOException("tried to deserialize " + newLength +
          " bytes of data!  newLength must be non-negative.");
    } else if (newLength >= maxLength) {
      throw new IOException("tried to deserialize " + newLength +
          " bytes of data, but maxLength = " + maxLength);
    }
    readWithKnownLength(in, newLength);
  }

  /**
   * Skips over one Text in the input.
   * @param in input in.
   * @throws IOException raised on errors performing I/O.
   */
  public static void skip(DataInput in) throws IOException {
    int length = WritableUtils.readVInt(in);
    WritableUtils.skipFully(in, length);
  }

  /**
   * Read a Text object whose length is already known.
   * This allows creating Text from a stream which uses a different serialization

View on GitHub (pinned to 2add963021)