apache/beam · error · IOException

length overflow %d

Error message

length overflow %d

What it means

TFRecord stores record lengths as 64-bit values, but in-memory buffers use 32-bit ints. This IOException is thrown when the 64-bit record length cannot fit in an int (length64 != (int) length64), i.e. a record claims to be larger than 2^31-1 bytes, which is not supported.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java:706

      int headerBytes = read(inChannel, header);
      if (headerBytes == 0) {
        return null;
      }
      checkState(headerBytes == HEADER_LEN, "Not a valid TFRecord. Fewer than 12 bytes.");

      header.rewind();
      long length64 = header.getLong();
      long lengthHash = hashLong(length64);
      int maskedCrc32OfLength = header.getInt();
      if (lengthHash != maskedCrc32OfLength) {
        throw new IOException(
            String.format(
                "Mismatch of length mask when reading a record. Expected %d but received %d.",
                maskedCrc32OfLength, lengthHash));
      }
      int length = (int) length64;
      if (length != length64) {
        throw new IOException(String.format("length overflow %d", length64));
      }

      ByteBuffer data = ByteBuffer.allocate(length);
      readFully(inChannel, data);

      footer.clear();
      readFully(inChannel, footer);
      footer.rewind();

      int maskedCrc32OfData = footer.getInt();
      int dataHash = hashBytes(data.array());
      if (dataHash != maskedCrc32OfData) {
        throw new IOException(
            String.format(
                "Mismatch of data mask when reading a record. Expected %d but received %d.",
                maskedCrc32OfData, dataHash));
      }
      return data.array();

View on GitHub (pinned to 12126d8942)

Solutions

1. Treat the file as corrupt and regenerate or re-transfer it.
2. Validate the file with a local TFRecord reader (e.g. tf.data.TFRecordDataset) to pinpoint the corrupt offset.
3. Exclude the bad shard from the input filepattern.

catch (IOException e) { if (e.getMessage() != null && e.getMessage().startsWith("length overflow")) { log.error("TFRecord claims record > 2GB; file is corrupt: {}", file, e); } throw e; }

When it happens

Trigger: Reading a TFRecord file whose header declares a record length > Integer.MAX_VALUE — typically a corrupt header where random bytes were interpreted as a huge length.

Common situations: Corrupted or truncated files; byte-offset drift after earlier corruption; reading garbage data with a .tfrecord extension.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Mismatch of length mask when reading a record. Expected %d b
• Mismatch of data mask when reading a record. Expected %d but
• Need to set the filepattern of a TFRecordIO.Read transform
• Failed to validate %s
• expected %d, but got %d

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/00aa29d62fc7d6bc. Report an issue: GitHub.