apache/beam · error · IOException

Mismatch of length mask when reading a record. Expected %d b

Error message

Mismatch of length mask when reading a record. Expected %d but received %d.

What it means

While reading a TFRecord, each record's length field is protected by a masked CRC32C hash stored in the header. This IOException is thrown when the stored mask doesn't match the hash computed from the length actually read, meaning the file is corrupt, truncated, or not a valid TFRecord file at that offset.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java:699

    public int recordLength(byte[] data) {
      return HEADER_LEN + data.length + FOOTER_LEN;
    }

    public byte @Nullable [] read(ReadableByteChannel inChannel) throws IOException {
      header.clear();
      int headerBytes = read(inChannel, header);
      if (headerBytes == 0) {
        return null;
      }
      checkState(headerBytes == HEADER_LEN, "Not a valid TFRecord. Fewer than 12 bytes.");

      header.rewind();
      long length64 = header.getLong();
      long lengthHash = hashLong(length64);
      int maskedCrc32OfLength = header.getInt();
      if (lengthHash != maskedCrc32OfLength) {
        throw new IOException(
            String.format(
                "Mismatch of length mask when reading a record. Expected %d but received %d.",
                maskedCrc32OfLength, lengthHash));
      }
      int length = (int) length64;
      if (length != length64) {
        throw new IOException(String.format("length overflow %d", length64));
      }

      ByteBuffer data = ByteBuffer.allocate(length);
      readFully(inChannel, data);

      footer.clear();
      readFully(inChannel, footer);
      footer.rewind();

      int maskedCrc32OfData = footer.getInt();
      int dataHash = hashBytes(data.array());

View on GitHub (pinned to 12126d8942)

Solutions

Verify the input file is a genuine, uncorrupted TFRecord file (re-upload or regenerate it).
Check that the producer finished writing before the read (record files are complete).
Point the filepattern at the correct files; exclude partial/temp files from the glob.

Example fix

// before
TFRecordIO.read().from("dir/*.tfrecord"); // includes dir/part-0000-of-00002.tfrecord.tmp

// after
TFRecordIO.read().from("dir/*-of-00002.tfrecord"); // only completed shards

Defensive patterns

Strategy: try-catch

Validate before calling

// sanity-check the first record header offline with tf.data.TFRecordDataset before the Beam run

Try / catch

try { pipeline.apply(TFRecordIO.read().from(pattern)); } catch (Exception e) { if (e.getCause() instanceof IOException && e.getCause().getMessage().contains("length mask")) { log.error("Corrupt TFRecord input: {}", pattern, e); } throw e; }

Prevention

Verify object checksums (GCS/S3 CRC32C) after transfers.
Exclude partial/temp files from globs; only read completed shards.

When it happens

Trigger: Reading a .tfrecord file whose header bytes are corrupted, a file truncated mid-write, or pointing TFRecordIO at a non-TFRecord file (e.g. plain text or a different format).

Common situations: Upload/download corruption of GCS/S3 objects; a producer crashed mid-write; mistakenly pointing the reader at a JSON or CSV file.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f4728aec2bb40af2. Report an issue: GitHub.