apache/beam · error · IOException

Mismatch of data mask when reading a record. Expected %d but

Error message

Mismatch of data mask when reading a record. Expected %d but received %d.

What it means

After reading a TFRecord's data bytes, the reader checks the record's CRC32C data mask stored in the footer. This IOException is thrown when the computed hash of the data bytes doesn't match the stored mask, indicating the record payload is corrupted even though the header length was valid.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java:719

                "Mismatch of length mask when reading a record. Expected %d but received %d.",
                maskedCrc32OfLength, lengthHash));
      }
      int length = (int) length64;
      if (length != length64) {
        throw new IOException(String.format("length overflow %d", length64));
      }

      ByteBuffer data = ByteBuffer.allocate(length);
      readFully(inChannel, data);

      footer.clear();
      readFully(inChannel, footer);
      footer.rewind();

      int maskedCrc32OfData = footer.getInt();
      int dataHash = hashBytes(data.array());
      if (dataHash != maskedCrc32OfData) {
        throw new IOException(
            String.format(
                "Mismatch of data mask when reading a record. Expected %d but received %d.",
                maskedCrc32OfData, dataHash));
      }
      return data.array();
    }

    public void write(WritableByteChannel outChannel, byte[] data) throws IOException {
      int maskedCrc32OfLength = hashLong(data.length);
      int maskedCrc32OfData = hashBytes(data);

      header.clear();
      header.putLong(data.length).putInt(maskedCrc32OfLength);
      header.rewind();
      writeFully(outChannel, header);

      writeFully(outChannel, ByteBuffer.wrap(data));

View on GitHub (pinned to 12126d8942)

Solutions

1. Regenerate or re-download the TFRecord file and compare checksums at the object-storage level.
2. Confirm the compression setting matches how the file was written (withCompression).
3. Isolate the corrupt shard and exclude it from the filepattern.

Example fix

// before
TFRecordIO.read().from("gs://b/data.tfrecord.gz"); // plain file read as gzipped

// after
TFRecordIO.read().from("gs://b/data.tfrecord").withCompression(Compression.NONE);

catch (IOException e) { if (e.getMessage() != null && e.getMessage().contains("data mask")) { log.error("TFRecord payload checksum mismatch (corrupt file): {}", file, e); } throw e; }

When it happens

Trigger: Bit-flips or truncation inside a record's data section; reading a file whose payload was altered after writing (re-compression, editing) or whose channel returned wrong bytes.

Common situations: Network/disk corruption during transfer or storage; tools that modified the file in place; decompressing with the wrong compression type.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• Mismatch of length mask when reading a record. Expected %d b
• length overflow %d
• Need to set the filepattern of a TFRecordIO.Read transform
• Failed to validate %s
• expected %d, but got %d

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/bfaa749b8f6ba7bf. Report an issue: GitHub.