{"record":{"id":"bfaa749b8f6ba7bf","repo":"apache/beam","slug":"mismatch-of-data-mask-when-reading-a-record-expected-d-but","errorCode":null,"errorMessage":"Mismatch of data mask when reading a record. Expected %d but received %d.","messagePattern":"Mismatch of data mask when reading a record\\. Expected (.+?) but received (.+?)\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java","lineNumber":719,"sourceCode":"                \"Mismatch of length mask when reading a record. Expected %d but received %d.\",\n                maskedCrc32OfLength, lengthHash));\n      }\n      int length = (int) length64;\n      if (length != length64) {\n        throw new IOException(String.format(\"length overflow %d\", length64));\n      }\n\n      ByteBuffer data = ByteBuffer.allocate(length);\n      readFully(inChannel, data);\n\n      footer.clear();\n      readFully(inChannel, footer);\n      footer.rewind();\n\n      int maskedCrc32OfData = footer.getInt();\n      int dataHash = hashBytes(data.array());\n      if (dataHash != maskedCrc32OfData) {\n        throw new IOException(\n            String.format(\n                \"Mismatch of data mask when reading a record. Expected %d but received %d.\",\n                maskedCrc32OfData, dataHash));\n      }\n      return data.array();\n    }\n\n    public void write(WritableByteChannel outChannel, byte[] data) throws IOException {\n      int maskedCrc32OfLength = hashLong(data.length);\n      int maskedCrc32OfData = hashBytes(data);\n\n      header.clear();\n      header.putLong(data.length).putInt(maskedCrc32OfLength);\n      header.rewind();\n      writeFully(outChannel, header);\n\n      writeFully(outChannel, ByteBuffer.wrap(data));\n","sourceCodeStart":701,"sourceCodeEnd":737,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java#L701-L737","documentation":"After reading a TFRecord's data bytes, the reader checks the record's CRC32C data mask stored in the footer. This IOException is thrown when the computed hash of the data bytes doesn't match the stored mask, indicating the record payload is corrupted even though the header length was valid.","triggerScenarios":"Bit-flips or truncation inside a record's data section; reading a file whose payload was altered after writing (re-compression, editing) or whose channel returned wrong bytes.","commonSituations":"Network/disk corruption during transfer or storage; tools that modified the file in place; decompressing with the wrong compression type.","solutions":["Regenerate or re-download the TFRecord file and compare checksums at the object-storage level.","Confirm the compression setting matches how the file was written (withCompression).","Isolate the corrupt shard and exclude it from the filepattern."],"exampleFix":"// before\nTFRecordIO.read().from(\"gs://b/data.tfrecord.gz\"); // plain file read as gzipped\n\n// after\nTFRecordIO.read().from(\"gs://b/data.tfrecord\").withCompression(Compression.NONE);","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (IOException e) { if (e.getMessage() != null && e.getMessage().contains(\"data mask\")) { log.error(\"TFRecord payload checksum mismatch (corrupt file): {}\", file, e); } throw e; }","preventionTips":["Match the withCompression() setting to how the file was written.","Re-download corrupted files and compare storage-level checksums."],"tags":["java","beam","tfrecord","corruption","checksum"],"backgroundTag":"checksum-mismatch","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}