{"record":{"id":"f4728aec2bb40af2","repo":"apache/beam","slug":"mismatch-of-length-mask-when-reading-a-record-expected-d-but","errorCode":null,"errorMessage":"Mismatch of length mask when reading a record. Expected %d but received %d.","messagePattern":"Mismatch of length mask when reading a record\\. Expected (.+?) but received (.+?)\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java","lineNumber":699,"sourceCode":"\n    public int recordLength(byte[] data) {\n      return HEADER_LEN + data.length + FOOTER_LEN;\n    }\n\n    public byte @Nullable [] read(ReadableByteChannel inChannel) throws IOException {\n      header.clear();\n      int headerBytes = read(inChannel, header);\n      if (headerBytes == 0) {\n        return null;\n      }\n      checkState(headerBytes == HEADER_LEN, \"Not a valid TFRecord. Fewer than 12 bytes.\");\n\n      header.rewind();\n      long length64 = header.getLong();\n      long lengthHash = hashLong(length64);\n      int maskedCrc32OfLength = header.getInt();\n      if (lengthHash != maskedCrc32OfLength) {\n        throw new IOException(\n            String.format(\n                \"Mismatch of length mask when reading a record. Expected %d but received %d.\",\n                maskedCrc32OfLength, lengthHash));\n      }\n      int length = (int) length64;\n      if (length != length64) {\n        throw new IOException(String.format(\"length overflow %d\", length64));\n      }\n\n      ByteBuffer data = ByteBuffer.allocate(length);\n      readFully(inChannel, data);\n\n      footer.clear();\n      readFully(inChannel, footer);\n      footer.rewind();\n\n      int maskedCrc32OfData = footer.getInt();\n      int dataHash = hashBytes(data.array());","sourceCodeStart":681,"sourceCodeEnd":717,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java#L681-L717","documentation":"While reading a TFRecord, each record's length field is protected by a masked CRC32C hash stored in the header. This IOException is thrown when the stored mask doesn't match the hash computed from the length actually read, meaning the file is corrupt, truncated, or not a valid TFRecord file at that offset.","triggerScenarios":"Reading a .tfrecord file whose header bytes are corrupted, a file truncated mid-write, or pointing TFRecordIO at a non-TFRecord file (e.g. plain text or a different format).","commonSituations":"Upload/download corruption of GCS/S3 objects; a producer crashed mid-write; mistakenly pointing the reader at a JSON or CSV file.","solutions":["Verify the input file is a genuine, uncorrupted TFRecord file (re-upload or regenerate it).","Check that the producer finished writing before the read (record files are complete).","Point the filepattern at the correct files; exclude partial/temp files from the glob."],"exampleFix":"// before\nTFRecordIO.read().from(\"dir/*.tfrecord\"); // includes dir/part-0000-of-00002.tfrecord.tmp\n\n// after\nTFRecordIO.read().from(\"dir/*-of-00002.tfrecord\"); // only completed shards","handlingStrategy":"try-catch","validationCode":"// sanity-check the first record header offline with tf.data.TFRecordDataset before the Beam run","typeGuard":null,"tryCatchPattern":"try { pipeline.apply(TFRecordIO.read().from(pattern)); } catch (Exception e) { if (e.getCause() instanceof IOException && e.getCause().getMessage().contains(\"length mask\")) { log.error(\"Corrupt TFRecord input: {}\", pattern, e); } throw e; }","preventionTips":["Verify object checksums (GCS/S3 CRC32C) after transfers.","Exclude partial/temp files from globs; only read completed shards."],"tags":["java","beam","tfrecord","corruption","checksum"],"backgroundTag":"checksum-mismatch","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}