{"record":{"id":"00aa29d62fc7d6bc","repo":"apache/beam","slug":"length-overflow-d","errorCode":null,"errorMessage":"length overflow %d","messagePattern":"length overflow (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java","lineNumber":706,"sourceCode":"      int headerBytes = read(inChannel, header);\n      if (headerBytes == 0) {\n        return null;\n      }\n      checkState(headerBytes == HEADER_LEN, \"Not a valid TFRecord. Fewer than 12 bytes.\");\n\n      header.rewind();\n      long length64 = header.getLong();\n      long lengthHash = hashLong(length64);\n      int maskedCrc32OfLength = header.getInt();\n      if (lengthHash != maskedCrc32OfLength) {\n        throw new IOException(\n            String.format(\n                \"Mismatch of length mask when reading a record. Expected %d but received %d.\",\n                maskedCrc32OfLength, lengthHash));\n      }\n      int length = (int) length64;\n      if (length != length64) {\n        throw new IOException(String.format(\"length overflow %d\", length64));\n      }\n\n      ByteBuffer data = ByteBuffer.allocate(length);\n      readFully(inChannel, data);\n\n      footer.clear();\n      readFully(inChannel, footer);\n      footer.rewind();\n\n      int maskedCrc32OfData = footer.getInt();\n      int dataHash = hashBytes(data.array());\n      if (dataHash != maskedCrc32OfData) {\n        throw new IOException(\n            String.format(\n                \"Mismatch of data mask when reading a record. Expected %d but received %d.\",\n                maskedCrc32OfData, dataHash));\n      }\n      return data.array();","sourceCodeStart":688,"sourceCodeEnd":724,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java#L688-L724","documentation":"TFRecord stores record lengths as 64-bit values, but in-memory buffers use 32-bit ints. This IOException is thrown when the 64-bit record length cannot fit in an int (length64 != (int) length64), i.e. a record claims to be larger than 2^31-1 bytes, which is not supported.","triggerScenarios":"Reading a TFRecord file whose header declares a record length > Integer.MAX_VALUE — typically a corrupt header where random bytes were interpreted as a huge length.","commonSituations":"Corrupted or truncated files; byte-offset drift after earlier corruption; reading garbage data with a .tfrecord extension.","solutions":["Treat the file as corrupt and regenerate or re-transfer it.","Validate the file with a local TFRecord reader (e.g. tf.data.TFRecordDataset) to pinpoint the corrupt offset.","Exclude the bad shard from the input filepattern."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (IOException e) { if (e.getMessage() != null && e.getMessage().startsWith(\"length overflow\")) { log.error(\"TFRecord claims record > 2GB; file is corrupt: {}\", file, e); } throw e; }","preventionTips":["Regenerate or re-transfer files that report length overflow — valid TFRecords never exceed int range.","Validate inputs with an independent TFRecord reader before running pipelines."],"tags":["java","beam","tfrecord","corruption","overflow"],"backgroundTag":"value-out-of-range","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}