{"record":{"id":"7afda31f3c739fcf","repo":"apache/beam","slug":"expected-d-but-got-d","errorCode":null,"errorMessage":"expected %d, but got %d","messagePattern":"expected (.+?), but got (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java","lineNumber":749,"sourceCode":"      header.clear();\n      header.putLong(data.length).putInt(maskedCrc32OfLength);\n      header.rewind();\n      writeFully(outChannel, header);\n\n      writeFully(outChannel, ByteBuffer.wrap(data));\n\n      footer.clear();\n      footer.putInt(maskedCrc32OfData);\n      footer.rewind();\n      writeFully(outChannel, footer);\n    }\n\n    @VisibleForTesting\n    static void readFully(ReadableByteChannel in, ByteBuffer bb) throws IOException {\n      int expected = bb.remaining();\n      int actual = read(in, bb);\n      if (expected != actual) {\n        throw new IOException(String.format(\"expected %d, but got %d\", expected, actual));\n      }\n    }\n\n    private static int read(ReadableByteChannel in, ByteBuffer bb) throws IOException {\n      int expected = bb.remaining();\n      while (bb.hasRemaining() && in.read(bb) >= 0) {}\n      return expected - bb.remaining();\n    }\n\n    @VisibleForTesting\n    static void writeFully(WritableByteChannel channel, ByteBuffer buffer) throws IOException {\n      while (buffer.hasRemaining()) {\n        channel.write(buffer);\n      }\n    }\n  }\n}\n","sourceCodeStart":731,"sourceCodeEnd":767,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java#L731-L767","documentation":"readFully() requires the channel to deliver exactly bb.remaining() bytes in one read() call; if the single read returns fewer (or EOF ends the stream early), it throws this IOException showing expected vs actual byte counts. It indicates a truncated or short read while loading TFRecord headers, data, or footers.","triggerScenarios":"Reading a TFRecord file truncated mid-record (EOF before the full header/data/footer); a channel that returns a partial read where the code expects a complete one.","commonSituations":"Files still being written when the pipeline starts; interrupted uploads; a record cut off by an earlier length misparse from corruption.","solutions":["Ensure the file is fully written/uploaded before running the pipeline.","Regenerate the truncated file and exclude incomplete shards (e.g. *.tmp) from the filepattern.","Check the expected/actual numbers to determine how many bytes were missing and locate the truncation point."],"exampleFix":"// before\nTFRecordIO.read().from(\"gs://b/part-*.tfrecord*\"); // picks up part-0000.tfrecord.tmp being written\n\n// after\nTFRecordIO.read().from(\"gs://b/part-*-of-00005.tfrecord\");","handlingStrategy":"validation","validationCode":"// ensure files are complete before reading\nMatchResult mr = FileSystems.match(pattern);\nmr.metadata().forEach(m -> {\n  if (m.resourceId().toString().endsWith(\".tmp\") || m.sizeBytes() == 0) {\n    throw new IllegalStateException(\"Incomplete input file: \" + m.resourceId());\n  }\n});","typeGuard":null,"tryCatchPattern":"catch (IOException e) { if (e.getMessage() != null && e.getMessage().matches(\"expected \\\\d+, but got \\\\d+\")) { log.error(\"Truncated TFRecord read (short read): {}\", e.getMessage()); } throw e; }","preventionTips":["Only point filepatterns at fully written shards; producers should write to temp names and rename on completion.","Re-run with retries for transient network short reads on remote filesystems."],"tags":["java","beam","tfrecord","truncated-file","io"],"backgroundTag":"file-read-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}