apache/beam · error · IOException

expected %d, but got %d

Error message

expected %d, but got %d

What it means

readFully() requires the channel to deliver exactly bb.remaining() bytes in one read() call; if the single read returns fewer (or EOF ends the stream early), it throws this IOException showing expected vs actual byte counts. It indicates a truncated or short read while loading TFRecord headers, data, or footers.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java:749

      header.clear();
      header.putLong(data.length).putInt(maskedCrc32OfLength);
      header.rewind();
      writeFully(outChannel, header);

      writeFully(outChannel, ByteBuffer.wrap(data));

      footer.clear();
      footer.putInt(maskedCrc32OfData);
      footer.rewind();
      writeFully(outChannel, footer);
    }

    @VisibleForTesting
    static void readFully(ReadableByteChannel in, ByteBuffer bb) throws IOException {
      int expected = bb.remaining();
      int actual = read(in, bb);
      if (expected != actual) {
        throw new IOException(String.format("expected %d, but got %d", expected, actual));
      }
    }

    private static int read(ReadableByteChannel in, ByteBuffer bb) throws IOException {
      int expected = bb.remaining();
      while (bb.hasRemaining() && in.read(bb) >= 0) {}
      return expected - bb.remaining();
    }

    @VisibleForTesting
    static void writeFully(WritableByteChannel channel, ByteBuffer buffer) throws IOException {
      while (buffer.hasRemaining()) {
        channel.write(buffer);
      }
    }
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

1. Ensure the file is fully written/uploaded before running the pipeline.
2. Regenerate the truncated file and exclude incomplete shards (e.g. *.tmp) from the filepattern.
3. Check the expected/actual numbers to determine how many bytes were missing and locate the truncation point.

Example fix

// before
TFRecordIO.read().from("gs://b/part-*.tfrecord*"); // picks up part-0000.tfrecord.tmp being written

// after
TFRecordIO.read().from("gs://b/part-*-of-00005.tfrecord");

// ensure files are complete before reading
MatchResult mr = FileSystems.match(pattern);
mr.metadata().forEach(m -> {
  if (m.resourceId().toString().endsWith(".tmp") || m.sizeBytes() == 0) {
    throw new IllegalStateException("Incomplete input file: " + m.resourceId());
  }
});

catch (IOException e) { if (e.getMessage() != null && e.getMessage().matches("expected \\d+, but got \\d+")) { log.error("Truncated TFRecord read (short read): {}", e.getMessage()); } throw e; }

When it happens

Trigger: Reading a TFRecord file truncated mid-record (EOF before the full header/data/footer); a channel that returns a partial read where the code expects a complete one.

Common situations: Files still being written when the pipeline starts; interrupted uploads; a record cut off by an earlier length misparse from corruption.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Need to set the filepattern of a TFRecordIO.Read transform
• Failed to validate %s
• Mismatch of length mask when reading a record. Expected %d b
• length overflow %d
• Mismatch of data mask when reading a record. Expected %d but

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7afda31f3c739fcf. Report an issue: GitHub.