apache/beam · error · ValueError

Start position not 0:%s

Error message

Start position not 0:%s

What it means

The TFRecord source is not splittable: every file must be read from the very start. If the range tracker's start position is non-zero, read_records raises ValueError to enforce this invariant rather than silently mis-reading records.

Source

Thrown at sdks/python/apache_beam/io/tfrecordio.py:199

class _TFRecordSource(FileBasedSource):
  """A File source for reading files of TFRecords.

  For detailed TFRecords format description see:
    https://www.tensorflow.org/versions/r1.11/api_guides/python/python_io#TFRecords_Format_Details
  """
  def __init__(self, file_pattern, coder, compression_type, validate):
    """Initialize a TFRecordSource.  See ReadFromTFRecord for details."""
    super().__init__(
        file_pattern=file_pattern,
        compression_type=compression_type,
        splittable=False,
        validate=validate)
    self._coder = coder

  def read_records(self, file_name, offset_range_tracker):
    if offset_range_tracker.start_position():
      raise ValueError(
          'Start position not 0:%s' % offset_range_tracker.start_position())

    current_offset = offset_range_tracker.start_position()
    with self.open_file(file_name) as file_handle:
      while True:
        if not offset_range_tracker.try_claim(current_offset):
          raise RuntimeError('Unable to claim position: %s' % current_offset)
        record = _TFRecordUtil.read_record(file_handle)
        if record is None:
          return  # Reached EOF
        else:
          current_offset += _TFRecordUtil.encoded_num_bytes(record)
          yield self._coder.decode(record)


def _create_tfrecordio_source(
    file_pattern=None, coder=None, compression_type=None):
  # We intentionally disable validation for ReadAll pattern so that reading does

View on GitHub (pinned to 12126d8942)

Solutions

1. Read TFRecord sources without splitting (keep splittable=False, the default)
2. Start the range tracker at position 0 for TFRecord sources
3. Use a splittable format (e.g. Avro/Parquet) if fine-grained splitting is required

Example fix

// before
tracker.try_split(1024)  # mid-file split on TFRecord source
// after
# read whole file from offset 0 (default behavior)
beam.io.ReadFromTFRecord(pattern)

if offset_range_tracker.start_position() not in (0, None):
    raise ValueError('TFRecord source must start at offset 0')

try:
    records = source.read_records(file_name, tracker)
except ValueError as e:
    if 'Start position not 0' in str(e):
        tracker = source.get_range_tracker(0, source.DEFAULT_SIZE)
        records = source.read_records(file_name, tracker)
    else:
        raise

When it happens

Trigger: Dynamic work rebalancing or a runner attempting to split a TFRecord read at a non-zero offset.

Common situations: Custom runners or pipelines calling try_split on unsplittable TFRecord sources; resharding during reads.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Could not find python-snappy, google-crc32c, or crcmod. To a
• Not a valid TFRecord. Fewer than %d bytes: %s
• Not a valid TFRecord. Mismatch of length mask: %s
• Not a valid TFRecord. Mismatch of data mask: %s
• Need to set the filepattern of a TFRecordIO.Read transform

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5c679b8ee0a58aa1. Report an issue: GitHub.