apache/beam · error · ValueError

Record at a split point has same offset as the previous…

Error message

Record at a split point has same offset as the previous split point: %d

What it means

Dynamic work rebalancing (splitting) can only occur at split points. If two consecutive records claim to be at a split point with the same start offset, the tracker cannot distinguish them, so _validate_record_start raises this ValueError to preserve unique split-point offsets.

Solutions

1. Mark only genuinely distinct positions as split points; ensure each split point has a strictly new offset.
2. Only pass split_point=True for the position currently being split; use split_point=False for continuation records.
3. Fix source logic that advances offsets so split points move forward.

Example fix

// before
tracker.try_claim(100)
tracker.set_current_position(100, split_point=True)  # duplicate split point
// after
tracker.try_claim(100)
tracker.set_current_position(150, split_point=True)  # new offset

if split_point and tracker._offset_of_last_split_point == record_start:
    split_point = False  # same offset as previous split point

def is_new_split_point(record_start: int, last_split: int) -> bool:
    return last_split == -1 or record_start != last_split

try:
    tracker.set_current_position(offset, split_point=True)
except ValueError as e:
    if 'same offset as the previous split' in str(e):
        tracker.set_current_position(offset, split_point=False)

When it happens

Trigger: Calling try_claim(offset) then set_current_position(offset, split_point=True) — or two consecutive split-point claims — with the same offset as the previously recorded split point.

Common situations: Custom source incorrectly marking every record (or the same record) as a split point; a source that re-claims a boundary record after resuming.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• The first record [starting at ] must be at a split point
• Trying to return a record
• End offset must not be 'None'
• Start offset must not be 'None'
• This function must only be called under the lock self.lock.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d5fda942882c0027. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/range_trackers.py:108

    no matter `try_claim()` returns `True` or `False`.
    """
    return self._last_attempted_record_start

  def _validate_record_start(self, record_start, split_point):
    # This function must only be called under the lock self.lock.
    if not self._lock.locked():
      raise ValueError(
          'This function must only be called under the lock self.lock.')

    if record_start < self._last_record_start:
      raise ValueError(
          'Trying to return a record [starting at %d] which is before the '
          'last-returned record [starting at %d]' %
          (record_start, self._last_record_start))

    if (split_point and self._offset_of_last_split_point != -1 and
        record_start == self._offset_of_last_split_point):
      raise ValueError(
          'Record at a split point has same offset as the previous split '
          'point: %d' % record_start)

    if not split_point and self._last_record_start == -1:
      raise ValueError(
          'The first record [starting at %d] must be at a split point' %
          record_start)

  def try_claim(self, record_start):
    with self._lock:
      # Attempted claim should be monotonous.
      if record_start <= self._last_attempted_record_start:
        raise ValueError(
            'Trying to return a record [starting at %d] which is not greater'
            'than the last-attempted record [starting at %d]' %
            (record_start, self._last_attempted_record_start))
      self._validate_record_start(record_start, True)
      self._last_attempted_record_start = record_start

View on GitHub (pinned to 12126d8942)