apache/beam · error · ValueError

Trying to return a record

Error message

Trying to return a record [starting at %d] which is before the last-returned record [starting at %d]

What it means

In ordered-record mode, OffsetRangeTracker requires record start positions to be monotonically non-decreasing. When _validate_record_start sees a record starting before the last returned record's start, it raises this ValueError because resuming from such a position would duplicate or skip records.

Solutions

1. Ensure records are claimed in non-decreasing start-offset order (sort or resume from the last returned position).
2. Resume from the last attempted position (tracker._last_attempted_record_start) after a failure instead of restarting at 0.
3. Do not re-claim already-returned records; advance past them.

Example fix

// before
tracker.try_claim(current_record.offset)  # < last returned offset
// after
if current_record.offset >= tracker._last_attempted_record_start:
    tracker.try_claim(current_record.offset)

if tracker._last_record_start != -1 and record_start < tracker._last_record_start:
    raise ValueError("record start regressed")

def is_monotonic(record_start: int, last: int) -> bool:
    return last == -1 or record_start >= last

try:
    tracker.try_claim(record_start)
except ValueError as e:
    if 'before the last-returned record' in str(e):
        resume_from(tracker._last_attempted_record_start)

When it happens

Trigger: try_claim(record_start) with a record_start lower than the previously returned record's start; calling set_current_position with a regressed position.

Common situations: Custom sources that re-read a file from the beginning after failure without resuming from the last checkpoint; out-of-order record delivery when reading unsorted data; rewinding an iterator on retry.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Record at a split point has same offset as the previous…
• The first record [starting at ] must be at a split point
• End offset must not be 'None'
• Start offset must not be 'None'
• This function must only be called under the lock self.lock.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6758d0bb8f5cf705. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/range_trackers.py:101

  @property
  def last_attempted_record_start(self):
    """Return current value of last_attempted_record_start.

    last_attempted_record_start records a valid position that tried to be
    claimed by calling try_claim(). This value is only updated by `try_claim()`
    no matter `try_claim()` returns `True` or `False`.
    """
    return self._last_attempted_record_start

  def _validate_record_start(self, record_start, split_point):
    # This function must only be called under the lock self.lock.
    if not self._lock.locked():
      raise ValueError(
          'This function must only be called under the lock self.lock.')

    if record_start < self._last_record_start:
      raise ValueError(
          'Trying to return a record [starting at %d] which is before the '
          'last-returned record [starting at %d]' %
          (record_start, self._last_record_start))

    if (split_point and self._offset_of_last_split_point != -1 and
        record_start == self._offset_of_last_split_point):
      raise ValueError(
          'Record at a split point has same offset as the previous split '
          'point: %d' % record_start)

    if not split_point and self._last_record_start == -1:
      raise ValueError(
          'The first record [starting at %d] must be at a split point' %
          record_start)

  def try_claim(self, record_start):
    with self._lock:
      # Attempted claim should be monotonous.

View on GitHub (pinned to 12126d8942)