apache/beam · error · ValueError

Watermark must be monotonically increasing.Provided watermar

Error message

Watermark must be monotonically increasing.Provided watermark %s is less than current watermark %s

What it means

Watermarks must advance monotonically: the runner relies on watermarks never moving backward to trigger windows correctly. ManualWatermarkEstimator.set_watermark raises ValueError when the provided timestamp is earlier than the currently held watermark.

Source

Thrown at sdks/python/apache_beam/io/watermark_estimators.py:132

  def set_watermark(self, timestamp):
    # pylint: disable=line-too-long

    """Sets a timestamp before or at the timestamps of all future elements
    produced by the associated DoFn.

    This can be approximate. If records are output that violate this guarantee,
    they will be considered late, which will affect how they will be processed.
    See https://beam.apache.org/documentation/programming-guide/#watermarks-and-late-data
    for more information on late data and how to handle it.

    However, this value should be as late as possible. Downstream windows may
    not be able to close until this watermark passes their end.
    """
    if not isinstance(timestamp, Timestamp):
      raise ValueError('set_watermark expects a Timestamp as input')
    if self._watermark and self._watermark > timestamp:
      raise ValueError(
          'Watermark must be monotonically increasing.'
          'Provided watermark %s is less than '
          'current watermark %s',
          timestamp,
          self._watermark)
    self._watermark = timestamp

  @staticmethod
  def default_provider():
    """Provide a default WatermarkEstimatorProvider for
    WalltimeWatermarkEstimator.
    """
    class DefaultManualWatermarkEstimatorProvider(WatermarkEstimatorProvider):
      def initial_estimator_state(self, element, restriction):
        return None

      def create_watermark_estimator(self, estimator_state):
        return ManualWatermarkEstimator(estimator_state)

View on GitHub (pinned to 12126d8942)

Solutions

1. Only ever advance: estimator.set_watermark(max(current, new)) — check estimator.current_watermark() before setting
2. Derive the watermark as a monotone function (e.g. running max of event times) rather than per-element event time
3. If the earlier timestamp is legitimate, this estimator is the wrong choice — consider a different watermark estimator or let the runner estimate it

Example fix

// before
estimator.set_watermark(new_ts)
// after
if estimator.current_watermark() is None or new_ts > estimator.current_watermark():
    estimator.set_watermark(new_ts)

cur = estimator.current_watermark()
if cur is None or new_ts > cur:
    estimator.set_watermark(new_ts)

try:
    estimator.set_watermark(new_ts)
except ValueError:
    pass  # stale watermark; safely ignored since it must only move forward

When it happens

Trigger: Calling set_watermark with a Timestamp earlier than one set previously — e.g. processing out-of-order events and setting the watermark from each element's event time without taking the max, or recomputing watermarks from scratch on each bundle.

Common situations: Out-of-order streaming sources; restarting/replaying data causing older timestamps to arrive after newer ones; computing watermark per-record instead of as a running maximum.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Watermark must be monotonically increasing. Provided %s < cu
• To write a GlobalWindow unbounded PCollection, triggering_fr
• UnboundedSource restriction was neither finished nor checkpo
• set_watermark expects a Timestamp as input
• MLTransform only supports GlobalWindows when producing artif

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/535c1c5a648fb40f. Report an issue: GitHub.