apache/beam · error · ValueError

MatchContinuously(timestamp_cursor=True) deduplicates, so it

Error message

MatchContinuously(timestamp_cursor=True) deduplicates, so it requires has_deduplication=True.

What it means

MatchContinuously with timestamp_cursor=True relies on emitting files with timestamp-based cursors, which only works when duplicate files across polls are removed. The library therefore refuses the configuration unless has_deduplication is also enabled, since without deduplication the cursor semantics would emit duplicated matches.

Source

Thrown at sdks/python/apache_beam/io/fileio.py:432

        to True) bound the deduplication state by last-modified time. By
        default, all file modification history is tracked. If set to true, file
        modification history prior to the max(mtime of last poll result) are
        dropped, for better performance. A file that appears with an older
        last-modified time is then taken as already seen and skipped.
    """

    self.file_pattern = file_pattern
    self.interval = interval
    self.has_deduplication = has_deduplication
    self.start_ts = start_timestamp
    self.stop_ts = stop_timestamp
    self.match_upd = match_updated_files
    self.apply_windowing = apply_windowing
    self.empty_match_treatment = empty_match_treatment
    self.timestamp_cursor = timestamp_cursor
    if timestamp_cursor:
      if not has_deduplication:
        raise ValueError(
            'MatchContinuously(timestamp_cursor=True) deduplicates, so it '
            'requires has_deduplication=True.')
      if not match_updated_files:
        _LOGGER.warning(
            'MatchContinuously(timestamp_cursor=True) implies '
            'match_updated_files=True.')
        self.match_upd = True
    else:
      _LOGGER.warning(
          'Matching Continuously is stateful, and can scale poorly. '
          'Consider using Pub/Sub Notifications '
          '(https://cloud.google.com/storage/docs/pubsub-notifications) '
          'if possible')

  def expand(self, pbegin) -> beam.PCollection[filesystem.FileMetadata]:
    if Duration.of(self.interval).micros <= 0:
      raise ValueError('MatchContinuously interval must be positive.')
    if self.has_deduplication:

View on GitHub (pinned to 12126d8942)

Solutions

1. Pass has_deduplication=True when constructing MatchContinuously with timestamp_cursor=True.
2. If deduplication is not wanted, drop timestamp_cursor=False and use the default matching behavior.
3. Check the Beam version's MatchContinuously signature to confirm both flags exist.

Example fix

# before
MatchContinuously('/data/*.log', timestamp_cursor=True)
# after
MatchContinuously('/data/*.log', timestamp_cursor=True, has_deduplication=True)

if timestamp_cursor and not has_deduplication:
    raise ValueError('timestamp_cursor=True requires has_deduplication=True')

When it happens

Trigger: Constructing fileio.MatchContinuously(file_pattern=..., timestamp_cursor=True) while leaving has_deduplication=False (or omitted) in __init__.

Common situations: Developers enable timestamp_cursor to get growing-file support (e.g. continuous log files) without realizing it implies deduplication; often after copying an older snippet predating the has_deduplication flag.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

• Cannot create a temporary directory for root path prefix %s.
• Encountered an Atomic type that is not currently supported b
• Please specify a BigQuery table to read from.
• nrows not yet supported
• Found no files that match {self.path!r}

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/424056e71a63f208. Report an issue: GitHub.