{"record":{"id":"424056e71a63f208","repo":"apache/beam","slug":"matchcontinuously-timestamp-cursor-true-deduplicates-so-it","errorCode":null,"errorMessage":"MatchContinuously(timestamp_cursor=True) deduplicates, so it requires has_deduplication=True.","messagePattern":"MatchContinuously\\(timestamp_cursor=True\\) deduplicates, so it requires has_deduplication=True\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/fileio.py","lineNumber":432,"sourceCode":"        to True) bound the deduplication state by last-modified time. By\n        default, all file modification history is tracked. If set to true, file\n        modification history prior to the max(mtime of last poll result) are\n        dropped, for better performance. A file that appears with an older\n        last-modified time is then taken as already seen and skipped.\n    \"\"\"\n\n    self.file_pattern = file_pattern\n    self.interval = interval\n    self.has_deduplication = has_deduplication\n    self.start_ts = start_timestamp\n    self.stop_ts = stop_timestamp\n    self.match_upd = match_updated_files\n    self.apply_windowing = apply_windowing\n    self.empty_match_treatment = empty_match_treatment\n    self.timestamp_cursor = timestamp_cursor\n    if timestamp_cursor:\n      if not has_deduplication:\n        raise ValueError(\n            'MatchContinuously(timestamp_cursor=True) deduplicates, so it '\n            'requires has_deduplication=True.')\n      if not match_updated_files:\n        _LOGGER.warning(\n            'MatchContinuously(timestamp_cursor=True) implies '\n            'match_updated_files=True.')\n        self.match_upd = True\n    else:\n      _LOGGER.warning(\n          'Matching Continuously is stateful, and can scale poorly. '\n          'Consider using Pub/Sub Notifications '\n          '(https://cloud.google.com/storage/docs/pubsub-notifications) '\n          'if possible')\n\n  def expand(self, pbegin) -> beam.PCollection[filesystem.FileMetadata]:\n    if Duration.of(self.interval).micros <= 0:\n      raise ValueError('MatchContinuously interval must be positive.')\n    if self.has_deduplication:","sourceCodeStart":414,"sourceCodeEnd":450,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/fileio.py#L414-L450","documentation":"MatchContinuously with timestamp_cursor=True relies on emitting files with timestamp-based cursors, which only works when duplicate files across polls are removed. The library therefore refuses the configuration unless has_deduplication is also enabled, since without deduplication the cursor semantics would emit duplicated matches.","triggerScenarios":"Constructing fileio.MatchContinuously(file_pattern=..., timestamp_cursor=True) while leaving has_deduplication=False (or omitted) in __init__.","commonSituations":"Developers enable timestamp_cursor to get growing-file support (e.g. continuous log files) without realizing it implies deduplication; often after copying an older snippet predating the has_deduplication flag.","solutions":["Pass has_deduplication=True when constructing MatchContinuously with timestamp_cursor=True.","If deduplication is not wanted, drop timestamp_cursor=False and use the default matching behavior.","Check the Beam version's MatchContinuously signature to confirm both flags exist."],"exampleFix":"# before\nMatchContinuously('/data/*.log', timestamp_cursor=True)\n# after\nMatchContinuously('/data/*.log', timestamp_cursor=True, has_deduplication=True)","handlingStrategy":"validation","validationCode":"if timestamp_cursor and not has_deduplication:\n    raise ValueError('timestamp_cursor=True requires has_deduplication=True')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Build MatchContinuously via a small wrapper function that enforces flag dependencies.","Review the docstring for implied flag combinations before enabling growing-file features."],"tags":["python","apache-beam","io","configuration"],"backgroundTag":"conflicting-config-options","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}