{"record":{"id":"054011f851d35dc2","repo":"apache/beam","slug":"deduplicate-requires-at-least-providing-either-processing","errorCode":null,"errorMessage":"Deduplicate requires at least providing either processing_time_duration or event_time_duration.","messagePattern":"Deduplicate requires at least providing either processing_time_duration or event_time_duration\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/transforms/deduplicate.py","lineNumber":118,"sourceCode":"          self, seen_state=core.DoFn.StateParam(state_spec)):\n        seen_state.clear()\n\n    return DeduplicationFn()\n\n  def expand(self, pcoll):\n    return (\n        pcoll\n        | 'DeduplicateFn' >> core.ParDo(self._create_deduplicate_fn()))\n\n\nclass Deduplicate(ptransform.PTransform):\n  \"\"\"Similar to DeduplicatePerKey, the Deduplicate transform takes any arbitrary\n  value as input and uses value as key to deduplicate among certain amount of\n  time duration.\n  \"\"\"\n  def __init__(self, processing_time_duration=None, event_time_duration=None):\n    if processing_time_duration is None and event_time_duration is None:\n      raise ValueError(\n          'Deduplicate requires at least providing either '\n          'processing_time_duration or event_time_duration.')\n    self.processing_time_duration = processing_time_duration\n    self.event_time_duration = event_time_duration\n\n  def expand(self, pcoll):\n    return (\n        pcoll\n        | 'Use Value as Key' >> core.Map(lambda x: (x, None))\n        | 'DeduplicatePerKey' >> DeduplicatePerKey(\n            processing_time_duration=self.processing_time_duration,\n            event_time_duration=self.event_time_duration)\n        | 'Output Value' >> core.Map(lambda kv: kv[0]))\n","sourceCodeStart":100,"sourceCodeEnd":132,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/transforms/deduplicate.py#L100-L132","documentation":"The Deduplicate transform deduplicates values over a time duration and throws ValueError when constructed with neither processing_time_duration nor event_time_duration. Without a duration there is no window over which to remember seen values.","triggerScenarios":"Calling Deduplicate() with no arguments, or with both keyword arguments set to None, when applying it to a PCollection.","commonSituations":"Omitting the duration while migrating code between Deduplicate and Deduplicate.PerKey; assuming a default duration exists; passing config values that resolve to None.","solutions":["Provide a duration: pcoll | Deduplicate(processing_time_duration=timedelta(minutes=10)).","Or use event_time_duration for event-time-based dedup.","Verify any config-driven argument is not None before constructing the transform."],"exampleFix":"// before\npcoll | Deduplicate()\n// after\npcoll | Deduplicate(event_time_duration=timedelta(hours=1))","handlingStrategy":"validation","validationCode":"def build_dedup(processing_time_duration=None, event_time_duration=None):\n    if processing_time_duration is None and event_time_duration is None:\n        raise ValueError('configure one duration before constructing Deduplicate')\n    return Deduplicate(processing_time_duration=processing_time_duration,\n                       event_time_duration=event_time_duration)","typeGuard":"def has_dedup_duration(**kw) -> bool:\n    return kw.get('processing_time_duration') is not None or kw.get('event_time_duration') is not None","tryCatchPattern":"try:\n    out = pcoll | Deduplicate(**dedup_cfg)\nexcept ValueError as e:\n    if 'Deduplicate requires' in str(e):\n        out = pcoll | Deduplicate(event_time_duration=timedelta(hours=1))\n    else:\n        raise","preventionTips":["Always pass an explicit duration keyword","Assert config values are non-None before pipeline construction","Test pipeline construction (not just execution) in CI"],"tags":["apache-beam","python","deduplicate","configuration"],"backgroundTag":"missing-required-argument","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}