{"record":{"id":"578bb7cd2039d73f","repo":"apache/beam","slug":"matchcontinuously-deduplicates-by-last-modified-time-but-s","errorCode":null,"errorMessage":"MatchContinuously deduplicates by last-modified time, but %s reports none.","messagePattern":"MatchContinuously deduplicates by last-modified time, but (.+?) reports none\\.","errorType":"exception","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/fileio.py","lineNumber":300,"sourceCode":"\n  def on_poll_complete(self, state):\n    poll_micros = self._clock.last_poll_micros\n    if poll_micros is not None and poll_micros >= self._start_micros:\n      return state + 1\n    return state\n\n  def can_stop_polling(self, now, state):\n    return state >= self._max_polls\n\n  def state_coder(self):\n    return VarIntCoder()\n\n\ndef _ensure_mtime(metadata: filesystem.FileMetadata) -> float:\n  # A missing (zero) timestamp is rejected because every file would then carry\n  # the same one, and updates could never be told apart.\n  if not metadata.last_updated_in_seconds:\n    raise BeamIOError(\n        'MatchContinuously deduplicates by last-modified time, but %s reports '\n        'none.' % metadata.path)\n  return metadata.last_updated_in_seconds\n\n\ndef _file_path_key(metadata: filesystem.FileMetadata) -> str:\n  return metadata.path\n\n\ndef _file_path_and_mtime_key(\n    metadata: filesystem.FileMetadata) -> tuple[str, float]:\n  # Keying on the last-modified time makes a changed file look new again.\n  return metadata.path, _ensure_mtime(metadata)\n\n\nclass _MatchContinuouslyPollFn(PollFn):\n  \"\"\"Polls a file pattern, honoring empty-match rules.\n","sourceCodeStart":282,"sourceCodeEnd":318,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/fileio.py#L282-L318","documentation":"_ensure_mtime rejects FileMetadata whose last_updated_in_seconds is 0/None because MatchContinuously deduplicates and timestamps its output by last-modified time; a file without a timestamp could never be distinguished from other updates. It raises BeamIOError naming the offending path.","triggerScenarios":"A filesystem implementation (custom FileSystem subclass, or a backend like some local/HDFS listings) returns metadata with last_updated_in_seconds unset for a file being polled by MatchContinuously.","commonSituations":"Custom FileSystem connectors that forget to populate last_updated_in_seconds; storage backends that don't expose mtime; test doubles with zeroed metadata.","solutions":["Fix the FileSystem implementation to populate last_updated_in_seconds from the backend's mtime","Switch to a filesystem connector that reports modification times","If the backend truly has no mtime, use MatchFiles/polling logic that keys on path instead of MatchContinuously"],"exampleFix":"// before\nreturn filesystem.FileMetadata(path=p, size_in_bytes=sz)  # no mtime\n// after\nimport time\nreturn filesystem.FileMetadata(path=p, size_in_bytes=sz, last_updated_in_seconds=time.time())","handlingStrategy":"try-catch","validationCode":"from apache_beam.io import filesystem\nmetas = FileSystems.match([pattern])[0].metadata_list\nmissing = [m.path for m in metas if not m.last_updated_in_seconds]\nif missing:\n    raise ValueError(f'No mtime reported for: {missing}')","typeGuard":"def has_mtime(m) -> bool:\n    return bool(m.last_updated_in_seconds)","tryCatchPattern":"try:\n    updates = p | MatchContinuously(pattern)\nexcept Exception as e:\n    if 'reports none' in str(e):\n        logging.error('Filesystem %s does not report mtimes; MatchContinuous unusable', pattern)\n    else:\n        raise","preventionTips":["Verify your FileSystem connector populates last_updated_in_seconds","Test custom filesystems against MatchContinuously before production","Fall back to scheduled MatchFiles pipelines for backends without mtimes"],"tags":["io","file-metadata","streaming"],"backgroundTag":"missing-required-argument","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}