apache/beam · error · BeamIOError

MatchContinuously deduplicates by last-modified time, but

Error message

MatchContinuously deduplicates by last-modified time, but %s reports none.

What it means

_ensure_mtime rejects FileMetadata whose last_updated_in_seconds is 0/None because MatchContinuously deduplicates and timestamps its output by last-modified time; a file without a timestamp could never be distinguished from other updates. It raises BeamIOError naming the offending path.

Solutions

1. Fix the FileSystem implementation to populate last_updated_in_seconds from the backend's mtime
2. Switch to a filesystem connector that reports modification times
3. If the backend truly has no mtime, use MatchFiles/polling logic that keys on path instead of MatchContinuously

Example fix

// before
return filesystem.FileMetadata(path=p, size_in_bytes=sz)  # no mtime
// after
import time
return filesystem.FileMetadata(path=p, size_in_bytes=sz, last_updated_in_seconds=time.time())

from apache_beam.io import filesystem
metas = FileSystems.match([pattern])[0].metadata_list
missing = [m.path for m in metas if not m.last_updated_in_seconds]
if missing:
    raise ValueError(f'No mtime reported for: {missing}')

def has_mtime(m) -> bool:
    return bool(m.last_updated_in_seconds)

try:
    updates = p | MatchContinuously(pattern)
except Exception as e:
    if 'reports none' in str(e):
        logging.error('Filesystem %s does not report mtimes; MatchContinuous unusable', pattern)
    else:
        raise

When it happens

Trigger: A filesystem implementation (custom FileSystem subclass, or a backend like some local/HDFS listings) returns metadata with last_updated_in_seconds unset for a file being polled by MatchContinuously.

Common situations: Custom FileSystem connectors that forget to populate last_updated_in_seconds; storage backends that don't expose mtime; test doubles with zeroed metadata.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• chunk write failed
• Failed to read elements from the bounded reader.
• last_updated operation failed
• Metadata operation failed
• A schema is required to write non-schema'd data.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/578bb7cd2039d73f. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/fileio.py:300

  def on_poll_complete(self, state):
    poll_micros = self._clock.last_poll_micros
    if poll_micros is not None and poll_micros >= self._start_micros:
      return state + 1
    return state

  def can_stop_polling(self, now, state):
    return state >= self._max_polls

  def state_coder(self):
    return VarIntCoder()


def _ensure_mtime(metadata: filesystem.FileMetadata) -> float:
  # A missing (zero) timestamp is rejected because every file would then carry
  # the same one, and updates could never be told apart.
  if not metadata.last_updated_in_seconds:
    raise BeamIOError(
        'MatchContinuously deduplicates by last-modified time, but %s reports '
        'none.' % metadata.path)
  return metadata.last_updated_in_seconds


def _file_path_key(metadata: filesystem.FileMetadata) -> str:
  return metadata.path


def _file_path_and_mtime_key(
    metadata: filesystem.FileMetadata) -> tuple[str, float]:
  # Keying on the last-modified time makes a changed file look new again.
  return metadata.path, _ensure_mtime(metadata)


class _MatchContinuouslyPollFn(PollFn):
  """Polls a file pattern, honoring empty-match rules.

View on GitHub (pinned to 12126d8942)