apache/beam · error · BeamIOError

src and dst files do not exist. src: %s, dst: %s

Error message

src and dst files do not exist. src: %s, dst: %s

What it means

During finalize_write, FileBasedSink verifies each source temp shard or its final destination exists before renaming; if neither the src temp file nor the dst final file is present in the glob results, it raises BeamIOError. This usually means the temp output vanished or was never written.

Source

Thrown at sdks/python/apache_beam/io/filebasedsink.py:329

    dst_glob = self._get_final_name_glob(num_shards, window)
    src_glob_files = set(
        file_metadata.path for mr in FileSystems.match([src_glob])
        for file_metadata in mr.metadata_list)
    dst_glob_files = set(
        file_metadata.path for mr in FileSystems.match([dst_glob])
        for file_metadata in mr.metadata_list)

    src_files = []
    dst_files = []
    delete_files = []
    num_skipped = 0
    for shard_num, src in enumerate(writer_results):
      final_name = self._get_final_name(shard_num, num_shards, window)
      dst = final_name
      src_exists = src in src_glob_files
      dst_exists = dst in dst_glob_files
      if not src_exists and not dst_exists:
        raise BeamIOError(
            'src and dst files do not exist. src: %s, dst: %s' % (src, dst))
      if not src_exists and dst_exists:
        _LOGGER.debug('src: %s -> dst: %s already renamed, skipping', src, dst)
        num_skipped += 1
        continue
      if (src_exists and dst_exists and
          FileSystems.checksum(src) == FileSystems.checksum(dst)):
        _LOGGER.debug('src: %s == dst: %s, deleting src', src, dst)
        delete_files.append(src)
        continue

      src_files.append(src)
      dst_files.append(dst)

    self._report_sink_lineage(dst_glob, dst_files)
    return src_files, dst_files, delete_files, num_skipped

  def _report_sink_lineage(self, dst_glob, dst_files):

View on GitHub (pinned to 12126d8942)

Solutions

1. Check that the output prefix and temp files still exist in the storage system before finalizing
2. Disable/adjust bucket lifecycle or cleanup policies that delete beam-temp-* files mid-pipeline
3. Rerun the pipeline; transient GCS/FS consistency issues often resolve
4. Inspect _LOGGER.debug output for shards that were skipped as already renamed

import apache_beam.io.filesystem as fs
existing = set(FileSystems.match([src_glob, dst_glob]))
# ensure src or dst visible before finalize

try:
    result = pipeline.run()
    result.wait_until_finish()
except apache_beam.error.BeamIOError as e:
    if 'src and dst files do not exist' in str(e):
        # verify temp files, then retry the finalize/pipeline

When it happens

Trigger: Temp shard files deleted or expired before finalize (e.g. gs lifecycle cleanup), writer results referencing files in a different location than the glob, or the pipeline wrote zero files for a shard.

Common situations: Concurrent jobs or cleanup scripts removing beam-temp files; misconfigured output prefix so the glob check misses the files; retries racing with renames.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Found no files that match {self.path!r}
• Encountered an Atomic type that is not currently supported b
• File spec %s not found
• Error completing file copies with retries, sample: from %s t
• Error trying to delete %s: %s

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0faa4248d9a12566. Report an issue: GitHub.