{"record":{"id":"0faa4248d9a12566","repo":"apache/beam","slug":"src-and-dst-files-do-not-exist-src-s-dst-s","errorCode":null,"errorMessage":"src and dst files do not exist. src: %s, dst: %s","messagePattern":"src and dst files do not exist\\. src: (.+?), dst: (.+?)","errorType":"exception","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/filebasedsink.py","lineNumber":329,"sourceCode":"    dst_glob = self._get_final_name_glob(num_shards, window)\n    src_glob_files = set(\n        file_metadata.path for mr in FileSystems.match([src_glob])\n        for file_metadata in mr.metadata_list)\n    dst_glob_files = set(\n        file_metadata.path for mr in FileSystems.match([dst_glob])\n        for file_metadata in mr.metadata_list)\n\n    src_files = []\n    dst_files = []\n    delete_files = []\n    num_skipped = 0\n    for shard_num, src in enumerate(writer_results):\n      final_name = self._get_final_name(shard_num, num_shards, window)\n      dst = final_name\n      src_exists = src in src_glob_files\n      dst_exists = dst in dst_glob_files\n      if not src_exists and not dst_exists:\n        raise BeamIOError(\n            'src and dst files do not exist. src: %s, dst: %s' % (src, dst))\n      if not src_exists and dst_exists:\n        _LOGGER.debug('src: %s -> dst: %s already renamed, skipping', src, dst)\n        num_skipped += 1\n        continue\n      if (src_exists and dst_exists and\n          FileSystems.checksum(src) == FileSystems.checksum(dst)):\n        _LOGGER.debug('src: %s == dst: %s, deleting src', src, dst)\n        delete_files.append(src)\n        continue\n\n      src_files.append(src)\n      dst_files.append(dst)\n\n    self._report_sink_lineage(dst_glob, dst_files)\n    return src_files, dst_files, delete_files, num_skipped\n\n  def _report_sink_lineage(self, dst_glob, dst_files):","sourceCodeStart":311,"sourceCodeEnd":347,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/filebasedsink.py#L311-L347","documentation":"During finalize_write, FileBasedSink verifies each source temp shard or its final destination exists before renaming; if neither the src temp file nor the dst final file is present in the glob results, it raises BeamIOError. This usually means the temp output vanished or was never written.","triggerScenarios":"Temp shard files deleted or expired before finalize (e.g. gs lifecycle cleanup), writer results referencing files in a different location than the glob, or the pipeline wrote zero files for a shard.","commonSituations":"Concurrent jobs or cleanup scripts removing beam-temp files; misconfigured output prefix so the glob check misses the files; retries racing with renames.","solutions":["Check that the output prefix and temp files still exist in the storage system before finalizing","Disable/adjust bucket lifecycle or cleanup policies that delete beam-temp-* files mid-pipeline","Rerun the pipeline; transient GCS/FS consistency issues often resolve","Inspect _LOGGER.debug output for shards that were skipped as already renamed"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"import apache_beam.io.filesystem as fs\nexisting = set(FileSystems.match([src_glob, dst_glob]))\n# ensure src or dst visible before finalize","typeGuard":null,"tryCatchPattern":"try:\n    result = pipeline.run()\n    result.wait_until_finish()\nexcept apache_beam.error.BeamIOError as e:\n    if 'src and dst files do not exist' in str(e):\n        # verify temp files, then retry the finalize/pipeline","preventionTips":["Disable lifecycle rules that remove beam-temp-* files during runs","Keep output prefixes unique per job to avoid rename races","Monitor storage availability; rerun transient failures"],"tags":["python","apache-beam","io","gcs","file-not-found"],"backgroundTag":"file-not-found","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}