apache/beam · error · NotFound

source blob not found during copying

Error message

source blob %s not found during copying

What it means

GcsIO.copy raises google.cloud.exceptions.NotFound ('source blob %s not found during copying') when _use_blob_generation is enabled and the source blob cannot be fetched before copying. Beam reads the source generation to make the copy generation-safe; a missing source aborts the copy.

Solutions

1. Verify the source object exists (gcsio.exists / blob listing) before copying.
2. Correct the source gs:// path (bucket and object name).
3. Catch NotFound and skip/recreate the missing source as appropriate.
4. If generation safety is not required, disable _use_blob_generation so a stale-generation reference isn't needed (still fails on missing object at copy time).

Example fix

// before
gcsio.copy(src, dst)
// after
if gcsio.exists(src):
  gcsio.copy(src, dst)
else:
  logging.warning('skip missing %s', src)

if not gcsio.exists(src):
    raise FileNotFoundError(src)

def is_gcs_path(p):
    return isinstance(p, str) and re.match(r'^gs://[^/]+/.+$', p) is not None

from google.cloud.exceptions import NotFound
try:
    gcsio.copy(src, dst)
except NotFound:
    logging.warning('source missing, skipping: %s', src)

When it happens

Trigger: Calling GcsIO.copy (directly or via copytree/rename) with a source gs:// path whose object does not exist while blob-generation mode is enabled.

Common situations: Renaming files already moved/deleted by a concurrent job; typo'd source object names; copying objects deleted by lifecycle rules before the copy runs.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Rewrite from to has failed. Either source or sink not…
• Invalid state for Rewrite, from=
• Object not found
• %s
• Target object already exists and strategy is FAIL_IF_EXISTS

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/560a073e27499df0. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/gcsio.py:450

    return final_results

  def copy(self, src, dest):
    """Copies the given GCS object from src to dest.

    Args:
      src: GCS file path pattern in the form gs://<bucket>/<name>.
      dest: GCS file path pattern in the form gs://<bucket>/<name>.

    Raises:
      Any exceptions during copying
    """
    src_bucket_name, src_blob_name = parse_gcs_path(src)
    dest_bucket_name, dest_blob_name= parse_gcs_path(dest, object_optional=True)
    src_bucket = self.client.bucket(src_bucket_name)
    if self._use_blob_generation:
      src_blob = src_bucket.get_blob(src_blob_name)
      if src_blob is None:
        raise NotFound("source blob %s not found during copying" % src)
      src_generation = src_blob.generation
    else:
      src_blob = src_bucket.blob(src_blob_name)
      src_generation = None
    dest_bucket = self.client.bucket(dest_bucket_name)
    if not dest_blob_name:
      dest_blob_name = None
    src_bucket.copy_blob(
        src_blob,
        dest_bucket,
        new_name=dest_blob_name,
        source_generation=src_generation,
        retry=self._storage_client_retry)

  def _copy_batch_request(self, pair):
    src_bucket_name, src_blob_name = parse_gcs_path(pair[0])
    dest_bucket_name, dest_blob_name = parse_gcs_path(pair[1])
    src_bucket = self.client.bucket(src_bucket_name)

View on GitHub (pinned to 12126d8942)