apache/beam · error · ValueError

Destination %r must be GCS path.

Error message

Destination %r must be GCS path.

What it means

GCSFileSystem._copy_path copies files/trees within GCS and requires the destination path to start with 'gs://'. Passing a non-GCS destination raises this ValueError, since the GCS copy API cannot write outside GCS.

Solutions

1. Ensure every destination starts with 'gs://'.
2. Use matching schemes: copy local-to-local via LocalFileSystem, GCS-to-GCS via GCSFileSystem.
3. For local downloads, use the appropriate beam.io/gcsio download API instead of FileSystems.copy.

Example fix

# before
FileSystems.copy(src_gcs_list, ['/tmp/out'])
# after
FileSystems.copy(src_gcs_list, ['gs://my-bucket/out'])

def assert_gcs_destinations(destinations):
    bad = [d for d in destinations if not str(d).startswith('gs://')]
    if bad:
        raise ValueError('Destinations must be gs:// paths: %r' % bad)
    return destinations

def all_gcs_paths(paths):
    return all(isinstance(p, str) and p.startswith('gs://') for p in paths)

try:
    FileSystems.copy(srcs, dests)
except ValueError as e:
    logging.error('Copy destination scheme invalid: %s', e)
    raise

When it happens

Trigger: Calling FileSystems.copy([...gs source...], ['/local/dest']) or GCSFileSystem copy helpers with a local or other-scheme destination.

Common situations: Configuring a local output/temp destination while Beam resolves the copy through the GCS file system; mismatched source/destination schemes in custom copy code.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

• Basepath %r must be GCS path.
• Invalid path
• Path %r must be GCS path.
• cache_root GCS bucket path is invalid.
• combine_fn must be specified.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/416662184eda49c4. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/gcsfilesystem.py:210

    """Recursively copy the file tree from the source to the destination

    Args:
      source_file_names: list of source file objects that needs to be copied
      destination_file_names: list of destination of the new object

    Raises:
      ``BeamIOError``: if any of the copy operations fail
    """
    err_msg = (
        "source_file_names and destination_file_names should "
        "be equal in length")
    assert len(source_file_names) == len(destination_file_names), err_msg

    def _copy_path(source, destination):
      """Recursively copy the file tree from the source to the destination
      """
      if not destination.startswith(GCSFileSystem.GCS_PREFIX):
        raise ValueError('Destination %r must be GCS path.' % destination)
      # Use copy_tree if the path ends with / as it is a directory
      if source.endswith('/'):
        self._gcsIO().copytree(source, destination)
      else:
        self._gcsIO().copy(source, destination)

    exceptions = {}
    for source, destination in zip(source_file_names, destination_file_names):
      try:
        _copy_path(source, destination)
      except Exception as e:  # pylint: disable=broad-except
        exceptions[(source, destination)] = e

    if exceptions:
      raise BeamIOError("Copy operation failed", exceptions)

  def rename(self, source_file_names, destination_file_names):
    """Rename the files at the source list to the destination list.

View on GitHub (pinned to 12126d8942)