apache/beam · error · RuntimeError

The --staging_location option must be specified.

Error message

The --staging_location option must be specified.

What it means

Dataflow job staging requires a place to upload worker packages; neither --staging_location nor a GCS default could be resolved from options, so _stage_resources aborts before building the job description.

Source

Thrown at sdks/python/apache_beam/runners/dataflow/internal/apiclient.py:602

          to_path,
          cached_path)
    else:
      self._uncached_gcs_file_copy(from_path, cached_path)

    FileSystems.copy(
        source_file_names=[cached_path], destination_file_names=[to_path])
    _LOGGER.info('Copied cached artifact from %s to %s', from_path, to_path)

  def _uncached_gcs_file_copy(self, from_path, to_path):
    to_folder, to_name = os.path.split(to_path)
    total_size = os.path.getsize(from_path)
    self.stage_file_with_retry(
        to_folder, to_name, from_path, total_size=total_size)

  def _stage_resources(self, pipeline, options):
    google_cloud_options = options.view_as(GoogleCloudOptions)
    if google_cloud_options.staging_location is None:
      raise RuntimeError('The --staging_location option must be specified.')
    if google_cloud_options.temp_location is None:
      raise RuntimeError('The --temp_location option must be specified.')

    resources = []
    staged_paths = {}
    staged_hashes = {}
    for _, env in sorted(pipeline.components.environments.items(),
                         key=lambda kv: kv[0]):
      for dep in env.dependencies:
        if dep.type_urn != common_urns.artifact_types.FILE.urn:
          raise RuntimeError('unsupported artifact type %s' % dep.type_urn)
        type_payload = beam_runner_api_pb2.ArtifactFilePayload.FromString(
            dep.type_payload)

        if dep.role_urn == common_urns.artifact_roles.STAGING_TO.urn:
          remote_name = (
              beam_runner_api_pb2.ArtifactStagingToRolePayload.FromString(
                  dep.role_payload)).staged_name

View on GitHub (pinned to 12126d8942)

Solutions

1. Set --staging_location=gs://<bucket>/<path> in pipeline options
2. Or ensure --temp_location is set, since __init__ defaults staging_location to temp_location
3. Use GoogleCloudOptions(staging_location='gs://...') programmatically

Example fix

# before
options = PipelineOptions(['--project=p', '--job_name=j'])
# after
options = PipelineOptions([
    '--project=p', '--job_name=j',
    '--staging_location=gs://my-bucket/staging',
    '--temp_location=gs://my-bucket/temp'])

gco = options.view_as(GoogleCloudOptions)
if gco.staging_location is None and gco.temp_location is None:
    raise SystemExit('set --staging_location or --temp_location')

def has_staging(options) -> bool:
    return options.view_as(GoogleCloudOptions).staging_location is not None

When it happens

Trigger: Submitting a Dataflow job where staging_location was not set and the earlier __init__ defaulting (temp_location fallback) also left it None — typically reached via create_job_description.

Common situations: Older or unusual option-passing paths (e.g. constructing Job directly, template flows) that bypass the __init__ validation/defaulting; temp_location itself missing so no default could be applied.

Understand the failure class

Background: "--flag is required" and "must specify" CLI errors: how missing-required-flag validation works and how to fix it — this error's family across 20 libraries.

Related errors

• The --temp_location option must be specified.
• Error constructing default value for gcpTempLocation: tempLo
• Sorter doesn't support GCS temporary location.
• BigQuery temp location expected a valid 'gs://' path, but wa
• Unable to fetch file %s to be used locally to create a Kafka

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4d972a0dc1056453. Report an issue: GitHub.