apache/beam · error · RuntimeError

The --temp_location option must be specified.

Error message

The --temp_location option must be specified.

What it means

Raised during _stage_resources (invoked from create_job_description) when preparing a Dataflow job: staging the pipeline's resources to GCS requires a staging location, and the --temp_location option (or the stage/temp_location derived from it) was not set on the pipeline's GoogleCloudOptions. It fires at job-creation time when neither --temp_location nor an implicit temp/staging location could be resolved from the options, so the remote workflow cannot upload files.

Source

Thrown at sdks/python/apache_beam/runners/dataflow/internal/apiclient.py:604

    else:
      self._uncached_gcs_file_copy(from_path, cached_path)

    FileSystems.copy(
        source_file_names=[cached_path], destination_file_names=[to_path])
    _LOGGER.info('Copied cached artifact from %s to %s', from_path, to_path)

  def _uncached_gcs_file_copy(self, from_path, to_path):
    to_folder, to_name = os.path.split(to_path)
    total_size = os.path.getsize(from_path)
    self.stage_file_with_retry(
        to_folder, to_name, from_path, total_size=total_size)

  def _stage_resources(self, pipeline, options):
    google_cloud_options = options.view_as(GoogleCloudOptions)
    if google_cloud_options.staging_location is None:
      raise RuntimeError('The --staging_location option must be specified.')
    if google_cloud_options.temp_location is None:
      raise RuntimeError('The --temp_location option must be specified.')

    resources = []
    staged_paths = {}
    staged_hashes = {}
    for _, env in sorted(pipeline.components.environments.items(),
                         key=lambda kv: kv[0]):
      for dep in env.dependencies:
        if dep.type_urn != common_urns.artifact_types.FILE.urn:
          raise RuntimeError('unsupported artifact type %s' % dep.type_urn)
        type_payload = beam_runner_api_pb2.ArtifactFilePayload.FromString(
            dep.type_payload)

        if dep.role_urn == common_urns.artifact_roles.STAGING_TO.urn:
          remote_name = (
              beam_runner_api_pb2.ArtifactStagingToRolePayload.FromString(
                  dep.role_payload)).staged_name
          is_staged_role = True
        else:

View on GitHub (pinned to 12126d8942)

Solutions

1. Add --temp_location=gs://<bucket>/<path> to pipeline options
2. Set it via GoogleCloudOptions(temp_location='gs://...') before creating the Job
3. If staging_location is set but temp is not, remember __init__ defaults staging from temp, not the reverse — set both explicitly

Example fix

# before
job = Job(pipeline, options)  # options lacks temp_location
# after
options.view_as(GoogleCloudOptions).temp_location = 'gs://my-bucket/temp'
job = Job(pipeline, options)

if options.view_as(GoogleCloudOptions).temp_location is None:
    raise SystemExit('set --temp_location (gs://bucket/temp)')

def has_temp(options) -> bool:
    return options.view_as(GoogleCloudOptions).temp_location is not None

When it happens

Trigger: create_job_description() -> _stage_resources() with options lacking --temp_location (and no earlier validation catching it, e.g. Job constructed directly).

Common situations: Job objects built manually or by tooling that skips DataflowJob __init__ validation; template-based submissions missing temp_location; options parsed from a config file that dropped the key.

Understand the failure class

Background: "--flag is required" and "must specify" CLI errors: how missing-required-flag validation works and how to fix it — this error's family across 20 libraries.

Related errors

• The --staging_location option must be specified.
• Error constructing default value for gcpTempLocation: tempLo
• Sorter doesn't support GCS temporary location.
• BigQuery temp location expected a valid 'gs://' path, but wa
• Unable to fetch file %s to be used locally to create a Kafka

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/cf383bd6072bf48d. Report an issue: GitHub.