apache/beam · error · ValueError

Missing required configuration parameters

Error message

Missing required configuration parameters: %s

What it means

ValueError raised in DataflowPipelineOptions validation when any of the required Google Cloud options — project, job_name, or temp_location — is missing or falsy. Dataflow cannot identify the GCP project, name the job, or stage temporary files without these.

Solutions

  1. Pass all three options: --project, --job_name, and --temp_location (a GCS path like gs://bucket/temp)
  2. Set them programmatically via GoogleCloudOptions before calling pipeline.run()
  3. Verify the options object actually carries the values (e.g. print(options.view_as(GoogleCloudOptions).project)) — config-file keys may not map as expected

Example fix

# before
options = PipelineOptions(['--runner=DataflowRunner'])
# after
options = PipelineOptions([
    '--runner=DataflowRunner',
    '--project=my-gcp-project',
    '--job_name=my-job',
    '--temp_location=gs://my-bucket/temp'])
Defensive patterns

Strategy: validation

Validate before calling

gco = options.view_as(GoogleCloudOptions)
missing = [o for o in ('project', 'job_name', 'temp_location') if not getattr(gco, o)]
if missing:
    raise SystemExit('missing: %s' % missing)

Try / catch

try:
    pipeline.run()
except ValueError as e:
    if 'Missing required configuration' in str(e):
        print('Add --project/--job_name/--temp_location')
    raise

Prevention

When it happens

Trigger: Running with the DataflowRunner while PipelineOptions lacks --project, --job_name, or --temp_location (or they are set to empty strings).

Common situations: Local runs that worked with DirectRunner being submitted to Dataflow without adding GCP options; CI pipelines missing flags; job_name or project accidentally overridden to None by config loading.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4f647a20fa93f486. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/dataflow/internal/apiclient.py:415

      job_name = Job._build_default_job_name(getpass.getuser())
    return job_name

  def __init__(self, options, proto_pipeline):
    self.options = options
    validate_pipeline_graph(proto_pipeline)
    self.proto_pipeline = proto_pipeline
    self.google_cloud_options = options.view_as(GoogleCloudOptions)
    if not self.google_cloud_options.job_name:
      self.google_cloud_options.job_name = self.default_job_name(
          self.google_cloud_options.job_name)

    required_google_cloud_options = ['project', 'job_name', 'temp_location']
    missing = [
        option for option in required_google_cloud_options
        if not getattr(self.google_cloud_options, option)
    ]
    if missing:
      raise ValueError(
          'Missing required configuration parameters: %s' % missing)

    if not self.google_cloud_options.staging_location:
      _LOGGER.info(
          'Defaulting to the temp_location as staging_location: %s',
          self.google_cloud_options.temp_location)
      (
          self.google_cloud_options.staging_location
      ) = self.google_cloud_options.temp_location

    self.root_staging_location = self.google_cloud_options.staging_location

    # Make the staging and temp locations job name and time specific. This is
    # needed to avoid clashes between job submissions using the same staging
    # area or team members using same job names. This method is not entirely
    # foolproof since two job submissions with same name can happen at exactly
    # the same time. However the window is extremely small given that
    # time.time() has at least microseconds granularity. We add the suffix only

View on GitHub (pinned to 12126d8942)