apache/beam · error

no GCS staging location specified. Use…

Error message

no GCS staging location specified. Use --staging_location=gs://<bucket>/<path>

What it means

The Dataflow runner needs a GCS staging location (a gs:// bucket/path) to stage binaries, temp files, and job artifacts. getJobOptions returns this error when the --staging_location flag is empty. GCS is mandatory because Dataflow jobs run on Google-managed workers that read staged artifacts from the bucket.

Solutions

  1. Pass --staging_location=gs://<bucket>/<path> to the pipeline
  2. Create the bucket first (gsutil mb gs://<bucket>) if it does not exist
  3. Set the flag programmatically in the runner options before Execute

Example fix

// before
--runner=dataflow --project=p --region=us-central1
// after
--runner=dataflow --project=p --region=us-central1 --staging_location=gs://my-bucket/staging
Defensive patterns

Strategy: validation

Validate before calling

if !strings.HasPrefix(stagingLocation, "gs://") {
    return errors.New("--staging_location must be gs://<bucket>/<path>")
}

Prevention

When it happens

Trigger: Executing a Dataflow pipeline without --staging_location=gs://<bucket>/<path> (the flag left at its default empty string).

Common situations: New pipelines where the developer set project/region but forgot staging; copying example commands that omit the flag; buckets existing but the flag never wired through in the launch script.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors


AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2b700c259e01ef24. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/dataflow/dataflow.go:279

	flag.Visit(func(f *flag.Flag) {
		if f.Name == name {
			found = true
		}
	})
	return found
}

func getJobOptions(ctx context.Context, streaming bool) (*dataflowlib.JobOptions, error) {
	project := gcpopts.GetProjectFromFlagOrEnvironment(ctx)
	if project == "" {
		return nil, errors.New("no Google Cloud project specified. Use --project=<project>")
	}
	region := gcpopts.GetRegion(ctx)
	if region == "" {
		return nil, errors.New("no Google Cloud region specified. Use --region=<region>. See https://cloud.google.com/dataflow/docs/concepts/regional-endpoints")
	}
	if *stagingLocation == "" {
		return nil, errors.New("no GCS staging location specified. Use --staging_location=gs://<bucket>/<path>")
	}

	checkSoftDeletePolicyEnabled(ctx, *stagingLocation, "staging_location")

	var jobLabels map[string]string
	if *labels != "" {
		if err := json.Unmarshal([]byte(*labels), &jobLabels); err != nil {
			return nil, errors.Wrapf(err, "error reading --label flag as JSON")
		}
	}

	if *cpuProfiling != "" {
		perf.EnableProfCaptureHook("gcs_profile_writer", *cpuProfiling)
	}

	if *autoscalingAlgorithm != "" {
		if *autoscalingAlgorithm != "NONE" && *autoscalingAlgorithm != "THROUGHPUT_BASED" {
			return nil, errors.New("invalid autoscaling algorithm. Use --autoscaling_algorithm=(NONE|THROUGHPUT_BASED)")

View on GitHub (pinned to 12126d8942)