apache/beam · error

Both worker_harness_container_image and sdk_container_image…

Error message

Both worker_harness_container_image and sdk_container_image cannot both be set. Prefer sdk_container_image, worker_harness_container_image is deprecated.

What it means

getContainerImage enforces that only one of the deprecated worker_harness_container_image flag and the newer sdk_container_image flag is supplied. Setting both is ambiguous, so the function panics and asks the user to prefer sdk_container_image.

Solutions

1. Remove --worker_harness_container_image and keep only --sdk_container_image
2. If both come from different config layers, unset the deprecated flag in the launch script or template
3. Search the job submission code/Makefile/CI for worker_harness_container_image and delete it
4. Update documentation/templates to reference sdk_container_image only

Example fix

// before
go run . --worker_harness_container_image=gcr.io/x/y:1 --sdk_container_image=gcr.io/x/y:2
// after
go run . --sdk_container_image=gcr.io/x/y:2

if workerHarnessImage != "" && sdkContainerImage != "" {
    return errors.New("set only --sdk_container_image")
}

func buildArgs(flags map[string]string) error {
    if flags["worker_harness_container_image"] != "" && flags["sdk_container_image"] != "" {
        return errors.New("mutually exclusive flags: keep sdk_container_image")
    }
    return nil
}

When it happens

Trigger: Launching a Dataflow job with both --worker_harness_container_image and --sdk_container_image set to non-empty values.

Common situations: Migrating old job-launch scripts that still pass the deprecated flag while templating/pipeline code adds the new one, or CI configs accumulating both flags over time.

Related errors

• cannot use option zone with workerRegion; prefer either…
• Native sinks no longer implemented; ignoring…
• workerRegion and workerZone options are mutually exclusive
• A non-standard version of Beam SDK detected
• bad I

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d98c5269fe714adf. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/dataflow/dataflow.go:456

	if err != nil {
		panic(fmt.Sprintf("Invalid hook configuration for gcsRecorderHook: %s", opts))
	}

	return func(ctx context.Context, spec string, r io.Reader) error {
		client, err := gcsx.NewClient(ctx, storage.ScopeReadWrite)
		if err != nil {
			return errors.WithContext(err, "establishing GCS client")
		}
		return gcsx.WriteObject(ctx, client, bucket, path.Join(prefix, spec), r)
	}
}

func getContainerImage(ctx context.Context) string {
	urn := jobopts.GetEnvironmentUrn(ctx)
	if urn == "" || urn == "beam:env:docker:v1" {
		if *workerHarnessImage != "" {
			if *image != "" {
				panic("Both worker_harness_container_image and sdk_container_image cannot both be set. Prefer sdk_container_image, worker_harness_container_image is deprecated.")
			}
			return *workerHarnessImage
		}
		if *image != "" {
			return *image
		}
		envConfig := jobopts.GetEnvironmentConfig(ctx)
		if envConfig == core.DefaultDockerImage {
			// It's possible the user set the image exactly manually, but unlikely.
			// Prefer using the gcr.io image by default.
			// Note: This doesn't change the dev experience, which requires a user
			// to have a dev image.
			// However, RC versions should automatically be picked up, since
			// they are never tagged the RC number, just the main version.
			return "gcr.io/cloud-dataflow/v1beta3/beam_go_sdk:" + core.SdkVersion
		}
		return envConfig
	}

View on GitHub (pinned to 12126d8942)