What causes "num_workers (%d) cannot exceed max_num_workers (%d)" in apache/beam?

Lower num_workers so it is <= max_num_workers, or raise max_num_workers to at least num_workers

num_workers ( ) cannot exceed max_num_workers ( )

Error message

num_workers (%d) cannot exceed max_num_workers (%d)

What it means

validateWorkerSettings in the Dataflow runner translation layer checks that requested worker counts are sane before submitting a job to Google Cloud Dataflow. This error means the user asked for a number of workers larger than the declared maximum, which Dataflow would reject or which indicates contradictory options. The library fails fast at translation time instead of sending a bad job to the service.

Solutions

Lower num_workers so it is <= max_num_workers, or raise max_num_workers to at least num_workers
Remove the explicit num_workers option and rely on autoscaling with max_num_workers only
Check the flag/config source for both values and make the template set them consistently

Example fix

// before
pOpts = append(pOpts, beampipeoption.WithNumWorkers(100), beampipeoption.WithMaxNumWorkers(50))
// after
pOpts = append(pOpts, beampipeoption.WithNumWorkers(50), beampipeoption.WithMaxNumWorkers(50))

Defensive patterns

Strategy: validation

Validate before calling

if numWorkers > 0 && maxNumWorkers > 0 && numWorkers > maxNumWorkers {
    return fmt.Errorf("num_workers (%d) must be <= max_num_workers (%d)", numWorkers, maxNumWorkers)
}

Try / catch

if err := validateWorkerSettings(numWorkers, maxNumWorkers); err != nil {
    return fmt.Errorf("invalid worker settings: %w", err)
}

Prevention

Always set num_workers and max_num_workers together from one config source
Add a CI check asserting numWorkers <= maxNumWorkers in deploy configs
Prefer autoscaling (set only max_num_workers) unless a fixed worker count is required

When it happens

Trigger: Calling beam.Run with a Dataflow pipeline where the num_workers option (NumWorkers) is > 0 and max_num_workers (MaxNumWorkers) is also > 0 but smaller than num_workers, e.g. --num_workers=100 --max_num_workers=50.

Common situations: Autoscaling configs where MaxNumWorkers was lowered (or left from a copied pipeline config) but NumWorkers was not adjusted; scripts templating worker counts where num_workers > max; setting NumWorkers after another config layer already set MaxNumWorkers lower.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3c12126bc016ac2b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go:445

		return errors.New("experiment worker_region and option Zone are mutually exclusive")
	}

	if opts.Zone != "" {
		log.Warn(ctx, "Option --zone is deprecated. Please use --workerZone instead.")
		opts.WorkerZone = opts.Zone
		opts.Zone = ""
	}

	numWorkers := opts.NumWorkers
	maxNumWorkers := opts.MaxNumWorkers
	if numWorkers < 0 {
		return fmt.Errorf("num_workers (%d) cannot be negative", numWorkers)
	}
	if maxNumWorkers < 0 {
		return fmt.Errorf("max_num_workers (%d) cannot be negative", maxNumWorkers)
	}
	if numWorkers > 0 && maxNumWorkers > 0 && numWorkers > maxNumWorkers {
		return fmt.Errorf("num_workers (%d) cannot exceed max_num_workers (%d)", numWorkers, maxNumWorkers)
	}
	return nil
}

View on GitHub (pinned to 12126d8942)