apache/beam · error · ValueError

Exactly one of yaml_pipeline or yaml_pipeline_file must be s

Error message

Exactly one of yaml_pipeline or yaml_pipeline_file must be set.

What it means

Pipeline specification in apache_beam.yaml.main accepts the pipeline either inline via --yaml_pipeline or from a file via --yaml_pipeline_file, but not both. Supplying both is ambiguous, so _pipeline_spec_from_args raises this ValueError.

Source

Thrown at sdks/python/apache_beam/yaml/main.py:140

      'Requires --test_suite if the pipeline uses jinja formatting.')
  parser.add_argument(
      '--create_test',
      action=argparse.BooleanOptionalAction,
      help='Automatically creates a regression test for the given pipeline, '
      'adding it to the pipeline spec or test suite dependon on whether '
      '--test_suite is given. '
      'Requires --test_suite if the pipeline uses jinja formatting.')
  parser.add_argument(
      '--test_suite',
      help='Run the given tests against the given pipeline, rather than the '
      'pipeline itself. '
      'Should be a file containing a list of yaml test specifications.')
  return parser.parse_known_args(argv)


def _pipeline_spec_from_args(known_args):
  if known_args.yaml_pipeline_file and known_args.yaml_pipeline:
    raise ValueError(
        "Exactly one of yaml_pipeline or yaml_pipeline_file must be set.")
  elif known_args.yaml_pipeline_file:
    with FileSystems.open(known_args.yaml_pipeline_file) as fin:
      pipeline_yaml = fin.read().decode()
  elif known_args.yaml_pipeline:
    pipeline_yaml = known_args.yaml_pipeline
  else:
    raise ValueError(
        "Exactly one of yaml_pipeline or yaml_pipeline_file must be set.")

  return pipeline_yaml


def run(argv=None):
  options, constructor, display_data = build_pipeline_components_from_argv(argv)
  with beam.Pipeline(options=options, display_data=display_data) as p:
    print('Building pipeline...')
    constructor(p)

View on GitHub (pinned to 12126d8942)

Solutions

1. Remove --yaml_pipeline from the command and keep only --yaml_pipeline_file (or vice versa).
2. If both sources exist in a script, pick one programmatically before invoking: use the file when present, else the inline string.
3. Check environment-variable-driven flag assembly (e.g. ${YAML_PIPELINE:+--yaml_pipeline ...}) so only one flag is ever emitted.
4. Wrap the invocation and catch ValueError to give a clearer CLI error message to users.

Example fix

# before
python -m apache_beam.yaml.main --yaml_pipeline "$(cat p.yaml)" --yaml_pipeline_file p.yaml
# after
python -m apache_beam.yaml.main --yaml_pipeline_file p.yaml

import sys
flags = sys.argv
has_inline = '--yaml_pipeline' in flags
has_file = '--yaml_pipeline_file' in flags
assert not (has_inline and has_file), 'pass only one of --yaml_pipeline / --yaml_pipeline_file'

def has_conflicting_pipeline_sources(argv):
    return '--yaml_pipeline' in argv and '--yaml_pipeline_file' in argv

try:
    apache_beam.yaml.main.run(argv)
except ValueError as e:
    if 'Exactly one of yaml_pipeline' in str(e):
        parser.error('--yaml_pipeline and --yaml_pipeline_file are mutually exclusive')
    raise

When it happens

Trigger: Running `python -m apache_beam.yaml.main --yaml_pipeline <yaml> --yaml_pipeline_file <path>` (or _build_pipeline_yaml_from_argv with both argv flags populated).

Common situations: Shell scripts with a default --yaml_pipeline flag plus an explicitly passed --yaml_pipeline_file; CI configs merging multiple sources of the pipeline definition; copy-pasted command lines accumulating both flags.

Understand the failure class

Background: "mutually exclusive" flag errors: what "can't supply both nx and xx", "--raw is not compatible with -i" and "cannot be used with" mean, and how to fix them — this error's family across 29 libraries.

Related errors

• At most one of --create_test and --fix_tests may be specifie
• Node ID cannot be empty
• Edge source and target cannot be empty
• At least one of --render_port or --render_output must be pro
• Invalid feature store yaml file provided. Make sure the %s c

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/409b6ec3cab6b1ac. Report an issue: GitHub.