apache/beam · error · ValueError

Missing output in error_handling of

Error message

Missing output in error_handling of {identify_object(t)}

What it means

Transforms with error_handling must declare which output name receives the error records (config.error_handling.output). ensure_errors_consumed validates this during preprocessing; an error_handling block without an 'output' key leaves the error records unroutable, so the pipeline is rejected.

Solutions

Add output: <name> inside error_handling, naming the error output of that transform.
Consume that named output as the input of another transform (e.g. WriteToJson or a log sink) to also satisfy the unconsumed-error check.
Remove error_handling entirely if error capture is not intended.

Example fix

# before
config:
  error_handling: {}
# after
config:
  error_handling:
    output: errors

Defensive patterns

Strategy: validation

Validate before calling

for t in spec.get('transforms', []):
    cfg = t.get('config', t)
    if 'error_handling' in cfg and 'output' not in cfg['error_handling']:
        raise ValueError(f"{t.get('name')} error_handling needs an 'output'")

Type guard

def has_error_output(t):
    eh = t.get('config', t).get('error_handling')
    return eh is None or isinstance(eh.get('output'), str)

Try / catch

try:
    spec = ensure_errors_consumed(spec)
except ValueError as e:
    if 'Missing output in error_handling' in str(e):
        raise SystemExit(f'Fix your YAML: {e}')
    raise

Prevention

Always pair error_handling with an output name
Scaffold error handling from official examples
Validate specs with the Beam YAML schema before expansion

When it happens

Trigger: ensure_errors_consumed iterating spec['transforms'] and finding a transform t where config contains 'error_handling' but config['error_handling'] has no 'output' key — e.g. error_handling: {} or only error_handling: {input: ...}.

Common situations: Omitting the output name when enabling error capture; copying an error_handling snippet from docs that assumed a default; building specs programmatically and not setting the key.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/03be7a4aef1c2f20. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_transform.py:1270

def ensure_transforms_have_types(spec):
  if 'type' not in spec:
    raise ValueError(f'Missing type specification in {identify_object(spec)}')
  return spec


def ensure_errors_consumed(spec):
  if spec['type'] == 'composite':
    scope = LightweightScope(spec['transforms'])
    to_handle = {}
    consumed = set(
        scope.get_transform_id_and_output_name(output)
        for output in spec['output'].values())
    for t in spec['transforms']:
      config = t.get('config', t)
      if 'error_handling' in config:
        if 'output' not in config['error_handling']:
          raise ValueError(
              f'Missing output in error_handling of {identify_object(t)}')
        to_handle[t['__uuid__'], config['error_handling']['output']] = t
      for _, input in empty_if_explicitly_empty(t['input']).items():
        if input not in spec['input']:
          consumed.add(scope.get_transform_id_and_output_name(input))
    for error_pcoll, t in to_handle.items():
      if error_pcoll not in consumed:
        config = t.get('config', t)
        transform_name = t.get('name', t.get('type'))
        error_output_name = config['error_handling']['output']
        raise ValueError(
            f'Unconsumed error output for {identify_object(t)}. '
            f'The output named {transform_name}.{error_output_name} '
            'must be used as an input to some other transform. '
            'See https://beam.apache.org/documentation/sdks/yaml-errors')
  return spec

View on GitHub (pinned to 12126d8942)