apache/beam · error · ValueError

error_handling config is not supported directly in the outpu

Error message

error_handling config is not supported directly in the output_schema. Please use error_handling config in the transform, if possible, or use ValidateWithSchema transform instead.

What it means

expand_output_schema_transform validates a transform's outputs against an output_schema. Error handling (where failed rows go) must be configured on the transform itself, not nested inside the output_schema block; putting 'error_handling' in output_schema is rejected with this ValueError.

Source

Thrown at sdks/python/apache_beam/yaml/yaml_transform.py:580

  Args:
    spec (dict): The `output_schema` specification from the YAML config.
    outputs (beam.PCollection or dict[str, beam.PCollection]): The output(s)
      from the transform to be validated.
    error_handling_spec (dict): The `error_handling` configuration from the
      original transform.

  Returns:
    The validated PCollection(s). If error handling is enabled, this will be a
    dictionary containing the 'good' output and any error outputs.

  Raises:
    ValueError: If `error_handling` is incorrectly specified within the
      `output_schema` spec itself, or if the main output of a multi-output
      transform cannot be determined.
  """
  if 'error_handling' in spec:
    raise ValueError(
        'error_handling config is not supported directly in '
        'the output_schema. Please use error_handling config in '
        'the transform, if possible, or use ValidateWithSchema transform '
        'instead.')

  # Strip metadata such as __line__ and __uuid__ as these will interfere with
  # the validation downstream.
  clean_schema = SafeLineLoader.strip_metadata(spec)

  # If no error handling is specified for the main transform, warn the user
  # that the pipeline may fail if any output data fails the output schema
  # validation.
  if not error_handling_spec:
    _LOGGER.warning("Output_schema config is attached to a transform that has "\
    "no error_handling config specified. Any failures validating on output" \
    "schema will fail the pipeline unless the user specifies an" \
    "error_handling config on a capable transform. Alternatively, you can " \
    "remove the output_schema config on this transform and add a " \

View on GitHub (pinned to 12126d8942)

Solutions

Move the error_handling block out of output_schema to the transform's own config level (sibling of output_schema).
Alternatively, replace the transform with an explicit ValidateWithSchema transform that owns the error_handling config.
Re-run the pipeline after restructuring the YAML.

Example fix

// before
- type: MapToFields
  config:
    output_schema:
      schema: 'id: INTEGER'
      error_handling: {output: errors}
// after
- type: MapToFields
  config:
    error_handling: {output: errors}
    output_schema:
      schema: 'id: INTEGER'

Defensive patterns

Strategy: validation

Validate before calling

def check_output_schema_nesting(spec):
    os_spec = spec.get('config', {}).get('output_schema', {})
    if 'error_handling' in os_spec:
        raise ValueError("Move 'error_handling' to the transform config level, not inside output_schema")

Type guard

def error_handling_is_top_level(spec) -> bool:
    cfg = spec.get('config', {})
    return 'error_handling' not in cfg.get('output_schema', {})

Try / catch

try:
    expand_transform(spec, scope)
except ValueError as e:
    if 'error_handling config is not supported' in str(e):
        print('Relocate error_handling block in YAML')
    else:
        raise

Prevention

Keep error_handling as a sibling of output_schema, not nested in it.
Validate YAML structure in CI before running pipelines.
Refer to ValidateWithSchema examples for correct error_handling placement.

When it happens

Trigger: A YAML spec where 'error_handling:' is placed under 'output_schema:' of a transform, e.g. output_schema: {schema: ..., error_handling: {...}}, instead of at the transform's top-level config alongside output_schema.

Common situations: Copy-pasting an error_handling block from a ValidateWithSchema example into the wrong nesting level; misunderstanding docs that show error_handling as a sibling of output_schema.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9eb02f269ce329b4. Report an issue: GitHub.