apache/beam · error · ValueError

Error applying transform

Error message

Error applying transform {identify_object(spec)}: {exn}

What it means

expand_leaf_transform wraps any exception raised while applying a leaf PTransform (during construction/expand) into a ValueError prefixed with 'Error applying transform' and the transform's identified location. Beam does this to attribute generic transform failures back to the specific YAML spec that caused them.

Solutions

1. Read the chained inner exception (caused by) — the fix is almost always in the original exn message, not this wrapper.
2. Validate the transform's 'config' against the transform's documented schema before running the pipeline.
3. Check that input PCollections have the schema/fields the transform expects.
4. Use ValidateWithSchema or --dry-run style validation to catch config problems early.

Example fix

// before (YAML)
- type: MapToFields
  input: rows
  config:
    total: price * quantity_typo
// after
- type: MapToFields
  input: rows
  config:
    total: price * quantity

def check_transform_config(spec, known):
    t = spec.get('type')
    if t not in known:
        raise ValueError(f'Unknown transform type {t}')
    missing = set(known[t].get('required', [])) - set(spec.get('config', {}))
    if missing:
        raise ValueError(f'{t} missing config keys: {missing}')

def is_valid_config(spec) -> bool:
    return isinstance(spec.get('config', {}), dict)

try:
    pipeline.run()
except ValueError as e:
    if e.__cause__ is not None:
        print(f"Root cause: {e.__cause__}")
    raise

When it happens

Trigger: scope.create_ptransform(spec, inputs) or the `inputs | name >> ptransform` application raises — e.g. invalid config for the transform type, missing required config field, wrong input schema, or any exception thrown inside the underlying PTransform's expand().

Common situations: Misconfigured built-in transforms (bad path in ReadFromText, wrong field names in MapToFields), schema mismatches between chained transforms, or provider/transform constructor errors surfaced through YAML.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• At most one of --create_test and --fix_tests may be…
• Cannot convert element of type
• "Cannot specify 'callable' with 'path' and 'name' for…
• Chain at missing transforms property.
• Dependencies must be a list of strings, got

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5a6949ce37b9f596. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_transform.py:521

  if input_type == 'list':
    inputs = tuple(inputs_dict.values())
  elif input_type == 'map':
    inputs = inputs_dict
  else:
    if len(inputs_dict) == 0:
      inputs = scope.root
    elif len(inputs_dict) == 1:
      inputs = next(iter(inputs_dict.values()))
    else:
      inputs = inputs_dict
  _LOGGER.info("Expanding %s ", identify_object(spec))
  ptransform = scope.create_ptransform(spec, inputs_dict.values())
  try:
    # TODO: Move validation to construction?
    with FullyQualifiedNamedTransform.with_filter('*'):
      outputs = inputs | scope.unique_name(spec, ptransform) >> ptransform
  except Exception as exn:
    raise ValueError(
        f"Error applying transform {identify_object(spec)}: {exn}") from exn

  # Optional output_schema was found, so lets expand on that before returning.
  if output_schema_spec:
    error_handling_spec = {}
    # Obtain original transform error_handling_spec, so that all validate
    # schema errors use that.
    if 'error_handling' in spec.get('config', None):
      error_handling_spec = spec.get('config').get('error_handling', {})

    outputs = expand_output_schema_transform(
        spec=output_schema_spec,
        outputs=outputs,
        error_handling_spec=error_handling_spec)

  if isinstance(outputs, dict):
    # TODO: Handle (or at least reject) nested case.
    return outputs

View on GitHub (pinned to 12126d8942)