apache/beam · error · ValueError

Invalid transform specification at

Error message

Invalid transform specification at {identify_object(spec)}: {msg}

What it means

When the provider's transform expansion/validation raises an exception (e.g. an argument error from wrong config kwargs), create_ptransform wraps it in a cleaner ValueError: 'Invalid transform specification at {location}: {message}'. It is a normalized wrapper around any underlying config/argument error for the transform.

Solutions

  1. Read the wrapped message — it contains the original error text identifying the bad option or value.
  2. Compare your config keys against the transform's documented schema.
  3. Fix the option types/values (lists, ints, valid expressions) at the reported location.
  4. Align apache-beam version with the pipeline spec if option names changed between releases.

Example fix

# before
- type: MapToFields
  config:
    language: sql
    feilds:
      upper: UPPER(name)
# after
- type: MapToFields
  config:
    language: sql
    fields:
      upper: UPPER(name)
Defensive patterns

Strategy: try-catch

Validate before calling

KNOWN_OPTIONS = {'fields', 'language', 'append', 'error_handling'}  # per-transform schema
bad = set(spec.get('config', {})) - KNOWN_OPTIONS
if bad:
    raise ValueError(f'Unknown options {bad} for {spec["type"]}')

Type guard

def config_matches_schema(spec, schema):
    return all(k in schema for k in spec.get('config', {}))

Try / catch

try:
    run_pipeline(spec)
except ValueError as e:
    if 'Invalid transform specification' in str(e):
        raise UserPipelineError(str(e)) from e  # inner message names the bad option/value

Prevention

When it happens

Trigger: Passing config options that do not match the transform's signature — unknown/misspelled option names, wrong value types, missing required options — so the underlying callable raises, or any exception raised during provider validation of the spec.

Common situations: Typos in config option names; string where a list/int is expected; missing required options; option names that changed between Beam versions; invalid expression syntax in field expressions.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0cfc2d055015b79e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_transform.py:453

              record_providers(p)
          elif isinstance(pvalueish, beam.PCollection):
            if pvalueish not in self.input_providers:
              self.input_providers[pvalueish] = provider

        record_providers(result)
        return result

      ptransform.expand = recording_expand
      return ptransform
    except Exception as exn:
      if isinstance(exn, TypeError):
        # Create a slightly more generic error message for argument errors.
        msg = str(exn).replace('positional', '').replace('keyword', '')
        msg = re.sub(r'\S+lambda\S+', '', msg)
        msg = re.sub('  +', ' ', msg).strip()
      else:
        msg = str(exn)
      raise ValueError(
          f'Invalid transform specification at {identify_object(spec)}: {msg}'
      ) from exn

  def unique_name(self, spec, ptransform, strictness=0):
    if 'name' in spec:
      name = spec['name']
      strictness += 1
    elif ('ExternalTransform' not in ptransform.label and
          not ptransform.label.startswith('_')):
      # The label may have interesting information.
      name = ptransform.label
    else:
      name = spec['type']
    if name in self._seen_names:
      if strictness >= 2:
        raise ValueError(f'Duplicate name at {identify_object(spec)}: {name}')
      else:
        name = f'{name}@{SafeLineLoader.get_line(spec)}'

View on GitHub (pinned to 12126d8942)