apache/beam · error · ValueError

f"Mapping destinations {missing} for {type} are not in the u

Error message

f"Mapping destinations {missing} for {type} are not in the underlying config schema {list(underlying_schema_fields.keys())}"

What it means

Raised by RenamingProvider while validating mapped config fields against the underlying transform's schema: after adding any kwargs-style pass-through, destinations that don't exist in the underlying schema are rejected. Mapped config names must correspond to fields the underlying transform actually accepts.

Source

Thrown at sdks/python/apache_beam/yaml/yaml_provider.py:1542

    return self._transforms.keys()

  def config_schema(self, type):
    underlying_schema = self._underlying_provider.config_schema(
        self._transforms[type])
    if underlying_schema is None:
      return None
    defaults = self._defaults.get(type, {})
    underlying_schema_fields = {f.name: f for f in underlying_schema.fields}
    missing = set(self._mappings[type].values()) - set(
        underlying_schema_fields.keys())
    if missing:
      if 'kwargs' in underlying_schema_fields.keys():
        # These are likely passed by keyword argument dict rather than missing.
        for field_name in missing:
          underlying_schema_fields[field_name] = schema_pb2.Field(
              name=field_name, type=typing_to_runner_api(Any))
      else:
        raise ValueError(
            f"Mapping destinations {missing} for {type} are not in the "
            f"underlying config schema {list(underlying_schema_fields.keys())}")

    def with_name(
        original: schema_pb2.Field, new_name: str) -> schema_pb2.Field:
      result = schema_pb2.Field()
      result.CopyFrom(original)
      result.name = new_name
      return result

    return schema_pb2.Schema(
        fields=[
            with_name(underlying_schema_fields[dest], src)
            for (src, dest) in self._mappings[type].items()
            if dest not in defaults
        ])

  def description(self, typ):

View on GitHub (pinned to 12126d8942)

Solutions

Make the mapping destination match a real field of the underlying transform's config schema.
Update the mapping names after a Beam upgrade where the underlying transform's schema changed.
Check the error's listed schema fields and use one of them as the destination; or use an underlying transform that accepts a kwargs dict.

Example fix

# before
mappings:
  MyWrite:
    destination: WriteToText
    config_mapping:
      path: 'file_path'
# after (if underlying schema uses 'file_path')
mappings:
  MyWrite:
    destination: WriteToText
    config_mapping:
      file_path: 'file_path'

Defensive patterns

Strategy: validation

Validate before calling

def validate_mapping_destinations(mappings, underlying_schema_fields):
    missing = [d for d in mappings.get('config_mapping', {}) if d not in underlying_schema_fields and 'kwargs' not in underlying_schema_fields]
    if missing:
        raise ValueError(f'Destinations not in underlying schema: {missing}; valid: {list(underlying_schema_fields)}')

Try / catch

try:
    provider = RenamingProvider(base, transforms, mappings)
except ValueError as e:
    if 'not in the underlying config schema' in str(e):
        logging.error('Rename destinations must exist in the underlying schema: %s', e)
        raise
    raise

Prevention

Inspect the underlying transform's config schema before renaming fields.
After Beam upgrades, re-check that mapped destination names still exist.
Keep destination names identical to source names when only renaming transform names, not config fields.

When it happens

Trigger: RenamingProvider mappings rename a config field to a name the underlying transform's schema doesn't define (e.g. renaming 'url' to 'endpoint' for a transform expecting 'url'), and the underlying schema has no 'kwargs' field to absorb extras.

Common situations: Renaming config keys for readability while targeting a transform whose config schema was updated/renamed in a new Beam version, or simple typos in mapping destinations.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/99b0b7506cd62696. Report an issue: GitHub.