{"record":{"id":"27a27951305536bd","repo":"apache/beam","slug":"exploding-unknown-field-field","errorCode":null,"errorMessage":"Exploding unknown field \"{field}\"","messagePattern":"Exploding unknown field \"(.+?)\"","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/yaml/yaml_mapping.py","lineNumber":596,"sourceCode":"      else:\n        # Doesn't matter.\n        cross_product = True\n    self._fields = fields\n    self._cross_product = cross_product\n    # TODO(yaml):\n    # 1. Support standard error handling argument.\n    # 2. Supposedly error_handling parameter is not an accepted parameter when\n    #    executing.  Needs further investigation.\n    self._exception_handling_args = exception_handling_args(error_handling)\n\n  @maybe_with_exception_handling\n  def expand(self, pcoll):\n    all_fields = [\n        x for x, _ in named_fields_from_element_type(pcoll.element_type)\n    ]\n    for field in self._fields:\n      if field not in all_fields:\n        raise ValueError(f'Exploding unknown field \"{field}\"')\n    to_explode = self._fields\n\n    def explode_cross_product(base, fields):\n      if fields:\n        copy = dict(base)\n        for value in base[fields[0]]:\n          copy[fields[0]] = value\n          yield from explode_cross_product(copy, fields[1:])\n      else:\n        yield beam.Row(**base)\n\n    def explode_zip(base, fields):\n      to_zip = [base[field] for field in fields]\n      copy = dict(base)\n      for values in itertools.zip_longest(*to_zip, fillvalue=None):\n        for ix, field in enumerate(fields):\n          copy[field] = values[ix]\n        yield beam.Row(**copy)","sourceCodeStart":578,"sourceCodeEnd":614,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/yaml/yaml_mapping.py#L578-L614","documentation":"Beam YAML's Explode transform raises this when a field listed in the `explode` option does not exist in the input PCollection's element schema. It fails fast at pipeline construction time in `ExpandExplode.expand()` because exploding a non-existent field would silently produce wrong output. The field list is validated against `named_fields_from_element_type(pcoll.element_type)`.","triggerScenarios":"Calling the Explode transform (yaml_mapping.ExpandExplode) with self._fields containing a field name not present in the input PCollections's schema; typically via YAML `transform: Explode` with an `explode: [field]` config where `field` is misspelled, removed by an earlier step, or the input is not schema'd.","commonSituations":"Typo in field name in the YAML spec; referencing a field produced downstream instead of upstream; the input is a plain dict/row without a Beam schema so `named_fields_from_element_type` returns nothing; renaming fields in an earlier Map step without updating the explode list.","solutions":["Fix the field name in the `explode` config to match a field that exists in the input schema.","Inspect the input PCollection schema (e.g. with `pcoll.element_type` or `beam.schema` logging) and confirm the field exists before the Explode step.","If the field is removed upstream, move the Explode before the removal or remove it from the explode list.","If the input lacks a schema, add one (e.g. via a Cast or ToRow step) before exploding."],"exampleFix":"// before\n- type: Explode\n  input: rows\n  config:\n    explode: [tags, categoroies]\n// after\n- type: Explode\n  input: rows\n  config:\n    explode: [tags, categories]","handlingStrategy":"validation","validationCode":"from apache_beam.typehints import row_type\nfields_in_explode = {'tags', 'categories'}\nschema_fields = {name for name, _ in __import__('apache_beam.yaml.yaml_mapping', fromlist=['named_fields_from_element_type']).named_fields_from_element_type(pcoll.element_type)}\nunknown = fields_in_explode - set(schema_fields)\nif unknown:\n    raise ValueError(f'Explode fields not in input schema: {unknown}')","typeGuard":null,"tryCatchPattern":"try:\n    expanded = Explode(fields=['tags']).expand(pcoll)\nexcept ValueError as e:\n    if 'Exploding unknown field' in str(e):\n        logger.error('Fix explode config: %s', e)\n    raise","preventionTips":["Keep the explode field list in sync with upstream schema definitions","Log the input schema (named_fields_from_element_type) during pipeline development","Use schema-checked Cast transforms before Explode so typos surface earlier"],"tags":["apache-beam","yaml","schema","configuration"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}