{"record":{"id":"00d923245b3bde9d","repo":"apache/beam","slug":"can-only-append-fields-on-a-schema-d-input","errorCode":null,"errorMessage":"Can only append fields on a schema'd input.","messagePattern":"Can only append fields on a schema'd input\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/yaml/yaml_mapping.py","lineNumber":669,"sourceCode":"    error_handling: Whether and where to output records that throw errors when\n      the above expressions are evaluated.\n  \"\"\"  # pylint: disable=line-too-long\n  keep_fn = _as_callable_for_pcoll(pcoll, keep, \"keep\", language or 'generic')\n  return pcoll | beam.Filter(keep_fn)\n\n\ndef is_expr(v):\n  return isinstance(v, str) or (isinstance(v, dict) and 'expression' in v)\n\n\ndef normalize_fields(pcoll, fields, drop=(), append=False, language='generic'):\n  try:\n    input_schema = dict(named_fields_from_element_type(pcoll.element_type))\n  except (TypeError, ValueError) as exn:\n    if drop:\n      raise ValueError(\"Can only drop fields on a schema'd input.\") from exn\n    if append:\n      raise ValueError(\"Can only append fields on a schema'd input.\") from exn\n    elif any(is_expr(x) for x in fields.values()):\n      raise ValueError(\"Can only use expressions on a schema'd input.\") from exn\n    input_schema = {}\n\n  if drop and not append:\n    raise ValueError(\"Can only drop fields if append is true.\")\n  for name in drop:\n    if name not in input_schema:\n      raise ValueError(f'Dropping unknown field \"{name}\"')\n  if append:\n    for name in fields:\n      if name in input_schema and name not in drop:\n        raise ValueError(\n            f'Redefinition of field \"{name}\". '\n            'Cannot append a field that already exists in original input.')\n\n  if append:\n    return input_schema, {","sourceCodeStart":651,"sourceCodeEnd":687,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/yaml/yaml_mapping.py#L651-L687","documentation":"Raised by `normalize_fields` when the `append` option is set but the input PCollection has no element schema. Appending fields builds the new output schema from the existing input schema, so an untyped input cannot be appended to; the transform raises rather than guessing types.","triggerScenarios":"Using a YAML transform (AddFields / MapToFields path) with `append: true` where `named_fields_from_element_type(pcoll.element_type)` fails because the input is not a schema'd row (e.g. plain dict from a generic map).","commonSituations":"Chaining AddFields after an untyped Map or JSON source; forgetting that Beam YAML transforms require schema'd PCollections; a DoFn without output type hints feeding the transform.","solutions":["Ensure the input has a schema, e.g. insert a Cast or use `beam.Row(...)` with type annotations upstream.","Remove `append: true` if you intend to create a fresh set of fields instead of extending a schema.","Use MapToFields on the untyped input to produce a schema first, then append."],"exampleFix":"// before\n- type: AddFields\n  input: raw_json\n  config:\n    append: true\n    fields: {score: {type: double}}\n// after\n- type: Cast\n  input: raw_json\n  config:\n    fields: {id: int64, name: string}\n- type: AddFields\n  input: cast_output\n  config:\n    append: true\n    fields: {score: {type: double}}","handlingStrategy":"validation","validationCode":"try:\n    named_fields_from_element_type(pcoll.element_type)\nexcept (TypeError, ValueError):\n    raise ValueError('append requires a schema\\'d input; add a Cast or beam.Row step upstream.')","typeGuard":null,"tryCatchPattern":"try:\n    out = normalize_fields(pcoll, fields={'score': 0.0}, append=True)\nexcept ValueError as e:\n    if \"Can only append fields\" in str(e):\n        out = normalize_fields(cast_to_schema(pcoll), fields={'score': 0.0}, append=True)\n    else:\n        raise","preventionTips":["Verify the input PCollection has a schema before any AddFields with append: true","Prefer Cast transforms at boundaries between untyped and typed sections of the pipeline","Annotate DoFn output types so schema inference succeeds"],"tags":["apache-beam","yaml","schema"],"backgroundTag":"incompatible-source-type","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}