apache/beam · error · ValueError

Input schema has multiple fields

Error message

Input schema has multiple fields {field_names}. Please specify the 'file_pattern' parameter to select which field contains the file pattern.

What it means

Thrown by match_all in apache_beam/yaml/yaml_io.py when the input PCollection's schema has more than one field and no 'file_pattern' parameter was given, so the transform cannot tell which field holds the file path pattern to match. The library only auto-selects the pattern field when the schema has exactly one field.

Solutions

1. Set the file_pattern parameter to the name of the schema field containing the file path pattern.
2. Check pcoll.element_type._fields to see the available field names and pick the right one.
3. If only the path matters upstream, project the PCollection to a single-field schema before match_all.

Example fix

// before
out = yaml_match_all.transform(pcoll, {})  # pcoll schema: (path, size)
// after
out = yaml_match_all.transform(pcoll, {'file_pattern': 'path'})

fields = list(getattr(pcoll.element_type, '_fields', []))
assert fields, 'input has no schema'
if len(fields) > 1 and not options.get('file_pattern'):
    raise ValueError(f"must pass file_pattern; fields={fields}")

def has_single_schema_field(pcoll):
    return len(getattr(pcoll.element_type, '_fields', [])) == 1

When it happens

Trigger: Calling match_all (or the YAML MatchAll transform) on a PCollection whose element schema has 2+ fields while omitting the file_pattern parameter.

Common situations: YAML pipeline authors pipe rows from a previous transform (e.g. a database read or ParseTransform) into MatchAll without realizing the rows have multiple columns; forgetting the file_pattern after changing an upstream schema to add extra fields.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• Exploding unknown field
• Redefinition of field
• Can only append fields on a schema'd input.
• Can only drop fields if append is true.
• Can only drop fields on a schema'd input.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e7ad951f803758e4. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_io.py:897

  try:
    field_names = [
        name for name, _ in schemas.named_fields_from_element_type(
            pcoll.element_type)
    ]
  except Exception:
    field_names = None

  if field_names:
    if file_pattern is not None:
      if file_pattern not in field_names:
        raise ValueError(
            f"Field '{file_pattern}' not found in input schema fields: {field_names}"
        )
      pattern_field = file_pattern
    elif len(field_names) == 1:
      pattern_field = field_names[0]
    else:
      raise ValueError(
          f"Input schema has multiple fields {field_names}. "
          f"Please specify the 'file_pattern' parameter to select which field "
          f"contains the file pattern.")
    patterns = pcoll | beam.Map(lambda x: str(getattr(x, pattern_field)))
  else:
    patterns = pcoll

  matched = patterns | beam.io.fileio.MatchAll(
      empty_match_treatment=empty_match_treatment)

  return matched | beam.Map(
      lambda x: beam.Row(
          path=str(x.path), size_in_bytes=int(x.size_in_bytes),
          last_updated_in_seconds=float(x.last_updated_in_seconds)
          if x.last_updated_in_seconds is not None else None))


_DICOM_SEARCH_OUTPUT_SCHEMA = RowTypeConstraint.from_fields([

View on GitHub (pinned to 12126d8942)