apache/beam · error · ValueError

Field ' ' not found in input schema fields

Error message

Field '{file_pattern}' not found in input schema fields: {field_names}

What it means

match_all reads file patterns from a schema field of the input. When the user explicitly supplies file_pattern naming a field, that name must exist in the input schema; otherwise this ValueError lists the missing name and the actual fields.

Solutions

1. Pass the name of the input schema field that contains the file patterns (e.g. file_pattern='path')
2. Ensure the input schema has that field, or drop the argument so a single-field schema is used automatically
3. Print pcoll.element_type schema fields to confirm names

Example fix

// before
match_all(file_pattern='gs://bucket/*.json')  # glob mistaken for field name
// after
match_all(file_pattern='file_path')  # schema field holding the patterns

field_names = None
try:
    field_names = [n for n, _ in schemas.named_fields_from_element_type(pcoll.element_type)]
except Exception:
    pass
if file_pattern is not None and field_names and file_pattern not in field_names:
    raise ValueError(f'{file_pattern!r} is not a schema field: {field_names}')

try:
    pcoll | yaml_io.match_all(file_pattern=fp)
except ValueError as e:
    if 'not found in input schema fields' in str(e):
        pcoll = pcoll | beam.Map(lambda r: beam.Row(file_path=r.path))

When it happens

Trigger: Calling match_all(file_pattern='some_field') where 'some_field' is not among the input PCollection's schema field names (and is being interpreted as a field name, not a glob).

Common situations: Confusion between a glob pattern and a field name: file_pattern here selects which schema field holds per-record patterns, not the pattern itself; typos or schema changes also trigger it.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• A schema is required to write non-schema'd data.
• All dicts in batch must have the same keys. extra keys
• An explicit schema is required to write non-schema'd…
• Arrow map key field cannot be nullable
• Attempted to encode null for non-nullable field

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8eb2b8520a5632ec. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_io.py:890

      the file pattern string. If not specified and the input PCollection has
      exactly one field, that field will be used.
    empty_match_treatment (str): How to treat empty matches. Possible values are
      'ALLOW', 'DISALLOW', and 'ALLOW_IF_WILDCARD'. Defaults to 'ALLOW'.
  """
  from apache_beam.typehints import schemas

  try:
    field_names = [
        name for name, _ in schemas.named_fields_from_element_type(
            pcoll.element_type)
    ]
  except Exception:
    field_names = None

  if field_names:
    if file_pattern is not None:
      if file_pattern not in field_names:
        raise ValueError(
            f"Field '{file_pattern}' not found in input schema fields: {field_names}"
        )
      pattern_field = file_pattern
    elif len(field_names) == 1:
      pattern_field = field_names[0]
    else:
      raise ValueError(
          f"Input schema has multiple fields {field_names}. "
          f"Please specify the 'file_pattern' parameter to select which field "
          f"contains the file pattern.")
    patterns = pcoll | beam.Map(lambda x: str(getattr(x, pattern_field)))
  else:
    patterns = pcoll

  matched = patterns | beam.io.fileio.MatchAll(
      empty_match_treatment=empty_match_treatment)

  return matched | beam.Map(

View on GitHub (pinned to 12126d8942)