apache/beam · error · ValueError

WriteToTFRecord requires an input schema with exactly one…

Error message

WriteToTFRecord requires an input schema with exactly one field.

What it means

WriteToTFRecord writes raw bytes, so the YAML wrapper expects a one-field schema whose single value is the record content. When named_fields_from_element_type cannot resolve the element type (no/incompatible schema), it re-raises as this ValueError.

Solutions

1. Ensure the input PCollection has a typed schema with exactly one field (e.g. via beam.Map(lambda x: beam.Row(data=x)))
2. Verify the element type is schema-aware before writing
3. Use a different sink (e.g. WriteToJson) for multi-field data

Example fix

// before
pcoll | yaml_io.write_to_tfrecord(file_pattern_prefix='out')  # untyped rows
// after
pcoll | beam.Map(lambda x: beam.Row(data=x)) | yaml_io.write_to_tfrecord(file_pattern_prefix='out')

try:
    names = [n for n, _ in schemas.named_fields_from_element_type(pcoll.element_type)]
except Exception:
    raise ValueError('Input must have a schema with exactly one field')

def has_single_field_schema(pcoll):
    try:
        return len(schemas.named_fields_from_element_type(pcoll.element_type)) == 1
    except Exception:
        return False

try:
    pcoll | yaml_io.write_to_tfrecord(...)
except ValueError as e:
    if 'exactly one field' in str(e):
        pcoll = pcoll | beam.Map(lambda x: beam.Row(data=x))

When it happens

Trigger: Calling write_to_tfrecord on a PCollection whose element type has no named schema fields (untyped rows, non-schema elements), so schemas.named_fields_from_element_type throws.

Common situations: Passing plain dicts/bytes without a Beam schema; forgetting to apply as_json/as_dict schema typing upstream; using a Parse with no output schema.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• WriteToTFRecord requires an input schema with exactly one…
• A schema is required to write non-schema'd data.
• All dicts in batch must have the same keys. extra keys
• An explicit schema is required to write non-schema'd…
• Arrow map key field cannot be nullable

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/553b2858a1022897. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/yaml/yaml_io.py:759

      particular shard number, the upper-case letters 'S' and 'N' are
      replaced with the 0-padded shard number and shard count respectively.
      This argument can be '' in which case it behaves as if num_shards was
      set to 1 and only one file will be generated. The default pattern used
      is '-SSSSS-of-NNNNN' if None is passed as the shard_name_template.
    compression_type: Used to handle compressed output files. Typical value
      is CompressionTypes.AUTO, in which case the file_path's extension will
      be used to detect the compression.

  Returns:
    A WriteToTFRecord transform object.
  """
  try:
    field_names = [
        name for name, _ in schemas.named_fields_from_element_type(
            pcoll.element_type)
    ]
  except Exception as exn:
    raise ValueError(
        "WriteToTFRecord requires an input schema with exactly one field."
    ) from exn
  if len(field_names) != 1:
    raise ValueError(
        "WriteToTFRecord requires an input schema with exactly one field,got %s"
        % field_names)
  sole_field_name, = field_names

  return pcoll | beam.Map(
      lambda x: getattr(x, sole_field_name)) | WriteToTFRecord(
          file_path_prefix=file_path_prefix,
          coder=coder,
          file_name_suffix=file_name_suffix,
          num_shards=num_shards,
          shard_name_template=shard_name_template,
          compression_type=getattr(CompressionTypes, compression_type))

View on GitHub (pinned to 12126d8942)