apache/beam · error · ValueError

Unknown format: {format}

Error message

Unknown format: {format}

What it means

_create_parser supports a fixed set of Pub/Sub read formats: RAW, STRING, JSON, AVRO, and PROTO (the parser's format is normalized with .upper()). If the requested format matches none of these branches, it raises ValueError('Unknown format: {format}').

Source

Thrown at sdks/python/apache_beam/yaml/yaml_io.py:272

        schema_pb2.Schema(fields=[schemas.schema_field('payload', str)]),
        lambda payload: beam.Row(payload=payload.decode('utf-8')))
  elif format == 'JSON':
    _validate_schema()
    beam_schema = json_utils.json_schema_to_beam_schema(schema)
    return beam_schema, json_utils.json_parser(beam_schema, schema)
  elif format == 'AVRO':
    _validate_schema()
    beam_schema = avroio.avro_schema_to_beam_schema(schema)
    covert_to_row = avroio.avro_dict_to_beam_row(schema, beam_schema)
    return (
        beam_schema, lambda record: covert_to_row(
            fastavro.schemaless_reader(io.BytesIO(record), schema)))
  elif format == 'PROTO':
    _validate_schema()
    beam_schema = json_utils.json_schema_to_beam_schema(schema)
    return beam_schema, RowCoder(beam_schema).decode
  else:
    raise ValueError(f'Unknown format: {format}')


def _create_formatter(
    format, schema: Any,
    beam_schema: schema_pb2.Schema) -> Callable[[beam.Row], bytes]:

  if format.islower():
    format = format.upper()

  if format == 'RAW':
    if schema:
      raise ValueError('RAW format does not take a schema')
    field_names = [field.name for field in beam_schema.fields]
    if len(field_names) != 1:
      raise ValueError(f'Expecting exactly one field, found {field_names}')

    def convert_to_bytes(row):
      output = getattr(row, field_names[0])

View on GitHub (pinned to 12126d8942)

Solutions

1. Use one of the supported formats: RAW, STRING, JSON, AVRO, PROTO (case-insensitive).
2. Fix typos/whitespace in the format value in the YAML config.
3. For unsupported encodings, read RAW and decode/parse with a custom Map transform.

Example fix

// before
- type: ReadFromPubSub
  format: CSV
// after
- type: ReadFromPubSub
  format: RAW
# then parse CSV with a Map transform

SUPPORTED = {'RAW', 'STRING', 'JSON', 'AVRO', 'PROTO'}
if fmt.upper() not in SUPPORTED:
    raise ValueError(f"Unknown format: {fmt}; use one of {sorted(SUPPORTED)}")

try:
    read_from_pubsub(format=fmt)
except ValueError as e:
    if str(e).startswith('Unknown format'):
        fmt = 'JSON'

When it happens

Trigger: Calling read_from_pubsub with format set to anything outside RAW/STRING/JSON/AVRO/PROTO — e.g. 'CSV', 'PARQUET', 'utf-8', or lowercase variants are fine (uppercased) but unsupported names are not.

Common situations: Assuming all file formats are valid Pub/Sub formats; typos like 'AVRO ' with trailing space or 'jsn'; copying write-side formats (e.g. BYTES) to the read side.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

• Unknown enrichment source: {enrichment_handler}
• {format} format requires valid {format} schema to be passed
• RAW format does not take a schema
• STRING format does not take a schema
• Unknown ML transform type: %r

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8e78213f36374fc5. Report an issue: GitHub.