apache/beam · error · ValueError

Only AVRO and JSON are supported as intermediate formats…

Error message

Only AVRO and JSON are supported as intermediate formats for BigQuery WriteRecordsToFile, got: {}.

What it means

BigQuery batch loads write intermediate row files in either AVRO or JSON format (bigquery_tools.FileFormat). _make_new_file_writer only handles these two formats; any other file_format value reaches the else branch and raises this ValueError.

Solutions

Use temp_file_format=apache_beam.io.gcp.bigquery_tools.FileFormat.AVRO or FileFormat.JSON.
Omit temp_file_format entirely to use the JSON default.
Remove any custom string value like 'csv'/'parquet' — those formats are not supported.

Example fix

// before
beam.io.WriteToBigQuery(table, temp_file_format='parquet')
// after
from apache_beam.io.gcp.bigquery_tools import FileFormat
beam.io.WriteToBigQuery(table, temp_file_format=FileFormat.AVRO)

Defensive patterns

Strategy: type-guard

Validate before calling

from apache_beam.io.gcp.bigquery_tools import FileFormat
assert temp_file_format in (FileFormat.AVRO, FileFormat.JSON, None)

Type guard

def is_valid_intermediate_format(fmt) -> bool:
    from apache_beam.io.gcp.bigquery_tools import FileFormat
    return fmt in (FileFormat.AVRO, FileFormat.JSON)

Prevention

Only use FileFormat enum members, never raw strings
Omit temp_file_format to accept the default
Check the FileFormat enum for supported values before upgrading Beam

When it happens

Trigger: Constructing WriteToBigQuery / BigQueryBatchFileLoads with temp_file_format set to something other than bigquery_tools.FileFormat.AVRO or FileFormat.JSON.

Common situations: Passing a raw string like 'csv' or 'parquet' as temp_file_format, or a typo'd custom enum value, when trying to change the staging file format.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/317a753a060fdd5b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_file_loads.py:186

      if not fs.FileSystems.exists(directory):
        raise

  file_name = str(uuid.uuid4())
  file_path = fs.FileSystems.join(file_prefix, destination, file_name)

  if file_format == bigquery_tools.FileFormat.AVRO:
    if callable(schema):
      schema = schema(destination, *schema_side_inputs)
    elif isinstance(schema, vp.ValueProvider):
      schema = schema.get()

    writer = bigquery_tools.AvroRowWriter(
        fs.FileSystems.create(file_path, "application/avro"), schema)
  elif file_format == bigquery_tools.FileFormat.JSON:
    writer = bigquery_tools.JsonRowWriter(
        fs.FileSystems.create(file_path, "application/text"))
  else:
    raise ValueError((
        'Only AVRO and JSON are supported as intermediate formats for '
        'BigQuery WriteRecordsToFile, got: {}.').format(file_format))

  return file_path, writer


def _bq_uuid(seed=None):
  if not seed:
    return str(uuid.uuid4()).replace("-", "")
  else:
    return str(hashlib.md5(seed.encode('utf8')).hexdigest())


class _ShardDestinations(beam.DoFn):
  """Adds a shard number to the key of the KV element.

  Experimental; no backwards compatibility guarantees."""
  DEFAULT_SHARDING_FACTOR = 10

View on GitHub (pinned to 12126d8942)