apache/beam · error · ValueError

write_config must be provided with 'table' specified

Error message

write_config must be provided with 'table' specified

What it means

BigQueryVectorWriterConfig validates at construction time that its write_config dict includes a 'table' key identifying the destination BigQuery table. Without it the writer cannot build the BigQuery write sink, so a ValueError is raised.

Source

Thrown at sdks/python/apache_beam/ml/rag/ingestion/bigquery.py:130

      >>> config = BigQueryVectorWriterConfig(
      ...   write_config={'table': 'project.dataset.embeddings'},
      ...   schema_config=schema_config
      ... )

    Args:
        write_config: BigQuery write configuration dict. Must include 'table'.
            Other options like create_disposition, write_disposition can be
            specified.
        schema_config: Optional configuration for custom schema and row
            conversion.
            If not provided, uses default schema with id, embedding, content and
            metadata columns.
    
    Raises:
        ValueError: If write_config doesn't include table specification.
    """
    if 'table' not in write_config:
      raise ValueError("write_config must be provided with 'table' specified")

    self.write_config = write_config
    self.schema_config = schema_config

  def create_write_transform(self) -> beam.PTransform:
    """Creates transform to write to BigQuery."""
    return _WriteToBigQueryVectorDatabase(self)


def _default_embeddable_to_dict_fn(item: EmbeddableItem):
  if item.embedding is None or item.embedding.dense_embedding is None:
    raise ValueError("EmbeddableItem must contain dense embedding")
  return {
      'id': item.id,
      'embedding': item.embedding.dense_embedding,
      'content': item.content_string,
      'metadata': [{
          "key": k, "value": str(v)

View on GitHub (pinned to 12126d8942)

Solutions

1. Add 'table': 'project:dataset.table' (or dataset.table) to the write_config dict
2. Also add 'project' when not using the fully-qualified table form
3. Validate the write_config keys before constructing the config

Example fix

// before
write_config = {'project': 'my-project', 'create_disposition': 'CREATE_IF_NEEDED'}
// after
write_config = {'project': 'my-project', 'table': 'my-project:my_dataset.embeddings'}

assert 'table' in write_config and write_config['table'], "write_config['table'] required"

def has_table(write_config: dict) -> bool:
    return bool(write_config.get('table'))

try:
    cfg = BigQueryVectorWriterConfig(schema_config=sc, write_config=wc)
except ValueError as e:
    if 'table' in str(e): wc['table'] = f'{project}:{dataset}.{table}'; cfg = BigQueryVectorWriterConfig(sc, wc)
    else: raise

When it happens

Trigger: Constructing BigQueryVectorWriterConfig(schema_config=..., write_config={...}) where write_config lacks 'table' — e.g., only passing create_disposition or other options.

Common situations: Copy-pasted write_config dicts missing the table; building write_config dynamically where the table key is injected later; confusing write_config with SchemaConfig parameters.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• poll_interval_sec must be >= 15, got {poll_interval_sec}
• buffer_sec must be >= 0, got {buffer_sec}
• Item {item.id} missing embedding
• Unexpected keyword arguments: {', '.join(kwargs)}
• SchemaConfig requires embeddable_to_dict_fn

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/31730285adf6819c. Report an issue: GitHub.