apache/beam · error · ValueError

EmbeddableItem must contain dense embedding

Error message

EmbeddableItem must contain dense embedding

What it means

The default EmbeddableItem-to-BigQuery-row converter requires each item to carry a dense embedding, since the 'embedding' column stores item.embedding.dense_embedding. Items without one are rejected with this ValueError at write time.

Source

Thrown at sdks/python/apache_beam/ml/rag/ingestion/bigquery.py:142

            metadata columns.
    
    Raises:
        ValueError: If write_config doesn't include table specification.
    """
    if 'table' not in write_config:
      raise ValueError("write_config must be provided with 'table' specified")

    self.write_config = write_config
    self.schema_config = schema_config

  def create_write_transform(self) -> beam.PTransform:
    """Creates transform to write to BigQuery."""
    return _WriteToBigQueryVectorDatabase(self)


def _default_embeddable_to_dict_fn(item: EmbeddableItem):
  if item.embedding is None or item.embedding.dense_embedding is None:
    raise ValueError("EmbeddableItem must contain dense embedding")
  return {
      'id': item.id,
      'embedding': item.embedding.dense_embedding,
      'content': item.content_string,
      'metadata': [{
          "key": k, "value": str(v)
      } for k, v in item.metadata.items()]
  }


def _default_schema():
  return {
      'fields': [{
          'name': 'id', 'type': 'STRING'
      }, {
          'name': 'embedding', 'type': 'FLOAT64', 'mode': 'REPEATED'
      }, {
          'name': 'content', 'type': 'STRING'

View on GitHub (pinned to 12126d8942)

Solutions

1. Run an embedding transform on all items before VectorDatabaseWriteTransform
2. Filter out items without dense embeddings, or backfill them
3. Provide a custom embeddable_to_dict_fn if dense embeddings are not needed

Example fix

// before
rows = pcoll | VectorDatabaseWriteTransform(bq_config)  # items unembedded
// after
rows = (pcoll
    | beam.Filter(lambda i: i.embedding and i.embedding.dense_embedding)
    | VectorDatabaseWriteTransform(bq_config))

bad = [it.id for it in items if not (it.embedding and it.embedding.dense_embedding)]
assert not bad, f'items missing dense embedding: {bad}'

def has_dense(item) -> bool:
    return bool(item.embedding and item.embedding.dense_embedding)

try:
    pcoll | _WriteToBigQueryVectorDatabase(cfg)
except ValueError as e:
    if 'must contain dense embedding' in str(e): route_to_reembedding(e)
    else: raise

When it happens

Trigger: Using the default embeddable_to_dict_fn (SchemaConfig with no custom fn) while writing EmbeddableItems whose embedding is None or whose embedding.dense_embedding is None/empty.

Common situations: Pipeline omitted the embedding step before the BigQuery vector write; failed/empty embeddings from the model for some records; reusing the default fn for metadata-only writes.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• Item {item.id} missing embedding
• Item {embeddable_item.id} missing dense embedding required f
• Item {embeddable_item.id} missing both text content and spar
• Unexpected keyword arguments: {', '.join(kwargs)}
• SchemaConfig requires embeddable_to_dict_fn

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/125158c6d391f29d. Report an issue: GitHub.