apache/beam · error · ValueError

EmbeddableItem must contain dense embedding

Error message

EmbeddableItem must contain dense embedding

What it means

The default EmbeddableItem-to-BigQuery-row converter requires each item to carry a dense embedding, since the 'embedding' column stores item.embedding.dense_embedding. Items without one are rejected with this ValueError at write time.

Solutions

1. Run an embedding transform on all items before VectorDatabaseWriteTransform
2. Filter out items without dense embeddings, or backfill them
3. Provide a custom embeddable_to_dict_fn if dense embeddings are not needed

Example fix

// before
rows = pcoll | VectorDatabaseWriteTransform(bq_config)  # items unembedded
// after
rows = (pcoll
    | beam.Filter(lambda i: i.embedding and i.embedding.dense_embedding)
    | VectorDatabaseWriteTransform(bq_config))

bad = [it.id for it in items if not (it.embedding and it.embedding.dense_embedding)]
assert not bad, f'items missing dense embedding: {bad}'

def has_dense(item) -> bool:
    return bool(item.embedding and item.embedding.dense_embedding)

try:
    pcoll | _WriteToBigQueryVectorDatabase(cfg)
except ValueError as e:
    if 'must contain dense embedding' in str(e): route_to_reembedding(e)
    else: raise

When it happens

Trigger: Using the default embeddable_to_dict_fn (SchemaConfig with no custom fn) while writing EmbeddableItems whose embedding is None or whose embedding.dense_embedding is None/empty.

Common situations: Pipeline omitted the embedding step before the BigQuery vector write; failed/empty embeddings from the model for some records; reusing the default fn for metadata-only writes.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• Item missing both text content and sparse embedding…
• Item missing dense embedding required for vector search
• Item missing embedding
• SchemaConfig requires embeddable_to_dict_fn
• Unexpected keyword arguments

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/125158c6d391f29d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/rag/ingestion/bigquery.py:142

            metadata columns.
    
    Raises:
        ValueError: If write_config doesn't include table specification.
    """
    if 'table' not in write_config:
      raise ValueError("write_config must be provided with 'table' specified")

    self.write_config = write_config
    self.schema_config = schema_config

  def create_write_transform(self) -> beam.PTransform:
    """Creates transform to write to BigQuery."""
    return _WriteToBigQueryVectorDatabase(self)


def _default_embeddable_to_dict_fn(item: EmbeddableItem):
  if item.embedding is None or item.embedding.dense_embedding is None:
    raise ValueError("EmbeddableItem must contain dense embedding")
  return {
      'id': item.id,
      'embedding': item.embedding.dense_embedding,
      'content': item.content_string,
      'metadata': [{
          "key": k, "value": str(v)
      } for k, v in item.metadata.items()]
  }


def _default_schema():
  return {
      'fields': [{
          'name': 'id', 'type': 'STRING'
      }, {
          'name': 'embedding', 'type': 'FLOAT64', 'mode': 'REPEATED'
      }, {
          'name': 'content', 'type': 'STRING'

View on GitHub (pinned to 12126d8942)