apache/beam · error · ValueError

Item {item.id} missing embedding

Error message

Item {item.id} missing embedding

What it means

BigQueryVectorSearch builds a SQL subquery that inlines each item's dense_embedding as a float array for nearest-neighbor search. When formatting the query, format_query raises ValueError if any item's dense_embedding is empty/None, since the generated SQL would be invalid without a vector literal.

Source

Thrown at sdks/python/apache_beam/ml/rag/enrichment/bigquery_vector_search.py:225

    # Group items by their metadata conditions
    condition_groups = defaultdict(list)
    if self.metadata_restriction_template:
      for item in items:
        condition = self._format_restrict(item)
        condition_groups[condition].append(item)
    else:
      # No metadata filtering - all items in one group
      condition_groups[""] = items

    # Generate VECTOR_SEARCH subqueries for each condition group
    vector_searches = []
    for condition, group_items in condition_groups.items():
      # Create embeddings subquery for this group
      embedding_unions = []
      for item in group_items:
        if not item.dense_embedding:
          raise ValueError(f"Item {item.id} missing embedding")
        embedding_str = (
            f"SELECT '{item.id}' as id, "
            f"{[float(x) for x in item.dense_embedding]} "
            f"as embedding")
        embedding_unions.append(embedding_str)
      group_embeddings = " UNION ALL ".join(embedding_unions)

      where_clause = f"WHERE {condition}" if condition else ""
      # Create VECTOR_SEARCH for this condition group
      vector_search = f"""
            SELECT 
                query.id,
                ARRAY_AGG(
                    STRUCT({"distance, " if self.include_distance else ""}\
 {base_columns_str})
                ) as chunks
            FROM VECTOR_SEARCH(
                (SELECT {columns_str}, {self.embedding_column}

View on GitHub (pinned to 12126d8942)

Solutions

  1. Ensure an embedding transform runs before the BigQuery enrichment and populates dense_embedding for every item.
  2. Filter out items with empty dense_embedding before the enrichment stage.
  3. Check upstream embedding transform logs for failed batches.
  4. Verify the embedding manager's output field matches the field format_query reads.

Example fix

// before
result = pcoll | EnrichWithBigQueryVectorSearch(...)  # items have dense_embedding=None
// after
pcoll = pcoll | EmbedVertexAI(...)  # populate embeddings first
result = pcoll | EnrichWithBigQueryVectorSearch(...)
Defensive patterns

Strategy: validation

Validate before calling

missing = [it.id for it in items if not it.dense_embedding]
if missing:
    raise ValueError(f'Items missing dense_embedding before enrichment: {missing}')

Type guard

def has_embedding(item) -> bool:
    return bool(getattr(item, 'dense_embedding', None))

Try / catch

try:
    enriched = pcoll | EnrichWithBigQueryVectorSearch(...)
except ValueError as e:
    if 'missing embedding' in str(e):
        logging.error('Upstream embedding step failed or was skipped: %s', e)
    raise

Prevention

When it happens

Trigger: Using BigQueryVectorSearch enrichment where the upstream embedding step failed, was skipped, or produced items with dense_embedding=None before enrichment runs (e.g. embedding model error silently ignored, wrong field populated).

Common situations: Pipelines where EnrichWithBigQueryVectorSearch runs before or without a VertexAI/OpenAI embedding transform; embedding field naming mismatch so the field never gets set; partial batch failures in a prior step.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3c0f50d8d79a874a. Report an issue: GitHub.