{"record":{"id":"3c0f50d8d79a874a","repo":"apache/beam","slug":"item-item-id-missing-embedding","errorCode":null,"errorMessage":"Item {item.id} missing embedding","messagePattern":"Item (.+?) missing embedding","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/ml/rag/enrichment/bigquery_vector_search.py","lineNumber":225,"sourceCode":"\n    # Group items by their metadata conditions\n    condition_groups = defaultdict(list)\n    if self.metadata_restriction_template:\n      for item in items:\n        condition = self._format_restrict(item)\n        condition_groups[condition].append(item)\n    else:\n      # No metadata filtering - all items in one group\n      condition_groups[\"\"] = items\n\n    # Generate VECTOR_SEARCH subqueries for each condition group\n    vector_searches = []\n    for condition, group_items in condition_groups.items():\n      # Create embeddings subquery for this group\n      embedding_unions = []\n      for item in group_items:\n        if not item.dense_embedding:\n          raise ValueError(f\"Item {item.id} missing embedding\")\n        embedding_str = (\n            f\"SELECT '{item.id}' as id, \"\n            f\"{[float(x) for x in item.dense_embedding]} \"\n            f\"as embedding\")\n        embedding_unions.append(embedding_str)\n      group_embeddings = \" UNION ALL \".join(embedding_unions)\n\n      where_clause = f\"WHERE {condition}\" if condition else \"\"\n      # Create VECTOR_SEARCH for this condition group\n      vector_search = f\"\"\"\n            SELECT \n                query.id,\n                ARRAY_AGG(\n                    STRUCT({\"distance, \" if self.include_distance else \"\"}\\\n {base_columns_str})\n                ) as chunks\n            FROM VECTOR_SEARCH(\n                (SELECT {columns_str}, {self.embedding_column}","sourceCodeStart":207,"sourceCodeEnd":243,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/ml/rag/enrichment/bigquery_vector_search.py#L207-L243","documentation":"BigQueryVectorSearch builds a SQL subquery that inlines each item's dense_embedding as a float array for nearest-neighbor search. When formatting the query, format_query raises ValueError if any item's dense_embedding is empty/None, since the generated SQL would be invalid without a vector literal.","triggerScenarios":"Using BigQueryVectorSearch enrichment where the upstream embedding step failed, was skipped, or produced items with dense_embedding=None before enrichment runs (e.g. embedding model error silently ignored, wrong field populated).","commonSituations":"Pipelines where EnrichWithBigQueryVectorSearch runs before or without a VertexAI/OpenAI embedding transform; embedding field naming mismatch so the field never gets set; partial batch failures in a prior step.","solutions":["Ensure an embedding transform runs before the BigQuery enrichment and populates dense_embedding for every item.","Filter out items with empty dense_embedding before the enrichment stage.","Check upstream embedding transform logs for failed batches.","Verify the embedding manager's output field matches the field format_query reads."],"exampleFix":"// before\nresult = pcoll | EnrichWithBigQueryVectorSearch(...)  # items have dense_embedding=None\n// after\npcoll = pcoll | EmbedVertexAI(...)  # populate embeddings first\nresult = pcoll | EnrichWithBigQueryVectorSearch(...)","handlingStrategy":"validation","validationCode":"missing = [it.id for it in items if not it.dense_embedding]\nif missing:\n    raise ValueError(f'Items missing dense_embedding before enrichment: {missing}')","typeGuard":"def has_embedding(item) -> bool:\n    return bool(getattr(item, 'dense_embedding', None))","tryCatchPattern":"try:\n    enriched = pcoll | EnrichWithBigQueryVectorSearch(...)\nexcept ValueError as e:\n    if 'missing embedding' in str(e):\n        logging.error('Upstream embedding step failed or was skipped: %s', e)\n    raise","preventionTips":["Always place an embedding transform before the BigQuery enrichment stage.","Monitor the embedding stage's failed-element metrics.","Filter out items with empty embeddings as an explicit pipeline step."],"tags":["python","apache-beam","bigquery","rag","enrichment"],"backgroundTag":"empty-required-field","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}