apache/beam · error · ValueError

Item {embeddable_item.id} missing dense embedding required f

Error message

Item {embeddable_item.id} missing dense embedding required for vector search

What it means

When running vector (or hybrid) search, MilvusEnricher converts each EmbeddableItem to a query vector via _get_vector_search_data. If an item's dense_embedding is empty/None, it cannot be used as a Milvus query vector, so a ValueError names the offending item id.

Source

Thrown at sdks/python/apache_beam/ml/rag/enrichment/milvus_search.py:489

        data=vector_search_data,
        anns_field=self.search_strategy.vector.anns_field,
        param=self.search_strategy.vector.search_params,
        limit=self.search_strategy.vector.limit,
        expr=self.search_strategy.vector.filter)

    keyword_search_req = AnnSearchRequest(
        data=keyword_search_data,
        anns_field=self.search_strategy.keyword.anns_field,
        param=self.search_strategy.keyword.search_params,
        limit=self.search_strategy.keyword.limit,
        expr=self.search_strategy.keyword.filter)

    reqs = [vector_search_req, keyword_search_req]
    return reqs

  def _get_vector_search_data(self, embeddable_item: EmbeddableItem):
    if not embeddable_item.dense_embedding:
      raise ValueError(
          f"Item {embeddable_item.id} missing dense embedding required for"
          " vector search")
    return embeddable_item.dense_embedding

  def _get_keyword_search_data(self, embeddable_item: EmbeddableItem):
    has_no_text = not embeddable_item.content.text
    has_no_sparse = not embeddable_item.sparse_embedding
    if has_no_text and has_no_sparse:
      raise ValueError(
          f"Item {embeddable_item.id} missing both text content and sparse "
          "embedding required for keyword search")
    sparse_embedding = MilvusHelpers.sparse_embedding(
        embeddable_item.sparse_embedding)
    return embeddable_item.content.text or sparse_embedding

  def _get_call_response(
      self,
      embeddable_items: list[EmbeddableItem],

View on GitHub (pinned to 12126d8942)

Solutions

1. Ensure every item passes through an embedding transform (e.g., EmbedText) before MilvusEnricher in vector/hybrid mode
2. Filter out or re-embed items with empty dense_embedding before enrichment
3. Log/inspect which ids lack embeddings; fix upstream producer for those records

Example fix

// before
results = pcoll | MilvusEnricher(params)  # items not embedded
// after
def has_dense(item):
    return item.embedding and item.embedding.dense_embedding
results = (pcoll
    | 'FilterEmbedded' >> beam.Filter(has_dense)
    | MilvusEnricher(params))

bad = [it.id for it in items if not (it.embedding and it.embedding.dense_embedding)]
assert not bad, f'items missing dense embedding: {bad}'

def has_dense_embedding(item) -> bool:
    return bool(item.embedding and item.embedding.dense_embedding)

try:
    enriched = enricher(items)
except ValueError as e:
    if 'missing dense embedding' in str(e): re_embed_and_retry(items)
    else: raise

When it happens

Trigger: An EmbeddableItem flowing through MilvusEnricher with hybrid or vector search strategy whose dense_embedding field is None or empty — usually because the upstream embedding step did not run or failed for that item.

Common situations: Embedding transform skipped for some records (e.g., empty text failing silently); items loaded from a cache/source that lacks embeddings; mixing strategies where keyword-sourced items enter a vector search pipeline.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• Item {embeddable_item.id} missing both text content and spar
• Search strategy must be provided
• Not supported search strategy yet: {self.search_strategy}
• EmbeddableItem must contain dense embedding
• Subclasses must implement get_splitter_transform

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4f5d31469f6e1f65. Report an issue: GitHub.