apache/beam · error · ValueError

EmbeddableItem must contain embedding

Error message

EmbeddableItem must contain embedding: {embeddable}

What it means

The default extract_fn used by SpannerColumnSpec.with_embedding_spec requires the EmbeddableItem to have a dense_embedding; if it's empty/None there is no vector to convert for the Spanner column, so it raises with the offending item in the message.

Solutions

Ensure an embedding-generation transform populates dense_embedding before the Spanner sink
Supply a custom extract_fn to with_embedding_spec that reads a different field or raises later
Filter out items with empty dense_embedding before writing
Debug why the embedding step produced empty vectors for the offending item (logged in the message)

Example fix

// before
builder.with_embedding_spec(column_name='embedding')
// after (custom extraction)
builder.with_embedding_spec(
    column_name='embedding',
    extract_fn=lambda item: item.dense_embedding or fallback_embed(item.content.text))

Defensive patterns

Strategy: validation

Validate before calling

bad = [i for i in items if not i.dense_embedding]
if bad:
    raise ValueError(f"{len(bad)} items lack dense_embedding before Spanner write")

Type guard

def has_dense(item) -> bool:
    return bool(item.dense_embedding)

Try / catch

try:
    row = spec.to_row(item)
except ValueError:
    logging.warning("Skipping item without embedding: %s", item)
    return None

Prevention

Run the embedding transform before the Spanner sink in the pipeline graph
Filter items with empty dense_embedding lists (empty lists also fail)
Log un-embedded items upstream so gaps are visible before the write stage

When it happens

Trigger: Writing to Cloud Spanner via the SpannerVectorWriterBuilder without running an embedding transform first, or with items whose dense_embedding is None or an empty list (empty lists are falsy and also trigger this).

Common situations: Embedding model returning [] for empty text; records sourced before the embedding stage; supplying a custom convert_fn but relying on the default extract_fn for items that only carry sparse embeddings.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/efa5656cdd8154d7. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/rag/ingestion/spanner.py:224

    
    Examples:
        Default embedding:
        >>> builder.with_embedding_spec()
        
        Normalized embedding:
        >>> def normalize(vec):
        ...     norm = (sum(x**2 for x in vec) ** 0.5) or 1.0
        ...     return [x/norm for x in vec]
        >>> builder.with_embedding_spec(convert_fn=normalize)
        
        Rounded precision:
        >>> builder.with_embedding_spec(
        ...     convert_fn=lambda vec: [round(x, 4) for x in vec]
        ... )
    """
    def extract_fn(embeddable: EmbeddableItem) -> list[float]:
      if not embeddable.dense_embedding:
        raise ValueError(f'EmbeddableItem must contain embedding: {embeddable}')
      return embeddable.dense_embedding

    self._specs.append(
        SpannerColumnSpec(
            column_name=column_name,
            python_type=list[float],
            value_fn=functools.partial(
                _extract_and_convert, extract_fn, convert_fn)))
    return self

  def with_content_spec(
      self,
      column_name: str = "content",
      python_type: type = str,
      convert_fn: Optional[Callable[[str], Any]] = None
  ) -> 'SpannerColumnSpecsBuilder':
    """Add content column.

View on GitHub (pinned to 12126d8942)