What causes "Expected chunk to contain embedding. {chunk}" in apache/beam?

Ensure an embedding transform (e.g. MLTransform or an EmbeddingFn) runs on all chunks before the MySQL writer.

apache/beam · error · ValueError

Expected chunk to contain embedding.

Error message

Expected chunk to contain embedding. {chunk}

What it means

chunk_embedding_fn converts an EmbeddableItem's dense embedding into the MySQL vector string '[v1,v2,...]'. It requires chunk.embedding and chunk.embedding.dense_embedding to be present; if either is None it raises, because a vector column cannot be written without a dense vector.

Solutions

Ensure an embedding transform (e.g. MLTransform or an EmbeddingFn) runs on all chunks before the MySQL writer.
Check that your embedding function produces dense_embedding, not just sparse_embedding.
Log/inspect chunks upstream and filter out (or re-embed) items with chunk.embedding is None.

Example fix

// before
rows = chunks | MySqlVectorWriter(config)  # chunks not embedded
// after
embedded = chunks | "embed" >> MLTransform(...).with_transform(embedding_transform)
rows = embedded | MySqlVectorWriter(config)

Defensive patterns

Strategy: type-guard

Validate before calling

if chunk.embedding is None or chunk.embedding.dense_embedding is None:
    raise ValueError(f"chunk missing dense embedding: {chunk.id}")

Type guard

def has_dense_embedding(chunk) -> bool:
    return chunk.embedding is not None and chunk.embedding.dense_embedding is not None

Try / catch

try:
    vec = chunk_embedding_fn(chunk)
except ValueError:
    chunk = re_embed(chunk)  # or route to dead-letter
    vec = chunk_embedding_fn(chunk)

Prevention

Run the embedding stage immediately before the sink in the pipeline graph
Assert dense embeddings exist in a validation DoFn before writing
Make your embedding fn raise instead of returning None on failure

When it happens

Trigger: Writing to MySQL via a ColumnSpec whose value_fn is chunk_embedding_fn when a chunk was never embedded (embedding stage failed/skipped), produced only sparse embeddings, or chunk.embedding is None.

Common situations: Pipeline stage ordering errors where the MySQL sink runs before the embedding transform; an embedding model returning None for empty text; loading pre-existing chunks that lack embeddings.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5af7f5834d7a367e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/rag/ingestion/mysql_common.py:43


def chunk_embedding_fn(chunk: EmbeddableItem) -> str:
  """Convert embedding to MySQL vector string format.

    Formats dense embedding as a MySQL-compatible vector string.
    Example: [1.0, 2.0] -> '[1.0,2.0]'

    Args:
        chunk: Input EmbeddableItem object.
    
    Returns:
        str: MySQL vector string representation of the embedding.
    
    Raises:
        ValueError: If chunk has no dense embedding.
    """
  if chunk.embedding is None or chunk.embedding.dense_embedding is None:
    raise ValueError(f'Expected chunk to contain embedding. {chunk}')
  return '[' + ','.join(str(x) for x in chunk.embedding.dense_embedding) + ']'


@dataclass
class ColumnSpec:
  """Mapping of EmbeddableItem fields to SQL columns for insertion.

  Defines how to extract and format values from EmbeddableItems into MySQL
  database columns, handling the full pipeline from Python value to SQL
  insertion.

  The insertion process works as follows:
  - value_fn extracts a value from the EmbeddableItem and formats it as needed
  - The value is stored in a NamedTuple field with the specified python_type
  - During SQL insertion, the value is bound to a ? placeholder

  Attributes:
      column_name: The column name in the database table.

View on GitHub (pinned to 12126d8942)