apache/beam · error · ValueError

Expected chunk to contain embedding.

Error message

Expected chunk to contain embedding. {chunk}

What it means

chunk_embedding_fn in the PostgreSQL helper formats an EmbeddableItem's dense embedding as a Postgres array literal '{v1,v2,...}'. It requires chunk.embedding.dense_embedding to exist; when either chunk.embedding or dense_embedding is None it raises because a vector column cannot be written without a dense vector.

Solutions

1. Run the embedding transform on all chunks before the Postgres writer.
2. Ensure the embedder fills dense_embedding (not just sparse_embedding).
3. Filter or re-embed chunks with chunk.embedding is None before writing.

Example fix

// before
rows = chunks | PostgresVectorWriter(config)  # chunks unembedded
// after
embedded = chunks | "embed" >> embedding_transform
rows = embedded | PostgresVectorWriter(config)

if chunk.embedding is None or chunk.embedding.dense_embedding is None:
    raise ValueError(f"chunk missing dense embedding: {chunk.id}")

def has_dense_embedding(chunk) -> bool:
    return chunk.embedding is not None and chunk.embedding.dense_embedding is not None

try:
    vec = chunk_embedding_fn(chunk)
except ValueError:
    chunk = re_embed(chunk)
    vec = chunk_embedding_fn(chunk)

When it happens

Trigger: Writing to Postgres with a spec whose value_fn is chunk_embedding_fn while a chunk's embedding (or its dense_embedding) is None — un-embedded items reaching the sink.

Common situations: Sink placed before the embedding transform in the Beam pipeline; embedding model returning None for some inputs; reusing a pipeline that produces only sparse embeddings.

Related errors

• Duplicate column names found
• Expected chunk to contain embedding.
• Expected chunk to contain sparse embedding.
• Approximate Nearest Neighbor Search (ANNS) field must be…
• chunk_to_dict_fn is deprecated, use embeddable_to_dict_fn

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b548e8837ec5a7c4. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/rag/ingestion/postgres_common.py:44

def chunk_embedding_fn(chunk: EmbeddableItem) -> str:
  """Convert embedding to PostgreSQL array string.

    Formats dense embedding as a PostgreSQL-compatible array string.
    Example: [1.0, 2.0] -> '{1.0,2.0}'

    Args:
        chunk: Input EmbeddableItem object.

    Returns:
        str: PostgreSQL array string representation of the embedding.

    Raises:
        ValueError: If chunk has no dense embedding.
    """
  if chunk.embedding is None or chunk.embedding.dense_embedding is None:
    raise ValueError(f'Expected chunk to contain embedding. {chunk}')
  return '{' + ','.join(str(x) for x in chunk.embedding.dense_embedding) + '}'


@dataclass
class ColumnSpec:
  """Mapping of EmbeddableItem fields to SQL columns for insertion.

  Defines how to extract and format values from EmbeddableItems into
  database columns, handling the full pipeline from Python value to
  SQL insertion.

  The insertion process works as follows:
  - value_fn extracts a value from the EmbeddableItem and formats it as needed
  - The value is stored in a NamedTuple field with the specified python_type
  - During SQL insertion, the value is bound to a ? placeholder

  Attributes:
      column_name: The column name in the database table.

View on GitHub (pinned to 12126d8942)