apache/beam · error · ValueError

Expected chunk to contain sparse embedding.

Error message

Expected chunk to contain sparse embedding. {chunk}

What it means

In PostgresVectorWriterConfig.add_sparse_vector, the internal value_fn requires chunk.embedding.sparse_embedding to be present before converting it (via conv_fn or the default JSON dict of indices/values). Missing sparse embedding raises ValueError for that element.

Solutions

Add a sparse embedding stage (e.g. SPLADE or a sparse encoder) upstream so sparse_embedding is populated.
If you only need dense vectors, drop the sparse column from the schema/specs.
Filter chunks lacking sparse_embedding or write a default empty sparse value.

Example fix

// before
config.add_sparse_vector(column_name="sparse_vector")  # dense-only pipeline
// after
sparse = chunks | RunInference(SparseEmbedder())  # fills sparse_embedding
config.add_sparse_vector(column_name="sparse_vector")
rows = sparse | PostgresVectorWriter(config)

Defensive patterns

Strategy: type-guard

Validate before calling

if chunk.embedding is None or chunk.embedding.sparse_embedding is None:
    raise ValueError(f"chunk missing sparse embedding: {chunk.id}")

Type guard

def has_sparse_embedding(chunk) -> bool:
    return chunk.embedding is not None and chunk.embedding.sparse_embedding is not None

Try / catch

try:
    value = spec.value_fn(chunk)
except ValueError:
    value = json.dumps({})  # or dead_letter(chunk)

Prevention

Only use add_sparse_vector when your embedding model emits sparse embeddings
Verify sparse indices/values exist before building the pipeline
Keep dense and sparse column specs aligned with your model outputs

When it happens

Trigger: Calling PostgresVectorWriterConfig().add_sparse_vector(column_name=..., conv_fn=...) and writing chunks whose embedding is None or whose sparse_embedding is None — e.g. only dense embeddings were produced.

Common situations: Dense-only embedding pipelines (standard text embedders) fed into a sparse column; forgetting to add a sparse-embedding model like SPLADE to the pipeline.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ab162e1245fd88f4. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/rag/ingestion/postgres_common.py:338

      Args:
          column_name: Name for the sparse embedding column
            (defaults to "sparse_embedding")
          conv_fn: Optional function to convert the sparse embedding tuple
                      If None, converts to PostgreSQL-compatible JSON format

      Returns:
          Self for method chaining

      Example:
          >>> builder.with_sparse_embedding_spec(
          ...     column_name="sparse_vector",
          ...     convert_fn=lambda sparse: dict(zip(sparse[0], sparse[1]))
          ... )
      """
    def value_fn(chunk: EmbeddableItem) -> Any:
      if chunk.embedding is None or chunk.embedding.sparse_embedding is None:
        raise ValueError(f'Expected chunk to contain sparse embedding. {chunk}')
      sparse_embedding = chunk.embedding.sparse_embedding
      if conv_fn:
        return conv_fn(sparse_embedding)
      # Default: convert to dict format for JSON storage.
      indices, values = sparse_embedding
      return json.dumps(dict(zip(indices, values)))

    self._specs.append(
        ColumnSpec.jsonb(column_name=column_name, value_fn=value_fn))
    return self

  def add_metadata_field(
      self,
      field: str,
      python_type: type,
      column_name: Optional[str] = None,
      convert_fn: Optional[Callable[[Any], Any]] = None,
      default: Any = None,

View on GitHub (pinned to 12126d8942)