{"record":{"id":"ab162e1245fd88f4","repo":"apache/beam","slug":"expected-chunk-to-contain-sparse-embedding-chunk","errorCode":null,"errorMessage":"Expected chunk to contain sparse embedding. {chunk}","messagePattern":"Expected chunk to contain sparse embedding\\. (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/ml/rag/ingestion/postgres_common.py","lineNumber":338,"sourceCode":"\n      Args:\n          column_name: Name for the sparse embedding column\n            (defaults to \"sparse_embedding\")\n          conv_fn: Optional function to convert the sparse embedding tuple\n                      If None, converts to PostgreSQL-compatible JSON format\n\n      Returns:\n          Self for method chaining\n\n      Example:\n          >>> builder.with_sparse_embedding_spec(\n          ...     column_name=\"sparse_vector\",\n          ...     convert_fn=lambda sparse: dict(zip(sparse[0], sparse[1]))\n          ... )\n      \"\"\"\n    def value_fn(chunk: EmbeddableItem) -> Any:\n      if chunk.embedding is None or chunk.embedding.sparse_embedding is None:\n        raise ValueError(f'Expected chunk to contain sparse embedding. {chunk}')\n      sparse_embedding = chunk.embedding.sparse_embedding\n      if conv_fn:\n        return conv_fn(sparse_embedding)\n      # Default: convert to dict format for JSON storage.\n      indices, values = sparse_embedding\n      return json.dumps(dict(zip(indices, values)))\n\n    self._specs.append(\n        ColumnSpec.jsonb(column_name=column_name, value_fn=value_fn))\n    return self\n\n  def add_metadata_field(\n      self,\n      field: str,\n      python_type: type,\n      column_name: Optional[str] = None,\n      convert_fn: Optional[Callable[[Any], Any]] = None,\n      default: Any = None,","sourceCodeStart":320,"sourceCodeEnd":356,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/ml/rag/ingestion/postgres_common.py#L320-L356","documentation":"In PostgresVectorWriterConfig.add_sparse_vector, the internal value_fn requires chunk.embedding.sparse_embedding to be present before converting it (via conv_fn or the default JSON dict of indices/values). Missing sparse embedding raises ValueError for that element.","triggerScenarios":"Calling PostgresVectorWriterConfig().add_sparse_vector(column_name=..., conv_fn=...) and writing chunks whose embedding is None or whose sparse_embedding is None — e.g. only dense embeddings were produced.","commonSituations":"Dense-only embedding pipelines (standard text embedders) fed into a sparse column; forgetting to add a sparse-embedding model like SPLADE to the pipeline.","solutions":["Add a sparse embedding stage (e.g. SPLADE or a sparse encoder) upstream so sparse_embedding is populated.","If you only need dense vectors, drop the sparse column from the schema/specs.","Filter chunks lacking sparse_embedding or write a default empty sparse value."],"exampleFix":"// before\nconfig.add_sparse_vector(column_name=\"sparse_vector\")  # dense-only pipeline\n// after\nsparse = chunks | RunInference(SparseEmbedder())  # fills sparse_embedding\nconfig.add_sparse_vector(column_name=\"sparse_vector\")\nrows = sparse | PostgresVectorWriter(config)","handlingStrategy":"type-guard","validationCode":"if chunk.embedding is None or chunk.embedding.sparse_embedding is None:\n    raise ValueError(f\"chunk missing sparse embedding: {chunk.id}\")","typeGuard":"def has_sparse_embedding(chunk) -> bool:\n    return chunk.embedding is not None and chunk.embedding.sparse_embedding is not None","tryCatchPattern":"try:\n    value = spec.value_fn(chunk)\nexcept ValueError:\n    value = json.dumps({})  # or dead_letter(chunk)","preventionTips":["Only use add_sparse_vector when your embedding model emits sparse embeddings","Verify sparse indices/values exist before building the pipeline","Keep dense and sparse column specs aligned with your model outputs"],"tags":["python","postgres","rag","sparse-embedding"],"backgroundTag":"null-argument","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}