pathwaycom/pathway · error · NotImplementedError

vector column {name!r} has type {dtype}, which is a multivec

Error message

vector column {name!r} has type {dtype}, which is a multivector; a Pinecone record carries a single dense or sparse vector, so multivectors are not supported.

What it means

A Pinecone record carries a single dense or sparse vector; a List-of-List (or List-of-Array) dtype is a multivector and cannot be stored. pw.io.pinecone.write raises this NotImplementedError at call time when the vector column's inner type is itself a list/array, so multivector attempts fail before the pipeline starts rather than per-row at the sink.

Source

Thrown at python/pathway/io/pinecone/__init__.py:85

    A dense vector is a numeric list / array, a sparse one a
    ``list[tuple[int, float]]`` of ``(index, weight)`` pairs. Mirrors the runtime
    ``PineconeError::InvalidVector`` / ``InvalidSparseVector`` guards so a wrong
    column type fails at ``write()`` time rather than once data flows.
    """
    if _is_statically_unknown(dtype):
        return
    if isinstance(dtype, dt.Optional):
        raise ValueError(
            f"vector column {name!r} is nullable (type {dtype}); every row must "
            "carry a vector, so the column cannot be optional."
        )
    if isinstance(dtype, dt.List):
        inner = dtype.wrapped
        if _is_numeric(inner) or _is_sparse_pair(inner):
            return
        if isinstance(inner, (dt.List, dt.Array)):
            raise NotImplementedError(
                f"vector column {name!r} has type {dtype}, which is a multivector; "
                "a Pinecone record carries a single dense or sparse vector, so "
                "multivectors are not supported."
            )
    if isinstance(dtype, dt.Array) and _is_numeric(dtype.wrapped):
        return
    if isinstance(dtype, dt.Tuple) and all(_is_numeric(arg) for arg in dtype.args):
        return
    raise ValueError(
        f"vector column {name!r} has unsupported type {dtype}; a Pinecone vector "
        "must be a list[float] or a 1-D float array (dense), or a "
        "list[tuple[int, float]] of (index, weight) pairs (sparse)."
    )


def _check_metadata_dtype(name: str, dtype: dt.DType) -> None:
    """Reject a metadata column whose type Pinecone cannot store.

View on GitHub (pinned to fa2f74a464)

Solutions

1. Flatten one-record-per-vector before the sink: explode/flatten the table so each row holds one embedding and a stable id (e.g. f"{doc_id}-{i}").
2. If only one vector per record is intended, fix the upstream step that wrapped embeddings in an extra list dimension.
3. Consider a store that supports multivectors if per-record multiple embeddings are a hard requirement.

Example fix

# before
class Chunks(pw.Schema):
    doc_id: str
    embeddings: list[list[float]]  # multivector -> rejected
pw.io.pinecone.write(chunks, "idx", primary_key=chunks.doc_id, vector=chunks.embeddings)

# after
flat = chunks.flatten(pw.this.embeddings).with_columns(
    vec_id=pw.this.doc_id + "-" + pw.this.meta.index.to_string()
)
# then write one embedding per row with vector=<the inner list column>

import pathway as pw

def is_multivector(dtype: pw.dt.DType) -> bool:
    return (
        isinstance(dtype, pw.dt.List)
        and isinstance(dtype.wrapped, (pw.dt.List, pw.dt.Array))
    )

assert not is_multivector(table.schema[vector_col].dtype), (
    "Flatten to one embedding per row before Pinecone"
)

import pathway as pw

def is_single_vector_dtype(dtype: pw.dt.DType) -> bool:
    if isinstance(dtype, pw.dt.List):
        inner = dtype.wrapped
        return not isinstance(inner, (pw.dt.List, pw.dt.Array))
    return True

try:
    pw.io.pinecone.write(chunks, "idx", primary_key=chunks.doc_id, vector=chunks.embeddings)
except NotImplementedError as e:
    if "multivector" in str(e):
        flat = chunks.flatten(pw.this.embeddings)
        # assign one stable id per flattened row, then write
    else:
        raise

When it happens

Trigger: Passing vector=table.vecs where table.vecs has dtype list[list[float]] — e.g. batching multiple embeddings per row — to pw.io.pinecone.write.

Common situations: Multi-chunk document pipelines that group several chunk embeddings into one row; switching from a multivector-capable store (e.g. some Milvus/Qdrant modes) to Pinecone; np.ndarray of shape (n_chunks, dim) serialized as nested lists.

Related errors

• vector column {name!r} has unsupported type {dtype}; a Pinec
• primary_key column {name!r} has unsupported type {dtype}; a
• vector column {name!r} is nullable (type {dtype}); every row
• metadata column {name!r} has unsupported type {dtype}; Pinec
• Cannot flatten column of type {dtype}.

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/e7e6dcde3e10b5ba. Report an issue: GitHub.