pathwaycom/pathway · error · TypeError

Embedder is not a valid `pw.UDF`.

Error message

Embedder is not a valid `pw.UDF`.

What it means

When a KnnIndexFactory (e.g. UsearchKnnFactory, BruteForceKnnFactory) is given an `embedder` but no explicit `dimensions`, __post_init__ must infer the embedding dimension by calling the embedder. It accepts only pathway.xpacks.llm.embedders.BaseEmbedder instances or pw.UDF callables; anything else (a plain function, a class, an OpenAI client wrapper without the UDF decorator) raises this TypeError from _get_embed_dimensions.

Source

Thrown at python/pathway/stdlib/indexing/nearest_neighbors.py:420

            metadata_filter=metadata_filter,
        )


@dataclass(kw_only=True)
class KnnIndexFactory(InnerIndexFactory):
    dimensions: int | None = None
    embedder: pw.UDF | None = None

    def _get_embed_dimensions(self) -> int:
        # import is here to prevent cyclical imports
        from pathway.xpacks.llm.embedders import BaseEmbedder

        if isinstance(self.embedder, BaseEmbedder):
            return self.embedder.get_embedding_dimension()
        elif isinstance(self.embedder, pw.UDF):
            return len(_coerce_sync(self.embedder.__wrapped__)("."))
        else:
            raise TypeError("Embedder is not a valid `pw.UDF`.")

    def __post_init__(self):
        if self.dimensions is None and self.embedder is not None:
            self.dimensions: int = self._get_embed_dimensions()
        elif self.dimensions is None and self.embedder is None:
            raise ValueError(
                "Either `dimensions` or `embedder` must be provided to index factory."
            )


@dataclass(kw_only=True)
class UsearchKnnFactory(KnnIndexFactory):
    """
    Factory for creating UsearchKNN indices.

    Args:
        dimensions (int): number of dimensions of vectors that are used by the index and
            queries. This is only needed if the `embedder` is not provided.

View on GitHub (pinned to fa2f74a464)

Solutions

1. Wrap the embedding function in a UDF: embedder=pw.udf(my_embed_fn) (async supported), so the isinstance(self.embedder, pw.UDF) branch matches.
2. Or use a ready BaseEmbedder from pathway.xpacks.llm.embedders (e.g. OpenAIEmbedder), which exposes get_embedding_dimension().
3. Or bypass inference entirely by passing dimensions=384 (etc.) explicitly so _get_embed_dimensions is never called.

Example fix

# before
def embed(text: str) -> list[float]: ...
factory = UsearchKnnFactory(embedder=embed)

# after
@pw.udf
def embed(text: str) -> list[float]: ...
factory = UsearchKnnFactory(embedder=embed)

import pathway as pw
from pathway.xpacks.llm.embedders import BaseEmbedder

def is_valid_embedder(e) -> bool:
    return isinstance(e, (BaseEmbedder, pw.UDF))

try:
    factory = UsearchKnnFactory(embedder=embed)
except TypeError as e:
    if "not a valid `pw.UDF`" in str(e):
        factory = UsearchKnnFactory(embedder=pw.udf(embed), dimensions=None)
    else:
        raise

When it happens

Trigger: Passing embedder=some_plain_function or a non-UDF object to UsearchKnnFactory/BruteForceKnnFactory without also passing dimensions; the check fires in __post_init__ when dimensions is None, i.e. at factory construction time.

Common situations: Using a raw sentence-transformers encode function or an SDK client method as the embedder instead of wrapping it with @pw.udf or using a BaseEmbedder from pathway.xpacks.llm; refactoring code so the embedder is lazily imported/wrapped and accidentally passing the unwrapped callable.

Related errors

• direction argument of join should be of type asof_join.Direc
• The interval argument of a join should be of a type pathway.
• Expected a ColumnReference, found a string. Did you mean thi
• Cannot flatten column of type {dtype}.
• not supported type of debug data

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/af64c37273d2bcbd. Report an issue: GitHub.