pathwaycom/pathway · error · ValueError

schema does not match given dataframe

Error message

schema does not match given dataframe

What it means

Output connectors such as the ClickHouse/Delta writers accept a string init_mode that is mapped to an engine enum (DEFAULT, CREATE_IF_NOT_EXISTS, REPLACE). init_mode_from_str raises ValueError for any string outside the supported set.

Source

Thrown at python/pathway/debug/__init__.py:382

    _new_universe: bool = False,
) -> Table:
    """A function for creating a table from a pandas DataFrame. If it contains a special
    column ``__time__``, rows will be split into batches with timestamps from the column.
    A special column ``__diff__`` can be used to set an event type - with ``1`` treated
    as inserting the row and ``-1`` as removing it.
    """
    if id_from is not None and schema is not None:
        raise ValueError("parameters `schema` and `id_from` are mutually exclusive")

    ordinary_columns_names = [
        column for column in df.columns if column not in api.PANDAS_PSEUDOCOLUMNS
    ]
    if schema is None:
        schema = schema_from_pandas(
            df, id_from=id_from, exclude_columns=api.PANDAS_PSEUDOCOLUMNS
        )
    elif set(ordinary_columns_names) != set(schema.column_names()):
        raise ValueError("schema does not match given dataframe")

    _validate_dataframe(df, stacklevel=_stacklevel + 4)

    if id_from is None and schema is not None:
        id_from = schema.primary_key_columns()

    if id_from is None:
        ids_df = pd.DataFrame({"id": df.index})
        ids_df.index = df.index
    else:
        ids_df = df[id_from].copy()

    for column in api.PANDAS_PSEUDOCOLUMNS:
        if column in df.columns:
            ids_df[column] = df[column]

    as_hashes = [fingerprint(x) for x in ids_df.to_dict(orient="records")]
    key = fingerprint((unsafe_trusted_ids, sorted(as_hashes)))

View on GitHub (pinned to fa2f74a464)

Solutions

1. Use one of the exact supported strings: "default", "create_if_not_exists", or "replace" (lowercase).
2. Check the installed Pathway version's docstring for pw.io.clickhouse.write to see the accepted init_mode values.
3. Upgrade pathway if the mode you need is documented but missing in your version.

Example fix

# before
pw.io.clickhouse.write(t, ..., init_mode="CREATE_IF_NOT_EXISTS")

# after
pw.io.clickhouse.write(t, ..., init_mode="create_if_not_exists")

VALID_INIT_MODES = {"default", "create_if_not_exists", "replace"}
if init_mode not in VALID_INIT_MODES:
    raise ValueError(f"init_mode must be one of {sorted(VALID_INIT_MODES)}")
pw.io.clickhouse.write(t, ..., init_mode=init_mode)

from typing import Literal
InitMode = Literal["default", "create_if_not_exists", "replace"]

def is_init_mode(s: str) -> TypeGuard[InitMode]:
    return s in {"default", "create_if_not_exists", "replace"}

When it happens

Trigger: Calling pw.io.clickhouse.write (or a similar writer) with init_mode set to a value other than "default", "create_if_not_exists", or "replace" — including case variants like "REPLACE" or newer ClickHouse modes like "create".

Common situations: User copies a mode name from ClickHouse SQL documentation (e.g. CREATE TABLE ... REPLACE TABLE) instead of the Pathway enum; or typos/case-mismatches the value; or passes a mode added in a newer Pathway version while running an older one.

Related errors

• SchemaRegistryHeader.key must be a str, got {type(self.key).
• Failed to install dependencies
• Column {api.TIME_PSEUDOCOLUMN} cannot contain negative times
• negative timestamp cannot be used
• demo.range_stream error: nb_rows should be strictly positive

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/34162a8121409afb. Report an issue: GitHub.