pathwaycom/pathway · error · AssertionError

schema does not have columns {missing_columns}

Error message

schema does not have columns {missing_columns}

What it means

Schema.assert_same_schema_as(other) (and schema assertions used in tests / connector validation) first checks that self contains all columns of other. If self's column set is missing some of other's columns, the missing names are collected and reported with this AssertionError — this is the subset check, raised before any type comparison.

Source

Thrown at python/pathway/internals/schema.py:614

        with open(path, mode="w") as f:
            f.write(class_definition)

    def assert_matches_schema(
        self,
        other: type[Schema],
        *,
        allow_superset: bool = True,
        ignore_primary_keys: bool = True,
        allow_subtype: bool = True,
        ignore_properties: bool = True,
    ) -> None:
        self_dict = self._dtypes()
        other_dict = other._dtypes()

        # Check if self has all columns of other
        if self_dict.keys() < other_dict.keys():
            missing_columns = other_dict.keys() - self_dict.keys()
            raise AssertionError(f"schema does not have columns {missing_columns}")

        # Check if types of columns are the same
        for col in other_dict:
            assert other_dict[col] == self_dict[col] or (
                allow_subtype and dt.dtype_issubclass(self_dict[col], other_dict[col])
            ), (
                f"type of column {col} does not match - its type is {self_dict[col]} in {self.__name__}",
                f" and {other_dict[col]} in {other.__name__}",
            )

        # When allow_superset=False, check that self does not have extra columns
        if not allow_superset and self_dict.keys() > other_dict.keys():
            extra_columns = self_dict.keys() - other_dict.keys()
            raise AssertionError(
                f"there are extra columns: {extra_columns} which are not present in the provided schema"
            )

        # Check whether primary keys are the same

View on GitHub (pinned to fa2f74a464)

Solutions

  1. Add the missing columns (e.g. with_columns(missing_col=pw.const(None)) or restore them in select) so self covers other.
  2. If the smaller schema is intended, compare against it instead: swap arguments so the superset schema is 'self', or trim other with .without(...).
  3. For connector reads, ensure the declared schema only lists columns actually present in the data.

Example fix

# before
a.schema.assert_same_schema_as(expected.schema)  # a lacks 'extra'

# after
a_ext = a.with_columns(extra=pw.const(None).astype(int))
a_ext.schema.assert_same_schema_as(expected.schema)
Defensive patterns

Strategy: validation

Validate before calling

import pathway as pw

def covers_all_columns(self_schema, other_schema) -> bool:
    return set(other_schema.keys()) <= set(self_schema.keys())

Try / catch

try:
    actual.schema.assert_same_schema_as(expected.schema)
except AssertionError as e:
    if 'does not have columns' in str(e):
        missing = set(expected.column_names()) - set(actual.column_names())
        actual = actual.with_columns(**{c: pw.const(None) for c in missing})
    else:
        raise

Prevention

When it happens

Trigger: Calling table_a.schema.assert_same_schema_as(table_b.schema) where table_a lacks columns present in table_b; also assert_schema_correct style validation in io connectors with a user-declared schema missing declared fields.

Common situations: Test assertions after select/reduce that dropped columns; input data not containing all columns declared in a schema passed to pw.io.*.read; refactors that removed columns without updating the expected schema.

Related errors

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/666ee3542e9ef281. Report an issue: GitHub.