pathwaycom/pathway · error · AssertionError

there are extra columns

Error message

there are extra columns: {extra_columns} which are not present in the provided schema

What it means

The second phase of assert_same_schema_as: with allow_superset=False, self must not have columns beyond other's. Extra columns in self (present in the actual schema but not in the expected one) are collected and reported by this AssertionError.

Solutions

1. Drop the extra columns before comparing: table.select(*expected_columns) or schema.without(*extra).
2. If extras are acceptable, call with allow_superset=True (the default).
3. Update the expected schema to include the new columns if they are intentional.

Example fix

# before
actual.schema.assert_same_schema_as(expected.schema, allow_superset=False)
# AssertionError: extra columns: {'meta'}

# after
actual.select(*expected.column_names()).schema.assert_same_schema_as(
    expected.schema, allow_superset=False
)

import pathway as pw

def no_extra_columns(self_schema, other_schema) -> bool:
    return set(self_schema.keys()) <= set(other_schema.keys())

try:
    actual.schema.assert_same_schema_as(expected.schema, allow_superset=False)
except AssertionError as e:
    if 'extra columns' in str(e):
        actual = actual.select(*expected.column_names())
        actual.schema.assert_same_schema_as(expected.schema, allow_superset=False)
    else:
        raise

When it happens

Trigger: table.schema.assert_same_schema_as(expected.schema, allow_superset=False) while table has additional columns (e.g. an upstream connector added metadata columns, or select kept too many columns).

Common situations: Strict schema checks in tests after adding a derived column; connectors appending internal columns (like _timestamped or partition metadata); comparing a widened schema against a strict expected schema.

Related errors

• schema does not have columns
• Column name '_id' is reserved: MongoDB uses '_id' as the…
• Defining a primary key in the schema is not supported for…
• Failed to detect the region of S3 bucket
• 'json_field_paths' references field

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/0e61fb55eef52bcf. Report an issue: GitHub.

Appendix: source

Thrown at python/pathway/internals/schema.py:628

        # Check if self has all columns of other
        if self_dict.keys() < other_dict.keys():
            missing_columns = other_dict.keys() - self_dict.keys()
            raise AssertionError(f"schema does not have columns {missing_columns}")

        # Check if types of columns are the same
        for col in other_dict:
            assert other_dict[col] == self_dict[col] or (
                allow_subtype and dt.dtype_issubclass(self_dict[col], other_dict[col])
            ), (
                f"type of column {col} does not match - its type is {self_dict[col]} in {self.__name__}",
                f" and {other_dict[col]} in {other.__name__}",
            )

        # When allow_superset=False, check that self does not have extra columns
        if not allow_superset and self_dict.keys() > other_dict.keys():
            extra_columns = self_dict.keys() - other_dict.keys()
            raise AssertionError(
                f"there are extra columns: {extra_columns} which are not present in the provided schema"
            )

        # Check whether primary keys are the same
        if not ignore_primary_keys:
            assert self.primary_key_columns() == other.primary_key_columns(), (
                f"primary keys in the schemas do not match - they are {self.primary_key_columns()} in {self.__name__}",
                f" and {other.primary_key_columns()} in {other.__name__}",
            )
        if not ignore_properties:
            self_columns = self.columns()
            other_columns = other.columns()
            for column_name, column_schema in self_columns.items():
                other_column_schema = other_columns[column_name]
                assert column_schema == other_column_schema


def _schema_builder(

View on GitHub (pinned to fa2f74a464)