pathwaycom/pathway · error · ValueError

Optimization is based on the column '{table_optimizer.tracke

Error message

Optimization is based on the column '{table_optimizer.tracked_column}', which is not a partition column. Please include this column in partition_columns.

What it means

Error "Optimization is based on the column '{table_optimizer.tracked_column}', which is not a partition column. Please include this column in partition_columns." thrown in pathwaycom/pathway.

Source

Thrown at python/pathway/io/deltalake/__init__.py:681

        storage_type="deltalake",
        path=uri,
        aws_s3_settings=_engine_s3_connection_settings(
            uri, prepared_connection_settings
        ),
        min_commit_frequency=min_commit_frequency,
        partition_columns=prepared_partition_columns,
        snapshot_maintenance_on_output=output_table_type == SNAPSHOT_OUTPUT_TABLE_TYPE,
        delta_optimizer_rule=(table_optimizer.engine_rule if table_optimizer else None),
    )
    data_format = api.DataFormat(
        format_type="identity",
        key_field_names=None,
        value_fields=_format_output_value_fields(table),
    )

    if table_optimizer is not None:
        if table_optimizer.tracked_column.name not in prepared_partition_columns:
            raise ValueError(
                f"Optimization is based on the column '{table_optimizer.tracked_column}', "
                "which is not a partition column. Please include this column in partition_columns."
            )
        table_optimizer._start_compression(table_path=uri)

    table.to(
        datasink.GenericDataSink(
            data_storage,
            data_format,
            datasink_name="deltalake",
            unique_name=name,
            sort_by=sort_by,
            on_pipeline_finished=(
                table_optimizer._stop_compression
                if table_optimizer is not None
                else None
            ),
        )

View on GitHub (pinned to fa2f74a464)

Solutions

1. Include the tracked (min/max) optimization column in partition_columns.

Example fix

pw.io.deltalake.write(t, uri, partition_columns=[t.date, t.tracked_col], min_max_index=[t.tracked_col])

When it happens

Trigger: Thrown at python/pathway/io/deltalake/__init__.py:681 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/6567e40796916abb. Report an issue: GitHub.