What causes "Incorrect S3 path: {s3_path}" in pathwaycom/pathway?

Pass primary_key with the identifying column(s), e.g. primary_key=t.id or primary_key=[t.id, t.tenant_id].

pathwaycom/pathway · error · ValueError

Incorrect S3 path: {s3_path}

Error message

Incorrect S3 path: {s3_path}

What it means

With output_table_type="snapshot", pw.io.clickhouse.write maintains the current row state keyed by primary key columns (using ClickHouse ReplacingMergeTree-style versioning), so it must know which column(s) identify a row. primary_key is mandatory in that mode.

Source

Thrown at python/pathway/internals/_io_helpers.py:178

                f"https://s3.amazonaws.com/{bucket}",
                allow_redirects=False,
                timeout=S3_REGION_DETECTION_TIMEOUT_S,
            )
            region = response.headers.get("x-amz-bucket-region")
            if region is None:
                raise ValueError(
                    f"Failed to detect the region of S3 bucket {bucket!r} "
                    f"(HTTP status {response.status_code}): the bucket may not exist. "
                    "If it does, pass AwsS3Settings with an explicit region"
                )

            return cls(
                bucket_name=bucket,
                region=region,
            )

        # If it doesn't start with a valid S3 prefix, it's not a full S3 path
        raise ValueError(f"Incorrect S3 path: {s3_path}")

    def authorize(self):
        """Fills in the credentials that the downstream libraries can't deduce.

        The DeltaLake library resolves environment variables and instance
        credentials on its own, but does not read AWS profile files — those are
        resolved here, with the official AWS SDK chain (environment, profile
        files, SSO, assume-role, IMDS) built into the engine.
        """
        if self._access_key is not None and self._secret_access_key is not None:
            return
        env_access_key = os.environ.get("AWS_ACCESS_KEY_ID")
        env_secret_access_key = os.environ.get("AWS_SECRET_ACCESS_KEY")
        if env_access_key and env_secret_access_key:
            # pinned explicitly instead of left for the engine to rediscover:
            # this snapshot is immune to the credential vars that delta-rs
            # writes into the process env for previously opened tables
            self._access_key = env_access_key

View on GitHub (pinned to fa2f74a464)

Solutions

Pass primary_key with the identifying column(s), e.g. primary_key=t.id or primary_key=[t.id, t.tenant_id].
Ensure the key columns come from the same table being written.
If you actually wanted append-only output, drop output_table_type="snapshot" instead.

Example fix

# before
pw.io.clickhouse.write(t, "db.pets", output_table_type="snapshot")

# after
pw.io.clickhouse.write(t, "db.pets", output_table_type="snapshot", primary_key=t.id)

Defensive patterns

Strategy: validation

Validate before calling

if output_table_type == "snapshot":
    assert primary_key is not None and len(list(primary_key)) > 0, (
        "output_table_type='snapshot' requires primary_key"
    )
pw.io.clickhouse.write(t, ..., output_table_type=output_table_type, primary_key=primary_key)

Prevention

Model the writer config so snapshot mode cannot be built without a primary_key field.
Pick key columns that truly identify a row across updates (stable ids, not mutable attributes).

When it happens

Trigger: Calling pw.io.clickhouse.write(..., output_table_type="snapshot") with primary_key omitted, None, or an empty list.

Common situations: User switches from default append mode (where primary_key is not needed) to snapshot mode to get up-to-date rows in ClickHouse and forgets to add the key; or passes primary_key_columns (an older/different spelling) instead of primary_key.

Related errors

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/8a495bcd9244449a. Report an issue: GitHub.