pathwaycom/pathway · error · ValueError

Failed to detect the region of S3 bucket {bucket!r} (HTTP st

Error message

Failed to detect the region of S3 bucket {bucket!r} (HTTP status {response.status_code}): the bucket may not exist. If it does, pass AwsS3Settings with an explicit region

What it means

pw.io.clickhouse.write appends system metadata columns to every output row: {time, diff} in the default streaming mode, or {version, is_deleted} when output_table_type="snapshot". If the user's schema already contains columns with those names, the write would be ambiguous, so it is rejected up front.

Source

Thrown at python/pathway/internals/_io_helpers.py:166

            has_extra_chars = len(s3_path) > len(s3_path_prefix)
            if not starts_with_prefix or not has_extra_chars:
                continue
            bucket = s3_path[len(s3_path_prefix) :].split("/")[0]

            # the crate we use on the Rust-engine side can't detect the location of a
            # bucket, so it's done here; S3 reports the region of a bucket in a response
            # header, even for anonymous requests and even when it replies with a
            # redirect or an access denial
            import requests

            response = requests.head(
                f"https://s3.amazonaws.com/{bucket}",
                allow_redirects=False,
                timeout=S3_REGION_DETECTION_TIMEOUT_S,
            )
            region = response.headers.get("x-amz-bucket-region")
            if region is None:
                raise ValueError(
                    f"Failed to detect the region of S3 bucket {bucket!r} "
                    f"(HTTP status {response.status_code}): the bucket may not exist. "
                    "If it does, pass AwsS3Settings with an explicit region"
                )

            return cls(
                bucket_name=bucket,
                region=region,
            )

        # If it doesn't start with a valid S3 prefix, it's not a full S3 path
        raise ValueError(f"Incorrect S3 path: {s3_path}")

    def authorize(self):
        """Fills in the credentials that the downstream libraries can't deduce.

        The DeltaLake library resolves environment variables and instance
        credentials on its own, but does not read AWS profile files — those are

View on GitHub (pinned to fa2f74a464)

Solutions

1. Rename the colliding column(s) in the Pathway schema before writing (e.g. time -> event_time, diff -> change_diff).
2. If you must keep names in ClickHouse, rename in Pathway and map back with a ClickHouse VIEW or column alias downstream.
3. Check which pair applies to your output_table_type and only rename those names.

Example fix

# before
class Input(pw.Schema):
    time: int
    value: str
pw.io.clickhouse.write(t, "db.table")

# after
t = t.with_columns(event_time=t.time).without(pw.this.time)
pw.io.clickhouse.write(t, "db.table")

def check_clickhouse_names(table: pw.Table, output_table_type: str) -> None:
    reserved = {"version", "is_deleted"} if output_table_type == "snapshot" else {"time", "diff"}
    clash = reserved & set(table.schema.column_names())
    if clash:
        raise ValueError(f"rename columns {sorted(clash)}; they are reserved")

check_clickhouse_names(t, output_table_type)
pw.io.clickhouse.write(t, ..., output_table_type=output_table_type)

When it happens

Trigger: Calling pw.io.clickhouse.write with a table whose schema includes a column named time or diff (default mode), or version / is_deleted (snapshot mode).

Common situations: CDC-style or audit schemas that naturally carry time/diff/version columns; switching output_table_type between runs changes which pair is reserved, so a schema that worked in one mode fails in the other.

Related errors

• schema does not match given dataframe
• Incorrect S3 path: {s3_path}
• SchemaRegistryHeader.key must be a str, got {type(self.key).
• Schema.with_types() argument name has to be an existing colu
• Schema.without() argument {name!r} has to refer to an existi

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/72302c8045185b25. Report an issue: GitHub.