pathwaycom/pathway · error · ValueError

Failed to detect the region of S3 bucket

Error message

Failed to detect the region of S3 bucket {bucket!r} (HTTP status {response.status_code}): the bucket may not exist. If it does, pass AwsS3Settings with an explicit region

What it means

pw.io.clickhouse.write appends system metadata columns to every output row: {time, diff} in the default streaming mode, or {version, is_deleted} when output_table_type="snapshot". If the user's schema already contains columns with those names, the write would be ambiguous, so it is rejected up front.

Solutions

  1. Rename the colliding column(s) in the Pathway schema before writing (e.g. time -> event_time, diff -> change_diff).
  2. If you must keep names in ClickHouse, rename in Pathway and map back with a ClickHouse VIEW or column alias downstream.
  3. Check which pair applies to your output_table_type and only rename those names.

Example fix

# before
class Input(pw.Schema):
    time: int
    value: str
pw.io.clickhouse.write(t, "db.table")

# after
t = t.with_columns(event_time=t.time).without(pw.this.time)
pw.io.clickhouse.write(t, "db.table")
Defensive patterns

Strategy: validation

Validate before calling

def check_clickhouse_names(table: pw.Table, output_table_type: str) -> None:
    reserved = {"version", "is_deleted"} if output_table_type == "snapshot" else {"time", "diff"}
    clash = reserved & set(table.schema.column_names())
    if clash:
        raise ValueError(f"rename columns {sorted(clash)}; they are reserved")

check_clickhouse_names(t, output_table_type)
pw.io.clickhouse.write(t, ..., output_table_type=output_table_type)

Prevention

When it happens

Trigger: Calling pw.io.clickhouse.write with a table whose schema includes a column named time or diff (default mode), or version / is_deleted (snapshot mode).

Common situations: CDC-style or audit schemas that naturally carry time/diff/version columns; switching output_table_type between runs changes which pair is reserved, so a schema that worked in one mode fails in the other.

Related errors

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/72302c8045185b25. Report an issue: GitHub.

Appendix: source

Thrown at python/pathway/internals/_io_helpers.py:166

            has_extra_chars = len(s3_path) > len(s3_path_prefix)
            if not starts_with_prefix or not has_extra_chars:
                continue
            bucket = s3_path[len(s3_path_prefix) :].split("/")[0]

            # the crate we use on the Rust-engine side can't detect the location of a
            # bucket, so it's done here; S3 reports the region of a bucket in a response
            # header, even for anonymous requests and even when it replies with a
            # redirect or an access denial
            import requests

            response = requests.head(
                f"https://s3.amazonaws.com/{bucket}",
                allow_redirects=False,
                timeout=S3_REGION_DETECTION_TIMEOUT_S,
            )
            region = response.headers.get("x-amz-bucket-region")
            if region is None:
                raise ValueError(
                    f"Failed to detect the region of S3 bucket {bucket!r} "
                    f"(HTTP status {response.status_code}): the bucket may not exist. "
                    "If it does, pass AwsS3Settings with an explicit region"
                )

            return cls(
                bucket_name=bucket,
                region=region,
            )

        # If it doesn't start with a valid S3 prefix, it's not a full S3 path
        raise ValueError(f"Incorrect S3 path: {s3_path}")

    def authorize(self):
        """Fills in the credentials that the downstream libraries can't deduce.

        The DeltaLake library resolves environment variables and instance
        credentials on its own, but does not read AWS profile files — those are

View on GitHub (pinned to fa2f74a464)