pathwaycom/pathway · error · ValueError

database {path_str!r} is an existing directory, not a DuckDB

Error message

database {path_str!r} is an existing directory, not a DuckDB database file. DuckDB cannot open a directory as a database; pass a file path instead.

What it means

Raised by pw.io.duckdb.write when the `database` argument resolves to an existing directory rather than a DuckDB database file. DuckDB cannot open a directory as a database; without this preflight check DuckDB would fail later inside an engine worker with an opaque 'unable to open database' error. The check is skipped for the special ':memory:' path.

Source

Thrown at python/pathway/io/duckdb/__init__.py:33

from pathway.internals.trace import trace_user_frame
from pathway.io._utils import (
    SNAPSHOT_OUTPUT_TABLE_TYPE,
    get_column_index,
    init_mode_from_str,
)

IN_MEMORY_DATABASE = ":memory:"


def _reject_directory_path(path_str: str) -> None:
    """Reject a ``database`` path that resolves to an existing directory
    (directly or via a symlink). Without this preflight DuckDB fails later
    with an opaque "unable to open database" error inside an engine worker.
    """
    if path_str == IN_MEMORY_DATABASE:
        return
    if os.path.isdir(path_str):
        raise ValueError(
            f"database {path_str!r} is an existing directory, not a DuckDB "
            "database file. DuckDB cannot open a directory as a database; "
            "pass a file path instead."
        )


@check_arg_types
@trace_user_frame
def write(
    table: Table,
    *,
    table_name: str,
    database: PathLike | str,
    max_batch_size: int | None = None,
    init_mode: Literal["default", "create_if_not_exists", "replace"] = "default",
    output_table_type: Literal["stream_of_changes", "snapshot"] = "stream_of_changes",
    primary_key: list[ColumnReference] | None = None,
    detach_between_batches: bool = False,

View on GitHub (pinned to fa2f74a464)

Solutions

Pass a full file path, e.g. os.path.join(data_dir, 'out.duckdb') instead of data_dir itself.
Check the value with os.path.isdir(...) before calling write() to fail fast in your own code with your own message.
If you intended an in-memory database, pass database=":memory:" exactly — that string is explicitly exempted.
Inspect the variable/symlink chain feeding `database` (os.path.realpath) to find where the directory crept in.

Example fix

# before
pw.io.duckdb.write(t, table_name="t", database="/data/warehouse")

# after
pw.io.duckdb.write(t, table_name="t", database="/data/warehouse/out.duckdb")

Defensive patterns

Strategy: validation

Validate before calling

import os

 database = "/data/warehouse/out.duckdb"
 if database != ":memory:" and os.path.isdir(database):
    raise ValueError(f"{database} is a directory; pass a database file path")

Type guard

def is_duckdb_file_path(p: str) -> bool:
    return p == ":memory:" or (os.path.exists(p) and not os.path.isdir(p)) or not os.path.exists(p)

Try / catch

try:
    pw.io.duckdb.write(t, table_name="t", database=path)
except ValueError as e:
    if "existing directory" in str(e):
        path = os.path.join(path, "out.duckdb")
        pw.io.duckdb.write(t, table_name="t", database=path)
    else:
        raise

Prevention

Never feed a DATA_DIR-style variable directly as `database`; always os.path.join(dir, filename).
Add an integration test that runs write() against a temp directory path with a real filename.
Document ':memory:' as the only non-file value allowed.

When it happens

Trigger: Calling pw.io.duckdb.write(table, table_name=..., database=<path>) where <path> is an existing directory, either directly or via a symlink. Common when a config var pointing at a data directory is passed unmodified, or when the intended filename was never appended to the directory path.

Common situations: Reusing a DATA_DIR environment variable as the database path without joining a filename; typos where the .duckdb extension and file name are omitted; build pipelines that pre-create output directories; symlinked paths that point at folders.

Related errors

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/a6f07cccefc79d32. Report an issue: GitHub.