{"record":{"id":"43c934ac948fc6c5","repo":"pathwaycom/pathway","slug":"detach-between-batches-true-cannot-be-used-with-da","errorCode":null,"errorMessage":"detach_between_batches=True cannot be used with database=\":memory:\": an in-memory DuckDB database is dropped when its last connection closes, so all data would be lost after every batch. Use an on-disk database file instead.","messagePattern":"detach_between_batches=True cannot be used with database=\":memory:\": an in-memory DuckDB database is dropped when its last connection closes, so all data would be lost after every batch\\. Use an on-disk database file instead\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/pathway/io/duckdb/__init__.py","lineNumber":278,"sourceCode":"        # and can place the insertion before the deletion, so the upsert is\n        # immediately wiped by the delete — silently losing rows. sort_by has no\n        # effect on a snapshot's final (unordered, keyed) state anyway, so reject\n        # the combination instead of corrupting the output.\n        raise ValueError(\n            \"sort_by cannot be used with the snapshot table type: a snapshot \"\n            \"reflects the current state of the table regardless of the order \"\n            \"changes are applied within a minibatch, and reordering would corrupt \"\n            \"the upsert/delete sequence. Remove sort_by, or use \"\n            'output_table_type=\"stream_of_changes\".'\n        )\n\n    database_str = fspath(database)\n    _reject_directory_path(database_str)\n\n    if detach_between_batches and database_str == IN_MEMORY_DATABASE:\n        # An in-memory database ceases to exist when its last connection closes,\n        # so detaching after every batch would silently drop all written data.\n        raise ValueError(\n            'detach_between_batches=True cannot be used with database=\":memory:\": '\n            \"an in-memory DuckDB database is dropped when its last connection \"\n            \"closes, so all data would be lost after every batch. Use an on-disk \"\n            \"database file instead.\"\n        )\n\n    value_fields = _format_output_value_fields(table)\n\n    # DuckDB matches identifiers case-insensitively, so two schema columns whose\n    # names differ only in case would make CREATE TABLE fail with a raw\n    # \"Column with name ... already exists\" catalog error. Surface it here.\n    case_groups: dict[str, list[str]] = {}\n    for field in value_fields:\n        case_groups.setdefault(field.name.lower(), []).append(field.name)\n    case_collisions = [\n        sorted(names) for names in case_groups.values() if len(names) > 1\n    ]\n    if case_collisions:","sourceCodeStart":260,"sourceCodeEnd":296,"githubUrl":"https://github.com/pathwaycom/pathway/blob/fa2f74a4649b7c5908690cf60137263d8d80de5f/python/pathway/io/duckdb/__init__.py#L260-L296","documentation":"Raised by pw.io.duckdb.write when detach_between_batches=True is combined with database=\":memory:\". An in-memory DuckDB database is dropped when its last connection closes, so detaching after every batch would silently lose all written data — the connector rejects the combination instead of producing an empty database.","triggerScenarios":"pw.io.duckdb.write(t, table_name=\"t\", database=\":memory:\", detach_between_batches=True).","commonSituations":"Prototyping with :memory: and enabling detach_between_batches (often to free locks or work around file-lock issues seen on disk); copying settings between configs when moving from file-based to in-memory testing.","solutions":["Use an on-disk database file: pw.io.duckdb.write(t, table_name=\"t\", database=\"out.duckdb\", detach_between_batches=True).","Or keep database=\":memory:\" and drop detach_between_batches=True (leave it False/None)."],"exampleFix":"# before\npw.io.duckdb.write(t, table_name=\"t\", database=\":memory:\", detach_between_batches=True)\n\n# after\npw.io.duckdb.write(t, table_name=\"t\", database=\"out.duckdb\", detach_between_batches=True)","handlingStrategy":"validation","validationCode":"if detach_between_batches and database == \":memory:\":\n    raise ValueError(\"in-memory database cannot survive detach_between_batches\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use a temp file (tempfile.NamedTemporaryFile) instead of :memory: for tests that need detach_between_batches.","Keep detach_between_batches out of default configs; enable it only with an on-disk path."],"tags":["duckdb","connector","in-memory","data-loss"],"backgroundTag":null,"analyzedSha":"fa2f74a4649b7c5908690cf60137263d8d80de5f","analyzedAt":"2026-08-15T01:48:17.006Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}