pathwaycom/pathway · error · ValueError

Column name '_id' is reserved: MongoDB uses '_id' as the pri

Error message

Column name '_id' is reserved: MongoDB uses '_id' as the primary key for every document, so pw.io.mongodb.write cannot accept a column with this name. Rename the column before writing.

What it means

MongoDB reserves _id as the mandatory primary key of every document and the connector generates it itself, so pw.io.mongodb.write refuses an input table that has a column literally named '_id'. Passing one through would collide with the connector-managed identifier and corrupt upserts, so it raises ValueError asking you to rename.

Source

Thrown at python/pathway/io/mongodb/__init__.py:649

                "numCandidates": 100,
                "limit": 5,
            }},
            {"$project": {"_id": 0, "doc_id": 1,
                          "score": {"$meta": "vectorSearchScore"}}},
        ])

    **Note on parallelism.** When the program is run with multiple workers
    (``pathway spawn -n N``), the write is distributed across them, and write
    throughput grows with the worker count up to the capacity of the target
    MongoDB/Atlas deployment. Each document is written by a single worker, so the
    result is the same as with one worker. The exception is ``sort_by``: requesting
    a global order within a minibatch makes the connector write from a single
    worker, so a sorted output does not benefit from additional workers.
    """
    is_snapshot_mode = output_table_type == SNAPSHOT_OUTPUT_TABLE_TYPE
    column_names = set(table.schema.column_names())
    if "_id" in column_names:
        raise ValueError(
            "Column name '_id' is reserved: MongoDB uses '_id' as the primary key "
            "for every document, so pw.io.mongodb.write cannot accept a column with "
            "this name. Rename the column before writing."
        )
    if not is_snapshot_mode:
        reserved = {"diff", "time"} & column_names
        if reserved:
            raise ValueError(
                f"Column name(s) {sorted(reserved)!r} collide with the reserved "
                f"fields written by pw.io.mongodb.write in 'stream_of_changes' mode. "
                f"Rename the column(s) or use output_table_type='snapshot'."
            )
    data_storage = api.DataStorage(
        storage_type="mongodb",
        connection_string=connection_string,
        database=database,
        table_name=collection,
        max_batch_size=max_batch_size,

View on GitHub (pinned to fa2f74a464)

Solutions

1. Rename or drop the column before writing: table = table.rename(_id='mongo_id') or table.without(_id) if Mongo may regenerate ids
2. If you must preserve original ids, let the connector key on a renamed copy: table.rename(_id='source_id')
3. Strip _id when ingesting raw documents upstream (del doc['_id']) when identity does not matter

Example fix

# before
pw.io.mongodb.write(t, uri, db, coll)  # t has column '_id'

# after
t = t.rename(_id='source_doc_id')
pw.io.mongodb.write(t, uri, db, coll)

names = set(table.schema.column_names())
if "_id" in names:
    raise ValueError("Rename the '_id' column before pw.io.mongodb.write")
# or fix: table = table.rename(_id='source_doc_id')

def has_reserved_mongo_names(table: pw.Table) -> bool:
    return "_id" in table.schema.column_names()

try:
    pw.io.mongodb.write(table, uri, db, coll)
except ValueError as e:
    if "'_id' is reserved" in str(e):
        pw.io.mongodb.write(table.rename(_id='source_doc_id'), uri, db, coll)
    else:
        raise

When it happens

Trigger: pw.io.mongodb.write(table, uri, db, coll) where table's schema contains a column named '_id' — typically a table that was previously read from MongoDB or built from BSON documents verbatim.

Common situations: Round-tripping: read from Mongo, transform, write back with the original _id still present; converting raw dicts (from pymongo) to a Pathway table without stripping _id.

Related errors

• Defining a primary key in the schema is not supported for pw
• Column name(s) {sorted(reserved)!r} collide with the reserve
• Failed to detect the region of S3 bucket {bucket!r} (HTTP st
• Schema.with_types() argument name has to be an existing colu
• Schema.without() argument {name!r} has to refer to an existi

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/667e98fbbb543d02. Report an issue: GitHub.