pathwaycom/pathway · error · ValueError

id_column {id_column!r} is not present in the schema

Error message

id_column {id_column!r} is not present in the schema

What it means

Raised by pw.io.elasticsearch.read when the id_column argument does not name a column of the provided schema. The connector uses that column as the document id to key the resulting Pathway table, so it must exist in schema.

Source

Thrown at python/pathway/io/elasticsearch/__init__.py:316

    ...     host="http://localhost:9200",
    ...     auth=pw.io.elasticsearch.ElasticSearchAuth.basic("admin", "admin"),
    ...     index_name="logs",
    ...     schema=LogSchema,
    ...     timestamp_column="ts",
    ...     id_column="doc_id",
    ...     max_transaction_duration=datetime.timedelta(minutes=5),
    ... )
    """

    _check_entitlements("elasticsearch")

    column_names = schema.column_names()
    if timestamp_column not in column_names:
        raise ValueError(
            f"timestamp_column {timestamp_column!r} is not present in the schema"
        )
    if id_column not in column_names:
        raise ValueError(f"id_column {id_column!r} is not present in the schema")
    if schema.primary_key_columns():
        raise ValueError(
            "Defining a primary key in the schema is not supported for "
            "pw.io.elasticsearch.read. The connector keys the resulting table by "
            "id_column. If you need to reindex the table by a different column, use "
            "pw.Table.with_id_from() after reading."
        )

    max_transaction_duration_ms = round(
        as_duration_seconds(max_transaction_duration, "max_transaction_duration") * 1000
    )
    poll_interval_ms = round(as_duration_seconds(poll_interval, "poll_interval") * 1000)
    if read_batch_size <= 0:
        raise ValueError("read_batch_size must be positive")

    data_storage = api.DataStorage(
        storage_type="elasticsearch",
        mode=internal_connector_mode(mode),

View on GitHub (pinned to fa2f74a464)

Solutions

1. Fix id_column to name an existing schema column and ensure that field is declared in the schema class.
2. Check DocSchema.column_names() for the exact spelling/case of the id field.
3. If the index only has ES's internal _id, materialize an explicit id field in the documents or pick a unique existing field.

Example fix

# before
class DocSchema(pw.Schema):
    _id: str
    text: str
pw.io.elasticsearch.read(..., schema=DocSchema, id_column="doc_id")

# after
class DocSchema(pw.Schema):
    doc_id: str
    text: str
pw.io.elasticsearch.read(..., schema=DocSchema, id_column="doc_id")

column_names = schema.column_names()
assert id_column in column_names, (
    f"id_column {id_column!r} not in schema columns {column_names}"
)

When it happens

Trigger: pw.io.elasticsearch.read(..., schema=DocSchema, id_column="doc_id") where DocSchema declares no 'doc_id' field — typo, renamed field, or a schema written for a different index.

Common situations: ES documents keyed by '_id' with no materialized id field in the mapping; copy-pasted schema classes; case or spelling mismatches between the mapping and the schema class.

Related errors

• timestamp_column {timestamp_column!r} is not present in the
• Defining a primary key in the schema is not supported for pw
• `{self.name}` has no attribute `{name}`
• Schemas should not be called. Use `table.schema` not `table.
• Schema.with_types() argument name has to be an existing colu

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/51132910fb06551d. Report an issue: GitHub.