{"record":{"id":"a04a8087218d27b4","repo":"pathwaycom/pathway","slug":"timestamp-column-timestamp-column-r-is-not-prese","errorCode":null,"errorMessage":"timestamp_column {timestamp_column!r} is not present in the schema","messagePattern":"timestamp_column (.+?) is not present in the schema","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/pathway/io/elasticsearch/__init__.py","lineNumber":312,"sourceCode":"    ...     doc_id: str\n    ...     ts: int\n    ...     message: str\n    >>> table = pw.io.elasticsearch.read(\n    ...     host=\"http://localhost:9200\",\n    ...     auth=pw.io.elasticsearch.ElasticSearchAuth.basic(\"admin\", \"admin\"),\n    ...     index_name=\"logs\",\n    ...     schema=LogSchema,\n    ...     timestamp_column=\"ts\",\n    ...     id_column=\"doc_id\",\n    ...     max_transaction_duration=datetime.timedelta(minutes=5),\n    ... )\n    \"\"\"\n\n    _check_entitlements(\"elasticsearch\")\n\n    column_names = schema.column_names()\n    if timestamp_column not in column_names:\n        raise ValueError(\n            f\"timestamp_column {timestamp_column!r} is not present in the schema\"\n        )\n    if id_column not in column_names:\n        raise ValueError(f\"id_column {id_column!r} is not present in the schema\")\n    if schema.primary_key_columns():\n        raise ValueError(\n            \"Defining a primary key in the schema is not supported for \"\n            \"pw.io.elasticsearch.read. The connector keys the resulting table by \"\n            \"id_column. If you need to reindex the table by a different column, use \"\n            \"pw.Table.with_id_from() after reading.\"\n        )\n\n    max_transaction_duration_ms = round(\n        as_duration_seconds(max_transaction_duration, \"max_transaction_duration\") * 1000\n    )\n    poll_interval_ms = round(as_duration_seconds(poll_interval, \"poll_interval\") * 1000)\n    if read_batch_size <= 0:\n        raise ValueError(\"read_batch_size must be positive\")","sourceCodeStart":294,"sourceCodeEnd":330,"githubUrl":"https://github.com/pathwaycom/pathway/blob/fa2f74a4649b7c5908690cf60137263d8d80de5f/python/pathway/io/elasticsearch/__init__.py#L294-L330","documentation":"Raised by pw.io.elasticsearch.read when the timestamp_column argument does not name a column of the provided schema. The connector uses that column to do incremental reads (polling for documents newer than the last seen timestamp), so it must exist in schema.","triggerScenarios":"pw.io.elasticsearch.read(..., schema=LogSchema, timestamp_column=\"ts\") where LogSchema has no 'ts' field — e.g. a renamed field, a typo, or the schema was written for a different index mapping.","commonSituations":"Index mappings evolve and the timestamp field is renamed (e.g. '@timestamp' vs 'ts'); schema class copy-pasted from another connector; casing mismatches ('Time' vs 'time').","solutions":["Set timestamp_column to an existing schema column: pw.io.elasticsearch.read(..., schema=LogSchema, timestamp_column=\"@timestamp\") and add/fix that field in the schema class.","Verify with LogSchema.column_names() that the name matches exactly, including case.","If the index lacks a timestamp field, add one to the schema corresponding to the actual ES mapping field used for sequencing."],"exampleFix":"# before\nclass LogSchema(pw.Schema):\n    @timestamp: int\n    level: str\npw.io.elasticsearch.read(..., schema=LogSchema, timestamp_column=\"ts\")\n\n# after\nclass LogSchema(pw.Schema):\n    @timestamp: int\n    level: str\npw.io.elasticsearch.read(..., schema=LogSchema, timestamp_column=\"@timestamp\")","handlingStrategy":"validation","validationCode":"column_names = schema.column_names()\nassert timestamp_column in column_names, (\n    f\"timestamp_column {timestamp_column!r} not in schema columns {column_names}\"\n)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Derive timestamp_column/id_column from the schema class itself (e.g. a constant next to the schema) instead of free strings.","Add a smoke test that constructs the schema and checks both column names before any ES call."],"tags":["elasticsearch","schema","connector","api-misuse"],"backgroundTag":null,"analyzedSha":"fa2f74a4649b7c5908690cf60137263d8d80de5f","analyzedAt":"2026-08-15T01:48:17.006Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}