{"record":{"id":"96944b375eea27eb","repo":"apache/beam","slug":"dropping-output-field-before-writing-to-pubsub-because-this","errorCode":null,"errorMessage":"Dropping output field '{}' before writing to PubSub because this is a read-only column. To preserve this information you must configure a timestamp attribute.","messagePattern":"Dropping output field '(.+?)' before writing to PubSub because this is a read-only column\\. To preserve this information you must configure a timestamp attribute\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/AddTimestampAttribute.java","lineNumber":64,"sourceCode":"    // element's event time. PubSubIO will populate the attribute from there.\n    PCollection<Row> withTimestamp =\n        useTimestampAttribute\n            ? input.apply(\n                WithTimestamps.of(\n                    (row) ->\n                        checkArgumentNotNull(\n                                row.getDateTime(TIMESTAMP_FIELD),\n                                \"Field '%s' must be present and non-null in the input row when writing to PubSub with a timestamp attribute.\",\n                                TIMESTAMP_FIELD)\n                            .toInstant()))\n            : input;\n\n    PCollection<Row> rows;\n    if (withTimestamp.getSchema().hasField(TIMESTAMP_FIELD)) {\n      if (!useTimestampAttribute) {\n        // Warn the user if they're writing data to TIMESTAMP_FIELD, but event timestamp is mapped\n        // to publish time. The data will be dropped.\n        LOG.warn(\n            \"Dropping output field '{}' before writing to PubSub because this is a read-only \"\n                + \"column. To preserve this information you must configure a timestamp attribute.\",\n            TIMESTAMP_FIELD);\n      }\n      rows = withTimestamp.apply(DropFields.fields(TIMESTAMP_FIELD));\n    } else {\n      rows = withTimestamp;\n    }\n\n    return rows;\n  }\n}\n","sourceCodeStart":46,"sourceCodeEnd":77,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/AddTimestampAttribute.java#L46-L77","documentation":"In AddTimestampAttribute.expand, the output Row schema contains a TIMESTAMP_FIELD column, but the Pub/Sub sink was not configured with a timestamp attribute (useTimestampAttribute is false). Since Pub/Sub event timestamps map to publish time in that configuration, the read-only TIMESTAMP_FIELD column would be misleading, so it is dropped and a warning is logged. Data in that column is silently removed from the written output.","triggerScenarios":"Writing a PCollection<Row> with a field named TIMESTAMP_FIELD (event_timestamp) to Pub/Sub via PubsubIO.write() without calling .withTimestampAttribute(); the expand() path drops the field before serialization.","commonSituations":"Users migrating from BigQuery sinks (where TIMESTAMP_FIELD is a real column) to Pub/Sub; SQL pipelines whose output schema includes event_timestamp but whose write config lacks withTimestampAttribute; copy-pasted sink configs that omit the attribute mapping.","solutions":["Call .withTimestampAttribute(\"<attribute-name>\") on the Pub/Sub write so the event timestamp is preserved as a message attribute.","Remove TIMESTAMP_FIELD from the output schema if publish-time semantics are acceptable.","Rename the field if you intend it as ordinary data and want it kept (only TIMESTAMP_FIELD is dropped).","Review logs for this warning in CI to catch misconfigured Pub/Sub writes before production."],"exampleFix":"// before\nPubsubIO.writeRows().to(topic);\n// after\nPubsubIO.writeRows().to(topic).withTimestampAttribute(\"event_timestamp_attr\");","handlingStrategy":"validation","validationCode":"// before writing, check schema/config consistency\nboolean ok = !withTimestamp.getSchema().hasField(\"event_timestamp\")\n          || writeTransform instanceof HasTimestampAttribute;","typeGuard":"boolean preservesTimestamp(PubsubIO.Write<Row> w) { return w != null && w.hasTimestampAttribute(); }","tryCatchPattern":null,"preventionTips":["Always pair a TIMESTAMP_FIELD column in the output schema with .withTimestampAttribute().","Assert schema/write-config consistency in pipeline unit tests.","Grep pipelines for TIMESTAMP_FIELD usages and audit each sink.","Treat this warning in CI logs as a build failure signal."],"tags":["java","pubsub","schema","data-loss","timestamp"],"backgroundTag":"conflicting-config-options","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-15T02:17:10.978Z"}