apache/beam · error · ValueError

Bad timestamp value for message %s: %s

Error message

Bad timestamp value for message %s: %s

What it means

When converting the message's publish_time datetime via Timestamp.from_utc_datetime, a failure is re-raised as ValueError('Bad timestamp value for message %s: %s'). Note the raise uses %s formatting with a tuple-style arg list in the source, so the message string itself may not interpolate correctly, but the cause is an unparseable publish_time datetime.

Source

Thrown at sdks/python/apache_beam/runners/direct/transform_evaluator.py:738

        try:
          timestamp = Timestamp(micros=int(rfc3339_or_milli) * 1000)
        except ValueError:
          try:
            timestamp = Timestamp.from_rfc3339(rfc3339_or_milli)
          except ValueError as e:
            raise ValueError('Bad timestamp value: %s' % e)
        if timestamp.precision() > Timestamp.MICROS_PRECISION:
          # Element timestamps are limited to microsecond resolution, so
          # ignore sub-microsecond digits, as the Dataflow service does.
          timestamp = timestamp.to_precision(
              Timestamp.MICROS_PRECISION, allow_lossy_conversion=True)
      else:
        if message.publish_time is None:
          raise ValueError('No publish time present in message: %s' % message)
        try:
          timestamp = Timestamp.from_utc_datetime(message.publish_time)
        except ValueError as e:
          raise ValueError('Bad timestamp value for message %s: %s', message, e)

      return timestamp, parsed_message

    # Because of the AutoAck, we are not able to reread messages if this
    # evaluator fails with an exception before emitting a bundle. However,
    # the DirectRunner currently doesn't retry work items anyway, so the
    # pipeline would enter an inconsistent state on any error.
    sub_client = self._get_subscriber_client(self._applied_ptransform)
    response = sub_client.pull(
        subscription=self._sub_name, max_messages=10, timeout=30)
    results = [_get_element(rm.message) for rm in response.received_messages]
    ack_ids = [rm.ack_id for rm in response.received_messages]
    if ack_ids:
      sub_client.acknowledge(subscription=self._sub_name, ack_ids=ack_ids)

    return results

  def finish_bundle(self) -> TransformResult:

View on GitHub (pinned to 12126d8942)

Solutions

1. Set publish_time to a timezone-aware UTC datetime: datetime.now(timezone.utc).
2. Clamp datetime values to the range Timestamp supports (avoid year 1 / far-future values).
3. Use timestamp_attribute with a valid RFC 3339 string to bypass publish_time parsing.
4. Inspect the wrapped cause (e) to see exactly why from_utc_datetime rejected the value.

Example fix

// before
msg.publish_time = datetime(1000, 1, 1)  # naive, out of range
// after
msg.publish_time = datetime.now(timezone.utc)

from datetime import datetime, timezone
if msg.publish_time is not None:
    assert msg.publish_time.tzinfo is not None, 'publish_time must be timezone-aware UTC'

def is_valid_publish_time(pt) -> bool:
    from datetime import datetime, timezone
    return isinstance(pt, datetime) and pt.tzinfo is not None and datetime(1970, 1, 1, tzinfo=timezone.utc) <= pt

try:
    read_pubsub()
except ValueError as e:
    if 'Bad timestamp value for message' in str(e):
        quarantine_message_and_log(e)

When it happens

Trigger: A Pub/Sub message (read without timestamp_attribute) whose publish_time is set to a value Timestamp.from_utc_datetime rejects — e.g. a naive datetime, out-of-range year, or a non-datetime object placed in publish_time.

Common situations: Manual message construction with naive datetimes (no tzinfo) or datetime values outside Timestamp's representable range; mocks supplying wrong types for publish_time; data injected from other systems without normalization.

Related errors

• Bad timestamp value: %s
• No publish time present in message: %s
• set_watermark expects a Timestamp as input
• Either data (%r) or attributes (%r) must be set.
• A pubsub message data field must not exceed 10MB

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d13ce08b7a4e1cd0. Report an issue: GitHub.