apache/beam · error · ValueError

Bad timestamp value: %s

Error message

Bad timestamp value: %s

What it means

When parsing a Pub/Sub message attribute (timestamp_attribute), the evaluator first tries to interpret the value as milliseconds since epoch, then as RFC 3339. If both parses fail, it re-raises as ValueError('Bad timestamp value: ...') wrapping the original error.

Source

Thrown at sdks/python/apache_beam/runners/direct/transform_evaluator.py:726

  def process_element(self, element):
    pass

  def _read_from_pubsub(
      self, timestamp_attribute) -> list[tuple[Timestamp, 'PubsubMessage']]:
    from apache_beam.io.gcp.pubsub import PubsubMessage

    def _get_element(message):
      parsed_message = PubsubMessage._from_message(message)
      if (timestamp_attribute and
          timestamp_attribute in parsed_message.attributes):
        rfc3339_or_milli = parsed_message.attributes[timestamp_attribute]
        try:
          timestamp = Timestamp(micros=int(rfc3339_or_milli) * 1000)
        except ValueError:
          try:
            timestamp = Timestamp.from_rfc3339(rfc3339_or_milli)
          except ValueError as e:
            raise ValueError('Bad timestamp value: %s' % e)
        if timestamp.precision() > Timestamp.MICROS_PRECISION:
          # Element timestamps are limited to microsecond resolution, so
          # ignore sub-microsecond digits, as the Dataflow service does.
          timestamp = timestamp.to_precision(
              Timestamp.MICROS_PRECISION, allow_lossy_conversion=True)
      else:
        if message.publish_time is None:
          raise ValueError('No publish time present in message: %s' % message)
        try:
          timestamp = Timestamp.from_utc_datetime(message.publish_time)
        except ValueError as e:
          raise ValueError('Bad timestamp value for message %s: %s', message, e)

      return timestamp, parsed_message

    # Because of the AutoAck, we are not able to reread messages if this
    # evaluator fails with an exception before emitting a bundle. However,
    # the DirectRunner currently doesn't retry work items anyway, so the

View on GitHub (pinned to 12126d8942)

Solutions

1. Publish attributes as RFC 3339 UTC strings, e.g. '2024-01-01T12:00:00Z', or integer milliseconds since epoch.
2. Use Python: datetime.now(timezone.utc).isoformat() or similar when publishing.
3. Validate/scrub the attribute at publish time; drop or default invalid values.
4. Confirm the correct attribute name is passed to timestamp_attribute (a wrong attribute may surface unrelated data).

Example fix

// before
msg['event_time'] = '2024-01-01 12:00:00'
// after
msg['event_time'] = '2024-01-01T12:00:00Z'

import re
RFC3339 = re.compile(r'^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?Z$')
MILLIS = re.compile(r'^\d+$')
assert RFC3339.match(attr) or MILLIS.match(attr), f'bad timestamp_attribute value: {attr!r}'

def is_valid_event_time_attr(value: str) -> bool:
    import re
    return bool(re.fullmatch(r'\d+', value)) or _is_rfc3339(value)

try:
    pipeline.run()
except ValueError as e:
    if 'Bad timestamp value' in str(e):
        log_publisher_schema_violation(e)

When it happens

Trigger: A Pub/Sub message whose timestamp_attribute value is neither an integer/numeric millisecond string nor a valid RFC 3339 timestamp (e.g. 'yesterday', '2024/01/01', empty string) while reading with timestamp_attribute set.

Common situations: Publisher writes human-readable dates or localized formats into the attribute; attribute accidentally empty; timezone-omitted timestamps ('2024-01-01 12:00:00') that aren't RFC 3339.

Related errors

• No publish time present in message: %s
• Bad timestamp value for message %s: %s
• set_watermark expects a Timestamp as input
• Either data (%r) or attributes (%r) must be set.
• A pubsub message data field must not exceed 10MB

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c2519a61b86e8be2. Report an issue: GitHub.