apache/beam · error · SizeLimitExceededException

Pubsub message ordering key of length {orderingKeySize} exce

Error message

Pubsub message ordering key of length {orderingKeySize} exceeds maximum of {ORDERING_KEY_MAX_BYTE_SIZE} bytes. See https://cloud.google.com/pubsub/quotas#resource_limits

What it means

validatePubsubMessage checks the Pub/Sub ordering key quota (ORDERING_KEY_MAX_BYTE_SIZE); ordering keys longer than that in UTF-8 bytes throw SizeLimitExceededException. Pub/Sub rejects ordering keys larger than the service limit, so the DoFn fails fast client-side.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PreparePubsubWriteDoFn.java:80

  static int validatePubsubMessage(PubsubMessage message, int maxPublishBatchSize)
      throws SizeLimitExceededException {
    int payloadSize = message.getPayload().length;
    if (payloadSize > PUBSUB_MESSAGE_DATA_MAX_BYTES) {
      throw new SizeLimitExceededException(
          "Pubsub message data field of length "
              + payloadSize
              + " exceeds maximum of "
              + PUBSUB_MESSAGE_DATA_MAX_BYTES
              + " bytes. See https://cloud.google.com/pubsub/quotas#resource_limits");
    }
    int totalSize = payloadSize;

    @Nullable String orderingKey = message.getOrderingKey();
    if (orderingKey != null) {
      int orderingKeySize = orderingKey.getBytes(StandardCharsets.UTF_8).length;
      if (orderingKeySize > ORDERING_KEY_MAX_BYTE_SIZE) {
        throw new SizeLimitExceededException(
            "Pubsub message ordering key of length "
                + orderingKeySize
                + " exceeds maximum of "
                + ORDERING_KEY_MAX_BYTE_SIZE
                + " bytes. See https://cloud.google.com/pubsub/quotas#resource_limits");
      }
      totalSize += orderingKeySize;
    }

    final @Nullable Map<String, String> attributes = message.getAttributeMap();
    if (payloadSize == 0 && (attributes == null || attributes.isEmpty())) {
      throw new IllegalArgumentException(
          "Pubsub message must contain a non-empty payload or at least one attribute.");
    }

    if (attributes != null) {
      if (attributes.size() > PUBSUB_MESSAGE_MAX_ATTRIBUTES) {
        throw new SizeLimitExceededException(

View on GitHub (pinned to 12126d8942)

Solutions

1. Shorten the ordering key: hash or truncate it to a fixed short value (e.g. SHA-256 prefix) while keeping ordering semantics
2. Validate key length in producer code before building the message
3. Restrict keys to ASCII and a fixed max length
4. Keep high-cardinality info in attributes, not the ordering key

Example fix

// before
String key = userId + ":" + fullUrl;
message = new PubsubMessage(payload, attrs).withOrderingKey(key);
// after
String key = Hashing.sha256().hashString(userId, UTF_8).toString().substring(0, 16);
message = new PubsubMessage(payload, attrs).withOrderingKey(key);

java
if (orderingKey != null && orderingKey.getBytes(StandardCharsets.UTF_8).length > 1024) {
  throw new IllegalArgumentException("Ordering key too long: " + orderingKey);
}

java
try {
  validatePubsubMessage(msg, maxBatchSize);
} catch (SizeLimitExceededException e) {
  msg = new PubsubMessage(msg.getPayload(), msg.getAttributeMap())
      .withOrderingKey(shorten(msg.getOrderingKey()));
}

When it happens

Trigger: Writing a PubsubMessage with a non-null ordering key whose UTF-8 byte length exceeds ORDERING_KEY_MAX_BYTE_SIZE through PubsubIO write with ordering keys enabled.

Common situations: Using long entity IDs, full URLs, or concatenated multi-field keys as ordering keys; forgetting that non-ASCII characters inflate UTF-8 byte length.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• Pubsub message data field of length {payloadSize} exceeds ma
• Pubsub message contains {attributes.size()} attributes which
• Pubsub message attribute key '{key}' exceeds the maximum of
• Pubsub message attribute value for key '{key}' starting with
• Pubsub message of length {totalSize} exceeds maximum of {max

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2075591b8147e767. Report an issue: GitHub.