apache/beam · error · SizeLimitExceededException

Pubsub message of length {totalSize} exceeds maximum of {max

Error message

Pubsub message of length {totalSize} exceeds maximum of {maxPublishBatchSize} bytes, when considering the payload and attributes. See https://cloud.google.com/pubsub/quotas#resource_limits

What it means

Before writing, PreparePubsubWriteDoFn sums the payload plus all attribute keys and values and rejects messages whose total size exceeds maxPublishBatchSize, enforcing the Pub/Sub per-message request size quota (10MB by default). This fails fast client-side instead of getting a rejected publish from the service.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PreparePubsubWriteDoFn.java:139

        String value = attribute.getValue();
        int valueSize = value.getBytes(StandardCharsets.UTF_8).length;
        if (valueSize > PUBSUB_MESSAGE_ATTRIBUTE_MAX_VALUE_BYTES) {
          throw new SizeLimitExceededException(
              "Pubsub message attribute value for key '"
                  + key
                  + "' starting with '"
                  + value.substring(0, Math.min(256, value.length()))
                  + "' exceeds the maximum of "
                  + PUBSUB_MESSAGE_ATTRIBUTE_MAX_VALUE_BYTES
                  + " bytes. See https://cloud.google.com/pubsub/quotas#resource_limits");
        }
        totalSize += valueSize;
      }
    }

    if (totalSize > maxPublishBatchSize) {
      throw new SizeLimitExceededException(
          "Pubsub message of length "
              + totalSize
              + " exceeds maximum of "
              + maxPublishBatchSize
              + " bytes, when considering the payload and attributes. "
              + "See https://cloud.google.com/pubsub/quotas#resource_limits");
    }
    return totalSize;
  }

  PreparePubsubWriteDoFn(
      SerializableFunction<ValueInSingleWindow<InputT>, PubsubMessage> formatFunction,
      @Nullable SerializableFunction<ValueInSingleWindow<InputT>, PubsubIO.PubsubTopic>
          topicFunction,
      boolean usesOrderingKey,
      int maxPublishBatchSize,
      BadRecordRouter badRecordRouter,
      Coder<InputT> inputCoder,

View on GitHub (pinned to 12126d8942)

Solutions

1. Reduce the message payload size (compress, chunk, or write large data to GCS and send a reference)
2. Trim attribute keys and values to the minimum needed
3. Enable Pub/Sub topic message storage / use a dead-letter style pattern with a reference pointer
4. Increase batching configuration only if within actual service quota, or raise maxPublishBatchSize explicitly to match the topic quota

Example fix

// before
PCollection<String> big = ...; // 15MB strings
big.apply(PubsubIO.writeStrings().to(topic));
// after
PCollection<String> refs = big.apply("toGcs", new WriteRefsToGcs());
refs.apply(PubsubIO.writeStrings().to(topic));

long total = payload.length + attributes.keySet().stream().mapToLong(k -> k.getBytes(StandardCharsets.UTF_8).length).sum()
    + attributes.values().stream().mapToLong(v -> v.getBytes(StandardCharsets.UTF_8).length).sum();
if (total > 10_000_000) throw new IllegalArgumentException("Message total " + total + " bytes exceeds Pub/Sub limit");

try { pipeline.run(); } catch (SizeLimitExceededException e) { log.error("Pub/Sub message too large: {}", e.getMessage()); }

When it happens

Trigger: process() calls validatePubsubMessage(); computed totalSize (payload + attribute keys + attribute values) exceeds maxPublishBatchSize when writing to Pub/Sub via PubsubIO.

Common situations: Publishing very large files or serialized objects as a single message; unbounded accumulation of attributes; misconfigured or reduced maxPublishBatchSize while sending large batches.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• Pubsub message attribute value for key '{key}' starting with
• Pubsub message data field of length {payloadSize} exceeds ma
• Pubsub message ordering key of length {orderingKeySize} exce
• Pubsub message contains {attributes.size()} attributes which
• Pubsub message attribute key '{key}' exceeds the maximum of

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0b5d9e006b3e3a2d. Report an issue: GitHub.