apache/beam · error · SizeLimitExceededException

Pubsub message data field of length {payloadSize} exceeds ma

Error message

Pubsub message data field of length {payloadSize} exceeds maximum of {PUBSUB_MESSAGE_DATA_MAX_BYTES} bytes. See https://cloud.google.com/pubsub/quotas#resource_limits

What it means

PreparePubsubWriteDoFn.validatePubsubMessage enforces the Cloud Pub/Sub quota that the message data field can be at most PUBSUB_MESSAGE_DATA_MAX_BYTES (10MB). It throws SizeLimitExceededException when the serialized payload exceeds this limit before publishing.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PreparePubsubWriteDoFn.java:67

  private int maxPublishBatchSize;
  private boolean logOrderingKeyUnconfigured = false;
  private SerializableFunction<ValueInSingleWindow<InputT>, PubsubMessage> formatFunction;
  @Nullable SerializableFunction<ValueInSingleWindow<InputT>, PubsubIO.PubsubTopic> topicFunction;

  /** Last TopicPath that reported Lineage. */
  private transient @Nullable String reportedLineage;

  private final BadRecordRouter badRecordRouter;

  private final Coder<InputT> inputCoder;

  private final TupleTag<PubsubMessage> outputTag;

  static int validatePubsubMessage(PubsubMessage message, int maxPublishBatchSize)
      throws SizeLimitExceededException {
    int payloadSize = message.getPayload().length;
    if (payloadSize > PUBSUB_MESSAGE_DATA_MAX_BYTES) {
      throw new SizeLimitExceededException(
          "Pubsub message data field of length "
              + payloadSize
              + " exceeds maximum of "
              + PUBSUB_MESSAGE_DATA_MAX_BYTES
              + " bytes. See https://cloud.google.com/pubsub/quotas#resource_limits");
    }
    int totalSize = payloadSize;

    @Nullable String orderingKey = message.getOrderingKey();
    if (orderingKey != null) {
      int orderingKeySize = orderingKey.getBytes(StandardCharsets.UTF_8).length;
      if (orderingKeySize > ORDERING_KEY_MAX_BYTE_SIZE) {
        throw new SizeLimitExceededException(
            "Pubsub message ordering key of length "
                + orderingKeySize
                + " exceeds maximum of "
                + ORDERING_KEY_MAX_BYTE_SIZE
                + " bytes. See https://cloud.google.com/pubsub/quotas#resource_limits");

View on GitHub (pinned to 12126d8942)

Solutions

1. Reduce payload size: compress (gzip) before publishing or split into chunks
2. Move large blobs to GCS/Cloud Storage and publish a reference URL instead
3. Increase aggregation sizing downstream and send multiple smaller messages
4. Validate payload lengths at source before running the pipeline

Example fix

// before
PubsubMessage msg = new PubsubMessage(fileBytes, attrs); // 12 MB
// after
byte[] compressed = gzip(fileBytes);
if (compressed.length > PUBSUB_MESSAGE_DATA_MAX_BYTES) {
  String gcsRef = uploadToGcs(fileBytes);
  msg = new PubsubMessage(gcsRef.getBytes(UTF_8), attrs);
}

java
if (message.getPayload() != null && message.getPayload().length > 10 * 1024 * 1024) {
  throw new IllegalArgumentException("Payload exceeds Pub/Sub 10MB data limit: " + message.getPayload().length);
}

java
try {
  validatePubsubMessage(msg, maxBatchSize);
} catch (SizeLimitExceededException e) {
  log.warn("Oversized pubsub message, offloading to GCS: {}", e.getMessage());
  msg = new PubsubMessage(uploadToGcs(msg.getPayload()), msg.getAttributeMap());
}

When it happens

Trigger: Publishing a PubsubMessage via PubsubIO write pipeline (PreparePubsubWriteDoFn.process) whose getPayload().length exceeds the Pub/Sub data limit — e.g. writing a large Avro/JSON record or file blob as a single message.

Common situations: Streaming whole files or images as one message; encoding large records without compression; misreading the 10MB quota as applying per-batch rather than per-message.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• Pubsub message ordering key of length {orderingKeySize} exce
• Pubsub message contains {attributes.size()} attributes which
• Pubsub message attribute key '{key}' exceeds the maximum of
• Pubsub message attribute value for key '{key}' starting with
• Pubsub message of length {totalSize} exceeds maximum of {max

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7913860a5eab0592. Report an issue: GitHub.