{"record":{"id":"7913860a5eab0592","repo":"apache/beam","slug":"pubsub-message-data-field-of-length-payloadsize-exceeds","errorCode":null,"errorMessage":"Pubsub message data field of length {payloadSize} exceeds maximum of {PUBSUB_MESSAGE_DATA_MAX_BYTES} bytes. See https://cloud.google.com/pubsub/quotas#resource_limits","messagePattern":"Pubsub message data field of length (.+?) exceeds maximum of (.+?) bytes\\. See https://cloud\\.google\\.com/pubsub/quotas#resource_limits","errorType":"validation","errorClass":"SizeLimitExceededException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PreparePubsubWriteDoFn.java","lineNumber":67,"sourceCode":"  private int maxPublishBatchSize;\n  private boolean logOrderingKeyUnconfigured = false;\n  private SerializableFunction<ValueInSingleWindow<InputT>, PubsubMessage> formatFunction;\n  @Nullable SerializableFunction<ValueInSingleWindow<InputT>, PubsubIO.PubsubTopic> topicFunction;\n\n  /** Last TopicPath that reported Lineage. */\n  private transient @Nullable String reportedLineage;\n\n  private final BadRecordRouter badRecordRouter;\n\n  private final Coder<InputT> inputCoder;\n\n  private final TupleTag<PubsubMessage> outputTag;\n\n  static int validatePubsubMessage(PubsubMessage message, int maxPublishBatchSize)\n      throws SizeLimitExceededException {\n    int payloadSize = message.getPayload().length;\n    if (payloadSize > PUBSUB_MESSAGE_DATA_MAX_BYTES) {\n      throw new SizeLimitExceededException(\n          \"Pubsub message data field of length \"\n              + payloadSize\n              + \" exceeds maximum of \"\n              + PUBSUB_MESSAGE_DATA_MAX_BYTES\n              + \" bytes. See https://cloud.google.com/pubsub/quotas#resource_limits\");\n    }\n    int totalSize = payloadSize;\n\n    @Nullable String orderingKey = message.getOrderingKey();\n    if (orderingKey != null) {\n      int orderingKeySize = orderingKey.getBytes(StandardCharsets.UTF_8).length;\n      if (orderingKeySize > ORDERING_KEY_MAX_BYTE_SIZE) {\n        throw new SizeLimitExceededException(\n            \"Pubsub message ordering key of length \"\n                + orderingKeySize\n                + \" exceeds maximum of \"\n                + ORDERING_KEY_MAX_BYTE_SIZE\n                + \" bytes. See https://cloud.google.com/pubsub/quotas#resource_limits\");","sourceCodeStart":49,"sourceCodeEnd":85,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PreparePubsubWriteDoFn.java#L49-L85","documentation":"PreparePubsubWriteDoFn.validatePubsubMessage enforces the Cloud Pub/Sub quota that the message data field can be at most PUBSUB_MESSAGE_DATA_MAX_BYTES (10MB). It throws SizeLimitExceededException when the serialized payload exceeds this limit before publishing.","triggerScenarios":"Publishing a PubsubMessage via PubsubIO write pipeline (PreparePubsubWriteDoFn.process) whose getPayload().length exceeds the Pub/Sub data limit — e.g. writing a large Avro/JSON record or file blob as a single message.","commonSituations":"Streaming whole files or images as one message; encoding large records without compression; misreading the 10MB quota as applying per-batch rather than per-message.","solutions":["Reduce payload size: compress (gzip) before publishing or split into chunks","Move large blobs to GCS/Cloud Storage and publish a reference URL instead","Increase aggregation sizing downstream and send multiple smaller messages","Validate payload lengths at source before running the pipeline"],"exampleFix":"// before\nPubsubMessage msg = new PubsubMessage(fileBytes, attrs); // 12 MB\n// after\nbyte[] compressed = gzip(fileBytes);\nif (compressed.length > PUBSUB_MESSAGE_DATA_MAX_BYTES) {\n  String gcsRef = uploadToGcs(fileBytes);\n  msg = new PubsubMessage(gcsRef.getBytes(UTF_8), attrs);\n}","handlingStrategy":"validation","validationCode":"java\nif (message.getPayload() != null && message.getPayload().length > 10 * 1024 * 1024) {\n  throw new IllegalArgumentException(\"Payload exceeds Pub/Sub 10MB data limit: \" + message.getPayload().length);\n}","typeGuard":null,"tryCatchPattern":"java\ntry {\n  validatePubsubMessage(msg, maxBatchSize);\n} catch (SizeLimitExceededException e) {\n  log.warn(\"Oversized pubsub message, offloading to GCS: {}\", e.getMessage());\n  msg = new PubsubMessage(uploadToGcs(msg.getPayload()), msg.getAttributeMap());\n}","preventionTips":["Check payload byte length before constructing PubsubMessage","Compress or chunk large payloads","Use GCS references for blobs instead of inline data","Remember the limit is per-message, not per-batch"],"tags":["java","pubsub","payload-size","quota"],"backgroundTag":"payload-too-large","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-15T02:17:10.978Z"}