apache/seatunnel · error · AzureQueueConnectorException

MESSAGE_TOO_LARGE

MESSAGE_TOO_LARGE

Error message

Serialized message is ${encodedSize} bytes after ${messageEncoding} encoding, exceeding the Azure Queue limit of ${MAX_ENCODED_MESSAGE_BYTES} bytes

What it means

validateMessageSize() computes the encoded size of a serialized message (Base64 inflates by ~4/3) and throws AzureQueueConnectorException(MESSAGE_TOO_LARGE) when it exceeds MAX_ENCODED_MESSAGE_BYTES — Azure Queue Storage's 64KB message limit. Called from write() before sending, so oversized messages fail the write rather than the Azure API call.

Solutions

  1. Reduce record size upstream (truncate/split/aggregate large fields)
  2. Switch message_encoding from BASE64 to NONE if possible to avoid 33% overhead
  3. Pre-filter oversized records before the sink with a transform
  4. Compress payload if the consumer supports it

Example fix

// before
message_encoding = BASE64  // 48KB payload -> 64KB+ encoded
// after
message_encoding = NONE    // or reduce payload size
Defensive patterns

Strategy: validation

Validate before calling

int encoded = encoding == MessageEncoding.BASE64
    ? 4 * ((payload.length + 2) / 3)
    : payload.length;
if (encoded > 65536) throw new IllegalArgumentException("message exceeds Azure 64KB limit");

Try / catch

try {
    writer.write(record);
} catch (AzureQueueConnectorException e) {
    if (e.getErrorCode() == AzureQueueConnectorErrorCode.MESSAGE_TOO_LARGE) {
        // route to dead-letter / split the record
    }
}

Prevention

When it happens

Trigger: write() produces a payload whose raw size, or Base64-encoded size (4 * ceil(payloadSize/3)), exceeds the Azure Queue message limit.

Common situations: Large rows/JSON documents from the upstream source, enabling BASE64 message encoding which adds ~33% overhead, or wide tables with big blob columns.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/ca16172835b7c9a9. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-connectors-v2/connector-azure-queue-storage/src/main/java/org/apache/seatunnel/connectors/seatunnel/azure/queue/sink/AzureQueueStorageSinkWriter.java:174

                                "Failed to close Azure Queue Storage sender",
                                e);
            } else {
                failure.addSuppressed(e);
            }
        }

        if (failure != null) {
            throw failure;
        }
    }

    private void validateMessageSize(int payloadSize) {
        long encodedSize = payloadSize;
        if (messageEncoding == MessageEncoding.BASE64) {
            encodedSize = 4L * ((payloadSize + 2L) / 3L);
        }
        if (encodedSize > MAX_ENCODED_MESSAGE_BYTES) {
            throw new AzureQueueConnectorException(
                    AzureQueueConnectorErrorCode.MESSAGE_TOO_LARGE,
                    "Serialized message is "
                            + encodedSize
                            + " bytes after "
                            + messageEncoding.name().toLowerCase(Locale.ROOT)
                            + " encoding, exceeding the Azure Queue limit of "
                            + MAX_ENCODED_MESSAGE_BYTES
                            + " bytes");
        }
    }

    private void checkSendError() {
        Throwable failure = sendError.get();
        if (failure != null) {
            throw writeFailure(failure);
        }
    }

View on GitHub (pinned to cf67b549a7)