apache/shardingsphere · error · BatchTooBigException

Batch is too big: accumulated %d + incoming %d data bytes ex

Error message

Batch is too big: accumulated %d + incoming %d data bytes exceeds buffer size limit %d bytes

What it means

BatchTooBigException thrown by FirebirdBatchMessageCommandExecutor when accumulated batch bytes plus the incoming MESSAGE data exceed the batch buffer size negotiated at CREATE BATCH (from TAG_BUFFER_BYTES_SIZE, capped at MAX_BUFFER_SIZE, default DEFAULT_BUFFER_SIZE). This is Firebird's native 'batch is too big' condition surfaced by the proxy.

Source

Thrown at proxy/frontend/dialect/firebird/src/main/java/org/apache/shardingsphere/proxy/frontend/firebird/command/query/batch/FirebirdBatchMessageCommandExecutor.java:54

/**
 * Batch send message command executor for Firebird.
 */
@RequiredArgsConstructor
public final class FirebirdBatchMessageCommandExecutor implements CommandExecutor {
    
    private final FirebirdBatchMessageCommandPacket packet;
    
    private final ConnectionSession connectionSession;
    
    @Override
    public Collection<DatabasePacket> execute() throws SQLException {
        int connectionId = connectionSession.getConnectionId();
        FirebirdBatchStatement batchStatement = FirebirdBatchRegistry.getInstance().getBatchStatement(connectionId, packet.getStatementHandle());
        if (null == batchStatement) {
            throw new InvalidBatchHandleException(packet.getStatementHandle());
        }
        if (batchStatement.getAccumulatedSize() + packet.getDataLength() > batchStatement.getBufferSize()) {
            throw new BatchTooBigException(packet.getStatementHandle(), batchStatement.getAccumulatedSize(), packet.getDataLength(), batchStatement.getBufferSize());
        }
        for (List<Object> each : packet.readParameterValues(batchStatement.getColumnDescriptors())) {
            batchStatement.addParameterValues(each);
        }
        batchStatement.addSize(packet.getDataLength());
        return Collections.singleton(new FirebirdGenericResponsePacket());
    }
}

View on GitHub (pinned to e952770a21)

Solutions

Call executeBatch periodically (flush) so accumulated size stays below the buffer limit, then continue with a fresh batch.
Request a larger TAG_BUFFER_BYTES_SIZE at CREATE BATCH (up to the protocol maximum) if the default is too small.
Reduce per-row payload size (shorter strings, fewer columns) when batching huge rows.

Example fix

// before
for (Row r : millionRows) { batch.add(r); }
batch.execute(); // BatchTooBigException

// after
int i = 0;
for (Row r : millionRows) {
    batch.add(r);
    if (++i % 10_000 == 0) { batch.execute(); batch.clear(); }
}
batch.execute();

Defensive patterns

Strategy: validation

Validate before calling

// Flush before the buffer overflows
if (accumulatedBytes + rowBytes > negotiatedBufferSize) { sendExecuteBatch(stmtHandle); accumulatedBytes = 0; }

Try / catch

catch (SQLException e) {
    if (e.getMessage().contains("Batch is too big")) { sendExecuteBatch(stmtHandle); resendLastMessage(stmtHandle); } else throw e;
}

Prevention

Count accumulated bytes and flush at ~80% of the negotiated buffer size.
Request a larger TAG_BUFFER_BYTES_SIZE for bulk loads.

When it happens

Trigger: batchStatement.getAccumulatedSize() + packet.getDataLength() > batchStatement.getBufferSize() inside execute(); clients queueing more rows than the negotiated buffer holds before executing, or negotiating a small TAG_BUFFER_BYTES_SIZE.

Common situations: bulk loaders queuing millions of rows without periodic executeBatch; client requesting a tiny buffer size; rows with large VARCHAR payloads filling the buffer quickly.

Related errors

AI-assisted analysis of apache/shardingsphere@e952770a21 (2026-08-14). Data as JSON: /api/errors/e8033d2722d7ae12. Report an issue: GitHub.