apache/druid · error · MSQException

BroadcastTablesTooLarge

BroadcastTablesTooLarge

Error message

BroadcastTablesTooLargeFault: broadcast tables exceed reserved memory %s

What it means

Raised when the total byte size of broadcast tables read into memory during a broadcast join exceeds the memory reserved for the broadcast join (BroadcastTablesTooLargeFault wrapped in MSQException). Broadcast joins materialize the entire right-hand tables on every worker, so Druid enforces a reserved-memory cap and aborts the query rather than risk heap exhaustion.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/querykit/BroadcastJoinSegmentMapFnProcessor.java:252

   * Reads up to one frame from each readable side channel, and uses them to incrementally build up joinable
   * broadcast tables.
   *
   * @param readableInputs all readable input channel numbers, including non-side-channels
   * @return whether side channels have been fully read
   */
  boolean buildBroadcastTablesIncrementally(final IntSet readableInputs)
  {
    final IntIterator inputChannelIterator = readableInputs.iterator();

    while (inputChannelIterator.hasNext()) {
      final int channelNumber = inputChannelIterator.nextInt();
      if (sideChannelNumbers.contains(channelNumber) && channels.get(channelNumber).canRead()) {
        final Frame frame = channels.get(channelNumber).readFrame();

        memoryUsed += frame.numBytes();

        if (memoryUsed > memoryReservedForBroadcastJoin) {
          throw new MSQException(
              new BroadcastTablesTooLargeFault(
                  memoryReservedForBroadcastJoin,
                  Optional.ofNullable(query)
                          .map(q -> q.context().getString(PlannerContext.CTX_SQL_JOIN_ALGORITHM))
                          .map(JoinAlgorithm::fromString)
                          .orElse(null)
              )
          );
        }

        addFrame(channelNumber, frame);
      }
    }

    for (int channelNumber : sideChannelNumbers) {
      if (!channels.get(channelNumber).isFinished()) {
        return false;
      }

View on GitHub (pinned to 9b90983fd2)

Solutions

  1. Reduce the size of the broadcast side of the join (filter rows/columns before joining).
  2. Force a sort-merge (partitioned) join instead: set "sqlJoinAlgorithm":"sortMerge" in the query context.
  3. Increase memory reserved for broadcast joins (druid.msq.memory.reservedForBroadcastJoin / stage memory tuning) or add more workers so per-worker load drops.
  4. Pre-materialize and downcast the small table, or filter the large table upstream.

Example fix

// before
SELECT * FROM big_events t JOIN dims d ON t.dim_id = d.id; // broadcast join
// after
SELECT * FROM big_events t
JOIN dims d ON t.dim_id = d.id
-- context: {"sqlJoinAlgorithm": "sortMerge"}
Defensive patterns

Strategy: try-catch

Validate before calling

// estimate broadcast size before query: sum of bytes of small-side tables
long broadcastBytes = estimateTableBytes(smallSideTables);
if (broadcastBytes > reservedBroadcastMemory) { /* switch to sortMerge */ }

Try / catch

try {
  result = runMsqQuery(query);
} catch (MSQException e) {
  if (e.getFault() instanceof BroadcastTablesTooLargeFault) {
    query.context().put("sqlJoinAlgorithm", "sortMerge");
    result = runMsqQuery(query); // retry with partitioned join
  } else throw e;
}

Prevention

When it happens

Trigger: Running a broadcast (hash) join in MSQ where the sum of broadcast frame sizes read in BroadcastTablesTooLargeFault.buildBroadcastTablesIncrementally exceeds memoryReservedForBroadcastJoin (from memory reserve config / cluster capacity).

Common situations: Joining large dimension tables that were assumed small; context join algorithm set to broadcast inadvertently (default for equi-joins with small tables); cluster with low memory reserve for broadcast joins.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/0410b28d9943936d. Report an issue: GitHub.