{"record":{"id":"8b19d07d008bdebb","repo":"apache/druid","slug":"query-s-url-s-total-bytes-gathered-d-exceed","errorCode":null,"errorMessage":"Query[%s] url[%s] total bytes gathered[%,d] exceeds maxScatterGatherBytes[%,d]","messagePattern":"Query\\[(.+?)\\] url\\[(.+?)\\] total bytes gathered\\[%,d\\] exceeds maxScatterGatherBytes\\[%,d\\]","errorType":"exception","errorClass":"ResourceLimitExceededException","httpStatus":null,"severity":"error","filePath":"server/src/main/java/org/apache/druid/client/DirectDruidClient.java","lineNumber":500,"sourceCode":"            throw new QueryTimeoutException(msg);\n          } else {\n            return timeLeft;\n          }\n        }\n\n        private void checkTotalBytesLimit(long bytes)\n        {\n          final long currentTotalBytesGathered = totalBytesGathered.addAndGet(bytes);\n          if (currentTotalBytesGathered > maxScatterGatherBytes) {\n            String msg = StringUtils.format(\n                \"Query[%s] url[%s] total bytes gathered[%,d] exceeds maxScatterGatherBytes[%,d]\",\n                query.getId(),\n                url,\n                currentTotalBytesGathered,\n                maxScatterGatherBytes\n            );\n            setupResponseReadFailure(msg, null);\n            throw new ResourceLimitExceededException(msg);\n          }\n        }\n      };\n\n      long timeLeft = timeoutAt - System.currentTimeMillis();\n\n      if (timeLeft <= 0) {\n        throw new QueryTimeoutException(StringUtils.nonStrictFormat(\"Query[%s] url[%s] timed out.\", query.getId(), url));\n      }\n\n      // increment is moved up so that if future initialization is queued by some other process,\n      // we can increment the count earlier so that we can route the request to a different server\n      openConnections.getAndIncrement();\n      try {\n        future = httpClient.go(\n            new Request(\n                HttpMethod.POST,\n                new URL(url)","sourceCodeStart":482,"sourceCodeEnd":518,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/server/src/main/java/org/apache/druid/client/DirectDruidClient.java#L482-L518","documentation":"Thrown by checkTotalBytesLimit() in DirectDruidClient when the cumulative bytes gathered across all data nodes for a query exceed the maxScatterGatherBytes limit (from query context, bounded by druid.server.http.maxScatterGatherBytes). It is a ResourceLimitExceededException protecting the broker from unbounded memory use in scatter-gather.","triggerScenarios":"Running a query whose total response bytes from all historicals exceed maxScatterGatherBytes: unbounded scans/selects, huge group-by cardinalities, or select-all column queries over large intervals.","commonSituations":"Exploratory 'SELECT *' style queries over large time ranges; per-query limit left at a low default (1GiB) while users run large exports; limit set in cluster config too small for reporting workloads.","solutions":["Raise maxScatterGatherBytes in the query context (bounded by the broker's druid.server.http.maxScatterGatherBytes) or increase the broker config.","Narrow the query: reduce intervals, add filters, project only needed columns, or page through results.","Export results incrementally rather than materializing everything in one query.","If the limit was hit unexpectedly, check for runaway queries (e.g. missing time filter)."],"exampleFix":"// before\n// context: {} (uses low default maxScatterGatherBytes)\n// after\n// context: {\"maxScatterGatherBytes\": 1073741824}","handlingStrategy":"try-catch","validationCode":"// estimate response size before issuing\nlong estBytes = estimateQueryResponseBytes(query); // columns * rows * avg width\nlong limit = query.context().get(\"maxScatterGatherBytes\");\nif (estBytes > limit) { narrowQuery(query); }","typeGuard":null,"tryCatchPattern":"try {\n  return client.run(query, ctx).toList();\n} catch (ResourceLimitExceededException e) {\n  // split query into smaller intervals or raise maxScatterGatherBytes\n  throw e;\n}","preventionTips":["Always include time filters; avoid unbounded scans.","Project only the columns needed.","Set maxScatterGatherBytes consciously per workload.","Break very large queries into interval chunks."],"tags":["druid","broker","resource-limit","scatter-gather"],"backgroundTag":"payload-too-large","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}