{"record":{"id":"bad83398edaca02d","repo":"apache/druid","slug":"estimated-numshards-s-exceeds-integer-bounds","errorCode":null,"errorMessage":"Estimated numShards [%s] exceeds integer bounds.","messagePattern":"Estimated numShards \\[(.+?)\\] exceeds integer bounds\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java","lineNumber":1031,"sourceCode":"          } else {\n            // determine numShards based on maxRowsPerSegment and the cardinality\n            estimatedNumShards = Math.round(estimatedCardinality / maxRowsPerSegment);\n          }\n          LOG.info(\"estimatedNumShards %d given estimated cardinality %.2f and maxRowsPerSegment %d\",\n                    estimatedNumShards, estimatedCardinality, maxRowsPerSegment\n          );\n          // We have seen this before in the wild in situations where more shards should have been created,\n          // log it if it happens with some information & context\n          if (estimatedNumShards == 1) {\n            LOG.info(\"estimatedNumShards is ONE (%d) given estimated cardinality %.2f and maxRowsPerSegment %d\",\n                      estimatedNumShards, estimatedCardinality, maxRowsPerSegment\n            );\n          }\n          try {\n            return Math.max(Math.toIntExact(estimatedNumShards), 1);\n          }\n          catch (ArithmeticException ae) {\n            throw new ISE(\"Estimated numShards [%s] exceeds integer bounds.\", estimatedNumShards);\n          }\n        }\n    );\n  }\n\n  /**\n   * Creates a map from partition (interval + bucketId) to the corresponding\n   * PartitionLocations. Note that the bucketId maybe different from the final\n   * partitionId (refer to {@link BuildingShardSpec} for more details).\n   */\n  static Map<Partition, List<PartitionLocation>> getPartitionToLocations(\n      Map<String, GeneratedPartitionsReport> subTaskIdToReport\n  )\n  {\n    // Create a map from partition to list of reports (PartitionStat and subTaskId)\n    final Map<Partition, List<PartitionReport>> partitionToReports = new TreeMap<>(\n        // Sort by (interval, bucketId) to maintain order of partitionIds within interval\n        Comparator","sourceCodeStart":1013,"sourceCodeEnd":1049,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java#L1013-L1049","documentation":"Thrown when the estimated number of hash shards (numShards) computed during multi-phase parallel ingestion overflows Integer.MAX_VALUE. Math.toIntExact throws ArithmeticException, which is translated into this ISE so callers get a clear message instead of an arithmetic crash.","triggerScenarios":"Auto-estimated numShards (from row counts / rows per shard) exceeding ~2.1 billion — extremely large datasets with a very small target rows-per-shard, or a miscomputed estimate from skewed statistics.","commonSituations":"Setting an absurdly small targetPartitionSize/rowsPerSegment on a huge dataset; ingestion reports with inflated row counts due to duplicate sampling; bugs in the shard-count estimator fed by skewed sampling.","solutions":["Increase target rows per shard (targetPartitionSize/rowsPerSegment) so the estimate fits an int.","Set numShards explicitly in the hash partitionsSpec instead of letting it be estimated.","Review sampling/row-count statistics feeding the estimate for duplication or skew.","Split the ingestion into multiple tasks by time interval to reduce shard counts per task."],"exampleFix":"// before\n\"partitionsSpec\": { \"type\": \"hash\", \"maxNumPartitions\": 2147483647 }\n// after\n\"partitionsSpec\": { \"type\": \"hash\", \"numShards\": 64 }","handlingStrategy":"validation","validationCode":"long estimated = estimateShards(rowCount, rowsPerShard);\nif (estimated > Integer.MAX_VALUE) {\n  throw new IllegalArgumentException(\"estimated numShards too large; increase rowsPerShard or split the task by interval\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep target rows per shard in a sane range (millions)","Set numShards explicitly for very large datasets","Split ingestion by time interval rather than shrinking shard size","Sanity-check sampling statistics feeding auto-estimates"],"tags":["partitioning","integer-overflow","multi-phase-ingestion"],"backgroundTag":"value-out-of-range","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}