apache/druid · error · IllegalStateException

Estimated numShards [%s] exceeds integer bounds.

Error message

Estimated numShards [%s] exceeds integer bounds.

What it means

Thrown when the estimated number of hash shards (numShards) computed during multi-phase parallel ingestion overflows Integer.MAX_VALUE. Math.toIntExact throws ArithmeticException, which is translated into this ISE so callers get a clear message instead of an arithmetic crash.

Source

Thrown at indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java:1031

          } else {
            // determine numShards based on maxRowsPerSegment and the cardinality
            estimatedNumShards = Math.round(estimatedCardinality / maxRowsPerSegment);
          }
          LOG.info("estimatedNumShards %d given estimated cardinality %.2f and maxRowsPerSegment %d",
                    estimatedNumShards, estimatedCardinality, maxRowsPerSegment
          );
          // We have seen this before in the wild in situations where more shards should have been created,
          // log it if it happens with some information & context
          if (estimatedNumShards == 1) {
            LOG.info("estimatedNumShards is ONE (%d) given estimated cardinality %.2f and maxRowsPerSegment %d",
                      estimatedNumShards, estimatedCardinality, maxRowsPerSegment
            );
          }
          try {
            return Math.max(Math.toIntExact(estimatedNumShards), 1);
          }
          catch (ArithmeticException ae) {
            throw new ISE("Estimated numShards [%s] exceeds integer bounds.", estimatedNumShards);
          }
        }
    );
  }

  /**
   * Creates a map from partition (interval + bucketId) to the corresponding
   * PartitionLocations. Note that the bucketId maybe different from the final
   * partitionId (refer to {@link BuildingShardSpec} for more details).
   */
  static Map<Partition, List<PartitionLocation>> getPartitionToLocations(
      Map<String, GeneratedPartitionsReport> subTaskIdToReport
  )
  {
    // Create a map from partition to list of reports (PartitionStat and subTaskId)
    final Map<Partition, List<PartitionReport>> partitionToReports = new TreeMap<>(
        // Sort by (interval, bucketId) to maintain order of partitionIds within interval
        Comparator

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Increase target rows per shard (targetPartitionSize/rowsPerSegment) so the estimate fits an int.
2. Set numShards explicitly in the hash partitionsSpec instead of letting it be estimated.
3. Review sampling/row-count statistics feeding the estimate for duplication or skew.
4. Split the ingestion into multiple tasks by time interval to reduce shard counts per task.

Example fix

// before
"partitionsSpec": { "type": "hash", "maxNumPartitions": 2147483647 }
// after
"partitionsSpec": { "type": "hash", "numShards": 64 }

long estimated = estimateShards(rowCount, rowsPerShard);
if (estimated > Integer.MAX_VALUE) {
  throw new IllegalArgumentException("estimated numShards too large; increase rowsPerShard or split the task by interval");
}

When it happens

Trigger: Auto-estimated numShards (from row counts / rows per shard) exceeding ~2.1 billion — extremely large datasets with a very small target rows-per-shard, or a miscomputed estimate from skewed statistics.

Common situations: Setting an absurdly small targetPartitionSize/rowsPerSegment on a huge dataset; ingestion reports with inflated row counts due to duplicate sampling; bugs in the shard-count estimator fed by skewed sampling.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• forceGuaranteedRollup is set but partitionsSpec [%s] is not
• %s is too large
• Root partition range[%d, %d] of new segments doesn't match t
• forceGuaranteedRollup is incompatible with partitionsSpec: %
• partitionDimensions must be specified

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/bad83398edaca02d. Report an issue: GitHub.