apache/druid · error · TooManyInputFilesFault

TooManyInputFiles

TooManyInputFiles

Error message

Too many input files/segments [%d] encountered. Maximum input files/segments per worker is set to [%d]. Try increasing the limit using the %s query context parameter, breaking your query up into smaller queries, or increasing the number of workers to at least [%d] by setting %s in your query context.

What it means

During createWorkOrders, the controller sums input files/segments per worker and throws an MSQException wrapping a TooManyInputFilesFault when a worker would exceed maxInputFilesPerWorker. The fault reports total files, the configured limit, and the minimum number of workers required. It protects workers from being assigned unmanageable numbers of splits.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernel.java:332

          queryDef,
          stageNumber,
          workerNumber,
          workerInputs.inputsForWorker(workerNumber),
          extraInfoHolder,
          config.getWorkerIds(),
          outputChannelMode,
          config.getWorkerContextMap()
      );

      final int numInputFiles = Ints.checkedCast(workOrder.getInputs().stream().mapToLong(InputSlice::fileCount).sum());
      fault = fault || IntMath.divide(numInputFiles, maxInputFilesPerWorker, RoundingMode.CEILING) > 1;
      totalFileCount += numInputFiles;
      workerToWorkOrder.put(workerNumber, workOrder);
    }

    final int requiredWorkers = IntMath.divide(totalFileCount, maxInputFilesPerWorker, RoundingMode.CEILING);
    if (fault) {
      throw new MSQException(new TooManyInputFilesFault(totalFileCount, maxInputFilesPerWorker, requiredWorkers));
    }
    stageWorkOrders.put(new StageId(queryDef.getQueryId(), stageNumber), workerToWorkOrder);
    return workerToWorkOrder;
  }

  private void createNewKernels(
      final InputSpecSlicerFactory slicerFactory,
      final WorkerAssignmentStrategy assignmentStrategy,
      final FrameType rowBasedFrameType,
      final int maxInputFilesPerWorker,
      final long maxInputBytesPerWorker,
      final int maxPartitions
  )
  {
    StageGroup stageGroup;

    while ((stageGroup = stageGroupQueue.peek()) != null) {
      if (readyToRunStages.contains(stageGroup.first())

View on GitHub (pinned to 9b90983fd2)

Solutions

Increase the number of workers (e.g. via the numTasks / task context parameter or cluster autoscaling) as the fault message suggests
Raise maxInputFilesPerWorker in the query context (accepting larger per-worker memory/CPU load)
Split the query into smaller queries covering fewer segments each

Example fix

// before
{"query": "..."}
// after
{"query": "...", "context": {"maxInputFilesPerWorker": 1000, "numTasks": 4}}

Defensive patterns

Strategy: try-catch

Validate before calling

long totalFiles = inputs.stream().mapToLong(InputSpec::getNumFiles).sum();
int workers = context.getInt("numTasks");
int limit = context.getInt("maxInputFilesPerWorker");
if (totalFiles > (long) workers * limit) { planMoreWorkers(totalFiles, limit); }

Try / catch

try { runMsqQuery(query); } catch (MSQException e) { if (e.getFault() instanceof TooManyInputFilesFault f) { rerunWithWorkers(f.getRequiredWorkers()); } else { throw e; } }

Prevention

Estimate input segment/file counts before submitting large batch queries
Configure enough MSQ workers for expected data volume
Raise maxInputFilesPerWorker consciously when workers handle bigger splits
Split very large backfills into chunked queries

When it happens

Trigger: Running an MSQ input-stage query over a number of segments/files that exceeds maxInputFilesPerWorker (default 100) for the available worker count; a query that aggregates too many historical segments or external files into one stage.

Common situations: Large batch re-indexing jobs over thousands of segments with few workers; context parameter maxInputFilesPerWorker left at default; undersized MSQ task/worker allocation in the overlord config.

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/33c6e19457bd5fba. Report an issue: GitHub.