{"record":{"id":"33c6e19457bd5fba","repo":"apache/druid","slug":"toomanyinputfiles","errorCode":"TooManyInputFiles","errorMessage":"Too many input files/segments [%d] encountered. Maximum input files/segments per worker is set to [%d]. Try increasing the limit using the %s query context parameter, breaking your query up into smaller queries, or increasing the number of workers to at least [%d] by setting %s in your query context.","messagePattern":"Too many input files/segments \\[(.+?)\\] encountered\\. Maximum input files/segments per worker is set to \\[(.+?)\\]\\. Try increasing the limit using the (.+?) query context parameter, breaking your query up into smaller queries, or increasing the number of workers to at least \\[(.+?)\\] by setting (.+?) in your query context\\.","errorType":"error_code","errorClass":"TooManyInputFilesFault","httpStatus":null,"severity":"error","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernel.java","lineNumber":332,"sourceCode":"          queryDef,\n          stageNumber,\n          workerNumber,\n          workerInputs.inputsForWorker(workerNumber),\n          extraInfoHolder,\n          config.getWorkerIds(),\n          outputChannelMode,\n          config.getWorkerContextMap()\n      );\n\n      final int numInputFiles = Ints.checkedCast(workOrder.getInputs().stream().mapToLong(InputSlice::fileCount).sum());\n      fault = fault || IntMath.divide(numInputFiles, maxInputFilesPerWorker, RoundingMode.CEILING) > 1;\n      totalFileCount += numInputFiles;\n      workerToWorkOrder.put(workerNumber, workOrder);\n    }\n\n    final int requiredWorkers = IntMath.divide(totalFileCount, maxInputFilesPerWorker, RoundingMode.CEILING);\n    if (fault) {\n      throw new MSQException(new TooManyInputFilesFault(totalFileCount, maxInputFilesPerWorker, requiredWorkers));\n    }\n    stageWorkOrders.put(new StageId(queryDef.getQueryId(), stageNumber), workerToWorkOrder);\n    return workerToWorkOrder;\n  }\n\n  private void createNewKernels(\n      final InputSpecSlicerFactory slicerFactory,\n      final WorkerAssignmentStrategy assignmentStrategy,\n      final FrameType rowBasedFrameType,\n      final int maxInputFilesPerWorker,\n      final long maxInputBytesPerWorker,\n      final int maxPartitions\n  )\n  {\n    StageGroup stageGroup;\n\n    while ((stageGroup = stageGroupQueue.peek()) != null) {\n      if (readyToRunStages.contains(stageGroup.first())","sourceCodeStart":314,"sourceCodeEnd":350,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernel.java#L314-L350","documentation":"During createWorkOrders, the controller sums input files/segments per worker and throws an MSQException wrapping a TooManyInputFilesFault when a worker would exceed maxInputFilesPerWorker. The fault reports total files, the configured limit, and the minimum number of workers required. It protects workers from being assigned unmanageable numbers of splits.","triggerScenarios":"Running an MSQ input-stage query over a number of segments/files that exceeds maxInputFilesPerWorker (default 100) for the available worker count; a query that aggregates too many historical segments or external files into one stage.","commonSituations":"Large batch re-indexing jobs over thousands of segments with few workers; context parameter maxInputFilesPerWorker left at default; undersized MSQ task/worker allocation in the overlord config.","solutions":["Increase the number of workers (e.g. via the numTasks / task context parameter or cluster autoscaling) as the fault message suggests","Raise maxInputFilesPerWorker in the query context (accepting larger per-worker memory/CPU load)","Split the query into smaller queries covering fewer segments each"],"exampleFix":"// before\n{\"query\": \"...\"}\n// after\n{\"query\": \"...\", \"context\": {\"maxInputFilesPerWorker\": 1000, \"numTasks\": 4}}","handlingStrategy":"try-catch","validationCode":"long totalFiles = inputs.stream().mapToLong(InputSpec::getNumFiles).sum();\nint workers = context.getInt(\"numTasks\");\nint limit = context.getInt(\"maxInputFilesPerWorker\");\nif (totalFiles > (long) workers * limit) { planMoreWorkers(totalFiles, limit); }","typeGuard":null,"tryCatchPattern":"try { runMsqQuery(query); } catch (MSQException e) { if (e.getFault() instanceof TooManyInputFilesFault f) { rerunWithWorkers(f.getRequiredWorkers()); } else { throw e; } }","preventionTips":["Estimate input segment/file counts before submitting large batch queries","Configure enough MSQ workers for expected data volume","Raise maxInputFilesPerWorker consciously when workers handle bigger splits","Split very large backfills into chunked queries"],"tags":["msq","limits","distributed"],"backgroundTag":"file-size-limit-exceeded","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}