{"record":{"id":"f3da028921c74977","repo":"apache/hadoop","slug":"this-job-has-exceeded-the-maximum-number-of-submit","errorCode":null,"errorMessage":"This job has exceeded the maximum number of submitted resources (Max: {}).","messagePattern":"This job has exceeded the maximum number of submitted resources \\(Max: (.+?)\\)\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java","lineNumber":601,"sourceCode":"    private boolean hasLimits() {\n      return maxNumOfResources > 0 || maxSizeMB > 0 || maxSizeOfResourceMB > 0;\n    }\n\n    private void addFile(Path p, long fileSizeBytes) throws IOException {\n      totalNumberOfResources++;\n      totalSizeBytes += fileSizeBytes;\n      if (fileSizeBytes > currentMaxSizeOfFileBytes) {\n        currentMaxSizeOfFileBytes = fileSizeBytes;\n      }\n\n      if (totalConfigSizeBytes > 0 && totalSizeBytes > totalConfigSizeBytes) {\n        throw new IOException(MAX_TOTAL_RESOURCE_MB_ERR_MSG + \" (Max: \"\n            + maxSizeMB + \"MB).\");\n      }\n\n      if (maxNumOfResources > 0 &&\n          totalNumberOfResources > maxNumOfResources) {\n        throw new IOException(MAX_RESOURCE_ERR_MSG + \" (Max: \"\n            + maxNumOfResources + \").\");\n      }\n\n      if (totalConfigSizeOfResourceBytes > 0\n          && currentMaxSizeOfFileBytes > totalConfigSizeOfResourceBytes) {\n        throw new IOException(MAX_SINGLE_RESOURCE_MB_ERR_MSG + \" (Max: \"\n            + maxSizeOfResourceMB + \"MB, Violating resource: \" + p + \").\");\n      }\n    }\n  }\n\n  /**\n   * Recursively explore the given path and enforce the limits for resource\n   * localization. This method assumes that there are no symlinks in the\n   * directory structure.\n   */\n  private void explorePath(Configuration job, Path p,\n      LimitChecker limitChecker, Map<URI, FileStatus> statCache)","sourceCodeStart":583,"sourceCodeEnd":619,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java#L583-L619","documentation":"Same LimitChecker pass, but the counting check: every staged file/archive/libjar/job-jar entry increments totalNumberOfResources, and once it exceeds the configured cap (mapreduce.job.cache.limit.max-resources, MRJobConfig.MAX_RESOURCES; default 0 = unlimited), addFile throws IOException('This job has exceeded the maximum number of submitted resources (Max: <N>).').","triggerScenarios":"Submitting a job with hundreds of -libjars entries on a cluster where the count limit is set; -files pointing at directories that explorePath expands recursively into many individual resources; frameworks (Pig/Hive/Cascading) auto-adding per-UDF jars until the count crosses the cap.","commonSituations":"Quota enforcement on multi-tenant clusters; dependency-heavy applications; -files of a directory (each contained file counts separately) rather than one archive.","solutions":["Consolidate: one shaded uber jar instead of many libjars; one tar.gz archive instead of -files of a directory","Count your arguments before submitting: number of -files/-libjars/-archives entries plus job jar must stay under the configured cap","Raise the limit with admin approval via mapreduce.job.cache.limit.max-resources","Audit for accidental recursive expansion — a directory argument becomes one resource per file"],"exampleFix":"# before\nhadoop jar app.jar Driver -files dir1,dir2,dir3 -libjars \"$(paste -sd, jars/*.jar)\" in out\n# IOException: exceeded the maximum number of submitted resources (Max: 100).\n\n# after\ntar czf bundle.tgz dir1 dir2 dir3\nhadoop jar app-uber.jar Driver -files bundle.tgz in out","handlingStrategy":"validation","validationCode":"// pre-count resources: -files/-libjars/-archives entries + job jar\nint count = cacheFiles.length + libjars.length + archives.length + 1;\nint cap = conf.getInt(\"mapreduce.job.cache.limit.max-resources\", 0);\nif (cap > 0 && count > cap) {\n  throw new IllegalStateException(\"Resource count \" + count + \" exceeds cap \" + cap + \"; consolidate into an archive or uber jar\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Bundle directory trees into a single tar.gz instead of -files of a directory (expansion counts each file)","Use a shaded uber jar to collapse dozens of libjars into one resource","Track the cluster's configured resource-count cap in your team's runbook"],"tags":["resource-limits","distributed-cache","quota","job-submission","mapreduce"],"backgroundTag":"resource-quota-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}