apache/hadoop · error · IOException

This job has exceeded the maximum number of submitted resour

Error message

This job has exceeded the maximum number of submitted resources (Max: {}).

What it means

Same LimitChecker pass, but the counting check: every staged file/archive/libjar/job-jar entry increments totalNumberOfResources, and once it exceeds the configured cap (mapreduce.job.cache.limit.max-resources, MRJobConfig.MAX_RESOURCES; default 0 = unlimited), addFile throws IOException('This job has exceeded the maximum number of submitted resources (Max: <N>).').

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java:601

    private boolean hasLimits() {
      return maxNumOfResources > 0 || maxSizeMB > 0 || maxSizeOfResourceMB > 0;
    }

    private void addFile(Path p, long fileSizeBytes) throws IOException {
      totalNumberOfResources++;
      totalSizeBytes += fileSizeBytes;
      if (fileSizeBytes > currentMaxSizeOfFileBytes) {
        currentMaxSizeOfFileBytes = fileSizeBytes;
      }

      if (totalConfigSizeBytes > 0 && totalSizeBytes > totalConfigSizeBytes) {
        throw new IOException(MAX_TOTAL_RESOURCE_MB_ERR_MSG + " (Max: "
            + maxSizeMB + "MB).");
      }

      if (maxNumOfResources > 0 &&
          totalNumberOfResources > maxNumOfResources) {
        throw new IOException(MAX_RESOURCE_ERR_MSG + " (Max: "
            + maxNumOfResources + ").");
      }

      if (totalConfigSizeOfResourceBytes > 0
          && currentMaxSizeOfFileBytes > totalConfigSizeOfResourceBytes) {
        throw new IOException(MAX_SINGLE_RESOURCE_MB_ERR_MSG + " (Max: "
            + maxSizeOfResourceMB + "MB, Violating resource: " + p + ").");
      }
    }
  }

  /**
   * Recursively explore the given path and enforce the limits for resource
   * localization. This method assumes that there are no symlinks in the
   * directory structure.
   */
  private void explorePath(Configuration job, Path p,
      LimitChecker limitChecker, Map<URI, FileStatus> statCache)

View on GitHub (pinned to 2add963021)

Solutions

Consolidate: one shaded uber jar instead of many libjars; one tar.gz archive instead of -files of a directory
Count your arguments before submitting: number of -files/-libjars/-archives entries plus job jar must stay under the configured cap
Raise the limit with admin approval via mapreduce.job.cache.limit.max-resources
Audit for accidental recursive expansion — a directory argument becomes one resource per file

Example fix

# before
hadoop jar app.jar Driver -files dir1,dir2,dir3 -libjars "$(paste -sd, jars/*.jar)" in out
# IOException: exceeded the maximum number of submitted resources (Max: 100).

# after
tar czf bundle.tgz dir1 dir2 dir3
hadoop jar app-uber.jar Driver -files bundle.tgz in out

Defensive patterns

Strategy: validation

Validate before calling

// pre-count resources: -files/-libjars/-archives entries + job jar
int count = cacheFiles.length + libjars.length + archives.length + 1;
int cap = conf.getInt("mapreduce.job.cache.limit.max-resources", 0);
if (cap > 0 && count > cap) {
  throw new IllegalStateException("Resource count " + count + " exceeds cap " + cap + "; consolidate into an archive or uber jar");
}

Prevention

Bundle directory trees into a single tar.gz instead of -files of a directory (expansion counts each file)
Use a shaded uber jar to collapse dozens of libjars into one resource
Track the cluster's configured resource-count cap in your team's runbook

When it happens

Trigger: Submitting a job with hundreds of -libjars entries on a cluster where the count limit is set; -files pointing at directories that explorePath expands recursively into many individual resources; frameworks (Pig/Hive/Cascading) auto-adding per-UDF jars until the count crosses the cap.

Common situations: Quota enforcement on multi-tenant clusters; dependency-heavy applications; -files of a directory (each contained file counts separately) rather than one archive.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f3da028921c74977. Report an issue: GitHub.