apache/hadoop · error · IOException

This job has exceeded the maximum size of submitted resource

Error message

This job has exceeded the maximum size of submitted resources (Max: {}MB).

What it means

At submission, JobResourceUploader's LimitChecker walks every resource staged for localization (files, libjars, archives, job jar) and accumulates total bytes. If the configured total cap is positive and exceeded, addFile throws IOException('This job has exceeded the maximum size of submitted resources (Max: <N>MB).'). The cap comes from mapreduce.job.cache.limit.max-resources-mb (MRJobConfig.MAX_RESOURCES_MB); the default is 0, meaning the check is disabled, so seeing this error means an admin (or job conf) enabled the limit.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java:595

    private final long maxSizeMB;
    private final int maxNumOfResources;
    private final long maxSizeOfResourceMB;
    private final long totalConfigSizeBytes;
    private final long totalConfigSizeOfResourceBytes;

    private boolean hasLimits() {
      return maxNumOfResources > 0 || maxSizeMB > 0 || maxSizeOfResourceMB > 0;
    }

    private void addFile(Path p, long fileSizeBytes) throws IOException {
      totalNumberOfResources++;
      totalSizeBytes += fileSizeBytes;
      if (fileSizeBytes > currentMaxSizeOfFileBytes) {
        currentMaxSizeOfFileBytes = fileSizeBytes;
      }

      if (totalConfigSizeBytes > 0 && totalSizeBytes > totalConfigSizeBytes) {
        throw new IOException(MAX_TOTAL_RESOURCE_MB_ERR_MSG + " (Max: "
            + maxSizeMB + "MB).");
      }

      if (maxNumOfResources > 0 &&
          totalNumberOfResources > maxNumOfResources) {
        throw new IOException(MAX_RESOURCE_ERR_MSG + " (Max: "
            + maxNumOfResources + ").");
      }

      if (totalConfigSizeOfResourceBytes > 0
          && currentMaxSizeOfFileBytes > totalConfigSizeOfResourceBytes) {
        throw new IOException(MAX_SINGLE_RESOURCE_MB_ERR_MSG + " (Max: "
            + maxSizeOfResourceMB + "MB, Violating resource: " + p + ").");
      }
    }
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

1. Cut the payload: build an uber jar (maven-shade-plugin) instead of many -libjars, and remove non-code files from -files
2. Split the job so each submission stays under the cap, or pre-stage large read-only data in HDFS and reference by path instead of the distributed cache
3. If policy allows, raise/disable the limit: set mapreduce.job.cache.limit.max-resources-mb to a larger value (or 0) in mapred-site.xml
4. Measure before submitting: sum the sizes of everything you pass so failures happen in your tooling, not on the cluster

Example fix

# before
hadoop jar app.jar Driver -libjars "a.jar,b.jar,...,z.jar" -files model.bin in out
# IOException: exceeded the maximum size of submitted resources (Max: 512MB).

# after
# shade dependencies into app.jar, ship only code
hadoop jar app-uber.jar Driver in out

# or admin-side:
# <property><name>mapreduce.job.cache.limit.max-resources-mb</name><value>2048</value></property>

// pre-compute total shipped bytes before submitting
long total = 0;
for (String p : allShippedPaths) {
  total += Files.walk(Paths.get(p)).mapToLong(f -> f.toFile().length()).sum();
}
long cap = conf.getLong("mapreduce.job.cache.limit.max-resources-mb", 0) * 1024L * 1024L;
if (cap > 0 && total > cap) {
  throw new IllegalStateException("Shipped resources " + (total / 1048576) + "MB exceed cluster cap " + (cap / 1048576) + "MB");
}

When it happens

Trigger: Submitting with many or large -files/-libjars/-archives while mapreduce.job.cache.limit.max-resources-mb is set on the cluster; shipping a fat job jar plus dozens of dependency jars whose sum crosses the cap; directories expanded recursively by explorePath counting more bytes than expected.

Common situations: Shared/managed clusters enforcing submission quotas; teams bundling entire dependency trees per job; accidentally including data files (not code) in -files after a change of layout.

Related errors

• This job has exceeded the maximum number of submitted resour
• This job has exceeded the maximum size of a single submitted
• Invalid specification for distributed-cache artifacts of typ
• Error parsing files argument. Argument must be a valid URI:
• Error parsing archives argument. Argument must be a valid UR

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/686238645e36a1ee. Report an issue: GitHub.