{"record":{"id":"ce9ecc4f01c2bd61","repo":"apache/hadoop","slug":"this-job-has-exceeded-the-maximum-size-of-a-single","errorCode":null,"errorMessage":"This job has exceeded the maximum size of a single submitted resource (Max: {}MB, Violating resource: {}).","messagePattern":"This job has exceeded the maximum size of a single submitted resource \\(Max: (.+?)MB, Violating resource: (.+?)\\)\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java","lineNumber":607,"sourceCode":"      totalSizeBytes += fileSizeBytes;\n      if (fileSizeBytes > currentMaxSizeOfFileBytes) {\n        currentMaxSizeOfFileBytes = fileSizeBytes;\n      }\n\n      if (totalConfigSizeBytes > 0 && totalSizeBytes > totalConfigSizeBytes) {\n        throw new IOException(MAX_TOTAL_RESOURCE_MB_ERR_MSG + \" (Max: \"\n            + maxSizeMB + \"MB).\");\n      }\n\n      if (maxNumOfResources > 0 &&\n          totalNumberOfResources > maxNumOfResources) {\n        throw new IOException(MAX_RESOURCE_ERR_MSG + \" (Max: \"\n            + maxNumOfResources + \").\");\n      }\n\n      if (totalConfigSizeOfResourceBytes > 0\n          && currentMaxSizeOfFileBytes > totalConfigSizeOfResourceBytes) {\n        throw new IOException(MAX_SINGLE_RESOURCE_MB_ERR_MSG + \" (Max: \"\n            + maxSizeOfResourceMB + \"MB, Violating resource: \" + p + \").\");\n      }\n    }\n  }\n\n  /**\n   * Recursively explore the given path and enforce the limits for resource\n   * localization. This method assumes that there are no symlinks in the\n   * directory structure.\n   */\n  private void explorePath(Configuration job, Path p,\n      LimitChecker limitChecker, Map<URI, FileStatus> statCache)\n      throws IOException {\n    Path pathWithScheme = p;\n    if (!pathWithScheme.toUri().isAbsolute()) {\n      // the path does not have a scheme, so we assume it is a path from the\n      // local filesystem\n      FileSystem localFs = FileSystem.getLocal(job);","sourceCodeStart":589,"sourceCodeEnd":625,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java#L589-L625","documentation":"The per-resource LimitChecker check: it tracks the largest single file seen (currentMaxSizeOfFileBytes) while walking staged resources, and if one resource exceeds the configured per-item cap (mapreduce.job.cache.limit.max-single-resource-mb, MRJobConfig.MAX_SINGLE_RESOURCE_MB; default 0 = unlimited), addFile throws IOException('This job has exceeded the maximum size of a single submitted resource (Max: <N>MB, Violating resource: <path>).') — conveniently naming the offending file.","triggerScenarios":"Including one large artifact (ML model, dictionary, native library bundle) in -files/-archives while a single-resource cap is configured; an uber jar grown past the cap after adding dependencies; a data file accidentally shipped as code.","commonSituations":"Clusters capping per-item localization to protect NodeManager disk; model-serving jobs shipping big binaries; builds whose shaded jar slowly grows release over release until it crosses the limit.","solutions":["Move the oversized artifact out of the distributed cache: upload it to HDFS once and read it by Path in the tasks","Split or shrink the offending resource named in the message (compress, trim unused deps from the shaded jar)","With admin approval, raise mapreduce.job.cache.limit.max-single-resource-mb above the size reported in the error","Add a build-time check that fails when any shipped artifact exceeds the cluster's per-item cap"],"exampleFix":"# before\nhadoop jar app.jar Driver -files \"model-2GB.bin\" in out\n# IOException: exceeded the maximum size of a single submitted resource\n# (Max: 512MB, Violating resource: /user/me/.staging/job_.../files/model-2GB.bin)\n\n# after\nhdfs dfs -put model-2GB.bin /data/models/\nhadoop jar app.jar Driver -Dmodel.path=/data/models/model-2GB.bin in out","handlingStrategy":"validation","validationCode":"// fail on any single oversized artifact before submit\nlong cap = conf.getLong(\"mapreduce.job.cache.limit.max-single-resource-mb\", 0) * 1024L * 1024L;\nif (cap > 0) {\n  for (Path p : shippedPaths) {\n    long size = p.getFileSystem(conf).getFileStatus(p).getLen();\n    if (size > cap) {\n      throw new IllegalStateException(p + \" is \" + (size / 1048576) + \"MB; cap is \" + (cap / 1048576) + \"MB — move it to HDFS and read by path\");\n    }\n  }\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep big models/binaries in HDFS and pass their Path, not the distributed cache","Add a build check that fails when any shipped artifact exceeds the per-item cap","When the error fires, read the 'Violating resource:' name — it identifies the exact file"],"tags":["resource-limits","distributed-cache","quota","large-file","mapreduce"],"backgroundTag":"resource-quota-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}