{"record":{"id":"6a2f5a476d996227","repo":"apache/hadoop","slug":"too-much-data-in-local-scratch-dir-largestwork","errorCode":null,"errorMessage":"too much data in local scratch dir=\" + largestWorkDir + \". current size is \" + localWritesSize + \" the limit is \" + fsLimit","messagePattern":"too much data in local scratch dir=\" \\+ largestWorkDir \\+ \"\\. current size is \" \\+ localWritesSize \\+ \" the limit is \" \\+ fsLimit","errorType":"exception","errorClass":"TaskLimitException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java","lineNumber":841,"sourceCode":"              MRJobConfig.DEFAULT_TASK_LOCAL_WRITE_LIMIT_BYTES);\n      if (limit >= 0) {\n        Counters.Counter localWritesCounter = null;\n        try {\n          LocalFileSystem localFS = FileSystem.getLocal(conf);\n          localWritesCounter = counters.findCounter(localFS.getScheme(),\n                  FileSystemCounter.BYTES_WRITTEN);\n        } catch (IOException e) {\n          LOG.warn(\"Could not get LocalFileSystem BYTES_WRITTEN counter\");\n        }\n        if (localWritesCounter != null\n                && localWritesCounter.getCounter() > limit) {\n          throw new TaskLimitException(\"too much write to local file system.\" +\n                  \" current value is \" + localWritesCounter.getCounter() +\n                  \" the limit is \" + limit);\n        }\n      }\n      if (diskLimitCheckStatus != null) {\n        throw new TaskLimitException(diskLimitCheckStatus);\n      }\n    }\n\n    /**\n     * The communication thread handles communication with the parent (Task\n     * Tracker). It sends progress updates if progress has been made or if\n     * the task needs to let the parent know that it's alive. It also pings\n     * the parent to see if it's alive.\n     */\n    public void run() {\n      final int MAX_RETRIES = 3;\n      int remainingRetries = MAX_RETRIES;\n      // get current flag value and reset it as well\n      boolean sendProgress = resetProgressFlag();\n\n      long taskProgressInterval = MRJobConfUtil.\n          getTaskProgressReportInterval(conf);\n","sourceCodeStart":823,"sourceCodeEnd":859,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java#L823-L859","documentation":"When mapreduce.job.local-fs.single-disk-limit.bytes is non-negative, a background DiskLimitCheck thread (default every 5 s) computes disk usage of each local dir in mapreduce.cluster.local.dir and tracks the largest one. If that largest scratch dir exceeds the limit and kill-limit-exceed is true (default), diskLimitCheckStatus is set and the next checkTaskLimits() throws TaskLimitException with this message. Setting mapreduce.job.local-fs.single-disk-limit.check.kill-limit-exceed=false downgrades it to a WARN log without killing.","triggerScenarios":"Job-level config enables mapreduce.job.local-fs.single-disk-limit.bytes=N while a task's work under mapreduce.cluster.local.dir (spills, intermediate data, user scratch) on one volume grows past N; the task is killed within one check interval (default 5000 ms).","commonSituations":"Cluster-wide disk protection after an incident; jobs with pathological spill growth; several concurrent tasks on one node whose combined local data trips a per-job limit set too tightly.","solutions":["Increase mapreduce.job.local-fs.single-disk-limit.bytes or leave it -1 (disabled) if the usage is expected.","Set mapreduce.job.local-fs.single-disk-limit.check.kill-limit-exceed=false to only log instead of killing the task while you investigate.","Reduce local footprint: combiners, larger io.sort.* buffers, fewer intermediate files per task.","Check the named scratch dir (largestWorkDir in the message) for orphaned attempt directories from previous jobs and clean them."],"exampleFix":"# before\nmapreduce.job.local-fs.single-disk-limit.bytes=10737418240\n\n# after\nmapreduce.job.local-fs.single-disk-limit.bytes=107374182400\nmapreduce.job.local-fs.single-disk-limit.check.kill-limit-exceed=false","handlingStrategy":"validation","validationCode":"// monitor the largest local scratch dir against the configured job limit\nlong fsLimit = conf.getLong(MRJobConfig.JOB_SINGLE_DISK_LIMIT_BYTES, -1);\nif (fsLimit >= 0) {\n  long used = FileUtil.getDU(new File(conf.getLocalDirs()[0]));\n  if (used > fsLimit) throw new IOException(\"Approaching single-disk limit: \" + used + \"/\" + fsLimit);\n}","typeGuard":null,"tryCatchPattern":"catch (TaskLimitException e) { // subclass of IOException\n  // killed by DiskLimitCheck: raise mapreduce.job.local-fs.single-disk-limit.bytes,\n  // or set ...check.kill-limit-exceed=false to log-only, then resubmit\n}","preventionTips":["Size mapreduce.job.local-fs.single-disk-limit.bytes against expected spill volume per task, with headroom.","Use kill-limit-exceed=false during tuning to observe usage before enforcing.","Keep mapreduce.cluster.local.dir volumes clean of orphaned attempt directories."],"tags":["hadoop","mapreduce","task-limits","local-disk","disk-usage","resource-limit"],"backgroundTag":"disk-quota-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}