apache/hadoop · error · TaskLimitException

too much data in local scratch dir=" + largestWorkDir + ". c

Error message

too much data in local scratch dir=" + largestWorkDir + ". current size is " + localWritesSize + " the limit is " + fsLimit

What it means

When mapreduce.job.local-fs.single-disk-limit.bytes is non-negative, a background DiskLimitCheck thread (default every 5 s) computes disk usage of each local dir in mapreduce.cluster.local.dir and tracks the largest one. If that largest scratch dir exceeds the limit and kill-limit-exceed is true (default), diskLimitCheckStatus is set and the next checkTaskLimits() throws TaskLimitException with this message. Setting mapreduce.job.local-fs.single-disk-limit.check.kill-limit-exceed=false downgrades it to a WARN log without killing.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java:841

              MRJobConfig.DEFAULT_TASK_LOCAL_WRITE_LIMIT_BYTES);
      if (limit >= 0) {
        Counters.Counter localWritesCounter = null;
        try {
          LocalFileSystem localFS = FileSystem.getLocal(conf);
          localWritesCounter = counters.findCounter(localFS.getScheme(),
                  FileSystemCounter.BYTES_WRITTEN);
        } catch (IOException e) {
          LOG.warn("Could not get LocalFileSystem BYTES_WRITTEN counter");
        }
        if (localWritesCounter != null
                && localWritesCounter.getCounter() > limit) {
          throw new TaskLimitException("too much write to local file system." +
                  " current value is " + localWritesCounter.getCounter() +
                  " the limit is " + limit);
        }
      }
      if (diskLimitCheckStatus != null) {
        throw new TaskLimitException(diskLimitCheckStatus);
      }
    }

    /**
     * The communication thread handles communication with the parent (Task
     * Tracker). It sends progress updates if progress has been made or if
     * the task needs to let the parent know that it's alive. It also pings
     * the parent to see if it's alive.
     */
    public void run() {
      final int MAX_RETRIES = 3;
      int remainingRetries = MAX_RETRIES;
      // get current flag value and reset it as well
      boolean sendProgress = resetProgressFlag();

      long taskProgressInterval = MRJobConfUtil.
          getTaskProgressReportInterval(conf);

View on GitHub (pinned to 2add963021)

Solutions

1. Increase mapreduce.job.local-fs.single-disk-limit.bytes or leave it -1 (disabled) if the usage is expected.
2. Set mapreduce.job.local-fs.single-disk-limit.check.kill-limit-exceed=false to only log instead of killing the task while you investigate.
3. Reduce local footprint: combiners, larger io.sort.* buffers, fewer intermediate files per task.
4. Check the named scratch dir (largestWorkDir in the message) for orphaned attempt directories from previous jobs and clean them.

Example fix

# before
mapreduce.job.local-fs.single-disk-limit.bytes=10737418240

# after
mapreduce.job.local-fs.single-disk-limit.bytes=107374182400
mapreduce.job.local-fs.single-disk-limit.check.kill-limit-exceed=false

// monitor the largest local scratch dir against the configured job limit
long fsLimit = conf.getLong(MRJobConfig.JOB_SINGLE_DISK_LIMIT_BYTES, -1);
if (fsLimit >= 0) {
  long used = FileUtil.getDU(new File(conf.getLocalDirs()[0]));
  if (used > fsLimit) throw new IOException("Approaching single-disk limit: " + used + "/" + fsLimit);
}

catch (TaskLimitException e) { // subclass of IOException
  // killed by DiskLimitCheck: raise mapreduce.job.local-fs.single-disk-limit.bytes,
  // or set ...check.kill-limit-exceed=false to log-only, then resubmit
}

When it happens

Trigger: Job-level config enables mapreduce.job.local-fs.single-disk-limit.bytes=N while a task's work under mapreduce.cluster.local.dir (spills, intermediate data, user scratch) on one volume grows past N; the task is killed within one check interval (default 5000 ms).

Common situations: Cluster-wide disk protection after an incident; jobs with pathological spill growth; several concurrent tasks on one node whose combined local data trips a per-job limit set too tightly.

Related errors

• too much write to local file system. current value is " + lo
• Unable to recover task %s, output: %s
• Changing job priority in LocalJobRunner is not supported.
• Killing tasks in LocalJobRunner is not supported
• Not supported

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6a2f5a476d996227. Report an issue: GitHub.