{"record":{"id":"9ab5e4c076a44fc9","repo":"apache/hadoop","slug":"too-much-write-to-local-file-system-current-value","errorCode":null,"errorMessage":"too much write to local file system. current value is \" + localWritesCounter.getCounter() + \" the limit is \" + limit","messagePattern":"too much write to local file system\\. current value is \" \\+ localWritesCounter\\.getCounter\\(\\) \\+ \" the limit is \" \\+ limit","errorType":"exception","errorClass":"TaskLimitException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java","lineNumber":835,"sourceCode":"     * limits.\n     * @throws TaskLimitException\n     */\n    protected void checkTaskLimits() throws TaskLimitException {\n      // check the limit for writing to local file system\n      long limit = conf.getLong(MRJobConfig.TASK_LOCAL_WRITE_LIMIT_BYTES,\n              MRJobConfig.DEFAULT_TASK_LOCAL_WRITE_LIMIT_BYTES);\n      if (limit >= 0) {\n        Counters.Counter localWritesCounter = null;\n        try {\n          LocalFileSystem localFS = FileSystem.getLocal(conf);\n          localWritesCounter = counters.findCounter(localFS.getScheme(),\n                  FileSystemCounter.BYTES_WRITTEN);\n        } catch (IOException e) {\n          LOG.warn(\"Could not get LocalFileSystem BYTES_WRITTEN counter\");\n        }\n        if (localWritesCounter != null\n                && localWritesCounter.getCounter() > limit) {\n          throw new TaskLimitException(\"too much write to local file system.\" +\n                  \" current value is \" + localWritesCounter.getCounter() +\n                  \" the limit is \" + limit);\n        }\n      }\n      if (diskLimitCheckStatus != null) {\n        throw new TaskLimitException(diskLimitCheckStatus);\n      }\n    }\n\n    /**\n     * The communication thread handles communication with the parent (Task\n     * Tracker). It sends progress updates if progress has been made or if\n     * the task needs to let the parent know that it's alive. It also pings\n     * the parent to see if it's alive.\n     */\n    public void run() {\n      final int MAX_RETRIES = 3;\n      int remainingRetries = MAX_RETRIES;","sourceCodeStart":817,"sourceCodeEnd":853,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java#L817-L853","documentation":"When mapreduce.task.local-fs.write-limit.bytes is set to a non-negative value, the task's progress thread periodically compares the LocalFileSystem BYTES_WRITTEN counter against that limit. Exceeding it raises TaskLimitException (an IOException) to deliberately fail the task — the mechanism exists to stop runaway tasks from filling local disks. Default is -1, i.e. disabled.","triggerScenarios":"Admin sets mapreduce.task.local-fs.write-limit.bytes=N on a cluster and a task writes more than N bytes through the LocalFileSystem (temp files, streaming pipes, heavy spill output); the next periodic check throws.","commonSituations":"Disk protection policies on shared clusters; streaming/hadoop-pipes jobs that stage large local files; jobs with little combiner use producing huge spill files in mapreduce.cluster.local.dir.","solutions":["Raise mapreduce.task.local-fs.write-limit.bytes (or set it to -1 to disable) if the write volume is legitimate.","Reduce local writes: add/verify combiners, increase io.sort.mb to spill less often, write final outputs to HDFS rather than local scratch.","Check the task's counters (File Systems/Local BYTES_WRITTEN in the job history) to see how close the task was to the limit."],"exampleFix":"# before\nmapreduce.task.local-fs.write-limit.bytes=1073741824\n\n# after\nmapreduce.task.local-fs.write-limit.bytes=10737418240","handlingStrategy":"validation","validationCode":"// before running a write-heavy job on a limited cluster, know your budget\nlong limit = conf.getLong(MRJobConfig.TASK_LOCAL_WRITE_LIMIT_BYTES, -1);\nif (limit >= 0) {\n  LOG.info(\"Task local write limit is \" + limit + \" bytes; monitor Local BYTES_WRITTEN\");\n}","typeGuard":null,"tryCatchPattern":"catch (TaskLimitException e) { // subclass of IOException\n  // task is failed by design; either raise mapreduce.task.local-fs.write-limit.bytes\n  // or reduce local writes and resubmit the job\n}","preventionTips":["Check the counter File Systems / Local / BYTES_WRITTEN in similar past jobs to size the limit.","Use combiners and adequate io.sort buffers to shrink local spill volume.","Set the limit to -1 on clusters where such caps are not needed."],"tags":["hadoop","mapreduce","task-limits","local-disk","configuration","resource-limit"],"backgroundTag":"disk-quota-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}