{"record":{"id":"db223a4830af4bdc","repo":"apache/hadoop","slug":"too-many-counters-size-max-countersmax","errorCode":null,"errorMessage":"Too many counters: ${size} max=${countersMax}","messagePattern":"Too many counters: (.+?) max=(.+?)","errorType":"validation","errorClass":"LimitExceededException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java","lineNumber":97,"sourceCode":"    }\n    return COUNTERS_MAX;\n  }\n  \n  public static String filterName(String name, int maxLen) {\n    return name.length() > maxLen ? name.substring(0, maxLen - 1) : name;\n  }\n\n  public static String filterCounterName(String name) {\n    return filterName(name, getCounterNameMax());\n  }\n\n  public static String filterGroupName(String name) {\n    return filterName(name, getGroupNameMax());\n  }\n\n  public synchronized void checkCounters(int size) {\n    if (firstViolation != null) {\n      throw new LimitExceededException(firstViolation);\n    }\n    int countersMax = getCountersMax();\n    if (size > countersMax) {\n      firstViolation = new LimitExceededException(\"Too many counters: \"+ size +\n                                                  \" max=\"+ countersMax);\n      throw firstViolation;\n    }\n  }\n\n  public synchronized void incrCounters() {\n    checkCounters(totalCounters + 1);\n    ++totalCounters;\n  }\n\n  public synchronized void checkGroups(int size) {\n    if (firstViolation != null) {\n      throw new LimitExceededException(firstViolation);\n    }","sourceCodeStart":79,"sourceCodeEnd":115,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java#L79-L115","documentation":"Limits enforces the configured maximum number of counters in a Counters object: mapreduce.job.counters.max, default 120 (MRJobConfig.COUNTERS_MAX_DEFAULT). checkCounters throws LimitExceededException(\"Too many counters: N max=M\") when the size crosses the cap, caches the violation in firstViolation, and rethrows the same exception on every subsequent limits check - so the job fails at the next counter operation, not just once. Note Limits.init is static-once per JVM, so the limits in effect are those of the first Configuration seen.","triggerScenarios":"incrCounters()/checkCounters() exceeding mapreduce.job.counters.max - e.g. a job whose user counters have dynamic names (per-key, per-minute, per-tenant), or counter aggregation on the ApplicationMaster merging per-task counters past the cap.","commonSituations":"Jobs using counter names with unbounded cardinality; stacked libraries each registering many counters; clusters that lowered the default limits; upgrades where jobs newly exceed 120 counters and fail during merge with 'Too many counters'.","solutions":["Raise the cap on the job configuration: conf.setInt(\"mapreduce.job.counters.max\", <needed>) (and mapreduce.job.counters.groups.max if groups also grow).","Replace dynamic counter names with a fixed vocabulary and aggregate in your own code or a metrics system.","Audit cardinality before submission: enumerate the counter names your mappers/reducers can emit.","Remove or gate third-party libraries that register unbounded counters."],"exampleFix":"// before: unbounded cardinality\ncontext.getCounter(\"stats\", \"key-\" + rawKey).increment(1);\n\n// after: fixed vocabulary (preferred)\ncontext.getCounter(\"stats\", \"distinct-keys\").increment(1);\n// or, if genuinely needed, raise the cap:\nconf.setInt(\"mapreduce.job.counters.max\", 1000);\nconf.setInt(\"mapreduce.job.counters.groups.max\", 200);","handlingStrategy":"validation","validationCode":"int countersMax = conf.getInt(\"mapreduce.job.counters.max\", 120);\nSet<String> plannedNames = /* enumerate every counter name your job can emit */;\nif (plannedNames.size() > countersMax) {\n  throw new IllegalStateException(\"Job emits \" + plannedNames.size()\n      + \" counters; raise mapreduce.job.counters.max or reduce cardinality\");\n}","typeGuard":null,"tryCatchPattern":"catch (LimitExceededException e) when merging/aggregating counters: stop merging, keep the already-aggregated partial counters, and log the overflow source - the limit re-throws on every later check until the Counters object is reset.","preventionTips":["Fix counter-name cardinality at design time; counters are for bounded aggregates, not per-record keys.","Set mapreduce.job.counters.max/groups.max explicitly in job conf instead of relying on the 120/50 defaults.","Call Limits.init(conf) early with the intended Configuration (limits are static-once per JVM)."],"tags":["hadoop","mapreduce","counters","limits","configuration"],"backgroundTag":"counter-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}