apache/hadoop · error · LimitExceededException

Too many counter groups: ${size} max=${groupsMax}

Error message

Too many counter groups: ${size} max=${groupsMax}

What it means

Limits.checkGroups enforces mapreduce.job.counters.groups.max (default 50, MRJobConfig.COUNTER_GROUPS_MAX_DEFAULT). Unlike checkCounters, this site does not throw: it only records a LimitExceededException("Too many counter groups") in firstViolation. The cached violation is thrown later by the next checkCounters()/incrCounters() call (or read via Limits.violation()), so the stack trace you see is usually at a counters increment, with this message as the cause.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java:114

    if (firstViolation != null) {
      throw new LimitExceededException(firstViolation);
    }
    int countersMax = getCountersMax();
    if (size > countersMax) {
      firstViolation = new LimitExceededException("Too many counters: "+ size +
                                                  " max="+ countersMax);
      throw firstViolation;
    }
  }

  public synchronized void incrCounters() {
    checkCounters(totalCounters + 1);
    ++totalCounters;
  }

  public synchronized void checkGroups(int size) {
    if (firstViolation != null) {
      throw new LimitExceededException(firstViolation);
    }
    int groupsMax = getGroupsMax();
    if (size > groupsMax) {
      firstViolation = new LimitExceededException("Too many counter groups: "+
                                                  size +" max="+ groupsMax);
    }
  }

  public synchronized LimitExceededException violation() {
    return firstViolation;
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Raise mapreduce.job.counters.groups.max (and mapreduce.job.counters.max) in the job configuration to cover your real group count.
2. Consolidate dynamic groups into one group with structured counter names.
3. Audit group-name cardinality the same way as counter-name cardinality.
4. Set the limits via Limits.init(conf) with the intended Configuration before any Counters object is created in the JVM (Limits are static-once).

Example fix

// before: one group per tenant
context.getCounter("tenant-" + tenantId, "records").increment(1);

// after: one group, structured names
context.getCounter("tenant-records", "TENANT" + tenantId).increment(1);
// and/or raise the cap:
conf.setInt("mapreduce.job.counters.groups.max", 200);

int groupsMax = conf.getInt("mapreduce.job.counters.groups.max", 50);
Set<String> plannedGroups = /* enumerate group names your job emits */;
if (plannedGroups.size() > groupsMax) {
  throw new IllegalStateException("Job emits " + plannedGroups.size()
      + " counter groups; raise mapreduce.job.counters.groups.max or consolidate");
}

catch (LimitExceededException e) with cause/message "Too many counter groups": the violation was recorded earlier by checkGroups - reduce group count and rebuild the Counters object; retrying on the same object keeps rethrowing.

When it happens

Trigger: Registering more distinct counter group names than mapreduce.job.counters.groups.max - e.g. getGroup("group-" + i) for many i - followed by any later counter add, which rethrows the cached groups violation. Also triggered during Counters deserialization (readFields calls checkGroups while loading groups).

Common situations: Dynamic group names from user code (per-module/per-tenant groups); aggregation of many libraries' counters; jobs migrated to clusters with default (50) group limit after using more on another cluster; history/audit tooling parsing large counters.

Related errors

• Too many counters: ${size} max=${countersMax}
• Too many spill files got created, control it with mapreduce.
• Invalid specification for distributed-cache artifacts of typ
• Unable to parse '{}' as a URI, check the setting for mapredu
• Could not locate MapReduce framework name '{}' in mapreduce.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9a09ff7dc88c42cb. Report an issue: GitHub.