{"record":{"id":"520fc2ff0be26a76","repo":"apache/hadoop","slug":"too-many-schemes-schemes-size-when-process-s","errorCode":null,"errorMessage":"too many schemes? ${schemes.size()} when process scheme: ${scheme}","messagePattern":"too many schemes\\? (.+?) when process scheme: (.+?)","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java","lineNumber":239,"sourceCode":"    Object[] counters = map.get(canonicalScheme);\n    int ord = key.ordinal();\n    if (counters == null) {\n      counters = new Object[FileSystemCounter.values().length];\n      map.put(canonicalScheme, counters);\n      counters[ord] = newCounter(canonicalScheme, key);\n    }\n    else if (counters[ord] == null) {\n      counters[ord] = newCounter(canonicalScheme, key);\n    }\n    return (C) counters[ord];\n  }\n\n  private String checkScheme(String scheme) {\n    String fixed = StringUtils.toUpperCase(scheme);\n    String interned = schemes.putIfAbsent(fixed, fixed);\n    if (schemes.size() > MAX_NUM_SCHEMES) {\n      // mistakes or abuses\n      throw new IllegalArgumentException(\"too many schemes? \"+ schemes.size() +\n                                         \" when process scheme: \"+ scheme);\n    }\n    return interned == null ? fixed : interned;\n  }\n\n  /**\n   * Abstract factory method to create a file system counter\n   * @param scheme of the file system\n   * @param key the enum of the file system counter\n   * @return a new file system counter\n   */\n  protected abstract C newCounter(String scheme, FileSystemCounter key);\n\n  @Override\n  public synchronized int size() {\n    int n = 0;\n    if (map != null) {\n      for (Object[] counters : map.values()) {","sourceCodeStart":221,"sourceCodeEnd":257,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java#L221-L257","documentation":"FileSystemCounterGroup interns every distinct filesystem scheme it sees in counter names (uppercased) and checkScheme enforces a sanity cap of MAX_NUM_SCHEMES = 100 (FileSystemCounterGroup.java:56). Exceeding it throws IllegalArgumentException(\"too many schemes?\") - the guard exists because malformed names create a pseudo-scheme per distinct prefix and blow up memory/serialization.","triggerScenarios":"More than 100 distinct scheme prefixes flowing through fs counter names - e.g. a generator emitting names like \"TENANT42_BYTES_READ\", \"JOB7_BYTES_READ\", each counted as a new scheme; merging counters from many synthetic filesystems into one Counters object.","commonSituations":"User/lib code fabricating dynamic first-segment names in the fs group instead of a fixed scheme vocabulary; counters aggregation across many jobs/tenants into one object; frameworks registering a filesystem per workload with unique scheme names.","solutions":["Normalize fs counter names to a fixed, small set of real schemes (HDFS, FILE, S3A, ...) before adding.","Fix the name generator: dynamic segments belong after the '_', or in a separate generic counter group.","Route high-cardinality metrics to a metrics system (Metrics2/Dropwizard) instead of Hadoop counters.","Audit with counters.countCounters()/group names before submission."],"exampleFix":"// before: every tenant looks like a new scheme\ncounters.getGroup(\"File System Counters\")\n    .findCounter(\"TENANT\" + tenantId + \"_BYTES_READ\").increment(n); // explodes past 100 schemes\n\n// after: fixed scheme, dynamic part in a generic group\ncounters.getGroup(\"File System Counters\").findCounter(\"HDFS_BYTES_READ\").increment(n);\ncounters.findCounter(\"tenant-bytes\", \"TENANT\" + tenantId).increment(n);","handlingStrategy":"validation","validationCode":"// before adding fs counters, keep your own scheme vocabulary bounded\nstatic final Set<String> ALLOWED_SCHEMES = Set.of(\"HDFS\", \"FILE\", \"S3A\");\nString scheme = name.substring(0, name.indexOf('_'));\nif (!ALLOWED_SCHEMES.contains(scheme)) {\n  // route to a generic group instead of creating a pseudo-scheme\n  counters.findCounter(\"fs-other\", name);\n}","typeGuard":null,"tryCatchPattern":"catch (IllegalArgumentException e) around counter merge: on \"too many schemes\" log the offending scheme name and quarantine the source feeding pseudo-schemes; do not retry unchanged.","preventionTips":["Keep the set of scheme prefixes fixed (real filesystems only).","Put dynamic segments after the '_' or in generic groups.","Cap your own emitter so it can never generate more than a handful of schemes."],"tags":["hadoop","mapreduce","counters","filesystem-counter","limit","illegalargument"],"backgroundTag":"counter-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}