apache/hadoop · error · IllegalArgumentException

Invalid -k argument. Must be of the form -k pos1,[pos2], whe

Error message

Invalid -k argument. Must be of the form -k pos1,[pos2], where pos is of the form f[.c]nr

What it means

KeyFieldHelper parses GNU-sort style -k key specifications for KeyFieldBasedComparator and Hadoop Streaming. In parseKey, once the optional end position (pos2, the part after the comma) has been read, only the modifier flags 'n' (numeric) and 'r' (reverse) may follow. This IllegalArgumentException fires when any other token trails the end position inside the do-while loop.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/KeyFieldHelper.java:263

          token = st.nextToken();
          if (token.equals(".")) {
            token = st.nextToken();
            key.endChar = Integer.parseInt(token);
            if (st.hasMoreTokens()) {
              token = st.nextToken();
            } else {
              return key;
            }
          }
          do {
            if (token.equals("n")) {
              key.numeric = true;
            }
            else if (token.equals("r")) {
              key.reverse = true;
            }
            else { 
              throw new IllegalArgumentException("Invalid -k argument. " +
               "Must be of the form -k pos1,[pos2], where pos is of the form " +
               "f[.c]nr");
            }
            if (st.hasMoreTokens()) {
              token = st.nextToken();
            } else {
              break;
            }
          } while (true);
        }
        return key;
      }
      throw new IllegalArgumentException("Invalid -k argument. " +
          "Must be of the form -k pos1,[pos2], where pos is of the form " +
          "f[.c]nr");
    }
    return key;
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Remove the invalid trailing token so pos2 is of the form f[.c] followed only by n/r flags, e.g. '-k 2,3n'
2. Strip GNU sort flags KeyFieldHelper does not implement ('b','d','f','i','M','h') — only 'n' and 'r' exist
3. Validate the -k string with a small KeyFieldHelper.parseOption() unit test before submitting the job

Example fix

// before
conf.set("mapreduce.partition.keycomparator.options", "-k 2,3nb"); // 'b' unsupported after pos2
// after
conf.set("mapreduce.partition.keycomparator.options", "-k 2,3n");

private static final Pattern K_SPEC =
    Pattern.compile("^\\d+(\\.\\d+)?[nr]*(,\\d+(\\.\\d+)?[nr]*)?$");

static boolean isValidKeySpec(String spec) {
  String arg = spec.startsWith("-k") ? spec.substring(2) : spec;
  return !arg.isEmpty() && K_SPEC.matcher(arg).matches();
}

try {
  helper.parseOption(keyArg);
} catch (IllegalArgumentException e) {
  throw new IllegalArgumentException("Bad -k spec '" + keyArg + "': " + e.getMessage(), e);
}

When it happens

Trigger: KeyFieldHelper.parseOption (or JobConf/streaming forwarding -k arguments) receives a spec such as '-k 2,3x', '-k 4.2,5.3b', or '-k 1,2,' — a token after pos2 that is neither 'n' nor 'r'. GNU sort flags like 'b' (skip blanks), 'd', 'f', 'i', 'M', 'h' are not supported and land here.

Common situations: Streaming jobs or mapreduce.partition.keycomparator.options values copied verbatim from a Linux 'sort' command line; adding an unsupported flag after the end field; a stray trailing comma or typo after the end position.

Related errors

• No more entry in " + f
• f + " is a directory"
• f + " already exists"
• Unsupported block buffer "{name}"
• Expecting arguments size of at most two, getting {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/03eec5919eeeee31. Report an issue: GitHub.