apache/hadoop · critical · IllegalArgumentException

numLevels must be at least 1

Error message

numLevels must be at least 1

What it means

CallQueueManager.parsePriorityLevels resolves the number of priority levels for the call queue: it first reads the deprecated ipc.<port>.callqueue.priority.levels, then ipc.<port>.scheduler.priority.levels (default 4). It throws IllegalArgumentException("numLevels must be at least 1") when the resolved value is < 1, which aborts RPC server startup.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java:408

   */
  @SuppressWarnings("deprecation")
  private static int parseNumLevels(String ns, Configuration conf) {
    // Fair call queue levels (IPC_CALLQUEUE_PRIORITY_LEVELS_KEY)
    // takes priority over the scheduler level key
    // (IPC_SCHEDULER_PRIORITY_LEVELS_KEY)
    int retval = conf.getInt(ns + "." +
        FairCallQueue.IPC_CALLQUEUE_PRIORITY_LEVELS_KEY, 0);
    if (retval == 0) { // No FCQ priority level configured
      retval = conf.getInt(ns + "." +
          CommonConfigurationKeys.IPC_SCHEDULER_PRIORITY_LEVELS_KEY,
          CommonConfigurationKeys.IPC_SCHEDULER_PRIORITY_LEVELS_DEFAULT_KEY);
    } else {
      LOG.warn(ns + "." + FairCallQueue.IPC_CALLQUEUE_PRIORITY_LEVELS_KEY +
          " is deprecated. Please use " + ns + "." +
          CommonConfigurationKeys.IPC_SCHEDULER_PRIORITY_LEVELS_KEY + ".");
    }
    if(retval < 1) {
      throw new IllegalArgumentException("numLevels must be at least 1");
    }
    return retval;
  }

  /**
   * Read the weights of capacity in callqueue and pass the value to
   * callqueue constructions.
   */
  private static int[] parseCapacityWeights(
      int priorityLevels, String ns, Configuration conf) {
    int[] weights = conf.getInts(ns + "." +
      CommonConfigurationKeys.IPC_CALLQUEUE_CAPACITY_WEIGHTS_KEY);
    if (weights.length == 0) {
      weights = getDefaultQueueCapacityWeights(priorityLevels);
    } else if (weights.length != priorityLevels) {
      throw new IllegalArgumentException(
          CommonConfigurationKeys.IPC_CALLQUEUE_CAPACITY_WEIGHTS_KEY + " must "
              + "specify " + priorityLevels + " capacity weights: one for each "

View on GitHub (pinned to 2add963021)

Solutions

1. Set ipc.<port>.scheduler.priority.levels to 1 for a single-level queue, or >= 2 for tiered scheduling.
2. Remove the deprecated ipc.<port>.callqueue.priority.levels key entirely — a stale 0 there takes precedence and triggers this too.
3. Search the effective config (hadoop conf -source or 'hdfs getconf -confKey') for priority.levels overrides you forgot about.
4. After fixing, restart the RPC server and confirm startup succeeds.

Example fix

# before
<property><name>ipc.8020.scheduler.priority.levels</name><value>0</value></property>

# after
<property><name>ipc.8020.scheduler.priority.levels</name><value>1</value></property>

int levels = conf.getInt("ipc.8020.scheduler.priority.levels", 4);
if (conf.get("ipc.8020.callqueue.priority.levels") != null) {
  LOG.warn("deprecated key set; value=" + conf.getInt("ipc.8020.callqueue.priority.levels", 0));
}
Preconditions.checkArgument(levels >= 1, "scheduler.priority.levels must be >= 1");

try {
  int levels = parsePriorityLevels(conf);
} catch (IllegalArgumentException e) {
  // config is bad; fail deployment loudly rather than retrying
}

When it happens

Trigger: Explicitly setting ipc.<port>.scheduler.priority.levels (or the deprecated ipc.<port>.callqueue.priority.levels) to 0 or a negative number in core-site.xml / programmatic Configuration; a typo like 'priority.levels=-1' or a property placeholder resolving to 0.

Common situations: Operators trying to disable FairCallQueue tiering by setting levels to 0 (not supported — 1 is the minimum, meaning a single level); configuration management (Ansible/Cloudera templates) defaulting the key to 0; upgrading configs that carried the deprecated key with a bad value (which also logs a deprecation warning).

Related errors

• callqueue.capacity.weights must specify ${priorityLevels} ca
• callqueue.capacity.weights only takes positive weights. ${w}
• ${theClass.getName()} could not be constructed.
• Illegal field separator: ${separator}
• value cannot be blank

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0d8467650869c0c0. Report an issue: GitHub.