apache/hadoop · error · IllegalArgumentException

responseTimeThresholds must match with the number of priorit

Error message

responseTimeThresholds must match with the number of priority levels

What it means

parseBackOffResponseTimeThreshold reads <ns>.decay-scheduler.backoff.responsetime.thresholds (via Configuration.getTimeDurations, milliseconds). When response-time backoff is configured, the server needs one average-response-time threshold per priority level to decide when each queue starts rejecting calls; if the array length differs from numLevels it throws IllegalArgumentException at scheduler construction. Empty config is fine (defaults of 10s, 20s, 30s... per level).

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DecayRpcScheduler.java:444

    for (int i = 0; i < ret.length; i++) {
      ret[i] = Math.pow(2, i)/div;
    }
    return ret;
  }

  private static long[] parseBackOffResponseTimeThreshold(String ns,
      Configuration conf, int numLevels) {
    long[] responseTimeThresholds = conf.getTimeDurations(ns + "." +
            IPC_DECAYSCHEDULER_BACKOFF_RESPONSETIME_THRESHOLDS_KEY,
        TimeUnit.MILLISECONDS);
    // backoff thresholds not specified
    if (responseTimeThresholds.length == 0) {
      return getDefaultBackOffResponseTimeThresholds(numLevels);
    }
    // backoff thresholds specified but not match with the levels
    if (responseTimeThresholds.length != numLevels) {
      throw new IllegalArgumentException(
          "responseTimeThresholds must match with the number of priority " +
          "levels");
    }
    // invalid thresholds
    for (long responseTimeThreshold: responseTimeThresholds) {
      if (responseTimeThreshold <= 0) {
        throw new IllegalArgumentException(
            "responseTimeThreshold millis must be >= 0");
      }
    }
    return responseTimeThresholds;
  }

  // 10s for level 0, 20s for level 1, 30s for level 2, ...
  private static long[] getDefaultBackOffResponseTimeThresholds(int numLevels) {
    long[] ret = new long[numLevels];
    for (int i = 0; i < ret.length; i++) {
      ret[i] = 10000*(i+1);

View on GitHub (pinned to 2add963021)

Solutions

1. Set exactly numLevels comma-separated durations, e.g., 4 levels -> '3s,10s,20s,30s'.
2. Or drop the property and rely on the defaults (10s * (level+1)).
3. Keep the count in sync whenever callqueue-priority-levels changes.

Example fix

// before (core-site.xml)
<property><name>ipc.8020.decay-scheduler.backoff.responsetime.enable</name><value>true</value></property>
<property><name>ipc.8020.decay-scheduler.backoff.responsetime.thresholds</name><value>10s,20s</value></property>
<property><name>ipc.8020.callqueue-priority-levels</name><value>4</value></property>
<!-- 2 thresholds for 4 levels -> IllegalArgumentException -->

// after
<property><name>ipc.8020.decay-scheduler.backoff.responsetime.thresholds</name><value>3s,10s,20s,30s</value></property>

int levels = conf.getInt("ipc." + port + ".callqueue-priority-levels", 1);
long[] th = conf.getTimeDurations("ipc." + port + ".decay-scheduler.backoff.responsetime.thresholds", TimeUnit.MILLISECONDS);
if (th.length > 0 && th.length != levels) {
  throw new ConfigException("backoff thresholds need exactly " + levels + " entries, got " + th.length);
}

When it happens

Trigger: Enabling decay-scheduler.backoff.responsetime.enable=true and providing fewer/more thresholds than ipc.<port>.callqueue-priority-levels; changing the level count without updating the thresholds list; mixing duration units ('10s','20s','5s' with 4 levels).

Common situations: Rolling out response-time-based callqueue backoff on a busy NameNode; reusing a 3-level threshold list after raising priority levels to 4; HDFSRouterFed/YARN configs copied between services with different level counts.

Understand the failure class

Background: Config validation failed: what "invalid value for {key}" and settings-rejection errors mean across 19 open-source libraries — this error's family across 19 libraries.

Related errors

• responseTimeThreshold millis must be >= 0
• Number of thresholds should be {}. Was: {}
• Number of Priority Levels must be at least 1
• Decay Factor must be between 0 and 1
• Period millis must be >= 0

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4d4026bd11bd178b. Report an issue: GitHub.