apache/hadoop · error · BandwidthLimitException

trigger bos rate limit for too many requests !!!

Error message

trigger bos rate limit for too many requests !!!

What it means

ZombieJob.getTaskInfo(loggedTask) computes aggregate TaskInfo (input/output bytes and records, resource metrics) from a task's successful attempts, but first requires the task type to be exactly MAP or REDUCE — otherwise it throws IllegalArgumentException('getTaskInfo only supports MAP or REDUCE tasks: <type> for task = <id>'). getTaskInfo() is called from many ZombieJob paths (missing attempts, killed attempts, locality scaling), so a non-MAP/REDUCE task record anywhere in a replayed trace can surface this error from several call sites.

Source

Thrown at hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosClientProxyImpl.java:526

    if (400 == e.getStatusCode()
        && (e.getErrorCode() == null
            || "null".equals(e.getErrorCode())
            || "InvalidSessionToken".equals(
                e.getErrorCode().trim()))) {
      LOG.error(
          "sts session expired or invalid,"
              + " need to update client");
      throw new SessionTokenExpireException(e);
    } else if (400 == e.getStatusCode()) {
      LOG.error("request unknown error : {}",
          e.getCause());
      throw new SessionTokenExpireException(e);
    } else if (BOS_REQUEST_LIMIT_CODE
        == e.getStatusCode()
        && (e.getErrorCode() == null
            || e.getErrorCode().trim()
                .equals("null"))) {
      throw new BandwidthLimitException(
          new IOException(
              "trigger bos rate limit"
                  + " for too many requests !!!"));
    } else if (BOS_REQUEST_LIMIT_CODE
        == e.getStatusCode()
        && e.getErrorCode().trim()
            .equals("RequestRateLimitExceeded")) {
      throw new BosHotObjectException(
          new IOException(
              "trigger bos object rate limit !!!"));
    } else if (BOS_REQUEST_LIMIT_CODE
        == e.getStatusCode()) {
      throw new BosHotObjectException(
          new IOException(
              "status code 429 !!!" + e.getCause()));
    } else if (e.getCause() instanceof IOException) {
      throw (IOException) e.getCause();
    } else {

View on GitHub (pinned to 2add963021)

Solutions

1. Filter tasks to MAP/REDUCE before replay: when iterating LoggedJob tasks, skip any whose getTaskType() is neither Values.MAP nor Values.REDUCE.
2. Regenerate or repair the trace so only map/reduce tasks carry simulation-relevant attempts.
3. Wrap ZombieJob in a defensive JobStory that returns a zero TaskInfo (new TaskInfo(0,0,0,0,0,0), the same default ZombieJob uses for null tasks) for non-MAP/REDUCE tasks instead of propagating the exception.

Example fix

// before
TaskInfo info = zombieJob.getTaskInfo(anyLoggedTask); // throws for SETUP

// after
if (anyLoggedTask.getTaskType() == Values.MAP
    || anyLoggedTask.getTaskType() == Values.REDUCE) {
  info = zombieJob.getTaskInfo(anyLoggedTask);
} else {
  info = new TaskInfo(0, 0, 0, 0, 0, 0); // neutral default for setup/cleanup
}

static boolean supportsTaskInfo(LoggedTask t) {
  Values type = (t == null) ? null : t.getTaskType();
  return type == Values.MAP || type == Values.REDUCE;
}

static boolean isMapOrReduce(LoggedTask t) {
  Values v = (t == null) ? null : t.getTaskType();
  return v == Values.MAP || v == Values.REDUCE;
}

When it happens

Trigger: A LoggedTask with taskType SETUP/CLEANUP/null reaching getTaskInfo() — via getMapTaskAttemptInfoAdjusted() when the task exists but an attempt is missing or killed, or via custom JobStory code calling getTaskInfo directly; damaged traces with null taskType.

Common situations: Simulating traces that contain job setup/cleanup tasks; custom simulator harnesses that walk all tasks of a LoggedJob; traces from Hadoop versions that record additional task kinds (e.g. ACTION) unknown to this rumen build.

Understand the failure class

• HTTP status errors: handling 4xx and 5xx responses — how to handle 4xx and 5xx responses properly.

Related errors

• Key '" + key + "' does not exist in BOS
• Failed to rename %s to %s, file already exists or not empty!
• Truncate is not supported by BaiduBosFileSystem
• user accessKey and secretAccessKey should not be null
• NoSuchUpload

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/86a4374949fa2ec0. Report an issue: GitHub.