apache/hadoop · error · BosHotObjectException

RequestRateLimitExceeded

RequestRateLimitExceeded

Error message

trigger bos object rate limit !!!

What it means

Thrown as BosHotObjectException (an IOException subclass) when Baidu BOS answers a request with HTTP 429 (BOS_REQUEST_LIMIT_CODE) and error code 'RequestRateLimitExceeded'. It marks a per-object (hot key) request-rate limit, distinct from the account bandwidth cap that raises BandwidthLimitException in the branch just above. Because it is a dedicated exception type, upper-layer retry policies can special-case it.

Source

Thrown at hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosClientProxyImpl.java:534

      throw new SessionTokenExpireException(e);
    } else if (400 == e.getStatusCode()) {
      LOG.error("request unknown error : {}",
          e.getCause());
      throw new SessionTokenExpireException(e);
    } else if (BOS_REQUEST_LIMIT_CODE
        == e.getStatusCode()
        && (e.getErrorCode() == null
            || e.getErrorCode().trim()
                .equals("null"))) {
      throw new BandwidthLimitException(
          new IOException(
              "trigger bos rate limit"
                  + " for too many requests !!!"));
    } else if (BOS_REQUEST_LIMIT_CODE
        == e.getStatusCode()
        && e.getErrorCode().trim()
            .equals("RequestRateLimitExceeded")) {
      throw new BosHotObjectException(
          new IOException(
              "trigger bos object rate limit !!!"));
    } else if (BOS_REQUEST_LIMIT_CODE
        == e.getStatusCode()) {
      throw new BosHotObjectException(
          new IOException(
              "status code 429 !!!" + e.getCause()));
    } else if (e.getCause() instanceof IOException) {
      throw (IOException) e.getCause();
    } else {
      LOG.debug(
          "BOS Error code: {}; BOS Error message: {}",
          e.getErrorCode(), e.getErrorMessage());
      if (5 == (e.getStatusCode() / 100)) {
        throw new BosServerException(e);
      }
      throw new BosException(e);
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Configure the BOS client retry policy with exponential backoff and jitter so 429s are retried with growing delay instead of surfacing
2. Eliminate the hot object: split it into more, smaller objects or partition the data so concurrent tasks hit different keys
3. Reduce per-task request rate: increase readahead/block size or buffer locally so fewer GETs are issued per byte read
4. Cache frequently-read objects (job distributed cache, Alluxio, local FileSystem cache) so BOS is not re-hit per task
5. Check the Baidu Cloud console for bucket/object QPS quotas and request a raise if the workload legitimately needs it

Example fix

// before: every task opens the same small file
for (task : tasks) { fs.open(new Path("bos://bucket/hot/dict.txt")); }

// after: pre-partition or cache so each key is read once
// distcp --input-format ... or copy to HDFS/local cache at job setup
job.addCacheFile(new Path("bos://bucket/hot/dict.txt").toUri());

static boolean isBosRateLimit(IOException e) {
  if (e instanceof org.apache.hadoop.fs.bos.exceptions.BosHotObjectException
      || e instanceof org.apache.hadoop.fs.bos.exceptions.BandwidthLimitException) {
    return true;
  }
  String m = e.getMessage();
  return m != null && m.contains("rate limit");
}

long delayMs = 500;
for (int attempt = 0; attempt < 5; attempt++) {
  try {
    return readObject(key);
  } catch (IOException e) {
    if (!isBosRateLimit(e) || attempt == 4) throw e;
    Thread.sleep(delayMs + new Random().nextInt(200)); // backoff + jitter
    delayMs *= 2;
  }
}

When it happens

Trigger: Any BosClientProxy operation on one key whose request rate exceeds that object's QPS quota: getObject from many mappers on the same file, repeated getObjectMetadata/list on one prefix, or upload parts hammering one upload session. BOS returns 429 + errorCode 'RequestRateLimitExceeded' and the proxy translates it here.

Common situations: Many MR/Spark tasks reading the same small 'hot' file (dict, index, small side table); tight polling loops on one key; skewed Hive/HBase partitions; scaling the cluster up without changing file layout; default connector config with no backoff on 429.

Related errors

• status code 429 !!!" + e.getCause()
• Invalid read parameters: buf.length=%d, off=%d, len=%d
• Retry " + retry + " times to read still exception: " + error
• Thread interrupted during retry
• Cannot seek to a negative offset " + targetPos

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/df0b07f8ff630cf3. Report an issue: GitHub.