{"record":{"id":"df0b07f8ff630cf3","repo":"apache/hadoop","slug":"requestratelimitexceeded","errorCode":"RequestRateLimitExceeded","errorMessage":"trigger bos object rate limit !!!","messagePattern":"trigger bos object rate limit !!!","errorType":"exception","errorClass":"BosHotObjectException","httpStatus":429,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosClientProxyImpl.java","lineNumber":534,"sourceCode":"      throw new SessionTokenExpireException(e);\n    } else if (400 == e.getStatusCode()) {\n      LOG.error(\"request unknown error : {}\",\n          e.getCause());\n      throw new SessionTokenExpireException(e);\n    } else if (BOS_REQUEST_LIMIT_CODE\n        == e.getStatusCode()\n        && (e.getErrorCode() == null\n            || e.getErrorCode().trim()\n                .equals(\"null\"))) {\n      throw new BandwidthLimitException(\n          new IOException(\n              \"trigger bos rate limit\"\n                  + \" for too many requests !!!\"));\n    } else if (BOS_REQUEST_LIMIT_CODE\n        == e.getStatusCode()\n        && e.getErrorCode().trim()\n            .equals(\"RequestRateLimitExceeded\")) {\n      throw new BosHotObjectException(\n          new IOException(\n              \"trigger bos object rate limit !!!\"));\n    } else if (BOS_REQUEST_LIMIT_CODE\n        == e.getStatusCode()) {\n      throw new BosHotObjectException(\n          new IOException(\n              \"status code 429 !!!\" + e.getCause()));\n    } else if (e.getCause() instanceof IOException) {\n      throw (IOException) e.getCause();\n    } else {\n      LOG.debug(\n          \"BOS Error code: {}; BOS Error message: {}\",\n          e.getErrorCode(), e.getErrorMessage());\n      if (5 == (e.getStatusCode() / 100)) {\n        throw new BosServerException(e);\n      }\n      throw new BosException(e);\n    }","sourceCodeStart":516,"sourceCodeEnd":552,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosClientProxyImpl.java#L516-L552","documentation":"Thrown as BosHotObjectException (an IOException subclass) when Baidu BOS answers a request with HTTP 429 (BOS_REQUEST_LIMIT_CODE) and error code 'RequestRateLimitExceeded'. It marks a per-object (hot key) request-rate limit, distinct from the account bandwidth cap that raises BandwidthLimitException in the branch just above. Because it is a dedicated exception type, upper-layer retry policies can special-case it.","triggerScenarios":"Any BosClientProxy operation on one key whose request rate exceeds that object's QPS quota: getObject from many mappers on the same file, repeated getObjectMetadata/list on one prefix, or upload parts hammering one upload session. BOS returns 429 + errorCode 'RequestRateLimitExceeded' and the proxy translates it here.","commonSituations":"Many MR/Spark tasks reading the same small 'hot' file (dict, index, small side table); tight polling loops on one key; skewed Hive/HBase partitions; scaling the cluster up without changing file layout; default connector config with no backoff on 429.","solutions":["Configure the BOS client retry policy with exponential backoff and jitter so 429s are retried with growing delay instead of surfacing","Eliminate the hot object: split it into more, smaller objects or partition the data so concurrent tasks hit different keys","Reduce per-task request rate: increase readahead/block size or buffer locally so fewer GETs are issued per byte read","Cache frequently-read objects (job distributed cache, Alluxio, local FileSystem cache) so BOS is not re-hit per task","Check the Baidu Cloud console for bucket/object QPS quotas and request a raise if the workload legitimately needs it"],"exampleFix":"// before: every task opens the same small file\nfor (task : tasks) { fs.open(new Path(\"bos://bucket/hot/dict.txt\")); }\n\n// after: pre-partition or cache so each key is read once\n// distcp --input-format ... or copy to HDFS/local cache at job setup\njob.addCacheFile(new Path(\"bos://bucket/hot/dict.txt\").toUri());","handlingStrategy":"retry","validationCode":null,"typeGuard":"static boolean isBosRateLimit(IOException e) {\n  if (e instanceof org.apache.hadoop.fs.bos.exceptions.BosHotObjectException\n      || e instanceof org.apache.hadoop.fs.bos.exceptions.BandwidthLimitException) {\n    return true;\n  }\n  String m = e.getMessage();\n  return m != null && m.contains(\"rate limit\");\n}","tryCatchPattern":"long delayMs = 500;\nfor (int attempt = 0; attempt < 5; attempt++) {\n  try {\n    return readObject(key);\n  } catch (IOException e) {\n    if (!isBosRateLimit(e) || attempt == 4) throw e;\n    Thread.sleep(delayMs + new Random().nextInt(200)); // backoff + jitter\n    delayMs *= 2;\n  }\n}","preventionTips":["Cache or replicate hot objects so per-key QPS stays under quota","Set exponential-backoff retry on the connector instead of failing on first 429","Monitor BOS throttling metrics and size cluster parallelism to the bucket quota"],"tags":["bos","rate-limit","http-429","hot-object","cloud-storage","hadoop"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}