{"record":{"id":"e9e0746bfe2855fd","repo":"apache/hadoop","slug":"status-code-429-e-getcause","errorCode":null,"errorMessage":"status code 429 !!!\" + e.getCause()","messagePattern":"status code 429 !!!\" \\+ e\\.getCause\\(\\)","errorType":"exception","errorClass":"BosHotObjectException","httpStatus":429,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosClientProxyImpl.java","lineNumber":539,"sourceCode":"    } else if (BOS_REQUEST_LIMIT_CODE\n        == e.getStatusCode()\n        && (e.getErrorCode() == null\n            || e.getErrorCode().trim()\n                .equals(\"null\"))) {\n      throw new BandwidthLimitException(\n          new IOException(\n              \"trigger bos rate limit\"\n                  + \" for too many requests !!!\"));\n    } else if (BOS_REQUEST_LIMIT_CODE\n        == e.getStatusCode()\n        && e.getErrorCode().trim()\n            .equals(\"RequestRateLimitExceeded\")) {\n      throw new BosHotObjectException(\n          new IOException(\n              \"trigger bos object rate limit !!!\"));\n    } else if (BOS_REQUEST_LIMIT_CODE\n        == e.getStatusCode()) {\n      throw new BosHotObjectException(\n          new IOException(\n              \"status code 429 !!!\" + e.getCause()));\n    } else if (e.getCause() instanceof IOException) {\n      throw (IOException) e.getCause();\n    } else {\n      LOG.debug(\n          \"BOS Error code: {}; BOS Error message: {}\",\n          e.getErrorCode(), e.getErrorMessage());\n      if (5 == (e.getStatusCode() / 100)) {\n        throw new BosServerException(e);\n      }\n      throw new BosException(e);\n    }\n  }\n\n  /** {@inheritDoc} */\n  public boolean isHierarchyBucket(String bucketName)\n      throws IOException {","sourceCodeStart":521,"sourceCodeEnd":557,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosClientProxyImpl.java#L521-L557","documentation":"The catch-all throttling branch of the 429 dispatch in BosClientProxyImpl: BOS returned HTTP 429 but the error code is absent or something other than 'RequestRateLimitExceeded' (and not the null-code bandwidth case), so the request is wrapped in BosHotObjectException with message 'status code 429 !!!' + e.getCause(). Note e.getCause() is frequently null, so the message often reads 'status code 429 !!!null' with no detail.","triggerScenarios":"Bucket- or account-level throttling (not a single object) where BOS returns 429 with a different or missing error-code string; also reached when the error code is non-null but blank after trim().","commonSituations":"Bucket-wide QPS exceeded during large listing/copy jobs; mixed read+write spikes; BOS-side throttling during maintenance windows; SDK/server versions that return different error-code strings than the connector expects.","solutions":["Enable exponential-backoff retries on the BOS client so transient 429s are absorbed before reaching this branch","Reduce request concurrency: lower mapreduce/Spark parallelism or connector upload-thread count against the same bucket","Check Baidu Cloud monitoring for the bucket's QPS/throttling metrics at the failure time to confirm bucket-level throttling","Request a quota increase for the bucket if the sustained request rate is legitimate"],"exampleFix":"// before\nfs.bos.retry.max=3\nfs.bos.retry.interval.seconds=1\n\n// after: larger budget, backoff doubles each attempt\nfs.bos.retry.max=8\nfs.bos.retry.interval.seconds=2","handlingStrategy":"retry","validationCode":null,"typeGuard":"static boolean isThrottled429(IOException e) {\n  return e instanceof org.apache.hadoop.fs.bos.exceptions.BosHotObjectException\n      || (e.getMessage() != null && e.getMessage().contains(\"429\"));\n}","tryCatchPattern":"catch (IOException e) {\n  if (isThrottled429(e)) {\n    // exponential backoff and retry the whole operation; surface only if budget exhausted\n  } else { throw e; }\n}","preventionTips":["Cap concurrent BOS requests per JVM below bucket QPS","Configure client retries with backoff for unspecified 429s","Watch bucket-level throttle metrics, not just per-object metrics"],"tags":["bos","rate-limit","http-429","throttling","cloud-storage","hadoop"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}