apache/hadoop · error · BosHotObjectException

status code 429 !!!" + e.getCause()

Error message

status code 429 !!!" + e.getCause()

What it means

The catch-all throttling branch of the 429 dispatch in BosClientProxyImpl: BOS returned HTTP 429 but the error code is absent or something other than 'RequestRateLimitExceeded' (and not the null-code bandwidth case), so the request is wrapped in BosHotObjectException with message 'status code 429 !!!' + e.getCause(). Note e.getCause() is frequently null, so the message often reads 'status code 429 !!!null' with no detail.

Source

Thrown at hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosClientProxyImpl.java:539

    } else if (BOS_REQUEST_LIMIT_CODE
        == e.getStatusCode()
        && (e.getErrorCode() == null
            || e.getErrorCode().trim()
                .equals("null"))) {
      throw new BandwidthLimitException(
          new IOException(
              "trigger bos rate limit"
                  + " for too many requests !!!"));
    } else if (BOS_REQUEST_LIMIT_CODE
        == e.getStatusCode()
        && e.getErrorCode().trim()
            .equals("RequestRateLimitExceeded")) {
      throw new BosHotObjectException(
          new IOException(
              "trigger bos object rate limit !!!"));
    } else if (BOS_REQUEST_LIMIT_CODE
        == e.getStatusCode()) {
      throw new BosHotObjectException(
          new IOException(
              "status code 429 !!!" + e.getCause()));
    } else if (e.getCause() instanceof IOException) {
      throw (IOException) e.getCause();
    } else {
      LOG.debug(
          "BOS Error code: {}; BOS Error message: {}",
          e.getErrorCode(), e.getErrorMessage());
      if (5 == (e.getStatusCode() / 100)) {
        throw new BosServerException(e);
      }
      throw new BosException(e);
    }
  }

  /** {@inheritDoc} */
  public boolean isHierarchyBucket(String bucketName)
      throws IOException {

View on GitHub (pinned to 2add963021)

Solutions

1. Enable exponential-backoff retries on the BOS client so transient 429s are absorbed before reaching this branch
2. Reduce request concurrency: lower mapreduce/Spark parallelism or connector upload-thread count against the same bucket
3. Check Baidu Cloud monitoring for the bucket's QPS/throttling metrics at the failure time to confirm bucket-level throttling
4. Request a quota increase for the bucket if the sustained request rate is legitimate

Example fix

// before
fs.bos.retry.max=3
fs.bos.retry.interval.seconds=1

// after: larger budget, backoff doubles each attempt
fs.bos.retry.max=8
fs.bos.retry.interval.seconds=2

static boolean isThrottled429(IOException e) {
  return e instanceof org.apache.hadoop.fs.bos.exceptions.BosHotObjectException
      || (e.getMessage() != null && e.getMessage().contains("429"));
}

catch (IOException e) {
  if (isThrottled429(e)) {
    // exponential backoff and retry the whole operation; surface only if budget exhausted
  } else { throw e; }
}

When it happens

Trigger: Bucket- or account-level throttling (not a single object) where BOS returns 429 with a different or missing error-code string; also reached when the error code is non-null but blank after trim().

Common situations: Bucket-wide QPS exceeded during large listing/copy jobs; mixed read+write spikes; BOS-side throttling during maintenance windows; SDK/server versions that return different error-code strings than the connector expects.

Understand the failure class

• HTTP status errors: handling 4xx and 5xx responses — how to handle 4xx and 5xx responses properly.

Related errors

• RequestRateLimitExceeded
• Invalid read parameters: buf.length=%d, off=%d, len=%d
• Retry " + retry + " times to read still exception: " + error
• Thread interrupted during retry
• Cannot seek to a negative offset " + targetPos

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e9e0746bfe2855fd. Report an issue: GitHub.