apache/hadoop · warning · TosClientException

tos: request interrupted.

Error message

tos: request interrupted.

What it means

DelegationClient.retry() retries TosException up to maxRetryTimes with exponential backoff via Thread.sleep(RetryableUtils.backoff(attempt)). If the thread is interrupted while sleeping between attempts, the InterruptedException is wrapped in TosClientException("tos: request interrupted.") and the original retry-worthy error is lost. The message therefore almost always means the calling thread was cancelled, not that TOS misbehaved.

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/tos/DelegationClient.java:1213

    int attempt = 0;
    while (true) {
      attempt++;
      try {
        refresh();
        return callable.call();
      } catch (TosException e) {
        if (attempt >= maxRetryTimes) {
          LOG.error("Retry exhausted after {} times.", maxRetryTimes);
          throw e;
        }
        if (isRetryableException(e, nonRetryable409ErrorCodes)) {
          LOG.warn("Retry TOS request in the {} times, error: {}", attempt,
              Throwables.getRootCause(e).getMessage());
          try {
            // last time does not need to sleep
            Thread.sleep(RetryableUtils.backoff(attempt));
          } catch (InterruptedException ex) {
            throw new TosClientException("tos: request interrupted.", ex);
          }
        } else {
          throw e;
        }
      } catch (Exception e) {
        throw new RuntimeException(e);
      }
    }
  }

  @VisibleForTesting
  static boolean isRetryableException(TosException e, List<String> nonRetryable409ErrorCodes) {
    return e.getStatusCode() >= HttpStatus.INTERNAL_SERVER_ERROR
        || e.getStatusCode() == HttpStatus.TOO_MANY_REQUESTS
        || e.getCause() instanceof SocketException
        || e.getCause() instanceof UnknownHostException
        || e.getCause() instanceof SSLException
        || e.getCause() instanceof SocketTimeoutException

View on GitHub (pinned to 2add963021)

Solutions

1. Treat it as cancellation: stop retrying, restore the interrupt flag (Thread.currentThread().interrupt()), and propagate an interrupted/IO error
2. Do not swallow the interrupt in callers; let the task terminate cleanly
3. If interrupts are unexpected, identify who interrupts the thread (job cancellation, stream close) and fix that lifecycle instead of the client

Example fix

// before
try {
  output = client.getObjectV2(input);
} catch (TosException e) {
  LOG.error("TOS failure", e);
}

// after
try {
  output = client.getObjectV2(input);
} catch (TosClientException e) {
  if (e.getCause() instanceof InterruptedException) {
    Thread.currentThread().interrupt();
    throw new InterruptedIOException("TOS request cancelled during retry");
  }
  throw e;
}

try {
  output = client.headObject(headObjectV2Input);
} catch (TosClientException e) {
  if (e.getCause() instanceof InterruptedException) {
    Thread.currentThread().interrupt();
    throw new InterruptedIOException("TOS request cancelled during retry backoff");
  }
  throw e;
}

When it happens

Trigger: The thread executing a TOS request is interrupted during backoff: YARN task kill/preemption, FSDataInputStream close from another thread while a request is stuck retrying, shutdown hooks interrupting workers, or explicit Thread.interrupt() by a framework.

Common situations: Killed or preempted MR/Spark tasks; thread pools sharing the client being shut down; cancellation racing a slow, retrying request.

Related errors

• Not supported
• Expected checksum is %s while actual checksum is %s
• Object %s doesn't exit
• %s is not appendable because append non-existed object with
• %s is not appendable.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/846b637a39538f61. Report an issue: GitHub.