{"record":{"id":"846b637a39538f61","repo":"apache/hadoop","slug":"tos-request-interrupted","errorCode":null,"errorMessage":"tos: request interrupted.","messagePattern":"tos: request interrupted\\.","errorType":"exception","errorClass":"TosClientException","httpStatus":null,"severity":"warning","filePath":"hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/tos/DelegationClient.java","lineNumber":1213,"sourceCode":"    int attempt = 0;\n    while (true) {\n      attempt++;\n      try {\n        refresh();\n        return callable.call();\n      } catch (TosException e) {\n        if (attempt >= maxRetryTimes) {\n          LOG.error(\"Retry exhausted after {} times.\", maxRetryTimes);\n          throw e;\n        }\n        if (isRetryableException(e, nonRetryable409ErrorCodes)) {\n          LOG.warn(\"Retry TOS request in the {} times, error: {}\", attempt,\n              Throwables.getRootCause(e).getMessage());\n          try {\n            // last time does not need to sleep\n            Thread.sleep(RetryableUtils.backoff(attempt));\n          } catch (InterruptedException ex) {\n            throw new TosClientException(\"tos: request interrupted.\", ex);\n          }\n        } else {\n          throw e;\n        }\n      } catch (Exception e) {\n        throw new RuntimeException(e);\n      }\n    }\n  }\n\n  @VisibleForTesting\n  static boolean isRetryableException(TosException e, List<String> nonRetryable409ErrorCodes) {\n    return e.getStatusCode() >= HttpStatus.INTERNAL_SERVER_ERROR\n        || e.getStatusCode() == HttpStatus.TOO_MANY_REQUESTS\n        || e.getCause() instanceof SocketException\n        || e.getCause() instanceof UnknownHostException\n        || e.getCause() instanceof SSLException\n        || e.getCause() instanceof SocketTimeoutException","sourceCodeStart":1195,"sourceCodeEnd":1231,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/tos/DelegationClient.java#L1195-L1231","documentation":"DelegationClient.retry() retries TosException up to maxRetryTimes with exponential backoff via Thread.sleep(RetryableUtils.backoff(attempt)). If the thread is interrupted while sleeping between attempts, the InterruptedException is wrapped in TosClientException(\"tos: request interrupted.\") and the original retry-worthy error is lost. The message therefore almost always means the calling thread was cancelled, not that TOS misbehaved.","triggerScenarios":"The thread executing a TOS request is interrupted during backoff: YARN task kill/preemption, FSDataInputStream close from another thread while a request is stuck retrying, shutdown hooks interrupting workers, or explicit Thread.interrupt() by a framework.","commonSituations":"Killed or preempted MR/Spark tasks; thread pools sharing the client being shut down; cancellation racing a slow, retrying request.","solutions":["Treat it as cancellation: stop retrying, restore the interrupt flag (Thread.currentThread().interrupt()), and propagate an interrupted/IO error","Do not swallow the interrupt in callers; let the task terminate cleanly","If interrupts are unexpected, identify who interrupts the thread (job cancellation, stream close) and fix that lifecycle instead of the client"],"exampleFix":"// before\ntry {\n  output = client.getObjectV2(input);\n} catch (TosException e) {\n  LOG.error(\"TOS failure\", e);\n}\n\n// after\ntry {\n  output = client.getObjectV2(input);\n} catch (TosClientException e) {\n  if (e.getCause() instanceof InterruptedException) {\n    Thread.currentThread().interrupt();\n    throw new InterruptedIOException(\"TOS request cancelled during retry\");\n  }\n  throw e;\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  output = client.headObject(headObjectV2Input);\n} catch (TosClientException e) {\n  if (e.getCause() instanceof InterruptedException) {\n    Thread.currentThread().interrupt();\n    throw new InterruptedIOException(\"TOS request cancelled during retry backoff\");\n  }\n  throw e;\n}","preventionTips":["Handle interrupts as cancellation: restore the flag, stop retrying","Do not share the calling thread with code that interrupts it mid-request","On task kill paths, expect this exception and shut down cleanly"],"tags":["hadoop-tos","volcengine-tos","retry","thread-interrupt","cancellation"],"backgroundTag":"thread-interrupted","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}