{"record":{"id":"5d58bb32c6470796","repo":"apache/hadoop","slug":"request-duration-exceeded-tail-latency-threshold","errorCode":null,"errorMessage":"Request Duration Exceeded Tail Latency Threshold.","messagePattern":"Request Duration Exceeded Tail Latency Threshold\\.","errorType":"exception","errorClass":"TailLatencyRequestTimeoutException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsApacheHttpClient.java","lineNumber":212,"sourceCode":"    RequestConfig.Builder requestConfigBuilder = RequestConfig\n        .custom()\n        .setConnectTimeout(connectTimeout)\n        .setSocketTimeout(readTimeout);\n    httpRequest.setConfig(requestConfigBuilder.build());\n    ExecutorService executor = Executors.newSingleThreadExecutor();\n    Future<HttpResponse> future = executor.submit(() ->\n        httpClient.execute(httpRequest, abfsHttpClientContext)\n    );\n\n    try {\n      return future.get(deadlineMillis, TimeUnit.MILLISECONDS);\n    } catch (TimeoutException e) {\n      /* Deadline exceeded, abort the request.\n       * This will also kill the underlying socket exception in the HttpClient.\n       * Connection will be marked stale and won't be returned back to KAC for reuse.\n       */\n      httpRequest.abort();\n      throw new TailLatencyRequestTimeoutException(e);\n    } catch (Exception e) {\n      // Any other exception from execution should be thrown as IOException.\n      throw new IOException(\"Request execution with deadline failed\", e);\n    } finally {\n      executor.shutdownNow();\n    }\n  }\n\n  /**\n   * Creates the socket factory registry for HTTP and HTTPS.\n   *\n   * @param sslSocketFactory SSL socket factory.\n   * @return Socket factory registry.\n   */\n  private Registry<ConnectionSocketFactory> createSocketFactoryRegistry(\n      ConnectionSocketFactory sslSocketFactory) {\n    if (sslSocketFactory == null) {\n      return RegistryBuilder.<ConnectionSocketFactory>create()","sourceCodeStart":194,"sourceCodeEnd":230,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsApacheHttpClient.java#L194-L230","documentation":"AbfsApacheHttpClient runs each REST call in a per-request executor and enforces a tail-latency deadline via future.get(deadlineMillis). When the Azure call exceeds that deadline the client aborts the request, discards the (stale) connection so it is never reused, and throws TailLatencyRequestTimeoutException (an AzureBlobFileSystemException). This is a deliberate client-side guard against pathological tail latency, driven by the tail-latency tracker configuration (fs.azure.enable.tail.latency.request.timeout, fs.azure.tail.latency.percentile, fs.azure.tail.latency.min.deviation).","triggerScenarios":"Any ABFS REST operation whose round trip exceeds the computed tail-latency deadline: storage throttling (429/503 bursts), slow listings of large directories, network degradation, or aggressively low percentile/min-deviation settings with the feature enabled. Aborts happen before the service response arrives.","commonSituations":"Heavy MapReduce/Spark jobs hitting ABFS throttling limits; VPN/ExpressRoute latency spikes; benchmark-tuned tail-latency defaults applied to high-latency links; transient Azure-side slowness during maintenance windows.","solutions":["Retry the operation — the failure is transient by design and the poisoned connection is already discarded","Relax the guard in core-site.xml: set fs.azure.enable.tail.latency.request.timeout=false, or raise fs.azure.tail.latency.min.deviation / adjust fs.azure.tail.latency.percentile","Check for throttling (ABFS metrics, 429/503 counters) and reduce request concurrency or rate","Investigate the network path (DNS, proxy, VPN MTU) if timeouts cluster at specific times"],"exampleFix":"<!-- before: aggressive tail-latency deadline -->\n<property>\n  <name>fs.azure.enable.tail.latency.request.timeout</name>\n  <value>true</value>\n</property>\n\n<!-- after: disable or soften the deadline for high-latency links -->\n<property>\n  <name>fs.azure.enable.tail.latency.request.timeout</name>\n  <value>false</value>\n</property>","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  return fs.open(path);\n} catch (TailLatencyRequestTimeoutException e) {\n  // deadline guard fired; connection already discarded — safe to retry idempotent ops\n  return retryWithBackoff(() -> fs.open(path), 3);\n}","preventionTips":["Size fs.azure.tail.latency.min.deviation and percentile to your real network profile","Disable fs.azure.enable.tail.latency.request.timeout on high-latency links","Watch ABFS throttling metrics (429/503) — throttling is the most common deadline breaker"],"tags":["azure","abfs","timeout","latency","network","throttling"],"backgroundTag":"request-timeout","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}