apache/hadoop · error · TailLatencyRequestTimeoutException

Request Duration Exceeded Tail Latency Threshold.

Error message

Request Duration Exceeded Tail Latency Threshold.

What it means

AbfsApacheHttpClient runs each REST call in a per-request executor and enforces a tail-latency deadline via future.get(deadlineMillis). When the Azure call exceeds that deadline the client aborts the request, discards the (stale) connection so it is never reused, and throws TailLatencyRequestTimeoutException (an AzureBlobFileSystemException). This is a deliberate client-side guard against pathological tail latency, driven by the tail-latency tracker configuration (fs.azure.enable.tail.latency.request.timeout, fs.azure.tail.latency.percentile, fs.azure.tail.latency.min.deviation).

Solutions

1. Retry the operation — the failure is transient by design and the poisoned connection is already discarded
2. Relax the guard in core-site.xml: set fs.azure.enable.tail.latency.request.timeout=false, or raise fs.azure.tail.latency.min.deviation / adjust fs.azure.tail.latency.percentile
3. Check for throttling (ABFS metrics, 429/503 counters) and reduce request concurrency or rate
4. Investigate the network path (DNS, proxy, VPN MTU) if timeouts cluster at specific times

Example fix

<!-- before: aggressive tail-latency deadline -->
<property>
  <name>fs.azure.enable.tail.latency.request.timeout</name>
  <value>true</value>
</property>

<!-- after: disable or soften the deadline for high-latency links -->
<property>
  <name>fs.azure.enable.tail.latency.request.timeout</name>
  <value>false</value>
</property>

try {
  return fs.open(path);
} catch (TailLatencyRequestTimeoutException e) {
  // deadline guard fired; connection already discarded — safe to retry idempotent ops
  return retryWithBackoff(() -> fs.open(path), 3);
}

When it happens

Trigger: Any ABFS REST operation whose round trip exceeds the computed tail-latency deadline: storage throttling (429/503 bursts), slow listings of large directories, network degradation, or aggressively low percentile/min-deviation settings with the feature enabled. Aborts happen before the service response arrives.

Common situations: Heavy MapReduce/Spark jobs hitting ABFS throttling limits; VPN/ExpressRoute latency spikes; benchmark-tuned tail-latency defaults applied to high-latency links; transient Azure-side slowness during maintenance windows.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• Request execution with deadline failed
• ABFS endpoint is not set correctly
• Both global key and encryption context are set, only one…
• Cannot create file because parent folder does not exist.
• Cannot remove user, group or other entry from access ACL.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5d58bb32c6470796. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsApacheHttpClient.java:212

    RequestConfig.Builder requestConfigBuilder = RequestConfig
        .custom()
        .setConnectTimeout(connectTimeout)
        .setSocketTimeout(readTimeout);
    httpRequest.setConfig(requestConfigBuilder.build());
    ExecutorService executor = Executors.newSingleThreadExecutor();
    Future<HttpResponse> future = executor.submit(() ->
        httpClient.execute(httpRequest, abfsHttpClientContext)
    );

    try {
      return future.get(deadlineMillis, TimeUnit.MILLISECONDS);
    } catch (TimeoutException e) {
      /* Deadline exceeded, abort the request.
       * This will also kill the underlying socket exception in the HttpClient.
       * Connection will be marked stale and won't be returned back to KAC for reuse.
       */
      httpRequest.abort();
      throw new TailLatencyRequestTimeoutException(e);
    } catch (Exception e) {
      // Any other exception from execution should be thrown as IOException.
      throw new IOException("Request execution with deadline failed", e);
    } finally {
      executor.shutdownNow();
    }
  }

  /**
   * Creates the socket factory registry for HTTP and HTTPS.
   *
   * @param sslSocketFactory SSL socket factory.
   * @return Socket factory registry.
   */
  private Registry<ConnectionSocketFactory> createSocketFactoryRegistry(
      ConnectionSocketFactory sslSocketFactory) {
    if (sslSocketFactory == null) {
      return RegistryBuilder.<ConnectionSocketFactory>create()

View on GitHub (pinned to 2add963021)