apache/hadoop · error · TailLatencyRequestTimeoutException

Request Duration Exceeded Tail Latency Threshold.

Error message

Request Duration Exceeded Tail Latency Threshold.

What it means

AbfsApacheHttpClient runs each REST call in a per-request executor and enforces a tail-latency deadline via future.get(deadlineMillis). When the Azure call exceeds that deadline the client aborts the request, discards the (stale) connection so it is never reused, and throws TailLatencyRequestTimeoutException (an AzureBlobFileSystemException). This is a deliberate client-side guard against pathological tail latency, driven by the tail-latency tracker configuration (fs.azure.enable.tail.latency.request.timeout, fs.azure.tail.latency.percentile, fs.azure.tail.latency.min.deviation).

Source

Thrown at hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsApacheHttpClient.java:212

    RequestConfig.Builder requestConfigBuilder = RequestConfig
        .custom()
        .setConnectTimeout(connectTimeout)
        .setSocketTimeout(readTimeout);
    httpRequest.setConfig(requestConfigBuilder.build());
    ExecutorService executor = Executors.newSingleThreadExecutor();
    Future<HttpResponse> future = executor.submit(() ->
        httpClient.execute(httpRequest, abfsHttpClientContext)
    );

    try {
      return future.get(deadlineMillis, TimeUnit.MILLISECONDS);
    } catch (TimeoutException e) {
      /* Deadline exceeded, abort the request.
       * This will also kill the underlying socket exception in the HttpClient.
       * Connection will be marked stale and won't be returned back to KAC for reuse.
       */
      httpRequest.abort();
      throw new TailLatencyRequestTimeoutException(e);
    } catch (Exception e) {
      // Any other exception from execution should be thrown as IOException.
      throw new IOException("Request execution with deadline failed", e);
    } finally {
      executor.shutdownNow();
    }
  }

  /**
   * Creates the socket factory registry for HTTP and HTTPS.
   *
   * @param sslSocketFactory SSL socket factory.
   * @return Socket factory registry.
   */
  private Registry<ConnectionSocketFactory> createSocketFactoryRegistry(
      ConnectionSocketFactory sslSocketFactory) {
    if (sslSocketFactory == null) {
      return RegistryBuilder.<ConnectionSocketFactory>create()

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the operation — the failure is transient by design and the poisoned connection is already discarded
2. Relax the guard in core-site.xml: set fs.azure.enable.tail.latency.request.timeout=false, or raise fs.azure.tail.latency.min.deviation / adjust fs.azure.tail.latency.percentile
3. Check for throttling (ABFS metrics, 429/503 counters) and reduce request concurrency or rate
4. Investigate the network path (DNS, proxy, VPN MTU) if timeouts cluster at specific times

Example fix

<!-- before: aggressive tail-latency deadline -->
<property>
  <name>fs.azure.enable.tail.latency.request.timeout</name>
  <value>true</value>
</property>

<!-- after: disable or soften the deadline for high-latency links -->
<property>
  <name>fs.azure.enable.tail.latency.request.timeout</name>
  <value>false</value>
</property>

try {
  return fs.open(path);
} catch (TailLatencyRequestTimeoutException e) {
  // deadline guard fired; connection already discarded — safe to retry idempotent ops
  return retryWithBackoff(() -> fs.open(path), 3);
}

When it happens

Trigger: Any ABFS REST operation whose round trip exceeds the computed tail-latency deadline: storage throttling (429/503 bursts), slow listings of large directories, network degradation, or aggressively low percentile/min-deviation settings with the feature enabled. Aborts happen before the service response arrives.

Common situations: Heavy MapReduce/Spark jobs hitting ABFS throttling limits; VPN/ExpressRoute latency spikes; benchmark-tuned tail-latency defaults applied to high-latency links; transient Azure-side slowness during maintenance windows.

Related errors

• Request execution with deadline failed
• {} millis timeout while waiting for channel to be ready for
• WASB Driver using wasb(s) schema is no longer supported. Ins
• "%s" must be set for user-bound SAS auth type.
• Unable to load user-bound SAS token provider class: {e}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5d58bb32c6470796. Report an issue: GitHub.