apache/hadoop · error · UncheckedIOException

Error while tracking Duration of an AbfsRestOperation call

Error message

Error while tracking Duration of an AbfsRestOperation call

What it means

AbfsRestOperation.execute runs the HTTP operation inside IOStatisticsBinding.trackDurationOfInvocation; completeExecute only declares AzureBlobFileSystemException, so any other IOException escaping that lambda is wrapped as UncheckedIOException("Error while tracking Duration of an AbfsRestOperation call", e). AzureBlobFileSystemExceptions are rethrown untouched — this wrapper appears only for unexpected IOExceptions on the per-call path (request construction/signing/audit or interruption), and the original IOException is always attached as the cause.

Source

Thrown at hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java:308

      throws AzureBlobFileSystemException {
    // Since this might be a sub-sequential or parallel rest operation
    // triggered by a single file system call, using a new tracing context.
    lastUsedTracingContext = createNewTracingContext(tracingContext);
    try {
      if (abfsCounters != null) {
        abfsCounters.getLastExecutionTime().set(now());
      }
      if (client.getAbfsMetricsManager() != null) {
        client.getAbfsMetricsManager()
            .timerOrchestrator(TimerFunctionality.RESUME, null);
      }
      IOStatisticsBinding.trackDurationOfInvocation(abfsCounters,
          AbfsStatistic.getStatNameFromHttpCall(method),
          () -> completeExecute(lastUsedTracingContext));
    } catch (AzureBlobFileSystemException aze) {
      throw aze;
    } catch (IOException e) {
      throw new UncheckedIOException("Error while tracking Duration of an "
          + "AbfsRestOperation call", e);
    }
  }

  /**
   * Executes the REST operation with retry, by issuing one or more
   * HTTP operations.
   * @param tracingContext TracingContext instance to track correlation IDs
   */
  void completeExecute(TracingContext tracingContext)
      throws AzureBlobFileSystemException {
    // see if we have latency reports from the previous requests
    String latencyHeader = getClientLatency();
    if (latencyHeader != null && !latencyHeader.isEmpty()) {
      AbfsHttpHeader httpHeader =
              new AbfsHttpHeader(HttpHeaderConfigurations.X_MS_ABFS_CLIENT_LATENCY, latencyHeader);
      requestHeaders.add(httpHeader);
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Unwrap the cause: catch UncheckedIOException and inspect getCause() — the real IOException names the actual problem
2. Fix the underlying I/O issue it names (TLS config, proxy, interruption, request size)
3. Avoid interrupting threads that are executing ABFS operations; shut pools down gracefully
4. If the cause is empty or the failure reproduces deterministically, collect debug logs and check for / file a hadoop-azure JIRA; upgrade to a patched release

Example fix

// before
try {
  fs.getFileStatus(path);
} catch (UncheckedIOException e) {
  throw e;   // cause (the real IOException) never inspected
}

// after
try {
  fs.getFileStatus(path);
} catch (UncheckedIOException e) {
  IOException cause = (IOException) e.getCause();
  log.error("ABFS REST call failed: {}", cause.getMessage(), cause);
  // handle the underlying condition (retry on transient IO, fix TLS, etc.)
}

try {
  fs.getFileStatus(path);
} catch (UncheckedIOException e) {
  IOException cause = (IOException) e.getCause();   // the real failure
  log.error("ABFS REST call failed: {}", cause, cause);
  // branch on the actual cause: TLS/proxy config, interruption, transient IO
}

When it happens

Trigger: An IOException that is not translated by the ABFS retry/error handling escaping completeExecute: thread interrupted during the REST call, SSL/TLS handshake failures raised outside retry handling, malformed request-building failures, or bugs in the statistics/tracking path itself.

Common situations: Thread pools shutting down and interrupting in-flight ABFS calls; restrictive TLS configurations on the JVM; upgrading hadoop-azure and hitting a regression in the REST/statistics path; the underlying IO error being masked by the wrapper so the real cause goes unread.

Related errors

• mark()/reset() not supported on this stream
• Lease desired but no lease threads configured, set fs.azure.
• - account name is not fully qualified.
• Exception while initializing metric credentials
• MD5 algorithm not available

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7e0873ed8ba25e28. Report an issue: GitHub.