{"record":{"id":"3ec2e691f117b489","repo":"apache/hadoop","slug":"failed-to-acquire-a-sas-token-for-s-on-s-due-to","errorCode":null,"errorMessage":"Failed to acquire a SAS token for %s on %s due to %s","messagePattern":"Failed to acquire a SAS token for (.+?) on (.+?) due to (.+?)","errorType":"exception","errorClass":"SASTokenProviderException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java","lineNumber":1181,"sourceCode":"          sasToken = sasTokenProvider.getSASToken(this.accountName,\n              this.filesystem, path, operation);\n          if ((sasToken == null) || sasToken.isEmpty()) {\n            throw new UnsupportedOperationException(\"SASToken received is empty or null\");\n          }\n        } else {\n          sasToken = cachedSasToken;\n          LOG.trace(\"Using cached SAS token.\");\n        }\n\n        // if SAS Token contains a prefix of ?, it should be removed\n        if (sasToken.charAt(0) == '?') {\n          sasToken = sasToken.substring(1);\n        }\n\n        queryBuilder.setSASToken(sasToken);\n        LOG.trace(\"SAS token fetch complete for {} on {}\", operation, path);\n      } catch (Exception ex) {\n        throw new SASTokenProviderException(String.format(\n            \"Failed to acquire a SAS token for %s on %s due to %s\", operation, path,\n            ex.toString()), ex);\n      }\n    }\n    return sasToken;\n  }\n\n  /**\n   * Creates REST operation URL with empty path for the given query.\n   * @param query to be added to the URL.\n   * @return URL for the REST operation.\n   * @throws AzureBlobFileSystemException if URL creation fails.\n   */\n  @VisibleForTesting\n  protected URL createRequestUrl(final String query) throws AzureBlobFileSystemException {\n    return createRequestUrl(EMPTY_STRING, query);\n  }\n","sourceCodeStart":1163,"sourceCodeEnd":1199,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java#L1163-L1199","documentation":"The catch-all for SAS token acquisition: any exception thrown while fetching or applying the SAS token for a given operation/path is wrapped in SASTokenProviderException with this formatted message. The fault is in the SASTokenProvider implementation or its backing secret store, not in Azure Storage itself — inspect the chained cause for the real error.","triggerScenarios":"The provider throws during getSASToken (KeyVault unreachable, missing secret, auth failure, timeout), or the inner empty-token UnsupportedOperationException is re-wrapped here; also a malformed token whose '?' stripping or query application fails.","commonSituations":"Custom providers hitting KeyVault outages or RBAC permission loss; expired service principals used to fetch tokens; network rules blocking the secret store from executor nodes; typos in fs.azure.account.sastokenprovider class name causing instantiation errors surfaced on first use.","solutions":["Read the cause (getCause()/toString() in the message) — it names the actual failure inside the provider.","Verify the provider class configured for fs.azure.account.sastokenprovider exists, is instantiable, and its dependencies are on the classpath of every node.","Check credentials and network access from all nodes to the secret store backing the provider.","Add retry/caching inside the provider for transient secret-store failures so single blips don't fail Hadoop jobs."],"exampleFix":"// before: provider bubbles raw KeyVault SDK errors to AbfsClient\n// after: provider handles transient failures itself\npublic String getSASToken(String account, String fs, String path, String op)\n    throws SASTokenProviderException {\n  return retryOnTransient(3,\n      () -> keyVault.getSecret(sasSecretName(account, op)),\n      ex -> ex instanceof SecretNotFoundException == false);\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (SASTokenProviderException ex) {\n  Throwable cause = ex.getCause();\n  if (isTransientSecretStoreError(cause)) { retryWithBackoff(op); }\n  else { alert(\"SAS provider failure: \" + cause); throw ex; }\n}","preventionTips":["Cache tokens inside the provider and refresh ahead of expiry to avoid per-call secret-store round trips.","Verify provider class and its dependencies ship on every node's classpath.","Add health checks for the KeyVault/secret backing the provider.","Distinguish transient vs permanent causes in the provider and only surface transients as retryable."],"tags":["azure","abfs","sas","auth","token-provider","keyvault"],"backgroundTag":"sas-token-provider-failure","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}