{"record":{"id":"9ad9c1708ae13438","repo":"argoproj/argo-workflows","slug":"unable-to-download-blob-s-w","errorCode":null,"errorMessage":"unable to download blob %s: %w","messagePattern":"unable to download blob (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/artifacts/azure/azure.go","lineNumber":142,"sourceCode":"\t// has HNS enabled (ADLS Gen 2), then there's an edge case with using the blob API to\n\t// access. The directory will be returned as an empty file, so check for that as well.\n\tvar isEmptyFile bool\n\torigErr := DownloadFile(ctx, containerClient, artifact.Azure.Blob, path)\n\tif origErr == nil {\n\t\tfileInfo, lstatErr := os.Lstat(path)\n\t\tif lstatErr != nil {\n\t\t\treturn fmt.Errorf(\"unable to retrieve stats for downloaded file %s: %w\", path, lstatErr)\n\t\t}\n\n\t\t// Empty file means it could be an ADLS Gen 2 account and we downloaded the\n\t\t// directory as an empty file -- we'll check below. If it's a non-empty file,\n\t\t// then we successfully downloaded a file blob.\n\t\tif fileInfo.Size() > 0 {\n\t\t\treturn nil\n\t\t}\n\t\tisEmptyFile = true\n\t} else if !bloberror.HasCode(origErr, bloberror.BlobNotFound) {\n\t\treturn fmt.Errorf(\"unable to download blob %s: %w\", artifact.Azure.Blob, origErr)\n\t}\n\n\tisDir, err := azblobDriver.IsDirectory(ctx, artifact)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"unable to determine if %s is a directory: %w\", artifact.Azure.Blob, err)\n\t}\n\n\t// It's not a directory and the file doesn't exist, Return the original NoSuchKey error.\n\tif !isDir && !isEmptyFile {\n\t\treturn argoerrors.New(argoerrors.CodeNotFound, origErr.Error())\n\t}\n\n\t// When we tried to download the blob as a file, we created an empty file for the\n\t// blob as a target. We need to delete that empty file so we can re-create as a directory.\n\terr = os.Remove(path)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"unable to remove attempted file download %s: %w\", path, err)\n\t}","sourceCodeStart":124,"sourceCodeEnd":160,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/artifacts/azure/azure.go#L124-L160","documentation":"This error wraps a non-BlobNotFound failure returned by the Azure SDK when DownloadFile tries to fetch the artifact blob as a single file during Load. The driver only tolerates BlobNotFound (it then probes for a directory); any other SDK error (auth, network, 403, container missing) surfaces here. It means the blob download itself failed for a reason other than the blob simply not existing.","triggerScenarios":"artifact.Azure.Blob exists as a key but blobClient.DownloadFile fails with e.g. AuthorizationFailure (403), ContainerNotFound (404 on container), InvalidAuthenticationInfo, SharedKeyCredential signature mismatch, network timeouts, or ADLS Gen2 HNS errors other than BlobNotFound.","commonSituations":"Wrong/mis-encoded accountKey secret, using a SAS token without the ? prefix being detected, container name typo (container not found), storage firewall blocking argoexec pods, clock skew breaking SharedKey signature, endpoint hostname not matching the account in the key.","solutions":["Check the wrapped origErr code in the message: fix the specific cause (403 => credentials, 404 on container => container name, network => egress/firewall).","Verify the azure accountKey secret matches the endpoint's storage account and that the key was not rotated.","If using useSDKCreds: true, ensure pod identity/workload identity federation is configured and the managed identity has Storage Blob Data Reader on the container.","Verify artifact.Azure.Endpoint is the full account endpoint (e.g. https://<account>.blob.core.windows.net/) and DNS-resolvable from the workflow pod.","Retry the workflow if the wrapped error is a transient network/timeout error; the SDK already retries but prolonged outages still fail."],"exampleFix":"// before: key rotated, stale secret\n# kubectl get secret my-azure-creds -o jsonpath='{.data.accountKey}'\n// after: refresh the secret with the current key\naz storage account keys renew -n myaccount -g myrg --key key1\nkubectl create secret generic my-azure-creds --from-literal=accountKey=$(az storage account keys list -n myaccount -g myrg --query '[0].value' -o tsv)","handlingStrategy":"try-catch","validationCode":"// pre-check blob readability with the same creds\nhead, err := containerClient.NewBlobClient(artifact.Azure.Blob).GetProperties(ctx, nil)\nif err != nil { return fmt.Errorf(\"precheck failed: %w\", err) }","typeGuard":"func isBlobNotFound(err error) bool {\n\tvar respErr *azcore.ResponseError\n\treturn errors.As(err, &respErr) && respErr.ErrorCode == \"BlobNotFound\"\n}","tryCatchPattern":"err := driver.Load(ctx, artifact, path)\nvar respErr *azcore.ResponseError\nif errors.As(err, &respErr) {\n\tswitch respErr.ErrorCode {\n\tcase \"AuthorizationFailure\":\n\t\t// fix credentials/RBAC\n\tcase \"ContainerNotFound\":\n\t\t// fix container name\n\tdefault:\n\t\tif isTransient(err) { time.Sleep(backoff); retry() }\n\t}\n}","preventionTips":["Validate the accountKey secret round-trips against the same endpoint before submitting workflows","Pin the storage account key rotation to a process that also updates the k8s secret","Allow-list <account>.blob.core.windows.net:443 from cluster egress","Smoke-test with Azurite or az CLI using the identical credentials"],"tags":["azure","blob-storage","authentication","network"],"backgroundTag":"azure-blob-download-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}