argoproj/argo-workflows · error

unable to download blob %s: %w

Error message

unable to download blob %s: %w

What it means

This error wraps a non-BlobNotFound failure returned by the Azure SDK when DownloadFile tries to fetch the artifact blob as a single file during Load. The driver only tolerates BlobNotFound (it then probes for a directory); any other SDK error (auth, network, 403, container missing) surfaces here. It means the blob download itself failed for a reason other than the blob simply not existing.

Source

Thrown at workflow/artifacts/azure/azure.go:142

	// has HNS enabled (ADLS Gen 2), then there's an edge case with using the blob API to
	// access. The directory will be returned as an empty file, so check for that as well.
	var isEmptyFile bool
	origErr := DownloadFile(ctx, containerClient, artifact.Azure.Blob, path)
	if origErr == nil {
		fileInfo, lstatErr := os.Lstat(path)
		if lstatErr != nil {
			return fmt.Errorf("unable to retrieve stats for downloaded file %s: %w", path, lstatErr)
		}

		// Empty file means it could be an ADLS Gen 2 account and we downloaded the
		// directory as an empty file -- we'll check below. If it's a non-empty file,
		// then we successfully downloaded a file blob.
		if fileInfo.Size() > 0 {
			return nil
		}
		isEmptyFile = true
	} else if !bloberror.HasCode(origErr, bloberror.BlobNotFound) {
		return fmt.Errorf("unable to download blob %s: %w", artifact.Azure.Blob, origErr)
	}

	isDir, err := azblobDriver.IsDirectory(ctx, artifact)
	if err != nil {
		return fmt.Errorf("unable to determine if %s is a directory: %w", artifact.Azure.Blob, err)
	}

	// It's not a directory and the file doesn't exist, Return the original NoSuchKey error.
	if !isDir && !isEmptyFile {
		return argoerrors.New(argoerrors.CodeNotFound, origErr.Error())
	}

	// When we tried to download the blob as a file, we created an empty file for the
	// blob as a target. We need to delete that empty file so we can re-create as a directory.
	err = os.Remove(path)
	if err != nil {
		return fmt.Errorf("unable to remove attempted file download %s: %w", path, err)
	}

View on GitHub (pinned to 35bff19146)

Solutions

1. Check the wrapped origErr code in the message: fix the specific cause (403 => credentials, 404 on container => container name, network => egress/firewall).
2. Verify the azure accountKey secret matches the endpoint's storage account and that the key was not rotated.
3. If using useSDKCreds: true, ensure pod identity/workload identity federation is configured and the managed identity has Storage Blob Data Reader on the container.
4. Verify artifact.Azure.Endpoint is the full account endpoint (e.g. https://<account>.blob.core.windows.net/) and DNS-resolvable from the workflow pod.
5. Retry the workflow if the wrapped error is a transient network/timeout error; the SDK already retries but prolonged outages still fail.

Example fix

// before: key rotated, stale secret
# kubectl get secret my-azure-creds -o jsonpath='{.data.accountKey}'
// after: refresh the secret with the current key
az storage account keys renew -n myaccount -g myrg --key key1
kubectl create secret generic my-azure-creds --from-literal=accountKey=$(az storage account keys list -n myaccount -g myrg --query '[0].value' -o tsv)

// pre-check blob readability with the same creds
head, err := containerClient.NewBlobClient(artifact.Azure.Blob).GetProperties(ctx, nil)
if err != nil { return fmt.Errorf("precheck failed: %w", err) }

func isBlobNotFound(err error) bool {
	var respErr *azcore.ResponseError
	return errors.As(err, &respErr) && respErr.ErrorCode == "BlobNotFound"
}

err := driver.Load(ctx, artifact, path)
var respErr *azcore.ResponseError
if errors.As(err, &respErr) {
	switch respErr.ErrorCode {
	case "AuthorizationFailure":
		// fix credentials/RBAC
	case "ContainerNotFound":
		// fix container name
	default:
		if isTransient(err) { time.Sleep(backoff); retry() }
	}
}

When it happens

Trigger: artifact.Azure.Blob exists as a key but blobClient.DownloadFile fails with e.g. AuthorizationFailure (403), ContainerNotFound (404 on container), InvalidAuthenticationInfo, SharedKeyCredential signature mismatch, network timeouts, or ADLS Gen2 HNS errors other than BlobNotFound.

Common situations: Wrong/mis-encoded accountKey secret, using a SAS token without the ? prefix being detected, container name typo (container not found), storage firewall blocking argoexec pods, clock skew breaking SharedKey signature, endpoint hostname not matching the account in the key.

Related errors

• unable to open stream for blob %s: %w
• unable to upload stream to Azure blob %s: %w
• unable to test if %s is a directory: %w
• unable to list files in %s: %w
• failed to obtain a credential: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/9ad9c1708ae13438. Report an issue: GitHub.