argoproj/argo-workflows · error

failed to check if key %s exists from bucket %s: %w

Error message

failed to check if key %s exists from bucket %s: %w

What it means

Wraps a failure from s3cli.KeyExists when listObjects finds no files under the artifact key and double-checks whether the key itself exists. The KeyExists S3 API call itself errored (as opposed to returning false), so load cannot distinguish missing vs broken; transient errors are retried.

Source

Thrown at workflow/artifacts/s3/s3.go:406

	return files, err
}

// listObjects returns the files inside the directory represented by the Artifact
// returns true if success or can't be retried (non-transient error)
// returns false if it can be retried (transient error)
func listObjects(ctx context.Context, s3cli Client, artifact *wfv1.Artifact) (bool, []string, error) {
	var files []string
	files, err := s3cli.ListDirectory(artifact.S3.Bucket, artifact.S3.Key)
	if err != nil {
		return !isTransientS3Err(ctx, err), files, fmt.Errorf("failed to list directory: %w", err)
	}
	log := logging.RequireLoggerFromContext(ctx)
	log.WithFields(logging.Fields{"bucket": artifact.S3.Bucket, "key": artifact.S3.Key, "files": files}).Debug(ctx, "successfully listing S3 directory")

	if len(files) == 0 {
		directoryExists, err := s3cli.KeyExists(artifact.S3.Bucket, artifact.S3.Key)
		if err != nil {
			return !isTransientS3Err(ctx, err), files, fmt.Errorf("failed to check if key %s exists from bucket %s: %w", artifact.S3.Key, artifact.S3.Bucket, err)
		}
		if !directoryExists {
			return true, files, argoerrs.New(argoerrs.CodeNotFound, fmt.Sprintf("no key found of name %s", artifact.S3.Key))
		}
	}
	return true, files, nil
}

func (s3Driver *ArtifactDriver) IsDirectory(ctx context.Context, artifact *wfv1.Artifact) (bool, error) {
	s3cli, err := s3Driver.newClient(ctx)
	if err != nil {
		return false, err
	}
	return s3cli.IsDirectory(artifact.S3.Bucket, artifact.S3.Key)
}

// Get AWS credentials based on default order from aws SDK
func getAWSCredentials(ctx context.Context, opts ClientOpts) (*credentials.Credentials, error) {

View on GitHub (pinned to 35bff19146)

Solutions

1. Check the wrapped cause (often HeadObject AccessDenied or throttling)
2. Ensure credentials allow s3:GetObject/HeadObject on the key prefix
3. Retry the workflow if the cause is transient throttling — the executor backoff should eventually succeed
4. Verify endpoint/TLS configuration if errors are connection-level

Example fix

// before (IAM)
// only s3:ListBucket granted
// after
{"Effect":"Allow","Action":["s3:ListBucket","s3:GetObject"],"Resource":["arn:aws:s3:::my-bucket","arn:aws:s3:::my-bucket/*"]}

aws --endpoint-url $ENDPOINT s3api head-object --bucket $BUCKET --key $KEY || aws s3api head-bucket --bucket $BUCKET

// treat as possibly-transient: rely on executor backoff; surface cause
if unwrapped := errors.Unwrap(err); unwrapped != nil { log.Printf("KeyExists check failed: %v", unwrapped) }

When it happens

Trigger: After ListDirectory returns an empty file list, listObjects calls s3cli.KeyExists(bucket, key) which fails: AccessDenied on HeadObject, throttling, network error.

Common situations: IAM policy permits ListBucket but denies GetObject/HeadObject; aggressive S3 rate limiting; intermittent connectivity to the endpoint.

Related errors

• new bucket reader: %w
• failed to test if %s is a directory: %w
• failed to get directory: %w
• failed to put directory: %w
• failed to put file: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/64e050edae84ea30. Report an issue: GitHub.