What causes "failed to create new S3 client: %w" in argoproj/argo-workflows?

Read the wrapped inner error to identify the root cause (credentials, endpoint, TLS)

failed to create new S3 client: %w

Error message

failed to create new S3 client: %w

What it means

Raised by the S3 artifact driver's Load when s3Driver.newClient fails to construct an S3 client during artifact download, inside the executor retry backoff. It wraps the underlying client-construction error (bad credentials, bad endpoint, invalid TLS/CA config, etc.). The backoff retries while isTransientS3Err classifies the error as transient; non-transient failures stop retrying.

Source

Thrown at workflow/artifacts/s3/s3.go:197

		}
		// Wrap transport with OpenTelemetry tracing
		opts.Transport = tracing.WrapS3Transport(tr)
	}

	return NewClient(ctx, opts)
}

// Load downloads artifacts from S3 compliant storage
func (s3Driver *ArtifactDriver) Load(ctx context.Context, inputArtifact *wfv1.Artifact, path string) error {
	ctx, cancel := context.WithCancel(ctx)
	defer cancel()
	log := logging.RequireLoggerFromContext(ctx)
	err := waitutil.Backoff(executorretry.ExecutorRetry(ctx),
		func() (bool, error) {
			log.WithFields(logging.Fields{"path": path, "key": inputArtifact.S3.Key}).Info(ctx, "S3 Load")
			s3cli, err := s3Driver.newClient(ctx)
			if err != nil {
				return !isTransientS3Err(ctx, err), fmt.Errorf("failed to create new S3 client: %w", err)
			}
			return loadS3Artifact(ctx, s3cli, inputArtifact, path)
		})

	return err
}

// loadS3Artifact downloads artifacts from an S3 compliant storage
// returns true if the download is completed or can't be retried (non-transient error)
// returns false if it can be retried (transient error)
func loadS3Artifact(ctx context.Context, s3cli Client, inputArtifact *wfv1.Artifact, path string) (bool, error) {
	origErr := s3cli.GetFile(inputArtifact.S3.Bucket, inputArtifact.S3.Key, path)
	if origErr == nil {
		return true, nil
	}
	if !IsS3ErrCode(origErr, "NoSuchKey") {
		return !isTransientS3Err(ctx, origErr), fmt.Errorf("failed to get file: %w", origErr)
	}

View on GitHub (pinned to 35bff19146)

Solutions

Read the wrapped inner error to identify the root cause (credentials, endpoint, TLS)
Verify the S3 credential secret exists and its keys match accessKeySecret/secretKeySecret in the artifact spec
Check endpoint URL, insecure flag, and addressing style for self-hosted S3 (MinIO etc.)
Ensure s3TrustedCA / TLS config: CA secret mounted and PEM valid if Secure with custom CA
If the error is transient (e.g. temporary credential-fetch failure), the executor retries automatically — check IRSA/service account configuration for persistent failures

Example fix

# before: secret key name typo
s3:
  accessKeySecret: {name: s3-creds, key: accessKey}
  secretKeySecret: {name: s3-creds, key: secretKey}
# after: correct key names matching the secret
s3:
  accessKeySecret: {name: s3-creds, key: accesskey}
  secretKeySecret: {name: s3-creds, key: secretkey}

Defensive patterns

Strategy: retry

Validate before calling

// validate S3 artifact config before submission
if art.S3 == nil || art.S3.Bucket == "" || art.S3.Endpoint == "" {
    return fmt.Errorf("s3 artifact must define bucket and endpoint")
}
if art.S3.AccessKeySecret != nil {
    if _, err := kubeClient.CoreV1().Secrets(ns).Get(ctx, art.S3.AccessKeySecret.Name, metav1.GetOptions{}); err != nil {
        return fmt.Errorf("s3 credential secret missing: %w", err)
    }
}

Try / catch

err := driver.Load(ctx, artifact, path)
if err != nil {
    if strings.Contains(err.Error(), "failed to create new S3 client") {
        // non-transient init failure: do NOT hot-retry blindly;
        // fix credentials/endpoint/TLS then re-run the step
    }
    return err
}

Prevention

Validate S3 credential secrets exist and key names match before submitting workflows
Test endpoint/insecure/addressing-style settings against your S3 provider (MinIO vs AWS)
Mount and verify custom CA certs when using Secure with s3TrustedCA
Keep transient-retry behavior in mind: isTransientS3Err controls backoff, so fix root causes rather than escalating retries

When it happens

Trigger: Executor downloading an S3 input artifact where the S3 client cannot be created: invalid accessKeySecret/secretKeySecret references, malformed endpoint URL, unreadable or invalid trusted CA (s3Driver.TrustedCA PEM), unsupported addressing style, or env/IRSA credential misconfiguration.

Common situations: Typo'd or missing Kubernetes secret keys for S3 credentials; self-hosted MinIO with wrong endpoint or missing insecure flag; custom CA not mounted into the executor pod; wrong s3AddressingStyle for the provider; region mismatch causing client init failure.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/c825763fbb8a7a07. Report an issue: GitHub.