argoproj/argo-workflows · error

GCS storage.NewClient with credential: %w

Error message

GCS storage.NewClient with credential: %w

What it means

After credentials parse successfully, newGCSClientWithCredential calls storage.NewClient with those credentials. This error wraps a failure constructing the Cloud Storage client. Because option.WithCredentials is used, failures here usually come from credential fields the client cannot use (e.g. malformed key data that parses but lacks valid fields) or client-option/environment conflicts.

Source

Thrown at workflow/artifacts/gcs/gcs.go:90

	return false
}

func (h *ArtifactDriver) newGCSClient(ctx context.Context) (*storage.Client, error) {
	if h.ServiceAccountKey != "" {
		return newGCSClientWithCredential(ctx, h.ServiceAccountKey)
	}
	// Assume it uses Workload Identity
	return newGCSClientDefault(ctx)
}

func newGCSClientWithCredential(ctx context.Context, serviceAccountJSON string) (*storage.Client, error) {
	creds, err := google.CredentialsFromJSONWithType(ctx, []byte(serviceAccountJSON), google.ServiceAccount, storage.ScopeReadWrite)
	if err != nil {
		return nil, fmt.Errorf("GCS client CredentialsFromJSONWithType: %w", err)
	}
	client, err := storage.NewClient(ctx, option.WithCredentials(creds))
	if err != nil {
		return nil, fmt.Errorf("GCS storage.NewClient with credential: %w", err)
	}
	return client, nil
}

func newGCSClientDefault(ctx context.Context) (*storage.Client, error) {
	client, err := storage.NewClient(ctx)
	if err != nil {
		return nil, fmt.Errorf("GCS storage.NewClient: %w", err)
	}
	return client, nil
}

// Load function downloads objects from GCS
func (h *ArtifactDriver) Load(ctx context.Context, inputArtifact *wfv1.Artifact, path string) error {
	err := waitutil.Backoff(defaultRetry,
		func() (bool, error) {
			key := filepath.Clean(inputArtifact.GCS.Key)
			logger := logging.RequireLoggerFromContext(ctx)

View on GitHub (pinned to 35bff19146)

Solutions

1. Re-download a fresh service-account key and update the secret
2. Unset conflicting env vars (GOOGLE_APPLICATION_CREDENTIALS, STORAGE_EMULATOR_HOST) in the workflow/executor pod
3. Retry with newGCSClientDefault (no explicit creds) if workload identity is configured
4. Check the storage client library version for known issues and upgrade the argo image

Example fix

// before
kubectl create secret generic gcs-creds --from-file=serviceAccountKey=partially-truncated-key.json
// after
kubectl delete secret gcs-creds
kubectl create secret generic gcs-creds --from-file=serviceAccountKey=key.json

// ensure the executor env has no conflicting client options:
// env:
// - name: STORAGE_EMULATOR_HOST
//   value: ""   # must not point at a dead emulator
// and validate the key fields after CredentialsFromJSON:
if creds == nil || creds.ProjectID == "" { /* key lacks project_id; regenerate */ }

_, err := newGCSClient(ctx, keyJSON)
if err != nil {
	if strings.Contains(err.Error(), "storage.NewClient with credential") {
		// fall back to default ADC client
		client, derr := storage.NewClient(ctx)
		if derr == nil { return use(client) }
	}
	return err
}

When it happens

Trigger: storage.NewClient(ctx, option.WithCredentials(creds)) failing — typically when the service-account JSON parsed but has invalid internal fields, or when environment variables like STORAGE_EMULATOR_HOST or GOOGLE_API_USE_MTLS_ENDPOINT produce an incompatible client configuration.

Common situations: Corrupted/partially rotated key; inconsistent GOOGLE_APPLICATION_CREDENTIALS and explicit creds mix; mTLS endpoint misconfiguration in restricted networks; broken STORAGE_EMULATOR_HOST setting in the executor environment.

Related errors

• GCS client CredentialsFromJSONWithType: %w
• GCS storage.NewClient: %w
• unable to create Azure shared key credential: %w
• unable to create Azure Blob Container client for %s: %w
• mkdir %s: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/625b075aee56615d. Report an issue: GitHub.