googleapis/mcp-toolbox · error

failed to create dataproc batch client: %w

Error message

failed to create dataproc batch client: %w

What it means

After resolving the regional endpoint, Initialize creates a Dataproc BatchControllerClient. Failure to construct this Google Cloud client (credential resolution, API enablement, endpoint problems) aborts source creation with this wrapped error. Google's client library returns the underlying cause, which is embedded in %w.

Source

Thrown at internal/sources/serverlessspark/serverlessspark.go:75

	Name     string `yaml:"name" validate:"required"`
	Type     string `yaml:"type" validate:"required"`
	Project  string `yaml:"project" validate:"required"`
	Location string `yaml:"location" validate:"required"`
}

func (r Config) SourceConfigType() string {
	return SourceType
}

func (r Config) Initialize(ctx context.Context, tracer trace.Tracer) (sources.Source, error) {
	ua, err := util.UserAgentFromContext(ctx)
	if err != nil {
		return nil, fmt.Errorf("error in User Agent retrieval: %s", err)
	}
	endpoint := fmt.Sprintf("%s-dataproc.googleapis.com:443", r.Location)
	batchClient, err := dataproc.NewBatchControllerClient(ctx, option.WithEndpoint(endpoint), option.WithUserAgent(ua))
	if err != nil {
		return nil, fmt.Errorf("failed to create dataproc batch client: %w", err)
	}
	sessionTemplateClient, err := dataproc.NewSessionTemplateControllerClient(ctx, option.WithEndpoint(endpoint), option.WithUserAgent(ua))
	if err != nil {
		batchClient.Close()
		return nil, fmt.Errorf("failed to create dataproc session template client: %w", err)
	}
	opsClient, err := longrunning.NewOperationsClient(ctx, option.WithEndpoint(endpoint), option.WithUserAgent(ua))
	if err != nil {
		batchClient.Close()
		sessionTemplateClient.Close()
		return nil, fmt.Errorf("failed to create longrunning client: %w", err)
	}
	sessionClient, err := dataproc.NewSessionControllerClient(ctx, option.WithEndpoint(endpoint), option.WithUserAgent(ua))
	if err != nil {
		batchClient.Close()
		sessionTemplateClient.Close()
		opsClient.Close()
		return nil, fmt.Errorf("failed to create dataproc session client: %w", err)

View on GitHub (pinned to 8cc6e09de2)

Solutions

1. Set up valid ADC: `gcloud auth application-default login` locally, or export GOOGLE_APPLICATION_CREDENTIALS to a service account key in non-GCP environments.
2. Enable the Dataproc API: `gcloud services enable dataproc.googleapis.com --project <project>`.
3. Verify `location` is a valid Dataproc region (e.g. us-central1); the endpoint is built as `<location>-dataproc.googleapis.com`.
4. Check outbound network access to `<region>-dataproc.googleapis.com:443` (proxies/firewalls).
5. Grant the identity `roles/dataproc.editor` (or similar) if errors indicate permission denial at request time.

Example fix

// before — no credentials configured
export LOCATION=us-central1
./toolbox
// after
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa-key.json
gcloud services enable dataproc.googleapis.com --project my-project
./toolbox

#!/usr/bin/env bash
# Pre-flight GCP checks before starting the toolbox
REGION=us-central1
[ -n "$GOOGLE_APPLICATION_CREDENTIALS" ] || [ -f "$HOME/.config/gcloud/application_default_credentials.json" ] \
  || { echo "No ADC credentials found"; exit 1; }
nc -zv "$REGION-dataproc.googleapis.com" 443 || { echo "endpoint unreachable"; exit 1; }
gcloud services list --enabled --project my-project | grep -q dataproc.googleapis.com \
  || { echo "Dataproc API not enabled"; exit 1; }

When it happens

Trigger: dataproc.NewBatchControllerClient(ctx, option.WithEndpoint(<region>-dataproc.googleapis.com:443), option.WithUserAgent(ua)) fails — typically because Application Default Credentials cannot be resolved, the Dataproc API is disabled for the project, or the region string is invalid.

Common situations: GOOGLE_APPLICATION_CREDENTIALS unset or pointing to an invalid key file; no metadata server when running outside GCP without a service account key; `location` set to a bogus region producing a bad endpoint; Dataproc API not enabled; network/proxy blocking googleapis.com:443.

Related errors

• failed to create dataproc session template client: %w
• error getting email from ADC: %v
• failed to initialize dataplex client: %w
• failed to create impersonated credentials for %q: %w
• failed to find default Google Cloud credentials with scopes

AI-assisted analysis of googleapis/mcp-toolbox@8cc6e09de2 (2026-09-05). Data as JSON: /api/errors/27e016536698e478. Report an issue: GitHub.