apache/beam · error

could not create data operations client

Error message

could not create data operations client: %v

What it means

During DoFn setup on each worker, writeFn.Setup creates a Cloud Bigtable data client with bigtable.NewClient. If client creation fails, the error is wrapped as "could not create data operations client". This aborts bundle processing on that worker.

Solutions

1. Verify Project and InstanceID values and that the instance exists in that project.
2. Enable the Bigtable API and grant the worker service account the Bigtable User role.
3. Check worker network egress/DNS to bigtable.googleapis.com:443.
4. Confirm ADC credentials are available on workers (e.g. --service_account or correct runner-provided credentials).

Example fix

// before
f := &writeFn{Project: "my-proj", InstanceID: "my-inst"}
// after — verify instance & permissions first
gcloud bigtable instances list --project=my-proj
// Project: "my-proj", InstanceID: "my-inst"

client, err := bigtable.NewClient(ctx, project, instance)
if err != nil {
    return fmt.Errorf("precheck failed: %w", err)
}
client.Close()

if err := fn.Setup(ctx); err != nil {
    if strings.Contains(err.Error(), "data operations client") {
        // check creds, API enablement, network before retrying
    }
}

When it happens

Trigger: writeFn.Setup failing because bigtable.NewClient returns an error — bad project/instance IDs, no network path to Bigtable endpoint, or missing credentials/API enablement.

Common situations: Typo in project or instance ID; Bigtable API not enabled in the project; workers running in a VPC without egress to bigtable.googleapis.com; wrong or missing service-account credentials; firewall/DNS failures.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• chunk send failed
• could not close data operations client
• failed to connect to state service
• failed to connect
• failed to send chunks for

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0b64d11d406684e2. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/bigtableio/bigtable.go:148

	// Project is the project
	Project string `json:"project"`
	// InstanceID is the bigtable instanceID
	InstanceID string `json:"instanceId"`
	// Client is the bigtable.Client
	client *bigtable.Client `json:"-"`
	// TableName is the qualified table identifier.
	TableName string `json:"tableName"`
	// Table is a bigtable.Table instance with an eventual open connection
	table *bigtable.Table `json:"-"`
	// Type is the encoded schema type.
	Type beam.EncodedType `json:"type"`
}

func (f *writeFn) Setup(ctx context.Context) error {
	var err error
	f.client, err = bigtable.NewClient(ctx, f.Project, f.InstanceID)
	if err != nil {
		return fmt.Errorf("could not create data operations client: %v", err)
	}

	f.table = f.client.Open(f.TableName)
	return nil
}

func (f *writeFn) Teardown() error {
	if err := f.client.Close(); err != nil {
		return fmt.Errorf("could not close data operations client: %v", err)
	}
	return nil
}

func (f *writeFn) ProcessElement(ctx context.Context, key int, values func(*Mutation) bool) error {

	var mutation Mutation
	for values(&mutation) {

View on GitHub (pinned to 12126d8942)