apache/beam · error

could not create data operations client: %v

Error message

could not create data operations client: %v

What it means

During DoFn setup on each worker, writeFn.Setup creates a Cloud Bigtable data client with bigtable.NewClient. If client creation fails, the error is wrapped as "could not create data operations client". This aborts bundle processing on that worker.

Source

Thrown at sdks/go/pkg/beam/io/bigtableio/bigtable.go:148

	// Project is the project
	Project string `json:"project"`
	// InstanceID is the bigtable instanceID
	InstanceID string `json:"instanceId"`
	// Client is the bigtable.Client
	client *bigtable.Client `json:"-"`
	// TableName is the qualified table identifier.
	TableName string `json:"tableName"`
	// Table is a bigtable.Table instance with an eventual open connection
	table *bigtable.Table `json:"-"`
	// Type is the encoded schema type.
	Type beam.EncodedType `json:"type"`
}

func (f *writeFn) Setup(ctx context.Context) error {
	var err error
	f.client, err = bigtable.NewClient(ctx, f.Project, f.InstanceID)
	if err != nil {
		return fmt.Errorf("could not create data operations client: %v", err)
	}

	f.table = f.client.Open(f.TableName)
	return nil
}

func (f *writeFn) Teardown() error {
	if err := f.client.Close(); err != nil {
		return fmt.Errorf("could not close data operations client: %v", err)
	}
	return nil
}

func (f *writeFn) ProcessElement(ctx context.Context, key int, values func(*Mutation) bool) error {

	var mutation Mutation
	for values(&mutation) {

View on GitHub (pinned to 12126d8942)

Solutions

1. Verify Project and InstanceID values and that the instance exists in that project.
2. Enable the Bigtable API and grant the worker service account the Bigtable User role.
3. Check worker network egress/DNS to bigtable.googleapis.com:443.
4. Confirm ADC credentials are available on workers (e.g. --service_account or correct runner-provided credentials).

Example fix

// before
f := &writeFn{Project: "my-proj", InstanceID: "my-inst"}
// after — verify instance & permissions first
gcloud bigtable instances list --project=my-proj
// Project: "my-proj", InstanceID: "my-inst"

client, err := bigtable.NewClient(ctx, project, instance)
if err != nil {
    return fmt.Errorf("precheck failed: %w", err)
}
client.Close()

if err := fn.Setup(ctx); err != nil {
    if strings.Contains(err.Error(), "data operations client") {
        // check creds, API enablement, network before retrying
    }
}

When it happens

Trigger: writeFn.Setup failing because bigtable.NewClient returns an error — bad project/instance IDs, no network path to Bigtable endpoint, or missing credentials/API enablement.

Common situations: Typo in project or instance ID; Bigtable API not enabled in the project; workers running in a VPC without egress to bigtable.googleapis.com; wrong or missing service-account credentials; firewall/DNS failures.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• Input Schema is invalid: Schema should be formatted in one
• table does not exist and create disposition is 'CreateNever'
• type must be bigtableio.Mutation but is: %v
• could not close data operations client: %v
• invalid bigtableio.Mutation: %s

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0b64d11d406684e2. Report an issue: GitHub.