one instance of bigtableio.Mutation must not have more than

Error message

one instance of bigtableio.Mutation must not have more than 100,000 operations/mutations, see https://cloud.google.com/bigtable/docs/writes#batch

What it means

validateMutation enforces the Cloud Bigtable constraint that one Mutation carries at most 100,000 operations. Exceeding it returns this static error (with a docs link). It runs before any network call, both in ProcessElement and unit tests.

Source

Thrown at sdks/go/pkg/beam/io/bigtableio/bigtable.go:262

		rowKeysInBatch = append(rowKeysInBatch, mutation.RowKey)
		mutationsInBatch = append(mutationsInBatch, getBigtableMutation(mutation))
		opsAddedToBatch += len(mutation.Ops)

	}

	if len(rowKeysInBatch) != 0 && len(mutationsInBatch) != 0 {
		err := tryApplyBulk(f.table.ApplyBulk(ctx, rowKeysInBatch, mutationsInBatch))
		if err != nil {
			return err
		}
	}

	return nil
}

func validateMutation(mutation Mutation) error {
	if len(mutation.Ops) > 100000 {
		return fmt.Errorf("one instance of bigtableio.Mutation must not have more than 100,000 operations/mutations, see https://cloud.google.com/bigtable/docs/writes#batch")
	}
	return nil
}

func tryApplyBulk(errs []error, processErr error) error {
	if processErr != nil {
		return fmt.Errorf("bulk apply procces failed: %v", processErr)
	}
	for _, err := range errs {
		if err != nil {
			return fmt.Errorf("could not apply mutation: %v", err)
		}
	}
	return nil
}

func getBigtableMutation(mutation Mutation) *bigtable.Mutation {
	bigtableMutation := bigtable.NewMutation()

View on GitHub (pinned to 12126d8942)

Solutions

Chunk the Ops slice into batches of ≤100,000 and emit one Mutation per chunk
Re-key the input so each group produces fewer ops (e.g. shard by hash of row key)
Pre-validate Mutation sizes in test fixtures with validateMutation before submitting the pipeline

Example fix

// before
ops := allOps // 250,000 ops in one Mutation
return Mutation{RowKey: key, Ops: ops}
// after
const chunk = 100000
for i := 0; i < len(ops); i += chunk {
	end := min(i+chunk, len(ops))
	emit(Mutation{RowKey: key, Ops: ops[i:end]})
}

Defensive patterns

Strategy: validation

Validate before calling

const bigtableBatchLimit = 100000
if len(m.Ops) > bigtableBatchLimit {
	return fmt.Errorf("%d ops exceeds Bigtable limit of %d", len(m.Ops), bigtableBatchLimit)
}

Prevention

Chunk mutation ops to ≤100,000 per Mutation before emitting
Add a pipeline-time assertion on op counts per group
Keep per-key fan-in bounded in joins/aggregations feeding the write

When it happens

Trigger: len(mutation.Ops) > 100000 in validateMutation (bigtable.go:262); triggered by ProcessElement and by TestValidateMutationFailsWhenGreaterThanHundredKOps.

Common situations: Aggregating a very large number of cell writes for one grouped key; auto-generated mutations from a wide join; backfill jobs emitting one Mutation per huge row group.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/573e1ac690205a95. Report an issue: GitHub.