{"record":{"id":"573e1ac690205a95","repo":"apache/beam","slug":"one-instance-of-bigtableio-mutation-must-not-have-more-than","errorCode":null,"errorMessage":"one instance of bigtableio.Mutation must not have more than 100,000 operations/mutations, see https://cloud.google.com/bigtable/docs/writes#batch","messagePattern":"one instance of bigtableio\\.Mutation must not have more than 100,000 operations/mutations, see https://cloud\\.google\\.com/bigtable/docs/writes#batch","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/go/pkg/beam/io/bigtableio/bigtable.go","lineNumber":262,"sourceCode":"\t\trowKeysInBatch = append(rowKeysInBatch, mutation.RowKey)\n\t\tmutationsInBatch = append(mutationsInBatch, getBigtableMutation(mutation))\n\t\topsAddedToBatch += len(mutation.Ops)\n\n\t}\n\n\tif len(rowKeysInBatch) != 0 && len(mutationsInBatch) != 0 {\n\t\terr := tryApplyBulk(f.table.ApplyBulk(ctx, rowKeysInBatch, mutationsInBatch))\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\n\treturn nil\n}\n\nfunc validateMutation(mutation Mutation) error {\n\tif len(mutation.Ops) > 100000 {\n\t\treturn fmt.Errorf(\"one instance of bigtableio.Mutation must not have more than 100,000 operations/mutations, see https://cloud.google.com/bigtable/docs/writes#batch\")\n\t}\n\treturn nil\n}\n\nfunc tryApplyBulk(errs []error, processErr error) error {\n\tif processErr != nil {\n\t\treturn fmt.Errorf(\"bulk apply procces failed: %v\", processErr)\n\t}\n\tfor _, err := range errs {\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"could not apply mutation: %v\", err)\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc getBigtableMutation(mutation Mutation) *bigtable.Mutation {\n\tbigtableMutation := bigtable.NewMutation()","sourceCodeStart":244,"sourceCodeEnd":280,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/io/bigtableio/bigtable.go#L244-L280","documentation":"validateMutation enforces the Cloud Bigtable constraint that one Mutation carries at most 100,000 operations. Exceeding it returns this static error (with a docs link). It runs before any network call, both in ProcessElement and unit tests.","triggerScenarios":"len(mutation.Ops) > 100000 in validateMutation (bigtable.go:262); triggered by ProcessElement and by TestValidateMutationFailsWhenGreaterThanHundredKOps.","commonSituations":"Aggregating a very large number of cell writes for one grouped key; auto-generated mutations from a wide join; backfill jobs emitting one Mutation per huge row group.","solutions":["Chunk the Ops slice into batches of ≤100,000 and emit one Mutation per chunk","Re-key the input so each group produces fewer ops (e.g. shard by hash of row key)","Pre-validate Mutation sizes in test fixtures with validateMutation before submitting the pipeline"],"exampleFix":"// before\nops := allOps // 250,000 ops in one Mutation\nreturn Mutation{RowKey: key, Ops: ops}\n// after\nconst chunk = 100000\nfor i := 0; i < len(ops); i += chunk {\n\tend := min(i+chunk, len(ops))\n\temit(Mutation{RowKey: key, Ops: ops[i:end]})\n}","handlingStrategy":"validation","validationCode":"const bigtableBatchLimit = 100000\nif len(m.Ops) > bigtableBatchLimit {\n\treturn fmt.Errorf(\"%d ops exceeds Bigtable limit of %d\", len(m.Ops), bigtableBatchLimit)\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Chunk mutation ops to ≤100,000 per Mutation before emitting","Add a pipeline-time assertion on op counts per group","Keep per-key fan-in bounded in joins/aggregations feeding the write"],"tags":["gcp","bigtable","validation","limit","beam-io"],"backgroundTag":"value-out-of-range","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}