{"record":{"id":"d236011d81a54c7b","repo":"apache/beam","slug":"failed-to-write-the-mutation-group","errorCode":null,"errorMessage":"Failed to write the mutation group: {}","messagePattern":"Failed to write the mutation group: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java","lineNumber":2879,"sourceCode":"          // fall through and retry individual mutationGroups.\n        } else if (failureMode == FailureMode.FAIL_FAST) {\n          mutationGroupsWriteFail.inc(mutations.size());\n          LOG.error(\"Failed to write a batch of mutation groups\", e);\n          throw e;\n        } else {\n          throw new IllegalArgumentException(\"Unknown failure mode \" + failureMode);\n        }\n      }\n\n      // If we are here, writing a batch has failed, retry individual mutations.\n      for (MutationGroup mg : mutations) {\n        try {\n          spannerWriteRetries.inc();\n          writeMutations(mg);\n          mutationGroupsWriteSuccess.inc();\n        } catch (SpannerException e) {\n          mutationGroupsWriteFail.inc();\n          LOG.warn(\"Failed to write the mutation group: {}\", mg, e);\n          c.output(failedTag, mg);\n        }\n      }\n    }\n\n    /*\n     Spanner aborts all inflight transactions during a schema change. Client is expected\n     to retry silently. These must not be counted against retry backoff.\n    */\n    private void spannerWriteWithRetryIfSchemaChange(List<Mutation> batch) throws SpannerException {\n      Set<String> tableNames = batch.stream().map(Mutation::getTable).collect(Collectors.toSet());\n      for (int retry = 1; ; retry++) {\n        try {\n          spannerAccessor\n              .getDatabaseClient()\n              .writeAtLeastOnceWithOptions(batch, getTransactionOptions());\n          // Get names of all tables in batch of mutations.\n          reportServiceCallMetricsForBatch(tableNames, \"ok\");","sourceCodeStart":2861,"sourceCodeEnd":2897,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java#L2861-L2897","documentation":"In the fault-tolerant write path of SpannerIO's grouping transform, a MutationGroup failed to be written to Cloud Spanner with a SpannerException after the configured retries. The failure is counted, logged, and the MutationGroup is emitted to the failure tag (withWriteFailureFn / failedTag output) instead of failing the pipeline. This is a warning-level signal that some writes were dropped from the main output and routed to the failures PCollection.","triggerScenarios":"Applying SpannerIO.write().to(...) in a pipeline configured with a failure output (failureTag), and writeMutations(mg) throws SpannerException — e.g. DeadlineExceeded, Aborted beyond retries, permission errors, or schema violations for a mutation group.","commonSituations":"Large mutation groups exceeding size/timeout limits; transient Spanner outages; writes violating the database schema (missing table/column); interleaved-key ordering violations.","solutions":["Inspect the returned failures PCollection (failedTag) and replay the MutationGroups after fixing the root cause.","Check the attached SpannerException for the specific cause (permissions, schema mismatch, timeouts) and fix accordingly.","Reduce mutation group size or batch count to avoid timeouts, or increase the retry deadline configuration.","Verify table schemas and that all mutations reference existing tables/columns with valid keys."],"exampleFix":"// before: failures silently dropped by ignoring the failed tag\nspannerWrites.apply(\"Write\", SpannerIO.write().to(config));\n\n// after: handle the failures output\nSpannerWriteResult result =\n    rows.apply(SpannerIO.write()\n        .withInstanceId(instanceId)\n        .withDatabaseId(databaseId)\n        .withFailureTag());\nresult.getFailedMutations().apply(\"LogFailures\", ParDo.of(new LogMutationGroupDoFn()));","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Consume the failure-tag output and persist/replay failed MutationGroups:\nSpannerWriteResult result = rows.apply(SpannerIO.write()\n    .withInstanceId(instanceId).withDatabaseId(dbId).withFailureTag());\nresult.getFailedMutations()\n    .apply(\"PersistFailures\", Write.to(failureSink));","preventionTips":["Always configure and monitor the failure-tag PCollection when using SpannerIO.write().","Keep mutation groups within Spanner size/timeout limits; split large batches.","Validate schemas and permissions before large write jobs.","Set up alerting on mutationGroupsWriteFail counters."],"tags":["java","spanner","write-failure","retries"],"backgroundTag":"database-write-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}