{"record":{"id":"d171c0697b0f390f","repo":"apache/beam","slug":"cancelled-mutaterow-request-after-exceeding-deadline","errorCode":null,"errorMessage":"Cancelled mutateRow request after exceeding deadline","messagePattern":"Cancelled mutateRow request after exceeding deadline","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/dao/MetadataTableDao.java","lineNumber":827,"sourceCode":"    mutateRowWithHardTimeout(rowMutation);\n  }\n\n  /**\n   * This adds a hard timeout of 40 seconds to mutate row futures. These requests already have a\n   * 30-second deadline. This is a workaround for an extremely rare issue we see with requests not\n   * respecting their deadlines. This can be removed once we've pinpointed the cause.\n   *\n   * @param rowMutation Bigtable RowMutation to apply\n   */\n  @VisibleForTesting\n  void mutateRowWithHardTimeout(RowMutation rowMutation) {\n    ApiFuture<Void> mutateRowFuture = dataClient.mutateRowAsync(rowMutation);\n    try {\n      mutateRowFuture.get(\n          BigtableChangeStreamAccessor.MUTATE_ROW_DEADLINE.getSeconds() + 10, TimeUnit.SECONDS);\n    } catch (TimeoutException timeoutException) {\n      mutateRowFuture.cancel(true);\n      throw new RuntimeException(\n          \"Cancelled mutateRow request after exceeding deadline\", timeoutException);\n    } catch (ExecutionException executionException) {\n      if (executionException.getCause() instanceof RuntimeException) {\n        throw (RuntimeException) executionException.getCause();\n      }\n      throw new RuntimeException(executionException);\n    } catch (InterruptedException interruptedException) {\n      Thread.currentThread().interrupt();\n      throw new RuntimeException(interruptedException);\n    }\n  }\n\n  /**\n   * Reads the raw bigtable StreamPartition rows. This is separate from {@link\n   * #readAllStreamPartitions()} only for testing purposes. {@link #readAllStreamPartitions()}\n   * should be used for all usage outside this file.\n   *\n   * @return {@link ServerStream} of StreamPartition bigtable rows","sourceCodeStart":809,"sourceCodeEnd":845,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/dao/MetadataTableDao.java#L809-L845","documentation":"MetadataTableDao wraps a mutateRow call with a hard deadline (MUTATE_ROW_DEADLINE + 10 seconds). If the ApiFuture doesn't complete in time, the request is cancelled and a RuntimeException \"Cancelled mutateRow request after exceeding deadline\" is thrown wrapping the TimeoutException. This protects the change-stream pipeline from hanging forever on metadata writes.","triggerScenarios":"mutateRowAsync against the Bigtable metadata table not completing within BigtableChangeStreamAccessor.MUTATE_ROW_DEADLINE.getSeconds() + 10 — e.g. sustained Bigtable latency, quota exhaustion, or network disruption to the Bigtable data endpoint.","commonSituations":"Bigtable instance under heavy load or throttled; network partition between the Dataflow worker and Bigtable; very large metadata rows or frequent watermark/duplicate detections causing contention.","solutions":["Retry the metadata write (the mutation was cancelled, so it is safe to retry idempotently).","Check Bigtable metrics/health: latency, error rates, and instance utilization; scale or fix underlying latency.","Verify network connectivity/VPC settings between workers and the Bigtable regional endpoint; increase MUTATE_ROW_DEADLINE only if latency is expected."],"exampleFix":"// before\nmetadataTableDao.writeCheckpoint(token); // may throw on transient latency spike\n// after\ntry {\n  metadataTableDao.writeCheckpoint(token);\n} catch (RuntimeException e) {\n  backoffRetry(() -> metadataTableDao.writeCheckpoint(token), 3); // idempotent retry\n}","handlingStrategy":"retry","validationCode":"// Pre-check endpoint health before bulk metadata writes\ngoogle::cloud::bigtable::DataClient status check / instance latency metrics via Cloud Monitoring API before starting metadata-heavy phases","typeGuard":null,"tryCatchPattern":"try {\n  metadataTableDao.mutateRow(mutation);\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Cancelled mutateRow\")) {\n    // safe to retry: the original mutation was cancelled\n    withExponentialBackoff(() -> metadataTableDao.mutateRow(mutation));\n  } else { throw e; }\n}","preventionTips":["Retry idempotently with exponential backoff — the cancelled request did not commit.","Monitor Bigtable instance latency/quota before metadata-heavy workloads.","Ensure network paths from workers to the regional Bigtable endpoint are healthy.","Keep metadata rows small to reduce mutation latency."],"tags":["bigtable","timeout","metadata","latency"],"backgroundTag":"request-timeout","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}