apache/beam · error · RuntimeException

Cancelled mutateRow request after exceeding deadline

Error message

Cancelled mutateRow request after exceeding deadline

What it means

MetadataTableDao wraps a mutateRow call with a hard deadline (MUTATE_ROW_DEADLINE + 10 seconds). If the ApiFuture doesn't complete in time, the request is cancelled and a RuntimeException "Cancelled mutateRow request after exceeding deadline" is thrown wrapping the TimeoutException. This protects the change-stream pipeline from hanging forever on metadata writes.

Solutions

Retry the metadata write (the mutation was cancelled, so it is safe to retry idempotently).
Check Bigtable metrics/health: latency, error rates, and instance utilization; scale or fix underlying latency.
Verify network connectivity/VPC settings between workers and the Bigtable regional endpoint; increase MUTATE_ROW_DEADLINE only if latency is expected.

Example fix

// before
metadataTableDao.writeCheckpoint(token); // may throw on transient latency spike
// after
try {
  metadataTableDao.writeCheckpoint(token);
} catch (RuntimeException e) {
  backoffRetry(() -> metadataTableDao.writeCheckpoint(token), 3); // idempotent retry
}

Defensive patterns

Strategy: retry

Validate before calling

// Pre-check endpoint health before bulk metadata writes
google::cloud::bigtable::DataClient status check / instance latency metrics via Cloud Monitoring API before starting metadata-heavy phases

Try / catch

try {
  metadataTableDao.mutateRow(mutation);
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Cancelled mutateRow")) {
    // safe to retry: the original mutation was cancelled
    withExponentialBackoff(() -> metadataTableDao.mutateRow(mutation));
  } else { throw e; }
}

Prevention

Retry idempotently with exponential backoff — the cancelled request did not commit.
Monitor Bigtable instance latency/quota before metadata-heavy workloads.
Ensure network paths from workers to the regional Bigtable endpoint are healthy.
Keep metadata rows small to reduce mutation latency.

When it happens

Trigger: mutateRowAsync against the Bigtable metadata table not completing within BigtableChangeStreamAccessor.MUTATE_ROW_DEADLINE.getSeconds() + 10 — e.g. sustained Bigtable latency, quota exhaustion, or network disruption to the Bigtable data endpoint.

Common situations: Bigtable instance under heavy load or throttled; network partition between the Dataflow worker and Bigtable; very large metadata rows or frequent watermark/duplicate detections causing contention.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d171c0697b0f390f. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/dao/MetadataTableDao.java:827

    mutateRowWithHardTimeout(rowMutation);
  }

  /**
   * This adds a hard timeout of 40 seconds to mutate row futures. These requests already have a
   * 30-second deadline. This is a workaround for an extremely rare issue we see with requests not
   * respecting their deadlines. This can be removed once we've pinpointed the cause.
   *
   * @param rowMutation Bigtable RowMutation to apply
   */
  @VisibleForTesting
  void mutateRowWithHardTimeout(RowMutation rowMutation) {
    ApiFuture<Void> mutateRowFuture = dataClient.mutateRowAsync(rowMutation);
    try {
      mutateRowFuture.get(
          BigtableChangeStreamAccessor.MUTATE_ROW_DEADLINE.getSeconds() + 10, TimeUnit.SECONDS);
    } catch (TimeoutException timeoutException) {
      mutateRowFuture.cancel(true);
      throw new RuntimeException(
          "Cancelled mutateRow request after exceeding deadline", timeoutException);
    } catch (ExecutionException executionException) {
      if (executionException.getCause() instanceof RuntimeException) {
        throw (RuntimeException) executionException.getCause();
      }
      throw new RuntimeException(executionException);
    } catch (InterruptedException interruptedException) {
      Thread.currentThread().interrupt();
      throw new RuntimeException(interruptedException);
    }
  }

  /**
   * Reads the raw bigtable StreamPartition rows. This is separate from {@link
   * #readAllStreamPartitions()} only for testing purposes. {@link #readAllStreamPartitions()}
   * should be used for all usage outside this file.
   *
   * @return {@link ServerStream} of StreamPartition bigtable rows

View on GitHub (pinned to 12126d8942)