apache/beam · error · IOException

BulkMutation took too long to close

Error message

BulkMutation took too long to close

What it means

BigtableServiceImpl.close flushes and waits for the in-flight BulkMutation batcher to complete. If the future returned by the batcher's close/flush does not finish before the configured timeout, a TimeoutException is wrapped into an IOException with this message, failing the write.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java:567

          // set a timeout waiting for the future.
          ApiFuture<Void> future = bulkMutation.closeAsync();
          if (Duration.ZERO.isShorterThan(closeWaitTimeout)) {
            future.get(closeWaitTimeout.getMillis(), TimeUnit.MILLISECONDS);
          } else {
            future.get();
          }
          bulkSize.update(outstandingMutations);
          outstandingMutations = 0;
          stopwatch.stop();
          latency.update(stopwatch.elapsed(TimeUnit.MILLISECONDS));
        } catch (BatchingException e) {
          // Ignore batching failures because element failures are tracked as is in
          // BigtableIOWriteFn.
          // TODO: Bigtable client already tracks BatchingExceptions, use BatchingExceptions
          // instead of tracking them separately in BigtableIOWriteFn.
        } catch (TimeoutException e) {
          // We fail because future.get() timed out
          throw new IOException("BulkMutation took too long to close", e);
        } catch (ExecutionException e) {
          throw new IOException("Failed to close batch", e.getCause());
        } catch (InterruptedException e) {
          Thread.currentThread().interrupt();
          // We fail since close() operation was interrupted.
          throw new IOException(e);
        }
        bulkMutation = null;
      }
    }

    @Override
    public CompletableFuture<MutateRowResponse> writeRecord(
        KV<ByteString, Iterable<Mutation>> record) throws IOException {

      com.google.cloud.bigtable.data.v2.models.Mutation mutation =
          com.google.cloud.bigtable.data.v2.models.Mutation.fromProtoUnsafe(record.getValue());

View on GitHub (pinned to 12126d8942)

Solutions

1. Increase the Bigtable write flush/close timeout (e.g. via BigtableIO write flow control / max outstanding elements options)
2. Reduce write throughput or batch size so flushes complete within the timeout
3. Check Cloud Bigtable instance CPU utilization and scale the cluster if throttling
4. Retry the affected bundle; partial failures are tracked per-element in BigtableIOWriteFn

Example fix

// before
pipeline.apply(BigtableIO.write().withBigtableOptions(opts)); // default timeouts
// after
opts = opts.toBuilder()
    .setBulkOptions(BulkOptions.newBuilder().setMaxElementCountPerRow(100).build())
    .build();
pipeline.apply(BigtableIO.write().withBigtableOptions(opts));

// size the flush budget against pending work
long pending = batcher.getOutstandingElementCount();
if (pending > threshold) { /* increase timeout or throttle writes */ }

try {
  writer.close();
} catch (IOException e) {
  if (e.getCause() instanceof TimeoutException) {
    // retry flush or extend timeout
  }
}

When it happens

Trigger: Closing the Bigtable writer while bulk mutations are still being flushed; the batcher's future.get() times out because Cloud Bigtable is slow, overloaded, or the network is degraded, or because the flush timeout is too short for the volume of pending mutations.

Common situations: Large bulkloads writing millions of rows where the final flush takes longer than the timeout; Bigtable cluster throttling; narrow per-shutdown timeout budgets in streaming pipelines.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• Error checking whether table %s exists
• Failed to close batch
• Failed to retrieve or create secret bytes
• Failed to retrieve secret bytes
• Unable to create dataset: %s, aborting after %d .

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ec280d54022987af. Report an issue: GitHub.