DNP: Updating watermark failed due to overlapping

Error message

DNP: Updating watermark failed due to overlapping: {}

What it means

DetectNewPartitionsAction.getNewWatermark logs 'DNP: Updating watermark failed due to overlapping: {}' when the set of StreamPartitions assembled for the watermark computation contains overlapping key ranges. Overlapping partitions indicate inconsistent metadata state, so the action returns Optional.empty() and the watermark is not advanced.

Solutions

Verify the metadata table (Bigtable change streams metadata) for overlapping partition rows and repair/consolidate them.
Ensure only one pipeline instance is processing the change stream for a given stream partition (exclusive app profile routing, single runner).
Restart the pipeline so DetectNewPartitions re-syncs partition state from the change stream.
Upgrade the connector, as partition reconciliation logic has been fixed in newer releases.

Example fix

// before
// two runners sharing one metadata table caused overlapping partitions
// after
// run a single streaming job per change stream, using an app profile with single-row transactions
BigtableIO.readChangeStream().withAppProfile("single-instance");

Defensive patterns

Strategy: validation

Validate before calling

// Before running: ensure exactly one streaming job per change stream and a healthy metadata table
// (Bigtable CLI) scan the metadata table and assert no two partition rows overlap

Prevention

Run a single pipeline instance per change stream (single-row-transaction app profile).
Never manually edit the change-stream metadata table.
Watch DNP warnings in logs; a stuck watermark signals metadata inconsistency to fix early.

When it happens

Trigger: getNewWatermark (called from maybeWatermark) detects that two or more ByteStringRange partitions in the metadata table overlap, typically after a partition split or merge was partially recorded.

Common situations: Concurrent split/merge operations on the Bigtable table while the change stream pipeline is running; stale or partially updated rows in the Bigtable change stream metadata table; a previous pipeline run crashed mid-partition-update.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c8c4911fa5b119ce. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/DetectNewPartitionsAction.java:134

      List<StreamPartitionWithWatermark> streamPartitionsWithWatermark,
      List<NewPartition> newPartitions) {
    // Get partitions with a watermark set but skip rows w a lock and no watermark yet
    List<StreamPartitionWithWatermark> slowPartitions = new ArrayList<>();
    Instant lowWatermark = Instant.ofEpochMilli(Long.MAX_VALUE);

    List<ByteStringRange> partitions = new ArrayList<>();
    for (StreamPartitionWithWatermark streamPartitionWithWatermark :
        streamPartitionsWithWatermark) {
      if (streamPartitionWithWatermark.getWatermark().plus(DEBUG_WATERMARK_DELAY).isBeforeNow()) {
        slowPartitions.add(streamPartitionWithWatermark);
      }
      if (streamPartitionWithWatermark.getWatermark().compareTo(lowWatermark) < 0) {
        lowWatermark = streamPartitionWithWatermark.getWatermark();
      }
      partitions.add(streamPartitionWithWatermark.getPartition());
    }
    if (!slowPartitions.isEmpty()) {
      LOG.warn(
          "DNP: Updating watermark is held back by {} partitions : {}",
          slowPartitions.size(),
          slowPartitions.stream()
              .map(e -> formatByteStringRange(e.getPartition()) + " => " + e.getWatermark())
              .collect(Collectors.joining(", ", "{", "}")));
    }

    // Only added StreamPartitions so far, check if there are any overlapping StreamPartitions. If
    // so, something is wrong, we should stop.
    List<ByteStringRange> overlappingStreamPartitions = getOverlappingPartitions(partitions);
    if (!overlappingStreamPartitions.isEmpty()) {
      LOG.warn(
          "DNP: Updating watermark failed due to overlapping: {}",
          partitionsToString(overlappingStreamPartitions));
      return Optional.empty();
    }

    for (NewPartition newPartition : newPartitions) {

View on GitHub (pinned to 12126d8942)