DNP: Updating watermark failed due to missing

Error message

DNP: Updating watermark failed due to missing {} partitions : {}.

What it means

DetectNewPartitionsAction.getNewWatermark logs 'DNP: Updating watermark failed due to missing {} partitions : {}.' when the union of known StreamPartitions does not cover the entire Bigtable keyspace. Since gaps mean unobserved row ranges, the watermark is held back (returns Optional.empty()) to preserve correctness.

Solutions

Wait/retry: the watermark will advance once DetectNewPartitionsAction registers the missing partitions.
Check the metadata table for missing or deleted partition rows and restore consistency.
Verify the app profile and change stream configuration so split/merge notifications are delivered.
Upgrade the connector if the gap persists across restarts (known reconciliation bugs fixed in later releases).

Example fix

// before
// parent partition deleted manually from metadata table -> keyspace gap
// after
// never manually edit the change-stream metadata table; let the connector reconcile splits/merges
// if corrupted, recreate the stream: BigtableIO.readChangeStream() on a fresh change stream

Defensive patterns

Strategy: retry

Try / catch

while (!watermarkAdvanced && attempts < MAX) { attemptWatermarkUpdate(); sleep(backoff); } // gaps usually close once new partitions are detected

Prevention

Let the connector reconcile splits/merges; avoid manual metadata edits.
Ensure the job is not stopped between a parent close and child registration.
Alert on watermark held back beyond SLA to catch persistent metadata gaps.

When it happens

Trigger: getMissingPartitionsFromEntireKeySpace(partitions) returns non-empty because partition metadata rows are missing or a new partition (from a split/merge) has not yet been registered/detected.

Common situations: A parent partition was closed but its children were not yet detected; metadata table rows were deleted or corrupted; pipeline start-up races where DetectNewPartitions runs before all partitions are registered.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c37b0389e3ce8275. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/DetectNewPartitionsAction.java:146

      if (streamPartitionWithWatermark.getWatermark().compareTo(lowWatermark) < 0) {
        lowWatermark = streamPartitionWithWatermark.getWatermark();
      }
      partitions.add(streamPartitionWithWatermark.getPartition());
    }
    if (!slowPartitions.isEmpty()) {
      LOG.warn(
          "DNP: Updating watermark is held back by {} partitions : {}",
          slowPartitions.size(),
          slowPartitions.stream()
              .map(e -> formatByteStringRange(e.getPartition()) + " => " + e.getWatermark())
              .collect(Collectors.joining(", ", "{", "}")));
    }

    // Only added StreamPartitions so far, check if there are any overlapping StreamPartitions. If
    // so, something is wrong, we should stop.
    List<ByteStringRange> overlappingStreamPartitions = getOverlappingPartitions(partitions);
    if (!overlappingStreamPartitions.isEmpty()) {
      LOG.warn(
          "DNP: Updating watermark failed due to overlapping: {}",
          partitionsToString(overlappingStreamPartitions));
      return Optional.empty();
    }

    for (NewPartition newPartition : newPartitions) {
      partitions.addAll(newPartition.getParentPartitions());
      if (newPartition.getLowWatermark().compareTo(lowWatermark) < 0) {
        lowWatermark = newPartition.getLowWatermark();
      }
    }

    List<ByteStringRange> missingPartitions = getMissingPartitionsFromEntireKeySpace(partitions);
    if (missingPartitions.isEmpty()) {
      LOG.info("DNP: Updating watermark: {}", lowWatermark);
      return Optional.of(lowWatermark);
    }
    LOG.warn(

View on GitHub (pinned to 12126d8942)