{"record":{"id":"c8c4911fa5b119ce","repo":"apache/beam","slug":"dnp-updating-watermark-failed-due-to-overlapping","errorCode":null,"errorMessage":"DNP: Updating watermark failed due to overlapping: {}","messagePattern":"DNP: Updating watermark failed due to overlapping: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/DetectNewPartitionsAction.java","lineNumber":134,"sourceCode":"      List<StreamPartitionWithWatermark> streamPartitionsWithWatermark,\n      List<NewPartition> newPartitions) {\n    // Get partitions with a watermark set but skip rows w a lock and no watermark yet\n    List<StreamPartitionWithWatermark> slowPartitions = new ArrayList<>();\n    Instant lowWatermark = Instant.ofEpochMilli(Long.MAX_VALUE);\n\n    List<ByteStringRange> partitions = new ArrayList<>();\n    for (StreamPartitionWithWatermark streamPartitionWithWatermark :\n        streamPartitionsWithWatermark) {\n      if (streamPartitionWithWatermark.getWatermark().plus(DEBUG_WATERMARK_DELAY).isBeforeNow()) {\n        slowPartitions.add(streamPartitionWithWatermark);\n      }\n      if (streamPartitionWithWatermark.getWatermark().compareTo(lowWatermark) < 0) {\n        lowWatermark = streamPartitionWithWatermark.getWatermark();\n      }\n      partitions.add(streamPartitionWithWatermark.getPartition());\n    }\n    if (!slowPartitions.isEmpty()) {\n      LOG.warn(\n          \"DNP: Updating watermark is held back by {} partitions : {}\",\n          slowPartitions.size(),\n          slowPartitions.stream()\n              .map(e -> formatByteStringRange(e.getPartition()) + \" => \" + e.getWatermark())\n              .collect(Collectors.joining(\", \", \"{\", \"}\")));\n    }\n\n    // Only added StreamPartitions so far, check if there are any overlapping StreamPartitions. If\n    // so, something is wrong, we should stop.\n    List<ByteStringRange> overlappingStreamPartitions = getOverlappingPartitions(partitions);\n    if (!overlappingStreamPartitions.isEmpty()) {\n      LOG.warn(\n          \"DNP: Updating watermark failed due to overlapping: {}\",\n          partitionsToString(overlappingStreamPartitions));\n      return Optional.empty();\n    }\n\n    for (NewPartition newPartition : newPartitions) {","sourceCodeStart":116,"sourceCodeEnd":152,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/DetectNewPartitionsAction.java#L116-L152","documentation":"DetectNewPartitionsAction.getNewWatermark logs 'DNP: Updating watermark failed due to overlapping: {}' when the set of StreamPartitions assembled for the watermark computation contains overlapping key ranges. Overlapping partitions indicate inconsistent metadata state, so the action returns Optional.empty() and the watermark is not advanced.","triggerScenarios":"getNewWatermark (called from maybeWatermark) detects that two or more ByteStringRange partitions in the metadata table overlap, typically after a partition split or merge was partially recorded.","commonSituations":"Concurrent split/merge operations on the Bigtable table while the change stream pipeline is running; stale or partially updated rows in the Bigtable change stream metadata table; a previous pipeline run crashed mid-partition-update.","solutions":["Verify the metadata table (Bigtable change streams metadata) for overlapping partition rows and repair/consolidate them.","Ensure only one pipeline instance is processing the change stream for a given stream partition (exclusive app profile routing, single runner).","Restart the pipeline so DetectNewPartitions re-syncs partition state from the change stream.","Upgrade the connector, as partition reconciliation logic has been fixed in newer releases."],"exampleFix":"// before\n// two runners sharing one metadata table caused overlapping partitions\n// after\n// run a single streaming job per change stream, using an app profile with single-row transactions\nBigtableIO.readChangeStream().withAppProfile(\"single-instance\");","handlingStrategy":"validation","validationCode":"// Before running: ensure exactly one streaming job per change stream and a healthy metadata table\n// (Bigtable CLI) scan the metadata table and assert no two partition rows overlap","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Run a single pipeline instance per change stream (single-row-transaction app profile).","Never manually edit the change-stream metadata table.","Watch DNP warnings in logs; a stuck watermark signals metadata inconsistency to fix early."],"tags":["java","apache-beam","bigtable","change-stream","watermark"],"backgroundTag":"internal-invariant-violation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}