{"record":{"id":"c37b0389e3ce8275","repo":"apache/beam","slug":"dnp-updating-watermark-failed-due-to-missing-partitions","errorCode":null,"errorMessage":"DNP: Updating watermark failed due to missing {} partitions : {}.","messagePattern":"DNP: Updating watermark failed due to missing (.+?) partitions : (.+?)\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/DetectNewPartitionsAction.java","lineNumber":146,"sourceCode":"      if (streamPartitionWithWatermark.getWatermark().compareTo(lowWatermark) < 0) {\n        lowWatermark = streamPartitionWithWatermark.getWatermark();\n      }\n      partitions.add(streamPartitionWithWatermark.getPartition());\n    }\n    if (!slowPartitions.isEmpty()) {\n      LOG.warn(\n          \"DNP: Updating watermark is held back by {} partitions : {}\",\n          slowPartitions.size(),\n          slowPartitions.stream()\n              .map(e -> formatByteStringRange(e.getPartition()) + \" => \" + e.getWatermark())\n              .collect(Collectors.joining(\", \", \"{\", \"}\")));\n    }\n\n    // Only added StreamPartitions so far, check if there are any overlapping StreamPartitions. If\n    // so, something is wrong, we should stop.\n    List<ByteStringRange> overlappingStreamPartitions = getOverlappingPartitions(partitions);\n    if (!overlappingStreamPartitions.isEmpty()) {\n      LOG.warn(\n          \"DNP: Updating watermark failed due to overlapping: {}\",\n          partitionsToString(overlappingStreamPartitions));\n      return Optional.empty();\n    }\n\n    for (NewPartition newPartition : newPartitions) {\n      partitions.addAll(newPartition.getParentPartitions());\n      if (newPartition.getLowWatermark().compareTo(lowWatermark) < 0) {\n        lowWatermark = newPartition.getLowWatermark();\n      }\n    }\n\n    List<ByteStringRange> missingPartitions = getMissingPartitionsFromEntireKeySpace(partitions);\n    if (missingPartitions.isEmpty()) {\n      LOG.info(\"DNP: Updating watermark: {}\", lowWatermark);\n      return Optional.of(lowWatermark);\n    }\n    LOG.warn(","sourceCodeStart":128,"sourceCodeEnd":164,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/DetectNewPartitionsAction.java#L128-L164","documentation":"DetectNewPartitionsAction.getNewWatermark logs 'DNP: Updating watermark failed due to missing {} partitions : {}.' when the union of known StreamPartitions does not cover the entire Bigtable keyspace. Since gaps mean unobserved row ranges, the watermark is held back (returns Optional.empty()) to preserve correctness.","triggerScenarios":"getMissingPartitionsFromEntireKeySpace(partitions) returns non-empty because partition metadata rows are missing or a new partition (from a split/merge) has not yet been registered/detected.","commonSituations":"A parent partition was closed but its children were not yet detected; metadata table rows were deleted or corrupted; pipeline start-up races where DetectNewPartitions runs before all partitions are registered.","solutions":["Wait/retry: the watermark will advance once DetectNewPartitionsAction registers the missing partitions.","Check the metadata table for missing or deleted partition rows and restore consistency.","Verify the app profile and change stream configuration so split/merge notifications are delivered.","Upgrade the connector if the gap persists across restarts (known reconciliation bugs fixed in later releases)."],"exampleFix":"// before\n// parent partition deleted manually from metadata table -> keyspace gap\n// after\n// never manually edit the change-stream metadata table; let the connector reconcile splits/merges\n// if corrupted, recreate the stream: BigtableIO.readChangeStream() on a fresh change stream","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"while (!watermarkAdvanced && attempts < MAX) { attemptWatermarkUpdate(); sleep(backoff); } // gaps usually close once new partitions are detected","preventionTips":["Let the connector reconcile splits/merges; avoid manual metadata edits.","Ensure the job is not stopped between a parent close and child registration.","Alert on watermark held back beyond SLA to catch persistent metadata gaps."],"tags":["java","apache-beam","bigtable","change-stream","watermark"],"backgroundTag":"internal-invariant-violation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}