{"record":{"id":"0cf06698d47c16d7","repo":"apache/beam","slug":"dnp-reconciling-missing-partition","errorCode":null,"errorMessage":"DNP: Reconciling missing partition: {}","messagePattern":"DNP: Reconciling missing partition: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"info","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/DetectNewPartitionsAction.java","lineNumber":164,"sourceCode":"      LOG.warn(\n          \"DNP: Updating watermark failed due to overlapping: {}\",\n          partitionsToString(overlappingStreamPartitions));\n      return Optional.empty();\n    }\n\n    for (NewPartition newPartition : newPartitions) {\n      partitions.addAll(newPartition.getParentPartitions());\n      if (newPartition.getLowWatermark().compareTo(lowWatermark) < 0) {\n        lowWatermark = newPartition.getLowWatermark();\n      }\n    }\n\n    List<ByteStringRange> missingPartitions = getMissingPartitionsFromEntireKeySpace(partitions);\n    if (missingPartitions.isEmpty()) {\n      LOG.info(\"DNP: Updating watermark: {}\", lowWatermark);\n      return Optional.of(lowWatermark);\n    }\n    LOG.warn(\n        \"DNP: Updating watermark failed due to missing {} partitions : {}.\",\n        missingPartitions.size(),\n        partitionsToString(missingPartitions));\n    return Optional.empty();\n  }\n\n  /**\n   * Uses PartitionReconciler to process any partitions that it has found to be missing for too long\n   * and restarts them. For more details on why this is necessary see {@link PartitionReconciler}\n   *\n   * @param receiver used to output reconciled partitions\n   * @param watermarkEstimator read the low watermark for all partitions\n   */\n  private void processReconcilerPartitions(\n      OutputReceiver<PartitionRecord> receiver,\n      ManualWatermarkEstimator<Instant> watermarkEstimator,\n      Instant startTime) {\n    for (PartitionRecord reconciledPartition :","sourceCodeStart":146,"sourceCodeEnd":182,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/DetectNewPartitionsAction.java#L146-L182","documentation":"DetectNewPartitionsAction logs 'DNP: Reconciling missing partition: {}' when computing the new watermark it finds gaps in the keyspace coverage and must reconcile (attempt to re-register) missing partitions before the watermark can advance. It is an informational reconciliation signal from the Bigtable change stream connector.","triggerScenarios":"getNewWatermark (via maybeWatermark) computes missingPartitions from the entire keyspace and each missing range must be reconciled against newPartitions/parent partitions.","commonSituations":"Ongoing split/merge operations; first runs after pipeline start; recovery after metadata inconsistency. Usually self-healing.","solutions":["No action usually needed; the connector reconciles automatically.","If the same partition is logged repeatedly, inspect the metadata table and parent partition coverage for that range.","Check that new partitions' parent lists cover the same keyspace (see ProcessNewPartitionsAction) to avoid repeated reconciliation failures."],"exampleFix":null,"handlingStrategy":"validation","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat this as informational; only investigate if the same partition is logged repeatedly.","Maintain healthy table split practices during streaming runs.","Keep connector versions current for robust reconciliation."],"tags":["java","apache-beam","bigtable","change-stream","watermark"],"backgroundTag":"internal-invariant-violation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}