apache/beam · warning

DNP: New partition does not have all the parents

Error message

DNP: New partition does not have all the parents: {}

What it means

ProcessNewPartitionsAction.processNewPartition logs 'DNP: New partition does not have all the parents: {}' and returns false when the parent partitions recorded for a NewPartition do not cover the same keyspace as the new partition itself. The new partition is not registered, preventing incorrect state in the change stream connector.

Solutions

  1. Inspect the metadata table row for the NewPartition and verify parentPartitions cover the same token range.
  2. Delete/repair the inconsistent NewPartition row so it can be re-detected from the change stream.
  3. Ensure a single job instance owns the stream and avoid manual metadata edits.
  4. Upgrade to a newer Beam release where split/merge parent tracking was fixed.

Example fix

// before
// metadata row: newPartition=[a,c) parents=[[a,b)]  (gap [b,c))
// after
// repair row so parents cover the full range: parents=[[a,b),[b,c)]
// or remove the row and let DetectNewPartitionsAction re-register it
Defensive patterns

Strategy: validation

Validate before calling

// Validate parent coverage before writing a NewPartition row: parents' union must equal newPartition range

Prevention

When it happens

Trigger: A NewPartition row read from the metadata table lists parent partitions whose union of ByteStringRanges does not exactly cover the new partition's range (e.g. split recorded with wrong parents, or parents already cleaned up).

Common situations: Corrupted or manually edited metadata table rows; races between merge operations and parent cleanup; connector bugs during concurrent split and merge of adjacent partitions.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8b1cec23b04a444c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/ProcessNewPartitionsAction.java:90

   *   <li>processNewPartition process A-C seeing that only A-B has been recorded and A-B does not
   *       cover A-C. Do Nothing
   *   <li>p2 writes to row A-C in metadata table
   *   <li>processNewPartition process A-C again, seeing that A-B and B-C has been recorded and
   *       outputs new partition A-C to be streamed.
   * </ol>
   *
   * <p>Note that, the algorithm to verify if a merge is valid, also correctly verifies if a split
   * is valid. A split is immediately valid as long as the row exists because there's only one
   * parent that needs to write to that row.
   *
   * @param newPartition new partition to be processed
   * @param receiver to output new partitions
   */
  public boolean processNewPartition(
      NewPartition newPartition, OutputReceiver<PartitionRecord> receiver) {
    List<ByteStringRange> parentPartitions = newPartition.getParentPartitions();
    if (!coverSameKeySpace(parentPartitions, newPartition.getPartition())) {
      LOG.warn("DNP: New partition does not have all the parents: {}", newPartition);
      return false;
    }

    String uuid = UniqueIdGenerator.getNextId();
    PartitionRecord partitionRecord =
        new PartitionRecord(
            newPartition.getPartition(),
            newPartition.getChangeStreamContinuationTokens(),
            uuid,
            newPartition.getLowWatermark(),
            Collections.singletonList(newPartition),
            endTime);
    if (parentPartitions.size() > 1) {
      metrics.incPartitionMergeCount();
    } else {
      metrics.incPartitionSplitCount();
    }
    LOG.info("DNP: Outputting new partition: {}", partitionRecord);

View on GitHub (pinned to 12126d8942)