apache/beam · error · IllegalStateException

Expected at least one overlapping task in bidirectional list

Error message

Expected at least one overlapping task in bidirectional list

What it means

During bidirectional changelog analysis, analyzeFiles computes global lower/upper record-identifier bounds over the overlapping insert and delete task lists. If any of the four global bounds is null, the assumed invariant — that both lists are non-empty and overlap — was violated, so it throws IllegalStateException rather than producing bogus bounds.

Source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/ChangelogScanner.java:639

      StructLike globalDeleteLower = null;
      StructLike globalDeleteUpper = null;
      for (TaskAndBounds t : deleteTasks) {
        if (t.overlaps) {
          if (globalDeleteLower == null || idComp.compare(t.lowerId, globalDeleteLower) < 0) {
            globalDeleteLower = t.lowerId;
          }
          if (globalDeleteUpper == null || idComp.compare(t.upperId, globalDeleteUpper) > 0) {
            globalDeleteUpper = t.upperId;
          }
        }
      }

      if (globalInsertLower == null
          || globalDeleteLower == null
          || globalInsertUpper == null
          || globalDeleteUpper == null) {
        throw new IllegalStateException(
            "Expected at least one overlapping task in bidirectional list");
      }

      overlapLower =
          idComp.compare(globalInsertLower, globalDeleteLower) > 0
              ? globalInsertLower
              : globalDeleteLower;
      overlapUpper =
          idComp.compare(globalInsertUpper, globalDeleteUpper) < 0
              ? globalInsertUpper
              : globalDeleteUpper;
    }

    return new AnalysisResult(unidirectional, bidirectional, overlapLower, overlapUpper);
  }

  /**
   * Routes bi-directional tasks from an {@link AnalysisResult} to either the in-memory local

View on GitHub (pinned to 12126d8942)

Solutions

Report this to the Beam maintainers with table snapshot details — it indicates a classification/overlap bug, not user misconfiguration.
Work around by restructuring the scan interval so insert and delete tasks genuinely overlap (e.g. adjust snapshot range/start strategy).
Ensure the table's changelog scan isn't split into degenerate windows with only one task kind (compact/delete old snapshots to normalize file metadata).
Verify sequence-number ordering in table metadata is sane (repair via Iceberg metadata validation tools) since bounds derive from file sequence bounds.

Example fix

null

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-check that both task lists are non-empty before bidirectional analysis
if (insertTasks.isEmpty() || deleteTasks.isEmpty()) {
  throw new IllegalStateException("Bidirectional analysis requires insert and delete tasks");
}

Type guard

boolean hasBothSides(List<ChangelogScanTask> tasks) {
  boolean ins = tasks.stream().anyMatch(t -> t instanceof AddedRowsScanTask);
  boolean del = tasks.stream().anyMatch(t -> t instanceof DeletedDataFileScanTask || t instanceof DeletedRowsScanTask);
  return ins && del;
}

Try / catch

try {
  AnalysisResult r = scanner.result(tasks);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("Expected at least one overlapping task")) {
    // fall back to conservative allBidirectional analysis
  } else { throw e; }
}

Prevention

Report occurrences upstream — indicates a scanner bug, not user error.
Avoid degenerate scan windows containing only one task kind.
Normalize table state via compaction/snapshot expiration before CDC scans.

When it happens

Trigger: analyzeFiles reaches the bidirectional branch with either the insert or delete task list empty (no overlapping tasks), so globalInsertLower/DeleteLower/Upper never get assigned, then the null check throws.

Common situations: A CDC scan window where only inserts or only deletes occur but the code path expects bidirectional overlap; boundary/boundary-metric edge cases in file sequence ordering; a bug triggered by unusual table states (e.g. all data deleted in one snapshot).

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5bb177df4992ca11. Report an issue: GitHub.