apache/iceberg · error · RuntimeException

Failed to discover new splits

Error message

Failed to discover new splits

What it means

The continuous enumerator's split discovery (planning) failed repeatedly. Failures are tolerated up to maxAllowedPlanningFailures (or indefinitely if negative); once consecutive failures exceed that limit, processDiscoveredSplits rethrows as RuntimeException and the enumerator fails.

Source

Thrown at flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/source/enumerator/ContinuousIcebergEnumerator.java:184

              result.toPosition());
        } else {
          LOG.info(
              "No new splits discovered between ({}, {}]",
              result.fromPosition(),
              result.toPosition());
        }
        // update the enumerator position even if there is no split discovered
        // or the toPosition is empty (e.g. for empty table).
        enumeratorPosition.set(result.toPosition());
        LOG.info("Update enumerator position to {}", result.toPosition());
      }
    } else {
      consecutiveFailures++;
      if (scanContext.maxAllowedPlanningFailures() < 0
          || consecutiveFailures <= scanContext.maxAllowedPlanningFailures()) {
        LOG.error("Failed to discover new splits", error);
      } else {
        throw new RuntimeException("Failed to discover new splits", error);
      }
    }
  }
}

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Fix the underlying planning error (check logs for the wrapped cause: catalog connectivity, permissions, expired snapshot).
2. Increase streaming-planning-failures (maxAllowedPlanningFailures) to tolerate transient outages.
3. Set maxAllowedPlanningFailures to -1 to retry indefinitely if failures are expected to be transient.
4. Extend snapshot retention / increase min-snapshots-to-keep so referenced snapshots are not expired.

Example fix

// before
table.updateProperties().set("streaming-planning-failures", "0");
// after
table.updateProperties().set("streaming-planning-failures", "10"); // tolerate transient planning failures

// Verify table is readable before starting the streaming source
try (org.apache.iceberg.TableScan scan = table.newScan()) {
  scan.useSnapshot(table.currentSnapshot().snapshotId()).planFiles();
}

When it happens

Trigger: Continuous streaming enumeration when table.scan throws (e.g., snapshot expired, catalog/network errors) and consecutiveFailures exceeds scanContext.maxAllowedPlanningFailures().

Common situations: See trigger scenarios.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• Failed to discover new splits
• Failed to process tasks iterable
• Failed to process task iterable:
• Unknown starting strategy:
• Failed to process tasks iterable

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/0444c419187887a5. Report an issue: GitHub.