What causes "Failed to discover new splits" in apache/iceberg?

Fix the root cause reported by the wrapped 'error' (see job logs) — usually storage access or table metadata health.

apache/iceberg · critical · RuntimeException

Failed to discover new splits

Error message

Failed to discover new splits

What it means

ContinuousIcebergEnumerator.processDiscoveredSplits wraps split discovery exceptions. Planning failures are tolerated up to scanContext.maxAllowedPlanningFailures() consecutive failures (logged as errors); once the limit is exceeded (and it is not negative = unlimited), the enumerator rethrows as RuntimeException('Failed to discover new splits'). This converts repeated transient scan-planning failures into a job failure to surface persistent table/storage problems.

Source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/source/enumerator/ContinuousIcebergEnumerator.java:184

              result.toPosition());
        } else {
          LOG.info(
              "No new splits discovered between ({}, {}]",
              result.fromPosition(),
              result.toPosition());
        }
        // update the enumerator position even if there is no split discovered
        // or the toPosition is empty (e.g. for empty table).
        enumeratorPosition.set(result.toPosition());
        LOG.info("Update enumerator position to {}", result.toPosition());
      }
    } else {
      consecutiveFailures++;
      if (scanContext.maxAllowedPlanningFailures() < 0
          || consecutiveFailures <= scanContext.maxAllowedPlanningFailures()) {
        LOG.error("Failed to discover new splits", error);
      } else {
        throw new RuntimeException("Failed to discover new splits", error);
      }
    }
  }
}

View on GitHub (pinned to 86d9c8fc54)

Solutions

Fix the root cause reported by the wrapped 'error' (see job logs) — usually storage access or table metadata health.
Raise 'stream-max-planning-failures' (ScanContext.maxAllowedPlanningFailures) or set it negative to tolerate longer outages while monitoring.
Refresh cloud credentials / Kerberos tickets and restart the job so enumeration resumes cleanly.
Pause concurrent table maintenance (expire_snapshots) that removes files needed by in-flight scans.

Example fix

// before
streaming-max-planning-failures: 20  // job dies after 20 consecutive failures during an outage
// after
streaming-max-planning-failures: -1  // keep retrying; monitor logs for 'Failed to discover new splits'

Defensive patterns

Strategy: retry

Validate before calling

// precheck storage reachability before relying on long retry budgets
boolean reachable = table.io().newInputFile(table.currentSnapshot().manifestListLocation()).exists();

Try / catch

try {
  enumerator.discoverSplits();
} catch (RuntimeException e) {
  if (e.getMessage().startsWith("Failed to discover new splits")) {
    // fix storage/metadata issue, then restart or rely on raised maxAllowedPlanningFailures
  }
}

Prevention

Set stream-max-planning-failures appropriately for your SLA (negative = unlimited retries).
Alert on the 'Failed to discover new splits' error log before the failure threshold is hit.
Refresh credentials proactively; avoid aggressive expireSnapshots during streaming consumption.

When it happens

Trigger: Streaming enumeration where table.newScan().planTasks() throws repeatedly — storage outages, expired credentials, or table metadata issues — until consecutive failures exceed stream-max-planning-failures (default 20).

Common situations: Prolonged S3/HDFS outage during streaming ingestion; credential rotation without job restart; manifests deleted by aggressive expireSnapshots running while a streaming job plans; consistently failing filter pushes due to corrupted stats.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/225a482e288fa881. Report an issue: GitHub.