{"record":{"id":"225a482e288fa881","repo":"apache/iceberg","slug":"failed-to-discover-new-splits-225a48","errorCode":null,"errorMessage":"Failed to discover new splits","messagePattern":"Failed to discover new splits","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/source/enumerator/ContinuousIcebergEnumerator.java","lineNumber":184,"sourceCode":"              result.toPosition());\n        } else {\n          LOG.info(\n              \"No new splits discovered between ({}, {}]\",\n              result.fromPosition(),\n              result.toPosition());\n        }\n        // update the enumerator position even if there is no split discovered\n        // or the toPosition is empty (e.g. for empty table).\n        enumeratorPosition.set(result.toPosition());\n        LOG.info(\"Update enumerator position to {}\", result.toPosition());\n      }\n    } else {\n      consecutiveFailures++;\n      if (scanContext.maxAllowedPlanningFailures() < 0\n          || consecutiveFailures <= scanContext.maxAllowedPlanningFailures()) {\n        LOG.error(\"Failed to discover new splits\", error);\n      } else {\n        throw new RuntimeException(\"Failed to discover new splits\", error);\n      }\n    }\n  }\n}\n","sourceCodeStart":166,"sourceCodeEnd":189,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/source/enumerator/ContinuousIcebergEnumerator.java#L166-L189","documentation":"ContinuousIcebergEnumerator.processDiscoveredSplits wraps split discovery exceptions. Planning failures are tolerated up to scanContext.maxAllowedPlanningFailures() consecutive failures (logged as errors); once the limit is exceeded (and it is not negative = unlimited), the enumerator rethrows as RuntimeException('Failed to discover new splits'). This converts repeated transient scan-planning failures into a job failure to surface persistent table/storage problems.","triggerScenarios":"Streaming enumeration where table.newScan().planTasks() throws repeatedly — storage outages, expired credentials, or table metadata issues — until consecutive failures exceed stream-max-planning-failures (default 20).","commonSituations":"Prolonged S3/HDFS outage during streaming ingestion; credential rotation without job restart; manifests deleted by aggressive expireSnapshots running while a streaming job plans; consistently failing filter pushes due to corrupted stats.","solutions":["Fix the root cause reported by the wrapped 'error' (see job logs) — usually storage access or table metadata health.","Raise 'stream-max-planning-failures' (ScanContext.maxAllowedPlanningFailures) or set it negative to tolerate longer outages while monitoring.","Refresh cloud credentials / Kerberos tickets and restart the job so enumeration resumes cleanly.","Pause concurrent table maintenance (expire_snapshots) that removes files needed by in-flight scans."],"exampleFix":"// before\nstreaming-max-planning-failures: 20  // job dies after 20 consecutive failures during an outage\n// after\nstreaming-max-planning-failures: -1  // keep retrying; monitor logs for 'Failed to discover new splits'","handlingStrategy":"retry","validationCode":"// precheck storage reachability before relying on long retry budgets\nboolean reachable = table.io().newInputFile(table.currentSnapshot().manifestListLocation()).exists();","typeGuard":null,"tryCatchPattern":"try {\n  enumerator.discoverSplits();\n} catch (RuntimeException e) {\n  if (e.getMessage().startsWith(\"Failed to discover new splits\")) {\n    // fix storage/metadata issue, then restart or rely on raised maxAllowedPlanningFailures\n  }\n}","preventionTips":["Set stream-max-planning-failures appropriately for your SLA (negative = unlimited retries).","Alert on the 'Failed to discover new splits' error log before the failure threshold is hit.","Refresh credentials proactively; avoid aggressive expireSnapshots during streaming consumption."],"tags":["flink","streaming","split-planning","retry-exhausted"],"backgroundTag":"retry-limit-exceeded","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}