{"record":{"id":"10f0e1ba1f522574","repo":"apache/iceberg","slug":"failed-to-discover-new-splits-10f0e1","errorCode":null,"errorMessage":"Failed to discover new splits","messagePattern":"Failed to discover new splits","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/source/enumerator/ContinuousIcebergEnumerator.java","lineNumber":184,"sourceCode":"              result.toPosition());\n        } else {\n          LOG.info(\n              \"No new splits discovered between ({}, {}]\",\n              result.fromPosition(),\n              result.toPosition());\n        }\n        // update the enumerator position even if there is no split discovered\n        // or the toPosition is empty (e.g. for empty table).\n        enumeratorPosition.set(result.toPosition());\n        LOG.info(\"Update enumerator position to {}\", result.toPosition());\n      }\n    } else {\n      consecutiveFailures++;\n      if (scanContext.maxAllowedPlanningFailures() < 0\n          || consecutiveFailures <= scanContext.maxAllowedPlanningFailures()) {\n        LOG.error(\"Failed to discover new splits\", error);\n      } else {\n        throw new RuntimeException(\"Failed to discover new splits\", error);\n      }\n    }\n  }\n}\n","sourceCodeStart":166,"sourceCodeEnd":189,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/source/enumerator/ContinuousIcebergEnumerator.java#L166-L189","documentation":"ContinuousIcebergEnumerator.processDiscoveredSplits wraps the underlying planning error in a RuntimeException once the number of consecutive incremental-scan failures exceeds scanContext.maxAllowedPlanningFailures. Until the threshold is reached the error is only logged, after which the enumerator fails the source instead of silently retrying forever.","triggerScenarios":"Continuous (streaming) enumeration repeatedly fails split discovery (table metadata reads, manifest scans) and consecutiveFailures exceeds maxAllowedPlanningFailures (default unlimited when negative).","commonSituations":"Table metadata unreadable due to permissions/object-store outages; concurrent table expiry deleting snapshots mid-scan; misconfigured catalog credentials; setting maxAllowedPlanningFailures to a small value like 0-3 makes transient outages fatal.","solutions":["Inspect the cause (the wrapped 'error') for the real root cause — catalog/auth/IO failure.","Increase 'max_allowed_planning-failures' (or the scanContext's maxAllowedPlanningFailures) to tolerate transient planning failures.","Verify catalog credentials and that the table location is reachable from the job manager.","Check whether snapshot expiration is concurrently deleting snapshots the planner needs; coordinate expiry schedules."],"exampleFix":"// before\nTableLoader loader = ...; // default low failure tolerance\n// after: tolerate transient failures\n.tableLoader(loader)\n.monitorInterval(Duration.ofSeconds(30))\n.streaming(true)\n// and set property\nproperties.put(\"max_allowed_planning-failures\", \"10\");","handlingStrategy":"retry","validationCode":"// validate catalog access before starting the job\ntableLoader.open(); Table table = tableLoader.loadTable();\ntable.refresh(); // throws early if metadata is unreachable\n// configure tolerated failures\nlong failures = Long.parseLong(properties.getProperty(\"max_allowed_planning-failures\", \"-1\"));","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set max_allowed_planning-failures generously for streaming jobs.","Avoid scheduling snapshot expiration while streaming scans are active on the same table.","Monitor catalog/object-store health and credentials expiry."],"tags":["flink","enumerator","planning","retries"],"backgroundTag":"upstream-api-error","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}