{"record":{"id":"7eb1bc4fac97d593","repo":"apache/iceberg","slug":"table-refresh-failed-7eb1bc","errorCode":null,"errorMessage":"Table refresh failed","messagePattern":"Table refresh failed","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java","lineNumber":216,"sourceCode":"              Pair<StreamingOffset, FileScanTask> nextElem = queue.peekFirst();\n              boolean endOffsetPeek = false;\n              if (nextElem != null) {\n                endOffsetPeek = endOffset.equals(nextElem.first());\n              }\n              // end offset may be synthetic and not exist in the queue\n              boolean endOffsetSynthetic =\n                  currentOffset.snapshotId() == endOffset.snapshotId()\n                      && (currentOffset.position() + 1) == endOffset.position();\n              shouldTerminate = endOffsetPeek || endOffsetSynthetic;\n            } else {\n              LOG.trace(\"planFiles hasn't reached {}, waiting\", endOffset);\n            }\n          } while (!shouldTerminate\n              && refreshFailedThrowable == null\n              && fillQueueFailedThrowable == null);\n\n          if (refreshFailedThrowable != null) {\n            throw new RuntimeException(\"Table refresh failed\", refreshFailedThrowable);\n          }\n\n          if (fillQueueFailedThrowable != null) {\n            throw new RuntimeException(\"Queue filling failed\", fillQueueFailedThrowable);\n          }\n\n          LOG.info(\n              \"completed planFiles for {}, startOffset: {}, endOffset: {}, files: {}, rows: {}\",\n              table().name(),\n              startOffset,\n              endOffset,\n              filesInPlan,\n              rowsInPlan);\n          return result;\n        });\n  }\n\n  /**","sourceCodeStart":198,"sourceCodeEnd":234,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java#L198-L234","documentation":"The async micro-batch planner refreshes the table and fills a planning queue on background tasks, recording any Throwable in refreshFailedThrowable. After the planning loop exits, if a refresh failure was captured, planFiles rethrows it as RuntimeException 'Table refresh failed' with the original cause attached. This surfaces background table-refresh failures to the streaming query.","triggerScenarios":"During planFiles' loop, the background table refresh task fails (e.g. metadata read error, catalog unreachable, permission revoked, invalid/expired credentials), the loop terminates, and the stored throwable is wrapped and thrown.","commonSituations":"S3/GCS/Azure credentials expiring mid-stream; catalog (REST/Hive/Hadoop) temporarily unavailable; table metadata corrupted or concurrently rewritten causing refresh errors; network partitions between the driver and object storage.","solutions":["Read the wrapped cause (refreshFailedThrowable) to identify the real failure and fix it (credentials, network, catalog).","Add retry/credentials refresh: configure a session token provider or re-login mechanism for the FileIO/catalog.","Restart the streaming query once the underlying storage/catalog issue is resolved.","Check table metadata health and catalog availability; verify the table hasn't been dropped or made unreadable."],"exampleFix":"// before: long-lived static creds\nspark.conf.set(\"s3.access-key\", ...);\n// after: refreshable provider\nspark.conf.set(\"s3.session-token-provider.type\", \"container\"); // or IAM role","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { planFiles(...); } catch (RuntimeException e) { if (\"Table refresh failed\".equals(e.getMessage())) { Throwable cause = e.getCause(); /* check creds/catalog and retry or restart the stream */ } else throw e; }","preventionTips":["Use refreshable credential providers for object storage.","Monitor catalog availability and table metadata health.","Add automatic restart for streaming queries on transient refresh failures."],"tags":["spark","streaming","table-refresh"],"backgroundTag":"http-request-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}