{"record":{"id":"b9b676402ae10215","repo":"apache/hadoop","slug":"wait-failed-for-get-d","errorCode":null,"errorMessage":"Wait failed for get(%d)","messagePattern":"Wait failed for get\\((.+?)\\)","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/CachingBlockManager.java","lineNumber":173,"sourceCode":"      if (closed) {\n        throw new IOException(\"this stream is already closed\");\n      }\n\n      data = bufferPool.acquire(blockNumber);\n      done = getInternal(data);\n\n      if (retryer.updateStatus()) {\n        LOG.warn(\"waiting to get block: {}\", blockNumber);\n        LOG.info(\"state = {}\", this.toString());\n      }\n    }\n    while (!done && retryer.continueRetry());\n\n    if (done) {\n      return data;\n    } else {\n      String message = String.format(\"Wait failed for get(%d)\", blockNumber);\n      throw new IllegalStateException(message);\n    }\n  }\n\n  private boolean getInternal(BufferData data) throws IOException {\n    Validate.checkNotNull(data, \"data\");\n\n    // Opportunistic check without locking.\n    if (data.stateEqualsOneOf(\n        BufferData.State.PREFETCHING,\n        BufferData.State.CACHING,\n        BufferData.State.DONE)) {\n      return false;\n    }\n\n    synchronized (data) {\n      // Reconfirm state after locking.\n      if (data.stateEqualsOneOf(\n          BufferData.State.PREFETCHING,","sourceCodeStart":155,"sourceCodeEnd":191,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/CachingBlockManager.java#L155-L191","documentation":"CachingBlockManager.get loops acquire + getInternal until the block reaches a terminal DONE state, retrying under a Retryer (10 attempts, max delay scaling with pool size: bufferPoolSize*120*1000 ms). If the block never completes — usually because the remote store keeps failing the fetch — it gives up with IllegalStateException(\"Wait failed for get(N)\"), after logging \"waiting to get block\" warnings that include the manager state.","triggerScenarios":"The underlying data source (object store / http backend) returning repeated errors for the block's byte range; a prefetch task dying without marking the block DONE; every attempt exceeding the retry window on a very slow link.","commonSituations":"Object-store credential/permission failures surfacing mid-file; throttling (429/503) during large parallel prefetches; prefetch configuration (block size, pool size) mismatched to the file and read pattern; network brownouts stretching every fetch past the budget.","solutions":["Read the earlier log lines: the first underlying store exception (auth, 403/404, throttle) is the root cause — fix that first","Inspect the \"waiting to get block\" / \"state =\" warnings to see which blocks stay PREFETCHING/CACHING and why","Raise the retry budget/delay or reduce read parallelism so attempts fit inside the window","Verify data-source health and prefetch settings (block size, pool size) against the file size being read"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch IllegalStateException(\"Wait failed for get\"); check logs for the underlying store failure and fix it, then reopen the stream at the last known position and retry the read once with reduced parallelism.","preventionTips":["Fix root-cause store errors (auth, 403/404, throttling) — the ISE is only the symptom","Size retry budget and pool to the slowest expected fetch, not the average","Watch the \"waiting to get block\" warnings as the leading indicator of exhaustion"],"tags":["hadoop","prefetch","retry-exhausted","object-store","illegal-state"],"backgroundTag":"retry-budget-exhausted","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}