apache/hadoop · error · IOException

GetImage failed. " + StringUtils.stringifyException(t)

Error message

GetImage failed. " + StringUtils.stringifyException(t)

What it means

Thrown by ImageServlet.doGet on the NameNode when any Throwable escapes while servicing a checkpoint transfer: the servlet streams the fsimage or edit log to the Secondary/Standby NameNode through TransferFsImage.copyFileToStream with a bandwidth throttler, and this catch-all wraps every failure of that pipeline. The peer receives HTTP 410 Gone with the full stack trace stringified into the message, and the servlet rethrows as IOException. The root cause is always in the nested exception text, not in this wrapper.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java:223

              // we've already opened the 'fis' stream.
              // It's also possible length could change, but this would be
              // detected by the client side as an inaccurate length header.
            }
            // send file
            DataTransferThrottler throttler = parsedParams.isBootstrapStandby ?
                getThrottlerForBootstrapStandby(conf) : getThrottler(conf);
            TransferFsImage.copyFileToStream(response.getOutputStream(),
               file, fis, throttler);
          } finally {
            IOUtils.closeStream(fis);
          }
        }
      });
      
    } catch (Throwable t) {
      String errMsg = "GetImage failed. " + StringUtils.stringifyException(t);
      sendError(response, HttpServletResponse.SC_GONE, errMsg);
      throw new IOException(errMsg);
    } finally {
      response.getOutputStream().close();
    }
  }

  private void validateRequest(ServletContext context, Configuration conf,
      HttpServletRequest request, HttpServletResponse response,
      FSImage nnImage, String theirStorageInfoString) throws IOException {

    if (UserGroupInformation.isSecurityEnabled()
        && !isValidRequestor(context, request.getUserPrincipal().getName(),
            conf)) {
      String errorMsg = "Only Namenode, Secondary Namenode, and administrators may access "
          + "this servlet";
      sendError(response, HttpServletResponse.SC_FORBIDDEN, errorMsg);
      LOG.warn("Received non-NN/SNN/administrator request for image or edits from "
          + request.getUserPrincipal().getName()
          + " at "

View on GitHub (pinned to 2add963021)

Solutions

Read the nested stack inside the message: StringUtils.stringifyException(t) names the root cause (FileNotFoundException, DiskErrorException, socket reset) - fix that first.
Verify the requested txid exists on disk: list fsimage_* and edits_* under dfs.namenode.name.dir/current and compare with what the 2NN/Standby asked for.
If the image rolled mid-transfer, no action is needed - the 2NN/Standby retries automatically on the next checkpoint cycle.
Check health and free space of every name/edits storage directory (df, dmesg for I/O errors); remount NFS journals if stale.
For persistent standby bootstrap failures, confirm the Active NN is healthy and re-run hdfs namenode -bootstrapStandby.

Defensive patterns

Strategy: try-catch

Validate before calling

// Client side, before requesting a specific txid from the peer
File current = new File(nameDir, "current");
File[] hit = current.listFiles((d, n) -> n.equals("fsimage_" + txid));
if (hit == null || hit.length == 0) {
  // txid is not served any more; wait for the next checkpoint instead of HTTP-failing
  LOG.warn("txid {} not present on peer storage; deferring transfer", txid);
  return;
}

Try / catch

try {
  TransferFsImage.downloadImageToStorage(host, port, storage, needDigest, txid);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("GetImage failed")) {
    LOG.warn("Checkpoint transfer failed, retrying next cycle: {}", e.getMessage());
    return; // checkpoint loop retries with a fresh txid
  }
  throw e; // different failure, do not mask it
}

Prevention

Keep checkpoint cadence healthy (dfs.namenode.checkpoint.period) so requested txids still exist on disk.
Monitor free space and health of all name/edits dirs; most root causes are disk-level.
Leave fsimage/edits retention wide enough for the slowest Standby to catch up before purge.
Keep NN and 2NN/Standby versions aligned so the transfer contract matches.

When it happens

Trigger: A Secondary or Standby issues GET /getimage or /getedit for a txid whose file cannot be opened or read (findImageFile raced with an image roll, disk read error), the client disconnects mid-copy producing an IOException inside copyFileToStream, or any earlier step in the wrapped callable (parameter parsing, validation, stream setup) throws.

Common situations: Checkpoint failing on a busy NameNode when fsimage is renamed/rolled concurrently; Standby running -bootstrapStandby while the Active rolls the image; failing or full name/edits disks (especially NFS-backed edits); throttler settings (dfs.image.transfer.bandwidthPerSec) interacting with slow storage.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ef87bb4ba16abcdf. Report an issue: GitHub.