apache/hadoop · error · IOException

File " + url + " received length " + received + " is not of

Error message

File " + url + " received length " + received + " is not of the advertised size " + advertisedSize + ". Fsimage name: " + fsImageName + " lastReceived: " + num

What it means

receiveFile counts bytes until the stream signals EOF; if the loop completed cleanly (finishedReceiving) but the byte count differs from the Content-Length advertised by the NameNode, the temp files are deleted and this mismatch is thrown. The finishedReceiving guard means a client-side read exception is never masked — this fires only when the server side delivered fewer bytes than promised with a clean end of stream.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java:312

        xferCombined += writeSec;
        xferStats.append(String
            .format(" Synchronous (fsync) write to disk of " +
                streamPathMap.get(fos).getAbsolutePath() +
                " took %.2fs.", writeSec));
      }

      // Something went wrong and did not finish reading.
      // Remove the temporary files.
      if (!finishedReceiving) {
        deleteTmpFiles(localPaths);
      }

      if (finishedReceiving && received != advertisedSize) {
        // only throw this exception if we think we read all of it on our end
        // -- otherwise a client-side IOException would be masked by this
        // exception that makes it look like a server-side problem!
        deleteTmpFiles(localPaths);
        throw new IOException("File " + url + " received length " + received +
            " is not of the advertised size " + advertisedSize +
            ". Fsimage name: " + fsImageName + " lastReceived: " + num);
      }
    }
    xferStats.insert(0, String.format("Combined time for file download and" +
        " fsync to all disks took %.2fs.", xferCombined));
    LOG.info(xferStats.toString());

    if (digester != null) {
      MD5Hash computedDigest = new MD5Hash(digester.digest());

      if (advertisedDigest != null &&
          !computedDigest.equals(advertisedDigest)) {
        deleteTmpFiles(localPaths);
        throw new IOException("File " + url + " computed digest " +
            computedDigest + " does not match advertised digest " +
            advertisedDigest);
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the checkpoint/download — the temp files are already cleaned up and the transfer is idempotent
2. Raise or remove proxy/LB max-response-size and idle timeouts on the NN HTTP path
3. Check NameNode logs around the transfer time for a dying servlet thread or failover
4. Confirm the image on the NN has a stable size (no concurrent checkpoint writing it) before retrying

int attempts = 0;
while (true) {
  try {
    Util.doGetUrl(url, localPaths, storage, true, timeoutMs, throttler);
    break;
  } catch (IOException e) {
    boolean truncated = e.getMessage() != null
        && e.getMessage().contains("is not of the advertised size");
    if (!truncated || ++attempts >= 3) throw e;
    // temp files already removed; safe to retry after backoff
    Thread.sleep(attempts * 5000L);
  }
}

When it happens

Trigger: The NN (or an intermediary) closes the connection early but cleanly: NN crashing or being restarted mid-transfer, a proxy/LB truncating a large fsimage body on an idle/max-body-size timeout, or a concurrent checkpoint replacing the file while it streams.

Common situations: Large images (tens of GB) cut off by an LB with a response-size cap; NN failover during the 2NN/standby download; flaky middleboxes that close streams without RST.

Related errors

• The length of the feature flag section was negative at {} by
• Content-Length header is not provided by the namenode when t
• Unable to download to any storage directory
• Safe mode should be turned ON in order to create namespace i
• Could not find image with txid " + txid

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ca1d6bf8d663a4dd. Report an issue: GitHub.