XX-net/XX-Net · warning

download size:%d, need size:%d, download fail.

Error message

download size:%d, need size:%d, download fail.

What it means

Logged after a download loop when the number of bytes written to disk does not match the expected file size reported by the server. The partial file is deleted and the retry loop continues; failure of all attempts is reported by the caller.

Source

Thrown at code/default/gae_proxy/local/download_gae_lib.py:88

                file_size = int(req.getheader('Content-Length', 0))

                left = file_size
                downloaded = 0
                with open(filename, 'wb') as fp:
                    while True:
                        chunk_len = min(65536, left)
                        if not chunk_len:
                            break

                        chunk = req.read(chunk_len)
                        if not chunk:
                            break
                        fp.write(chunk)
                        downloaded += len(chunk)
                        left -= len(chunk)

            if downloaded != file_size:
                xlog.warn("download size:%d, need size:%d, download fail.", downloaded, file_size)
                os.remove(filename)
                continue
            else:
                xlog.info("download %s to %s success.", org_url, filename)
                return True
        except Exception as e:
            xlog.warn("download %s to %s fail:%r", org_url, filename, e)
            continue
    xlog.warn("download %s fail", org_url)


def download_unzip(url, extract_path):
    if os.path.isdir(extract_path):
        return True

    data_root = os.path.join(top_path, 'data')
    download_path = os.path.join(data_root, 'downloads')
    if not os.path.isdir(download_path):

View on GitHub (pinned to cfa5bc17b6)

Solutions

  1. Check network stability / retry with a different proxy IP
  2. Verify the URL returns a correct Content-Length (curl -I)
  3. Upgrade/download the library files manually and place them in the target path
  4. Inspect xlog for the paired exception message to see the underlying socket error

Example fix

// before
if not download_file(url, dfn):
    xlog.warn("download file %s fail.", url)

// after - pre-verify and retry externally, or place file manually
if not download_file(url, dfn):
    xlog.warn("download file %s fail.", url)
    # manually download url and unzip into extract_path, then restart
Defensive patterns

Strategy: retry

Validate before calling

import os
# ensure target dir writable and disk has room before triggering lib download
assert os.access(download_path, os.W_OK) and shutil.disk_usage(download_path).free > file_size_expect * 2

Prevention

When it happens

Trigger: download_file(url, filename) completes its chunk loop but `downloaded != file_size`: server closed the connection early, Content-Length mismatch, or proxy interrupted the stream.

Common situations: Flaky network through the local proxy, GAE IP reset mid-transfer, antivirus/disk issues truncating writes, server misreporting Content-Length.

Related errors

AI-assisted analysis of XX-net/XX-Net@cfa5bc17b6 (2026-08-27). Data as JSON: /api/errors/1cf7dd91d1766931. Report an issue: GitHub.