code4craft/webmagic · error · NullPointerException

task or site can not be null

Error message

task or site can not be null

What it means

HttpClientDownloader.download() requires a non-null Task whose Site is non-null; it throws NullPointerException('task or site can not be null') otherwise, since the Site holds HTTP client configuration (timeouts, headers, user agent).

Solutions

1. Always pass a Task with a valid Site, e.g. new Site().toTask() or Request/Site-based task creation
2. Build the Task before download and check task != null && task.getSite() != null
3. If using Spider, rely on its internal task instead of a hand-rolled null task

Example fix

// before
downloader.download(request, null);
// after
Task task = new Task() {
    public Site getSite() { return Site.me().setDomain("example.com"); }
    public String getUUID() { return "my-task"; }
};
downloader.download(request, task);

if (request != null && task != null && task.getSite() != null) { downloader.download(request, task); }

boolean isValidTask(Task t) { return t != null && t.getSite() != null; }

try { downloader.download(request, task); } catch (NullPointerException e) { /* supply a task with a Site */ }

When it happens

Trigger: Calling downloader.download(request, null); passing a Task whose getSite() returns null; internally, Page/download helpers invoked before a task is bound (as in test_no_task_download).

Common situations: Using the downloader standalone without constructing a Task/Site; a custom Task implementation that never sets a Site; misconfigured test harnesses.

Related errors

• language and script must not be null!
• threadNum should be more than one!
• emptySleepTime should be more than zero!
• Illegal HTTP Method
• illegal encoding

AI-assisted analysis of code4craft/webmagic@67816a19d6 (2026-09-08). Data as JSON: /api/errors/14f9db06285ddede. Report an issue: GitHub.

Appendix: source

Thrown at webmagic-core/src/main/java/us/codecraft/webmagic/downloader/HttpClientDownloader.java:73

        }
        String domain = site.getDomain();
        CloseableHttpClient httpClient = httpClients.get(domain);
        if (httpClient == null) {
            synchronized (this) {
                httpClient = httpClients.get(domain);
                if (httpClient == null) {
                    httpClient = httpClientGenerator.getClient(site);
                    httpClients.put(domain, httpClient);
                }
            }
        }
        return httpClient;
    }

    @Override
    public Page download(Request request, Task task) {
        if (task == null || task.getSite() == null) {
            throw new NullPointerException("task or site can not be null");
        }
        CloseableHttpResponse httpResponse = null;
        CloseableHttpClient httpClient = getHttpClient(task.getSite());
        Proxy proxy = proxyProvider != null ? proxyProvider.getProxy(request, task) : null;
        HttpClientRequestContext requestContext = httpUriRequestConverter.convert(request, task.getSite(), proxy);
        Page page = null;
        try {
            httpResponse = httpClient.execute(requestContext.getHttpUriRequest(), requestContext.getHttpClientContext());
            page = handleResponse(request, request.getCharset() != null ? request.getCharset() : task.getSite().getCharset(), httpResponse, task);
            onSuccess(page, task);
            return page;
        } catch (IOException e) {
            page = Page.ofFailure(request);
            onError(page, task, e);
            return page;
        } finally {
            if (httpResponse != null) {
                //ensure the connection is released back to pool

View on GitHub (pinned to 67816a19d6)