{"record":{"id":"14f9db06285ddede","repo":"code4craft/webmagic","slug":"task-or-site-can-not-be-null","errorCode":null,"errorMessage":"task or site can not be null","messagePattern":"task or site can not be null","errorType":"exception","errorClass":"NullPointerException","httpStatus":null,"severity":"error","filePath":"webmagic-core/src/main/java/us/codecraft/webmagic/downloader/HttpClientDownloader.java","lineNumber":73,"sourceCode":"        }\n        String domain = site.getDomain();\n        CloseableHttpClient httpClient = httpClients.get(domain);\n        if (httpClient == null) {\n            synchronized (this) {\n                httpClient = httpClients.get(domain);\n                if (httpClient == null) {\n                    httpClient = httpClientGenerator.getClient(site);\n                    httpClients.put(domain, httpClient);\n                }\n            }\n        }\n        return httpClient;\n    }\n\n    @Override\n    public Page download(Request request, Task task) {\n        if (task == null || task.getSite() == null) {\n            throw new NullPointerException(\"task or site can not be null\");\n        }\n        CloseableHttpResponse httpResponse = null;\n        CloseableHttpClient httpClient = getHttpClient(task.getSite());\n        Proxy proxy = proxyProvider != null ? proxyProvider.getProxy(request, task) : null;\n        HttpClientRequestContext requestContext = httpUriRequestConverter.convert(request, task.getSite(), proxy);\n        Page page = null;\n        try {\n            httpResponse = httpClient.execute(requestContext.getHttpUriRequest(), requestContext.getHttpClientContext());\n            page = handleResponse(request, request.getCharset() != null ? request.getCharset() : task.getSite().getCharset(), httpResponse, task);\n            onSuccess(page, task);\n            return page;\n        } catch (IOException e) {\n            page = Page.ofFailure(request);\n            onError(page, task, e);\n            return page;\n        } finally {\n            if (httpResponse != null) {\n                //ensure the connection is released back to pool","sourceCodeStart":55,"sourceCodeEnd":91,"githubUrl":"https://github.com/code4craft/webmagic/blob/67816a19d68a4fec4657bf1336227e046e251df2/webmagic-core/src/main/java/us/codecraft/webmagic/downloader/HttpClientDownloader.java#L55-L91","documentation":"HttpClientDownloader.download() requires a non-null Task whose Site is non-null; it throws NullPointerException('task or site can not be null') otherwise, since the Site holds HTTP client configuration (timeouts, headers, user agent).","triggerScenarios":"Calling downloader.download(request, null); passing a Task whose getSite() returns null; internally, Page/download helpers invoked before a task is bound (as in test_no_task_download).","commonSituations":"Using the downloader standalone without constructing a Task/Site; a custom Task implementation that never sets a Site; misconfigured test harnesses.","solutions":["Always pass a Task with a valid Site, e.g. new Site().toTask() or Request/Site-based task creation","Build the Task before download and check task != null && task.getSite() != null","If using Spider, rely on its internal task instead of a hand-rolled null task"],"exampleFix":"// before\ndownloader.download(request, null);\n// after\nTask task = new Task() {\n    public Site getSite() { return Site.me().setDomain(\"example.com\"); }\n    public String getUUID() { return \"my-task\"; }\n};\ndownloader.download(request, task);","handlingStrategy":"validation","validationCode":"if (request != null && task != null && task.getSite() != null) { downloader.download(request, task); }","typeGuard":"boolean isValidTask(Task t) { return t != null && t.getSite() != null; }","tryCatchPattern":"try { downloader.download(request, task); } catch (NullPointerException e) { /* supply a task with a Site */ }","preventionTips":["Never call the downloader standalone without a Task backed by a Site","Use Site.me().toTask() style helpers","Unit-test custom Task implementations so getSite() is non-null"],"tags":["null-check","downloader","argument-validation"],"backgroundTag":"null-argument","analyzedSha":"67816a19d68a4fec4657bf1336227e046e251df2","analyzedAt":"2026-09-08T11:15:28.425Z","contentChangedAt":"2026-09-08T11:15:28.425Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}