{"record":{"id":"aef678c387c68bca","repo":"Tencent/WeKnora","slug":"create-download-request-w","errorCode":null,"errorMessage":"create download request: %w","messagePattern":"create download request: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/datasource/connector/notion/client.go","lineNumber":387,"sourceCode":"\t}\n\tvar block notionBlock\n\tif err := json.Unmarshal(respBody, &block); err != nil {\n\t\treturn nil, fmt.Errorf(\"unmarshal block: %w\", err)\n\t}\n\treturn &block, nil\n}\n\nconst maxDownloadSize = 100 * 1024 * 1024 // 100MB — prevent OOM from oversized files\n\n// DownloadFile downloads a file from the given URL (typically an S3 signed URL).\n// Does not go through the rate limiter since it's not a Notion API call.\nfunc (c *notionClient) DownloadFile(ctx context.Context, fileURL string) ([]byte, error) {\n\tif err := utils.ValidateURLForSSRF(fileURL); err != nil {\n\t\treturn nil, fmt.Errorf(\"attachment URL rejected: %w\", err)\n\t}\n\treq, err := http.NewRequestWithContext(ctx, http.MethodGet, fileURL, nil)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"create download request: %w\", err)\n\t}\n\n\tvar lastErr error\n\tfor attempt := 0; attempt <= maxRetries; attempt++ {\n\t\tresp, err := c.httpClient.Do(req)\n\t\tif err != nil {\n\t\t\tlastErr = err\n\t\t\tif attempt < maxRetries {\n\t\t\t\tif sErr := sleepWithContext(ctx, time.Duration(1<<attempt)*time.Second); sErr != nil {\n\t\t\t\t\treturn nil, sErr\n\t\t\t\t}\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tbreak\n\t\t}\n\n\t\tif resp.StatusCode != http.StatusOK {\n\t\t\tresp.Body.Close()","sourceCodeStart":369,"sourceCodeEnd":405,"githubUrl":"https://github.com/Tencent/WeKnora/blob/988cbb03305e055d8ebb7d46d9ac6cc0803cd074/internal/datasource/connector/notion/client.go#L369-L405","documentation":"DownloadFile builds the attachment download with http.NewRequestWithContext(ctx, GET, fileURL, nil); if http.NewRequestWithContext fails (it only fails on malformed URLs or a nil context), the error is wrapped as 'create download request'. Since the URL already passed SSRF validation, this almost always means the URL string itself is not parseable as an absolute http(s) URL.","triggerScenarios":"Calling DownloadFile with a URL missing its scheme (e.g. \"s3/bucket/file.pdf\" or \"/files/doc.pdf\"), containing spaces or control characters, or otherwise unparsable by net/url. Also fires if ctx is nil.","commonSituations":"Storing attachment URLs in a database that stripped the https:// prefix; hand-built relative paths from a mirror config; template output with whitespace/quotes embedded in the URL; passing an empty string when the block's file URL field was absent.","solutions":["Log the exact fileURL value in the wrapped error context and run url.Parse on it locally to see the parse failure.","Ensure the URL is absolute with an http/https scheme; re-fetch the block (ResolveBlock) to get Notion's canonical signed URL instead of a stored/reconstructed one.","Guard the call site: skip downloading when the block's file URL field is empty rather than passing \"\".","Trim whitespace and reject empty/relative strings before calling DownloadFile."],"exampleFix":"// before\nclient.DownloadFile(ctx, strings.TrimSpace(attachment.Path)) // \"s3/bucket/file.pdf\"\n// after\nif u, err := url.Parse(attachment.Path); err != nil || u.Scheme == \"\" {\n    return fmt.Errorf(\"invalid attachment url %q\", attachment.Path)\n}\nclient.DownloadFile(ctx, attachment.Path)","handlingStrategy":"validation","validationCode":"func validateDownloadURL(raw string) error {\n    u, err := url.Parse(strings.TrimSpace(raw))\n    if err != nil {\n        return fmt.Errorf(\"unparsable attachment url: %w\", err)\n    }\n    if u.Scheme == \"\" || u.Host == \"\" {\n        return fmt.Errorf(\"attachment url must be absolute: %q\", raw)\n    }\n    return nil\n}","typeGuard":"func isAbsoluteHTTPURL(raw string) bool {\n    u, err := url.Parse(raw)\n    return err == nil && (u.Scheme == \"http\" || u.Scheme == \"https\") && u.Host != \"\"\n}","tryCatchPattern":"data, err := client.DownloadFile(ctx, fileURL)\nif err != nil {\n    if strings.Contains(err.Error(), \"create download request\") {\n        logger.Warnf(ctx, \"malformed attachment url %q; re-fetching block\", fileURL)\n        return downloadAfterResolve(ctx, blockID)\n    }\n    return err\n}","preventionTips":["Always take the file URL verbatim from the Notion block payload — never store a truncated or scheme-stripped version.","Skip attachments with empty URL fields instead of passing \"\" to DownloadFile.","Trim whitespace and reject relative paths at your ingestion boundary."],"tags":["http","url","download","notion"],"backgroundTag":"invalid-url-format","analyzedSha":"988cbb03305e055d8ebb7d46d9ac6cc0803cd074","analyzedAt":"2026-09-02T14:41:08.344Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}