{"record":{"id":"ca643ba25ec867ac","repo":"Tencent/WeKnora","slug":"downloaded-data-is-not-an-image-sniffed-s","errorCode":null,"errorMessage":"downloaded data is not an image (sniffed: %s)","messagePattern":"downloaded data is not an image \\(sniffed: (.+?)\\)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/infrastructure/docparser/image_resolver.go","lineNumber":1053,"sourceCode":"\t}\n\n\t// Read body with size limit.\n\tlimited := io.LimitReader(resp.Body, maxRemoteImageSize+1)\n\tbody, err := io.ReadAll(limited)\n\tif err != nil {\n\t\treturn nil, \"\", fmt.Errorf(\"read body: %w\", err)\n\t}\n\tif len(body) > maxRemoteImageSize {\n\t\treturn nil, \"\", fmt.Errorf(\"image exceeds %d bytes limit\", maxRemoteImageSize)\n\t}\n\n\t// If MIME was octet-stream, sniff the real type from body.\n\tif mimeType == \"application/octet-stream\" {\n\t\tdetected := http.DetectContentType(body)\n\t\tif strings.HasPrefix(detected, \"image/\") {\n\t\t\tmimeType = detected\n\t\t} else {\n\t\t\treturn nil, \"\", fmt.Errorf(\"downloaded data is not an image (sniffed: %s)\", detected)\n\t\t}\n\t}\n\n\treturn body, mimeType, nil\n}\n\n// extFromURLPath extracts the image file extension from the URL path segment.\nfunc extFromURLPath(rawURL string) string {\n\tp := path.Ext(path.Base(rawURL))\n\tswitch strings.ToLower(p) {\n\tcase \".jpg\", \".jpeg\", \".png\", \".gif\", \".webp\", \".bmp\", \".svg\":\n\t\treturn strings.ToLower(p)\n\tdefault:\n\t\treturn \"\"\n\t}\n}\n","sourceCodeStart":1035,"sourceCodeEnd":1070,"githubUrl":"https://github.com/Tencent/WeKnora/blob/988cbb03305e055d8ebb7d46d9ac6cc0803cd074/internal/infrastructure/docparser/image_resolver.go#L1035-L1070","documentation":"When the server responds with Content-Type application/octet-stream, downloadImage sniffs the actual type with http.DetectContentType; if the sniffed type does not start with image/, the payload is rejected with this error. This prevents non-image data masquerading as an image from entering the pipeline.","triggerScenarios":"fetchAndStoreRemoteImage -> downloadImage got a 200 response with Content-Type application/octet-stream whose body sniffs as text/html, application/json, text/plain, etc. — typically an HTML error/login page or a JSON API response instead of the image bytes.","commonSituations":"Expired or auth-gated image URLs returning an HTML login/error page with 200; CDN bot protection returning a challenge page; misconfigured object storage serving wrong Content-Type; URL points to an API endpoint rather than the image file.","solutions":["Open the image URL in a browser/curl to confirm it actually serves image bytes; fix or replace the URL in the source document.","Fix the server/storage to send a correct image Content-Type (e.g. image/png) so sniffing is not needed.","Remove any auth or bot-protection requirement on the image URL so fetchers get the real bytes.","Check that the URL is not a redirect landing on an HTML error or JSON error body."],"exampleFix":"// before: S3 object stored with wrong content type\naws s3 cp bad.png s3://bucket/bad.png --content-type application/octet-stream\n// after\naws s3 cp bad.png s3://bucket/bad.png --content-type image/png","handlingStrategy":"validation","validationCode":"resp, err := http.Get(imgURL)\nif err != nil { return err }\nct := resp.Header.Get(\"Content-Type\")\nif ct == \"\" || ct == \"application/octet-stream\" {\n    head := make([]byte, 512)\n    n, _ := io.ReadFull(resp.Body, head)\n    detected := http.DetectContentType(head[:n])\n    if !strings.HasPrefix(detected, \"image/\") {\n        return fmt.Errorf(\"url %s serves %s, not an image\", imgURL, detected)\n    }\n}","typeGuard":"func isImageContentType(contentType string) bool {\n    mt, _, err := mime.ParseMediaType(contentType)\n    return err == nil && strings.HasPrefix(mt, \"image/\")\n}","tryCatchPattern":"body, mimeType, err := resolver.fetchAndStoreRemoteImage(ctx, req, imgURL)\nif err != nil {\n    if strings.Contains(err.Error(), \"not an image\") {\n        log.Warnf(\"image URL %s returned non-image data; skipping\", imgURL)\n        return nil\n    }\n    return err\n}","preventionTips":["Verify every image URL in source documents actually returns image bytes before conversion (curl -I).","Set correct Content-Type headers on object storage (image/png, image/jpeg).","Avoid hotlinking URLs behind auth walls or bot-protection pages.","Monitor for this error in bulk pipelines — it usually signals stale or hijacked URLs."],"tags":["image-download","content-type","network"],"backgroundTag":"invalid-content-type","analyzedSha":"988cbb03305e055d8ebb7d46d9ac6cc0803cd074","analyzedAt":"2026-09-02T14:41:08.344Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}