{"record":{"id":"a98f32bac0b0d00f","repo":"iawia002/lux","slug":"failed-to-parse-the-instagram-response-v","errorCode":null,"errorMessage":"failed to parse the Instagram response: %v","messagePattern":"failed to parse the Instagram response: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"extractors/instagram/instagram.go","lineNumber":108,"sourceCode":"\t\ts = strings.ReplaceAll(s, `\\\\/`, `/`)\n\t\ts = strings.ReplaceAll(s, `\\\\`, `\\`)\n\n\t\terr := json.Unmarshal([]byte(s), &embedResponse)\n\t\tif err != nil {\n\t\t\tcollectorErr = err\n\t\t}\n\t})\n\n\tcollector.OnRequest(func(r *colly.Request) {\n\t\tr.Headers.Set(\"User-Agent\", browser.Chrome())\n\t})\n\n\tif err := collector.Visit(URL); err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to send HTTP request to the Instagram: %v\", err)\n\t}\n\n\tif collectorErr != nil {\n\t\treturn nil, fmt.Errorf(\"failed to parse the Instagram response: %v\", collectorErr)\n\t}\n\n\t// If the method one which is JSON parsing didn't fail\n\tif !embedResponse.isEmpty() {\n\t\tresult := make([]string, 0, len(embedResponse.Media.SliderItems.Edges))\n\t\tfor _, item := range embedResponse.Media.SliderItems.Edges {\n\t\t\tresult = append(result, item.Node.extractMediaURL())\n\t\t}\n\n\t\treturn result, nil\n\t}\n\n\tif embeddedMediaImage != \"\" {\n\t\treturn []string{embeddedMediaImage}, nil\n\t}\n\n\t// If every two methods have failed, then return an error\n\treturn nil, errors.New(\"failed to fetch the post, the page might be \\\"private\\\", or the link is completely wrong\")","sourceCodeStart":90,"sourceCodeEnd":126,"githubUrl":"https://github.com/iawia002/lux/blob/dd00f6d258d80b6684a0b9402d7124e5c18ef42f/extractors/instagram/instagram.go#L90-L126","documentation":"Raised by lux's Instagram extractor when the embed page (https://www.instagram.com/p/<code>/embed/captioned/) loads but the inline JSON scraped out of a <script> tag fails json.Unmarshal in the colly OnHTML callback (instagram.go:93-96 stores it in collectorErr, re-raised at :108). The regex targeting \\\"gql_data\\\":...NavigationMetrics either captured a truncated/reformatted fragment or the unescaping pass (\\\" -> \", \\\\/ -> /) produced invalid JSON, so the response cannot be deserialized into instagramPayload. Note the check order: collectorErr is returned before the img.EmbeddedMediaImage fallback is consulted, so even posts whose image could be scraped fail hard when JSON parsing breaks.","triggerScenarios":"Running lux (or calling the instagram extractor's Extract) on an instagram.com/p|tv|reel URL where Instagram serves an embed page whose embedded gql_data blob no longer matches the hard-coded regex/shape: any Instagram frontend deploy that renames NavigationMetrics, reorders the script array, or changes the escaping. Also triggered when a login-wall/rate-limit page contains a partially matching script tag, yielding malformed JSON after unescaping.","commonSituations":"Instagram frontend changes (the most common cause; the brittle regex breaks silently and only this error reveals it), running an outdated lux version, datacenter IPs getting rate-limited or wall-screened HTML, private/deleted posts whose error page still matches partially, and heavy batch runs tripping anti-bot responses.","solutions":["Update lux to the latest release - Instagram embed-format fixes land frequently and this error is almost always fixed upstream.","Retry later or from a different network/IP - if the HTML served was a rate-limit or login wall, a retry often succeeds.","Open https://www.instagram.com/p/<code>/embed/captioned/ in an incognito browser to confirm the post is public and the embed renders.","If you maintain a fork: make the img.EmbeddedMediaImage fallback run before returning collectorErr, and log the raw matched string to diagnose the format drift."],"exampleFix":"// before (instagram.go:107-109)\nif collectorErr != nil {\n\treturn nil, fmt.Errorf(\"failed to parse the Instagram response: %v\", collectorErr)\n}\n\n// after: try the scraped <img> fallback before failing on JSON drift\nif embeddedMediaImage != \"\" {\n\treturn []string{embeddedMediaImage}, nil\n}\nif collectorErr != nil {\n\treturn nil, fmt.Errorf(\"failed to parse the Instagram response: %v\", collectorErr)\n}","handlingStrategy":"retry","validationCode":"var instagramPostRe = regexp.MustCompile(`^https?://(www\\.)?instagram\\.(com|net)/(p|tv|reel)/[A-Za-z0-9_-]+/?$`)\n\nfunc isSupportedInstagramURL(u string) bool {\n\treturn instagramPostRe.MatchString(u)\n}","typeGuard":null,"tryCatchPattern":"data, err := extractor.Extract(url, opts)\nif err != nil {\n\tif strings.Contains(err.Error(), \"failed to parse the Instagram response\") {\n\t\t// embed-page JSON drift or a transient rate-limit page: retry, then give up\n\t\tdata, err = retryExtract(extractor, url, opts, 3, 5*time.Second)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"instagram extraction failed (embed format may have changed, update lux): %w\", err)\n\t\t}\n\t} else {\n\t\treturn err\n\t}\n}","preventionTips":["Keep lux updated - Instagram's embed markup changes often and this error is usually fixed upstream first","Only enqueue public post/reel URLs you have verified render in an incognito browser","Rate-limit your own batch requests so Instagram does not serve login-wall HTML","Treat this error differently from 'failed to fetch the post...' - it means HTML arrived but parsing broke","Log the failing shortCode so you can re-check those URLs after upgrading lux"],"tags":["instagram","json","parsing","scraping","colly","api-drift"],"backgroundTag":null,"analyzedSha":"dd00f6d258d80b6684a0b9402d7124e5c18ef42f","analyzedAt":"2026-08-15T16:57:31.080Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}