iawia002/lux · error

failed to parse the Instagram response: %v

Error message

failed to parse the Instagram response: %v

What it means

Raised by lux's Instagram extractor when the embed page (https://www.instagram.com/p/<code>/embed/captioned/) loads but the inline JSON scraped out of a <script> tag fails json.Unmarshal in the colly OnHTML callback (instagram.go:93-96 stores it in collectorErr, re-raised at :108). The regex targeting \"gql_data\":...NavigationMetrics either captured a truncated/reformatted fragment or the unescaping pass (\" -> ", \\/ -> /) produced invalid JSON, so the response cannot be deserialized into instagramPayload. Note the check order: collectorErr is returned before the img.EmbeddedMediaImage fallback is consulted, so even posts whose image could be scraped fail hard when JSON parsing breaks.

Source

Thrown at extractors/instagram/instagram.go:108

		s = strings.ReplaceAll(s, `\\/`, `/`)
		s = strings.ReplaceAll(s, `\\`, `\`)

		err := json.Unmarshal([]byte(s), &embedResponse)
		if err != nil {
			collectorErr = err
		}
	})

	collector.OnRequest(func(r *colly.Request) {
		r.Headers.Set("User-Agent", browser.Chrome())
	})

	if err := collector.Visit(URL); err != nil {
		return nil, fmt.Errorf("failed to send HTTP request to the Instagram: %v", err)
	}

	if collectorErr != nil {
		return nil, fmt.Errorf("failed to parse the Instagram response: %v", collectorErr)
	}

	// If the method one which is JSON parsing didn't fail
	if !embedResponse.isEmpty() {
		result := make([]string, 0, len(embedResponse.Media.SliderItems.Edges))
		for _, item := range embedResponse.Media.SliderItems.Edges {
			result = append(result, item.Node.extractMediaURL())
		}

		return result, nil
	}

	if embeddedMediaImage != "" {
		return []string{embeddedMediaImage}, nil
	}

	// If every two methods have failed, then return an error
	return nil, errors.New("failed to fetch the post, the page might be \"private\", or the link is completely wrong")

View on GitHub (pinned to dd00f6d258)

Solutions

1. Update lux to the latest release - Instagram embed-format fixes land frequently and this error is almost always fixed upstream.
2. Retry later or from a different network/IP - if the HTML served was a rate-limit or login wall, a retry often succeeds.
3. Open https://www.instagram.com/p/<code>/embed/captioned/ in an incognito browser to confirm the post is public and the embed renders.
4. If you maintain a fork: make the img.EmbeddedMediaImage fallback run before returning collectorErr, and log the raw matched string to diagnose the format drift.

Example fix

// before (instagram.go:107-109)
if collectorErr != nil {
	return nil, fmt.Errorf("failed to parse the Instagram response: %v", collectorErr)
}

// after: try the scraped <img> fallback before failing on JSON drift
if embeddedMediaImage != "" {
	return []string{embeddedMediaImage}, nil
}
if collectorErr != nil {
	return nil, fmt.Errorf("failed to parse the Instagram response: %v", collectorErr)
}

var instagramPostRe = regexp.MustCompile(`^https?://(www\.)?instagram\.(com|net)/(p|tv|reel)/[A-Za-z0-9_-]+/?$`)

func isSupportedInstagramURL(u string) bool {
	return instagramPostRe.MatchString(u)
}

data, err := extractor.Extract(url, opts)
if err != nil {
	if strings.Contains(err.Error(), "failed to parse the Instagram response") {
		// embed-page JSON drift or a transient rate-limit page: retry, then give up
		data, err = retryExtract(extractor, url, opts, 3, 5*time.Second)
		if err != nil {
			return fmt.Errorf("instagram extraction failed (embed format may have changed, update lux): %w", err)
		}
	} else {
		return err
	}
}

When it happens

Trigger: Running lux (or calling the instagram extractor's Extract) on an instagram.com/p|tv|reel URL where Instagram serves an embed page whose embedded gql_data blob no longer matches the hard-coded regex/shape: any Instagram frontend deploy that renames NavigationMetrics, reorders the script array, or changes the escaping. Also triggered when a login-wall/rate-limit page contains a partially matching script tag, yielding malformed JSON after unescaping.

Common situations: Instagram frontend changes (the most common cause; the brittle regex breaks silently and only this error reveals it), running an outdated lux version, datacenter IPs getting rate-limited or wall-screened HTML, private/deleted posts whose error page still matches partially, and heavy batch runs tripping anti-bot responses.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• failed to fetch the post, the page might be "private", or th
• failed to send HTTP request to the Instagram: %v
• this page has no playlist
• could not find video in page
• Could not extract media URL from page

AI-assisted analysis of iawia002/lux@dd00f6d258 (2026-08-15). Data as JSON: /api/errors/a98f32bac0b0d00f. Report an issue: GitHub.