{"record":{"id":"7fbd9a94e1a66ebb","repo":"tonhowtf/omniget","slug":"could-not-extract-data-from-embed-mod","errorCode":null,"errorMessage":"Could not extract data from embed","messagePattern":"Could not extract data from embed","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"src-tauri/src/platforms/instagram/mod.rs","lineNumber":471,"sourceCode":"\n        if let Some(json_str) = Self::regex_extract(r#\"\"init\",\\[\\],\\[(.*?)\\]\\],\"#, &html) {\n            if let Ok(embed_data) = serde_json::from_str::<serde_json::Value>(&json_str) {\n                if let Some(context_json) = embed_data.get(\"contextJSON\").and_then(|v| v.as_str()) {\n                    let context: serde_json::Value = serde_json::from_str(context_json)?;\n                    return Ok(context);\n                }\n            }\n        }\n\n        if let Some(json_str) = Self::regex_extract(\n            r#\"window\\.__additionalDataLoaded\\('extra',\\s*(\\{.*?\\})\\s*\\)\"#,\n            &html,\n        ) {\n            let data: serde_json::Value = serde_json::from_str(&json_str)?;\n            return Ok(data);\n        }\n\n        Err(anyhow!(\"Could not extract data from embed\"))\n    }\n\n    async fn fallback_ytdlp(&self, url: &str, post_id: &str) -> anyhow::Result<MediaInfo> {\n        let ytdlp_path = crate::core::ytdlp::ensure_ytdlp().await?;\n        let json = crate::core::ytdlp::get_video_info(&ytdlp_path, url, &[]).await?;\n        let mut info =\n            crate::platforms::generic_ytdlp::GenericYtdlpDownloader::parse_video_info(&json)?;\n\n        info.title = format!(\"instagram_{}\", post_id);\n        info.platform = \"instagram\".to_string();\n\n        let post_url = format!(\"https://www.instagram.com/p/{}/\", post_id);\n        for q in &mut info.available_qualities {\n            q.format = \"ytdlp\".to_string();\n            q.url = post_url.clone();\n        }\n\n        Ok(info)","sourceCodeStart":453,"sourceCodeEnd":489,"githubUrl":"https://github.com/tonhowtf/omniget/blob/8600b91f4246848bac346874daa9e61c1fc5677a/src-tauri/src/platforms/instagram/mod.rs#L453-L489","documentation":"Raised by `request_embed` in InstagramDownloader after every JSON extraction strategy over the fetched embed page (regex-extracted JSON blobs) failed to parse as serde_json::Value. It means Instagram returned an embed page that did not contain any recognizable serialized data payload (e.g. no `shortcode_media`/`contextJSON` blob), so no structured post data could be produced. This is the top-level sentinel for 'embed fetch succeeded HTTP-wise but content was unusable'.","triggerScenarios":"Calling `request_embed(post_id)` when Instagram's embed endpoint returns an HTML page without parseable JSON — e.g. rate-limited/login-wall HTML, a removed or private post, or Instagram changing the embedded JSON shape so all regex extraction patterns miss.","commonSituations":"Instagram layout/API change breaking extraction regexes; the post is private, deleted, or age-restricted; datacenter IP gets served a consent/login interstitial; a typo'd post_id produces an empty embed page.","solutions":["Retry the download through the built-in fallback path (`fallback_ytdlp`), which callers already use when `request_embed` fails","Verify the post is public and renders at https://www.instagram.com/p/<post_id>/embed/ in a plain browser","Check for a login-wall or consent page in the fetched HTML and route requests through a residential proxy or add valid cookies","Update the extraction regexes/this crate to match Instagram's current embed JSON shape"],"exampleFix":"// before\nlet media = self.request_embed(&post_id).await.ok();\n// after\nlet media = match self.request_embed(&post_id).await {\n    Ok(m) => m,\n    Err(e) => {\n        tracing::warn!(\"embed extraction failed: {e:#}; falling back to yt-dlp\");\n        return self.fallback_ytdlp(url, &post_id).await;\n    }\n};","handlingStrategy":"fallback","validationCode":"// pre-check before relying on embed extraction\nasync fn embed_has_data(client: &reqwest::Client, post_id: &str) -> bool {\n    let html = client.get(format!(\"https://www.instagram.com/p/{post_id}/embed/captioned/\")).send().await.ok()?.text().await.ok()?;\n    html.contains(\"contextJSON\") || html.contains(\"shortcode_media\")\n}","typeGuard":"fn has_extractable_json(html: &str) -> bool {\n    html.contains(\"contextJSON\") || html.contains(\"shortcode_media\") || html.contains(\"display_url\")\n}","tryCatchPattern":"match downloader.request_embed(&post_id).await {\n    Ok(data) => data,\n    Err(e) if e.to_string().contains(\"Could not extract data from embed\") => {\n        // fall back to yt-dlp path\n        downloader.fallback_ytdlp(url, &post_id).await?\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Always wire the yt-dlp fallback for Instagram items instead of failing hard on embed extraction","Log the raw embed HTML on failure to detect login walls and markup changes early","Send Instagram requests through instagram_headers() and rotate IPs/proxies for datacenter traffic","Add an integration test that fetches a known public post's embed to catch Instagram markup changes"],"tags":["instagram","html-parsing","scraping","json"],"backgroundTag":"unexpected-response-shape","analyzedSha":"8600b91f4246848bac346874daa9e61c1fc5677a","analyzedAt":"2026-09-12T14:29:19.317Z","contentChangedAt":"2026-09-12T14:29:19.317Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}