{"record":{"id":"cc7ee702144ccbad","repo":"NanmiCoder/MediaCrawler","slug":"failed-to-parse-json-from-creator-notes-page-e","errorCode":null,"errorMessage":"Failed to parse JSON from creator notes page: {e}","messagePattern":"Failed to parse JSON from creator notes page: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"media_platform/tieba/client.py","lineNumber":750,"sourceCode":"            await self.playwright_page.goto(creator_url, wait_until=\"domcontentloaded\")\n\n            # Wait for page loading, using delay setting from config file\n            await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)\n\n            # Get page content (this API returns JSON)\n            page_content = await self.playwright_page.content()\n\n            # Extract JSON data (page will contain <pre> tag or is directly JSON)\n            try:\n                # Try to extract JSON from page\n                json_text = await self.playwright_page.evaluate(\"() => document.body.innerText\")\n                result = json.loads(json_text)\n                utils.logger.info(f\"[BaiduTieBaClient.get_notes_by_creator] Successfully retrieved creator post data\")\n                return result\n            except json.JSONDecodeError as e:\n                utils.logger.error(f\"[BaiduTieBaClient.get_notes_by_creator] JSON parsing failed: {e}\")\n                utils.logger.error(f\"[BaiduTieBaClient.get_notes_by_creator] Page content: {page_content[:500]}\")\n                raise Exception(f\"Failed to parse JSON from creator notes page: {e}\")\n\n        except Exception as e:\n            utils.logger.error(f\"[BaiduTieBaClient.get_notes_by_creator] Failed to get creator post list: {e}\")\n            raise\n\n    async def get_all_notes_by_creator_user_name(\n        self,\n        user_name: str,\n        crawl_interval: float = 1.0,\n        callback: Optional[Callable] = None,\n        max_note_count: int = 0,\n        creator_page_html_content: str = None,\n    ) -> List[TiebaNote]:\n        \"\"\"\n        Get all creator posts by creator username\n        Args:\n            user_name: Creator username\n            crawl_interval: Crawl delay interval in seconds","sourceCodeStart":732,"sourceCodeEnd":768,"githubUrl":"https://github.com/NanmiCoder/MediaCrawler/blob/d6f7c5bb906b6dac40ddf343ef9e26438a3de092/media_platform/tieba/client.py#L732-L768","documentation":"Raised by BaiduTieBaClient.get_notes_by_creator (media_platform/tieba/client.py:750) when the page body returned by the getthread endpoint is not valid JSON (json.JSONDecodeError). The method reads document.body.innerText of the navigated JSON endpoint and json.loads it; if Baidu returns an HTML error/verification page, a login redirect, or an empty body, parsing fails. The error chains the original JSONDecodeError and logs the first 500 chars of page content for diagnosis.","triggerScenarios":"Navigating to the getthread URL while the session is degraded: expired cookies produce a login/redirect HTML page; anti-crawler interception returns a captcha/verification page; a proxy error page replaces the JSON. All yield non-JSON innerText.","commonSituations":"Long crawls whose cookies expire mid-run; IP flagged between requests; endpoint format changes on Baidu's side; navigating too fast so the page hasn't finished rendering JSON.","solutions":["Re-login to refresh cookies, then retry the creator fetch","Check the logged page_content snippet to identify whether it is a login redirect, captcha, or proxy error and address that specific cause","Increase CRAWLER_MAX_SLEEP_SEC so the JSON page fully loads before content() is read","Rotate IP proxy if the content shows anti-crawl interception"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"from tenacity import retry, stop_after_attempt, wait_fixed\n\n@retry(stop=stop_after_attempt(3), wait=wait_fixed(5), reraise=True)\nasync def fetch_creator_page(client, user_name, page_number):\n    try:\n        return await client.get_notes_by_creator(user_name, page_number)\n    except Exception as e:\n        if \"Failed to parse JSON\" in str(e):\n            await refresh_login_cookies(client)  # expired session returns HTML\n            raise  # retry with fresh cookies\n        raise","preventionTips":["Refresh cookies proactively on long crawls before they expire","Check the logged page_content prefix — login redirects vs captcha need different fixes","Give JSON endpoints time to render before reading innerText (CRAWLER_MAX_SLEEP_SEC)"],"tags":["tieba","json-parsing","anti-crawl","playwright","creator"],"backgroundTag":null,"analyzedSha":"d6f7c5bb906b6dac40ddf343ef9e26438a3de092","analyzedAt":"2026-08-15T01:39:07.505Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}