NanmiCoder/MediaCrawler · error

Failed to parse JSON from creator notes page: {e}

Error message

Failed to parse JSON from creator notes page: {e}

What it means

Raised by BaiduTieBaClient.get_notes_by_creator (media_platform/tieba/client.py:750) when the page body returned by the getthread endpoint is not valid JSON (json.JSONDecodeError). The method reads document.body.innerText of the navigated JSON endpoint and json.loads it; if Baidu returns an HTML error/verification page, a login redirect, or an empty body, parsing fails. The error chains the original JSONDecodeError and logs the first 500 chars of page content for diagnosis.

Source

Thrown at media_platform/tieba/client.py:750

            await self.playwright_page.goto(creator_url, wait_until="domcontentloaded")

            # Wait for page loading, using delay setting from config file
            await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)

            # Get page content (this API returns JSON)
            page_content = await self.playwright_page.content()

            # Extract JSON data (page will contain <pre> tag or is directly JSON)
            try:
                # Try to extract JSON from page
                json_text = await self.playwright_page.evaluate("() => document.body.innerText")
                result = json.loads(json_text)
                utils.logger.info(f"[BaiduTieBaClient.get_notes_by_creator] Successfully retrieved creator post data")
                return result
            except json.JSONDecodeError as e:
                utils.logger.error(f"[BaiduTieBaClient.get_notes_by_creator] JSON parsing failed: {e}")
                utils.logger.error(f"[BaiduTieBaClient.get_notes_by_creator] Page content: {page_content[:500]}")
                raise Exception(f"Failed to parse JSON from creator notes page: {e}")

        except Exception as e:
            utils.logger.error(f"[BaiduTieBaClient.get_notes_by_creator] Failed to get creator post list: {e}")
            raise

    async def get_all_notes_by_creator_user_name(
        self,
        user_name: str,
        crawl_interval: float = 1.0,
        callback: Optional[Callable] = None,
        max_note_count: int = 0,
        creator_page_html_content: str = None,
    ) -> List[TiebaNote]:
        """
        Get all creator posts by creator username
        Args:
            user_name: Creator username
            crawl_interval: Crawl delay interval in seconds

View on GitHub (pinned to d6f7c5bb90)

Solutions

  1. Re-login to refresh cookies, then retry the creator fetch
  2. Check the logged page_content snippet to identify whether it is a login redirect, captcha, or proxy error and address that specific cause
  3. Increase CRAWLER_MAX_SLEEP_SEC so the JSON page fully loads before content() is read
  4. Rotate IP proxy if the content shows anti-crawl interception
Defensive patterns

Strategy: retry

Try / catch

from tenacity import retry, stop_after_attempt, wait_fixed

@retry(stop=stop_after_attempt(3), wait=wait_fixed(5), reraise=True)
async def fetch_creator_page(client, user_name, page_number):
    try:
        return await client.get_notes_by_creator(user_name, page_number)
    except Exception as e:
        if "Failed to parse JSON" in str(e):
            await refresh_login_cookies(client)  # expired session returns HTML
            raise  # retry with fresh cookies
        raise

Prevention

When it happens

Trigger: Navigating to the getthread URL while the session is degraded: expired cookies produce a login/redirect HTML page; anti-crawler interception returns a captcha/verification page; a proxy error page replaces the JSON. All yield non-JSON innerText.

Common situations: Long crawls whose cookies expire mid-run; IP flagged between requests; endpoint format changes on Baidu's side; navigating too fast so the page hasn't finished rendering JSON.

Understand the failure class

Related errors

AI-assisted analysis of NanmiCoder/MediaCrawler@d6f7c5bb90 (2026-08-15). Data as JSON: /api/errors/cc7ee702144ccbad. Report an issue: GitHub.