NanmiCoder/MediaCrawler · warning · HTTPException

Invalid JSON file

Error message

Invalid JSON file

What it means

HTTP 400 raised in the preview branch when json.load() raises JSONDecodeError for a .json-suffixed file. The file exists and passed the security check, but its content is not valid JSON (truncated, empty, BOM-corrupted, or NDJSON).

Source

Thrown at api/routers/data.py:154

                    return {"data": rows, "total": total}
            elif full_path.suffix.lower() in (".xlsx", ".xls"):
                import pandas as pd
                # Read first limit rows
                df = pd.read_excel(full_path, nrows=limit)
                # Get total row count (only read first column to save memory)
                df_count = pd.read_excel(full_path, usecols=[0])
                total = len(df_count)
                # Convert to list of dictionaries, handle NaN values
                rows = df.where(pd.notnull(df), None).to_dict(orient='records')
                return {
                    "data": rows,
                    "total": total,
                    "columns": list(df.columns)
                }
            else:
                raise HTTPException(status_code=400, detail="Unsupported file type for preview")
        except json.JSONDecodeError:
            raise HTTPException(status_code=400, detail="Invalid JSON file")
        except Exception as e:
            raise HTTPException(status_code=500, detail=str(e))
    else:
        # Return file download
        return FileResponse(
            path=full_path,
            filename=full_path.name,
            media_type="application/octet-stream"
        )


@router.get("/download/{file_path:path}")
async def download_file(file_path: str):
    """Download file"""
    full_path = DATA_DIR / file_path

    if not full_path.exists():
        raise HTTPException(status_code=404, detail="File not found")

View on GitHub (pinned to d6f7c5bb90)

Solutions

1. Wait for the crawler to finish (GET /crawler/status) before previewing output files.
2. Open the file directly and inspect it — json.tool or a validator will point at the exact offending position.
3. If the pipeline writes JSON Lines, rename to .jsonl and add a line-by-line reader, or emit proper JSON arrays.
4. Re-run or regenerate the file if it is an empty/truncated artifact of a failed run.

const st = await (await fetch('/api/crawler/status')).json();
if (!st.running) { /* safe to preview: files are fully written */ }

try { await fetch(`/api/data/files/${path}?preview=true`); } catch (e) { if (e.status === 400 && e.detail === 'Invalid JSON file') { /* fetch raw and inspect, or retry after crawl ends */ } }

When it happens

Trigger: A .json file still being written by the crawler when the preview request lands (truncated tail); an empty .json file from a crashed run; JSON Lines output saved with a .json extension, which json.load rejects because it expects a single document.

Common situations: Previewing data while a crawl is actively writing; file transfer that truncated the JSON; BOM from a Windows editor breaking the first token.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• {e}
• response error
• unknown error

AI-assisted analysis of NanmiCoder/MediaCrawler@d6f7c5bb90 (2026-08-15). Data as JSON: /api/errors/6e8e2cc7b28d6404. Report an issue: GitHub.