FoundationAgents/MetaGPT · error · ValueError

read jsonl file: {jsonl_file} failed

Error message

read jsonl file: {jsonl_file} failed

What it means

read_jsonl_file in metagpt/utils/common.py parses each line with json.loads inside one try block covering the whole loop. A single bad line — malformed JSON, a blank line with whitespace won't fail but stray text will, or an encoding error — aborts everything and is re-raised as ValueError with the file name.

Source

Thrown at metagpt/utils/common.py:622

        folder_path.mkdir(parents=True, exist_ok=True)

    custom_default = partial(to_jsonable_python, fallback=handle_unknown_serialization if use_fallback else None)

    with open(json_file, "w", encoding=encoding) as fout:
        json.dump(data, fout, ensure_ascii=False, indent=indent, default=custom_default)


def read_jsonl_file(jsonl_file: str, encoding="utf-8") -> list[dict]:
    if not Path(jsonl_file).exists():
        raise FileNotFoundError(f"json_file: {jsonl_file} not exist, return []")
    datas = []
    with open(jsonl_file, "r", encoding=encoding) as fin:
        try:
            for line in fin:
                data = json.loads(line)
                datas.append(data)
        except Exception:
            raise ValueError(f"read jsonl file: {jsonl_file} failed")
    return datas


def add_jsonl_file(jsonl_file: str, data: list[dict], encoding: str = None):
    folder_path = Path(jsonl_file).parent
    if not folder_path.exists():
        folder_path.mkdir(parents=True, exist_ok=True)

    with open(jsonl_file, "a", encoding=encoding) as fout:
        for json_item in data:
            fout.write(json.dumps(json_item) + "\n")


def read_csv_to_list(curr_file: str, header=False, strip_trail=True):
    """
    Reads in a csv file to a list of list. If header is True, it returns a
    tuple with (header row, all rows)
    ARGS:

View on GitHub (pinned to 11cdf466d0)

Solutions

  1. Find the bad line: iterate with enumerate and json.loads per line to get the line number and JSONDecodeError position.
  2. Skip or repair corrupt lines if partial data is acceptable.
  3. Ensure writers append complete lines atomically (single write() of line + newline, flush).
  4. If encoding differs, pass the correct encoding parameter.

Example fix

# before
rows = read_jsonl_file(p)  # one bad line kills the read

# after (tolerant read)
rows = []
with open(p, encoding='utf-8') as f:
    for i, line in enumerate(f):
        try:
            rows.append(json.loads(line))
        except json.JSONDecodeError:
            print(f'skipping bad line {i}')
Defensive patterns

Strategy: try-catch

Validate before calling

import json
from pathlib import Path

def find_bad_jsonl_lines(path: str):
    bad = []
    with open(path, encoding="utf-8") as f:
        for i, line in enumerate(f, 1):
            try:
                json.loads(line)
            except json.JSONDecodeError:
                bad.append(i)
    return bad

Try / catch

try:
    rows = read_jsonl_file(path)
except ValueError:
    # one bad line poisons the batch; fall back to per-line tolerant parsing
    rows = [json.loads(l) for l in open(path, encoding="utf-8") if l.strip() and _ok(l)]

Prevention

When it happens

Trigger: One line in the .jsonl is truncated (crash during append); a line contains single-quoted JSON or NaN/Infinity; file has a trailing partial line from a concurrent writer; wrong encoding causing UnicodeDecodeError mid-iteration.

Common situations: Append-only logs damaged by an interrupted run; multiple processes appending without line-atomic writes; hand-edited jsonl with a stray comma or prose line.

Related errors

AI-assisted analysis of FoundationAgents/MetaGPT@11cdf466d0 (2026-08-14). Data as JSON: /api/errors/0a233546e9c45621. Report an issue: GitHub.