{"record":{"id":"0a233546e9c45621","repo":"FoundationAgents/MetaGPT","slug":"read-jsonl-file-jsonl-file-failed","errorCode":null,"errorMessage":"read jsonl file: {jsonl_file} failed","messagePattern":"read jsonl file: (.+?) failed","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"metagpt/utils/common.py","lineNumber":622,"sourceCode":"        folder_path.mkdir(parents=True, exist_ok=True)\n\n    custom_default = partial(to_jsonable_python, fallback=handle_unknown_serialization if use_fallback else None)\n\n    with open(json_file, \"w\", encoding=encoding) as fout:\n        json.dump(data, fout, ensure_ascii=False, indent=indent, default=custom_default)\n\n\ndef read_jsonl_file(jsonl_file: str, encoding=\"utf-8\") -> list[dict]:\n    if not Path(jsonl_file).exists():\n        raise FileNotFoundError(f\"json_file: {jsonl_file} not exist, return []\")\n    datas = []\n    with open(jsonl_file, \"r\", encoding=encoding) as fin:\n        try:\n            for line in fin:\n                data = json.loads(line)\n                datas.append(data)\n        except Exception:\n            raise ValueError(f\"read jsonl file: {jsonl_file} failed\")\n    return datas\n\n\ndef add_jsonl_file(jsonl_file: str, data: list[dict], encoding: str = None):\n    folder_path = Path(jsonl_file).parent\n    if not folder_path.exists():\n        folder_path.mkdir(parents=True, exist_ok=True)\n\n    with open(jsonl_file, \"a\", encoding=encoding) as fout:\n        for json_item in data:\n            fout.write(json.dumps(json_item) + \"\\n\")\n\n\ndef read_csv_to_list(curr_file: str, header=False, strip_trail=True):\n    \"\"\"\n    Reads in a csv file to a list of list. If header is True, it returns a\n    tuple with (header row, all rows)\n    ARGS:","sourceCodeStart":604,"sourceCodeEnd":640,"githubUrl":"https://github.com/FoundationAgents/MetaGPT/blob/11cdf466d042aece04fc6cfd13b28e1a70341b1f/metagpt/utils/common.py#L604-L640","documentation":"read_jsonl_file in metagpt/utils/common.py parses each line with json.loads inside one try block covering the whole loop. A single bad line — malformed JSON, a blank line with whitespace won't fail but stray text will, or an encoding error — aborts everything and is re-raised as ValueError with the file name.","triggerScenarios":"One line in the .jsonl is truncated (crash during append); a line contains single-quoted JSON or NaN/Infinity; file has a trailing partial line from a concurrent writer; wrong encoding causing UnicodeDecodeError mid-iteration.","commonSituations":"Append-only logs damaged by an interrupted run; multiple processes appending without line-atomic writes; hand-edited jsonl with a stray comma or prose line.","solutions":["Find the bad line: iterate with enumerate and json.loads per line to get the line number and JSONDecodeError position.","Skip or repair corrupt lines if partial data is acceptable.","Ensure writers append complete lines atomically (single write() of line + newline, flush).","If encoding differs, pass the correct encoding parameter."],"exampleFix":"# before\nrows = read_jsonl_file(p)  # one bad line kills the read\n\n# after (tolerant read)\nrows = []\nwith open(p, encoding='utf-8') as f:\n    for i, line in enumerate(f):\n        try:\n            rows.append(json.loads(line))\n        except json.JSONDecodeError:\n            print(f'skipping bad line {i}')","handlingStrategy":"try-catch","validationCode":"import json\nfrom pathlib import Path\n\ndef find_bad_jsonl_lines(path: str):\n    bad = []\n    with open(path, encoding=\"utf-8\") as f:\n        for i, line in enumerate(f, 1):\n            try:\n                json.loads(line)\n            except json.JSONDecodeError:\n                bad.append(i)\n    return bad","typeGuard":null,"tryCatchPattern":"try:\n    rows = read_jsonl_file(path)\nexcept ValueError:\n    # one bad line poisons the batch; fall back to per-line tolerant parsing\n    rows = [json.loads(l) for l in open(path, encoding=\"utf-8\") if l.strip() and _ok(l)]","preventionTips":["Write each jsonl line with a single atomic write call and flush.","Validate per line during ingestion so corrupt lines are skipped early.","Avoid multiple unsynchronized appenders to one file."],"tags":["metagpt","jsonl","json","corrupt-file","file-io"],"backgroundTag":null,"analyzedSha":"11cdf466d042aece04fc6cfd13b28e1a70341b1f","analyzedAt":"2026-08-14T23:20:02.994Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}