{"record":{"id":"246eee40448b2b18","repo":"binary-husky/gpt_academic","slug":"remain-txt-to-cut-246eee","errorCode":null,"errorMessage":"存在一行极长的文本！{remain_txt_to_cut}","messagePattern":"存在一行极长的文本！(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"crazy_functions/pdf_fns/breakdown_txt.py","lineNumber":72,"sourceCode":"            cnt = 0\n            for cnt in reversed(range(estimated_line_cut)):\n                if must_break_at_empty_line:\n                    # 首先尝试用双空行（\\n\\n）作为切分点\n                    if lines[cnt] != \"\":\n                        continue\n                prev = \"\\n\".join(lines[:cnt])\n                post = \"\\n\".join(lines[cnt:])\n                if get_token_fn(prev) < limit:\n                    break\n\n            if cnt == 0:\n                # 如果没有找到合适的切分点\n                if break_anyway:\n                    # 是否允许暴力切分\n                    prev, post = force_breakdown(remain_txt_to_cut, limit, get_token_fn)\n                else:\n                    # 不允许直接报错\n                    raise RuntimeError(f\"存在一行极长的文本！{remain_txt_to_cut}\")\n\n            # 追加列表\n            res.append(prev); fin_len+=len(prev)\n            # 准备下一次迭代\n            remain_txt_to_cut = post\n            remain_txt_to_cut, remain_txt_to_cut_storage = maintain_storage(remain_txt_to_cut, remain_txt_to_cut_storage)\n            process = fin_len/total_len\n            logger.info(f'正在文本切分 {int(process*100)}%')\n            if len(remain_txt_to_cut.strip()) == 0:\n                break\n    return res\n\n\ndef breakdown_text_to_satisfy_token_limit_(txt, limit, llm_model=\"gpt-3.5-turbo\"):\n    \"\"\" 使用多种方式尝试切分文本，以满足 token 限制\n    \"\"\"\n    from request_llms.bridge_all import model_info\n    enc = model_info[llm_model]['tokenizer']","sourceCodeStart":54,"sourceCodeEnd":90,"githubUrl":"https://github.com/binary-husky/gpt_academic/blob/d6bde0fa54373309bd05823a49bda8da019d2c77/crazy_functions/pdf_fns/breakdown_txt.py#L54-L90","documentation":"The breakdown_txt.py sibling of errors 67/68: the generic (non-PDF-specific) token-limit splitter raises when cnt==0 — no newline split point yields a prefix under the token limit, i.e. the first line alone is over limit — and break_anyway is False. It refuses blind mid-line cuts in strict mode.","triggerScenarios":"Feeding plain text (not PDF-derived) with one line longer than the token limit; minified JSON/CSV/log lines with no newlines; strict mode requested by the caller.","commonSituations":"Chunking logs or minified data files for LLM input; users assuming newlines exist but the file uses \\r or no line breaks at all; limit set aggressively low.","solutions":["Enable break_anyway=True for unstructured inputs that legitimately lack newlines","Normalize line endings and wrap long lines before splitting","Raise the limit above the longest single line","If input is minified JSON, pretty-print it first (json.dumps(obj, indent=1))"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"def is_splittable(text: str, limit: int, get_token_fn) -> bool:\n    lines = text.split('\\n')\n    return bool(lines) and all(get_token_fn(l) < limit for l in lines)","typeGuard":null,"tryCatchPattern":"try:\n    chunks = breakdown_txt_to_token_limit(txt, limit, get_token_fn, break_anyway=False)\nexcept RuntimeError as e:\n    if '极长' in str(e):\n        chunks = breakdown_txt_to_token_limit(txt, limit, get_token_fn, break_anyway=True)\n    raise","preventionTips":["Pretty-print minified inputs (JSON, CSV single-line) before chunking","Choose limit based on measured max line length of the corpus","Default to break_anyway=True for machine-generated text where newlines are absent"],"tags":["text-splitting","token-limit","long-line","plain-text"],"backgroundTag":null,"analyzedSha":"d6bde0fa54373309bd05823a49bda8da019d2c77","analyzedAt":"2026-08-14T22:48:35.038Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}