binary-husky/gpt_academic · error · RuntimeError

存在一行极长的文本！{remain_txt_to_cut}

Error message

存在一行极长的文本！{remain_txt_to_cut}

What it means

The breakdown_txt.py sibling of errors 67/68: the generic (non-PDF-specific) token-limit splitter raises when cnt==0 — no newline split point yields a prefix under the token limit, i.e. the first line alone is over limit — and break_anyway is False. It refuses blind mid-line cuts in strict mode.

Source

Thrown at crazy_functions/pdf_fns/breakdown_txt.py:72

            cnt = 0
            for cnt in reversed(range(estimated_line_cut)):
                if must_break_at_empty_line:
                    # 首先尝试用双空行（\n\n）作为切分点
                    if lines[cnt] != "":
                        continue
                prev = "\n".join(lines[:cnt])
                post = "\n".join(lines[cnt:])
                if get_token_fn(prev) < limit:
                    break

            if cnt == 0:
                # 如果没有找到合适的切分点
                if break_anyway:
                    # 是否允许暴力切分
                    prev, post = force_breakdown(remain_txt_to_cut, limit, get_token_fn)
                else:
                    # 不允许直接报错
                    raise RuntimeError(f"存在一行极长的文本！{remain_txt_to_cut}")

            # 追加列表
            res.append(prev); fin_len+=len(prev)
            # 准备下一次迭代
            remain_txt_to_cut = post
            remain_txt_to_cut, remain_txt_to_cut_storage = maintain_storage(remain_txt_to_cut, remain_txt_to_cut_storage)
            process = fin_len/total_len
            logger.info(f'正在文本切分 {int(process*100)}%')
            if len(remain_txt_to_cut.strip()) == 0:
                break
    return res


def breakdown_text_to_satisfy_token_limit_(txt, limit, llm_model="gpt-3.5-turbo"):
    """ 使用多种方式尝试切分文本，以满足 token 限制
    """
    from request_llms.bridge_all import model_info
    enc = model_info[llm_model]['tokenizer']

View on GitHub (pinned to d6bde0fa54)

Solutions

1. Enable break_anyway=True for unstructured inputs that legitimately lack newlines
2. Normalize line endings and wrap long lines before splitting
3. Raise the limit above the longest single line
4. If input is minified JSON, pretty-print it first (json.dumps(obj, indent=1))

def is_splittable(text: str, limit: int, get_token_fn) -> bool:
    lines = text.split('\n')
    return bool(lines) and all(get_token_fn(l) < limit for l in lines)

try:
    chunks = breakdown_txt_to_token_limit(txt, limit, get_token_fn, break_anyway=False)
except RuntimeError as e:
    if '极长' in str(e):
        chunks = breakdown_txt_to_token_limit(txt, limit, get_token_fn, break_anyway=True)
    raise

When it happens

Trigger: Feeding plain text (not PDF-derived) with one line longer than the token limit; minified JSON/CSV/log lines with no newlines; strict mode requested by the caller.

Common situations: Chunking logs or minified data files for LLM input; users assuming newlines exist but the file uses \r or no line breaks at all; limit set aggressively low.

Related errors

• 存在一行极长的文本！{remain_txt_to_cut}
• 文本分割超时（5秒）

AI-assisted analysis of binary-husky/gpt_academic@d6bde0fa54 (2026-08-14). Data as JSON: /api/errors/246eee40448b2b18. Report an issue: GitHub.