ZhuLinsen/daily_stock_analysis · error · ValueError

无法识别文件编码，请使用 UTF-8 或 GBK

Error message

无法识别文件编码，请使用 UTF-8 或 GBK

What it means

Raised in the CSV/text path when the bytes cannot be decoded as either UTF-8 or GBK — the only two encodings attempted. UTF-16, Latin-1, Big5, etc. fail both decodes and hit this error.

Source

Thrown at src/services/import_parser.py:189

                    "若为 .xls 格式，请另存为 .xlsx 后重试。"
                )
                raise ValueError(f"Excel 解析失败: {e}。{hint}") from e
            # For extension-only mismatch (e.g. csv named .xlsx), fallback to text parsing.
            logger.warning(f"扩展名为 .xlsx 但未解析为 Excel，将回退文本解析: {e}")

    # .xls not supported
    if ext == ".xls":
        raise ValueError("仅支持 .xlsx 格式，请将 .xls 另存为 .xlsx 后重试")

    # CSV / text
    for encoding in ("utf-8", "gbk"):
        try:
            text = data.decode(encoding)
            break
        except UnicodeDecodeError:
            continue
    else:
        raise ValueError("无法识别文件编码，请使用 UTF-8 或 GBK")

    # Single-column (one value per line): bypass pandas to avoid sep=None inference issues
    # e.g. "00700\n600519" or "code\n00700" - pandas with sep=None can produce wrong results
    lines = [ln.strip() for ln in text.strip().splitlines() if ln.strip()]
    if _should_use_single_column_fast_path(lines):
        rows = [[ln] for ln in lines]
        df = pd.DataFrame(rows)
        first_row = [str(x).strip().lower() for x in df.iloc[0].tolist()]
        if any(c in _CODE_ALIASES or c in _NAME_ALIASES for c in first_row):
            df.columns = df.iloc[0]
            df = df.iloc[1:].reset_index(drop=True)
        return _parse_dataframe(df)

    # Try pandas for CSV-like; use dtype=str to preserve leading zeros (e.g. 00700)
    try:
        df = pd.read_csv(io.StringIO(text), sep=None, engine="python", header=None, dtype=str)
        if df is not None and not df.empty:
            df = df.fillna("")

View on GitHub (pinned to 5159bd72e8)

Solutions

1. Re-save the file as UTF-8 (Excel: 'CSV UTF-8 (Comma delimited)' option)
2. Or programmatically pre-convert: text = data.decode('utf-16'); then re-encode utf-8 and parse
3. Check for a BOM (\xff\xfe / \xfe\xff) as a UTF-16 telltale

Example fix

# before
items = parse_import_from_bytes(data, 'list.csv')  # UTF-16 -> ValueError
# after
items = parse_import_from_bytes(data.decode('utf-16').encode('utf-8'), 'list.csv')

def decode_text(data: bytes) -> str:
    for enc in ('utf-8', 'gbk', 'utf-16', 'big5'):
        try:
            return data.decode(enc)
        except UnicodeDecodeError:
            continue
    raise ValueError('unsupported encoding')

def is_decodable(data: bytes) -> bool:
    return any(_try(data, e) for e in ('utf-8', 'gbk'))
def _try(b, enc):
    try: b.decode(enc); return True
    except UnicodeDecodeError: return False

try:
    items = parse_import_from_bytes(data, fn)
except ValueError as e:
    if '无法识别文件编码' in str(e):
        text = data.decode('utf-16')  # most common Windows case
        items = parse_import_from_text(text)
    else:
        raise

When it happens

Trigger: File saved as UTF-16 (common for Windows 'Unicode Text' exports), Big5 (Traditional Chinese), Shift-JIS, or binary garbage with a text extension.

Common situations: Windows Notepad 'Unicode' save producing UTF-16 LE with BOM; Taiwanese/HK exports in Big5; Japanese CSVs in Shift-JIS.

Related errors

• CSV 解析失败：请检查分隔符是否一致、列数是否匹配。常见原因：引号未闭合、某行列数与其他行不一致。原始错误: {e}
• invalid_import_file
• 未安装 Futu OpenAPI SDK；请先执行 `pip install "futu-api==10.8.6808"
• 加载 Futu OpenAPI SDK 失败: {exc}
• 请安装 schedule 库: pip install schedule

AI-assisted analysis of ZhuLinsen/daily_stock_analysis@5159bd72e8 (2026-08-15). Data as JSON: /api/errors/98033f0d0a171b0e. Report an issue: GitHub.