zylon-ai/private-gpt · error · ValueError

Error reading delimited file: {e}

Error message

Error reading delimited file: {e}

What it means

The delimiter reader wraps its whole parse/emit block in try/except and re-raises any exception as ValueError('Error reading delimited file: ...') chained to the original. It is an aggregation wrapper: the informative part is the inner exception text and its __cause__ (parser errors, encoding errors, dtype problems). Any failure while chunk-reading a delimited file and formatting rows as Markdown surfaces here.

Source

Thrown at private_gpt/components/readers/text/delimiter_reader.py:86

                    separator_row = "| " + " | ".join("-" for _ in chunk.columns) + " |"
                    markdown_lines.append(header_row)
                    markdown_lines.append(separator_row)
                    first_chunk = False

                # Convert chunk to strings and format rows
                chunk_str = chunk.astype(str)
                for row in chunk_str.values.tolist():
                    markdown_lines.append(format_row(row))

            # Join all Markdown lines into a single string.
            markdown_content = "\n".join(markdown_lines)
            yield Document(
                text=markdown_content,
                extra_info=extra_info if extra_info is not None else {},
            )

        except Exception as e:
            raise ValueError(f"Error reading delimited file: {e}") from e

View on GitHub (pinned to 4a030776a3)

Solutions

1. Inspect `exc.__cause__` and the appended original message — it usually names the exact line/byte that failed.
2. Reproduce the parse directly: `pd.read_csv(path, chunksize=..., sep=<same>)` and apply the fix it suggests (encoding='utf-8-sig', on_bad_lines='skip', dtype=str, quoting=...).
3. Pre-clean or re-export the file (fix encoding, strip preamble rows) before ingestion.
4. If occasional bad lines are acceptable, configure the reader to skip them rather than fail the whole document.

Example fix

# before
for doc in reader.load_data(file=Path("export.csv")):
    ...  # ValueError: Error reading delimited file: ...

# after
# pre-check the file parses cleanly with the same options
preview = pd.read_csv("export.csv", nrows=100, sep=",", encoding="utf-8-sig")
for doc in reader.load_data(file=Path("export.csv")):
    ...

import pandas as pd

def delimited_file_parses(path: str, sep: str) -> bool:
    try:
        pd.read_csv(path, sep=sep, nrows=10, encoding="utf-8-sig")
        return True
    except Exception:
        return False

try:
    docs = list(reader.load_data(file=path))
except ValueError as e:
    cause = e.__cause__ or e
    logger.error("Delimited parse failed (%s): %s", path, cause)
    quarantine(path)
    raise

When it happens

Trigger: Running the delimiter reader on a .csv/.tsv/.psv file where pandas chunked reading or row formatting raises — malformed lines, undecodable bytes, mixed dtypes in a column, or a wrong delimiter setting.

Common situations: Files with embedded stray quotes or stray delimiters; non-UTF-8 encodings (latin-1 exports); Excel exports with BOM/metadata rows; inconsistent column counts; very large files where chunk boundaries expose dtype inference issues.

Related errors

• zpgt.ingest.parsing_failure.error
• zpgt.ingest.no_valid_files.error
• Header and content length mismatch: {len(value.header)} != {
• Invalid system item in list (dict): {item}
• Invalid system item in list: {item}

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/419e553487eff64c. Report an issue: GitHub.