{"record":{"id":"c8ba7b51df7897f9","repo":"stanford-oval/storm","slug":"url-column-url-column-not-found-in-the-csv-file","errorCode":null,"errorMessage":"URL column {url_column} not found in the csv file.","messagePattern":"URL column (.+?) not found in the csv file\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"knowledge_storm/utils.py","lineNumber":249,"sourceCode":"            )\n        else:\n            raise ValueError(\n                \"Invalid vector_db_mode. Please provide either 'online' or 'offline'.\"\n            )\n        if qdrant is None:\n            raise ValueError(\"Qdrant client is not initialized.\")\n\n        # read the csv file\n        import pandas as pd\n\n        df = pd.read_csv(file_path)\n        # check that content column exists and url column exists\n        if content_column not in df.columns:\n            raise ValueError(\n                f\"Content column {content_column} not found in the csv file.\"\n            )\n        if url_column not in df.columns:\n            raise ValueError(f\"URL column {url_column} not found in the csv file.\")\n\n        documents = [\n            Document(\n                page_content=row[content_column],\n                metadata={\n                    \"title\": row.get(title_column, \"\"),\n                    \"url\": row[url_column],\n                    \"description\": row.get(desc_column, \"\"),\n                },\n            )\n            for row in df.to_dict(orient=\"records\")\n        ]\n\n        # split the documents\n        from langchain_text_splitters import RecursiveCharacterTextSplitter\n\n        text_splitter = RecursiveCharacterTextSplitter(\n            chunk_size=chunk_size,","sourceCodeStart":231,"sourceCodeEnd":267,"githubUrl":"https://github.com/stanford-oval/storm/blob/fb951af7744dab086e34962e9bc6fe878e145f83/knowledge_storm/utils.py#L231-L267","documentation":"Raised by create_or_update_vector_store in knowledge_storm/utils.py when the DataFrame loaded from the CSV does not contain the column named by the url_column argument. The function requires both a content column and a URL column to build Documents (the URL becomes per-document metadata), so it validates the CSV schema up front and aborts if the URL column is missing. This is a data-contract error: the caller's configuration does not match the actual CSV header.","triggerScenarios":"Calling create_or_update_vector_store(csv_path, url_column='url') (or via main/CLI) where the CSV's header has no column literally named 'url' — e.g. the column is named 'link', 'URL ' (trailing space), or is absent entirely. Also triggered by case mismatches ('URL' vs 'url'), CSVs with a different delimiter so pandas parses one giant column, or passing the wrong csv_path so a different file's schema is checked.","commonSituations":"Swapping in a new crawl/export CSV whose link column is named differently; typos or trailing whitespace in the header row; passing the default url_column without checking a user-supplied file; reading a TSV or semicolon-delimited file without sep=';\\t'/sep=';'; locale/case differences in headers ('Url' vs 'url').","solutions":["Inspect the CSV header (pd.read_csv(path).columns.tolist()) and pass the exact existing column name as url_column.","Fix the CSV: rename the link column to the expected url_column value, or strip whitespace/BOM from headers.","If the file uses a different delimiter or encoding, load it correctly (pd.read_csv(path, sep=';', encoding='utf-8-sig')) before calling the function, or fix the source export.","If no URL column exists but URLs are derivable, add one (e.g. df['url'] = base_url) and re-save before ingestion."],"exampleFix":"// before\ncreate_or_update_vector_store(\"data.csv\", content_column=\"text\", url_column=\"url\")\n# ValueError: URL column url not found in the csv file.\n\n// after\nimport pandas as pd\ndf = pd.read_csv(\"data.csv\")\nprint(df.columns.tolist())  # e.g. ['text', 'link']\ncreate_or_update_vector_store(\"data.csv\", content_column=\"text\", url_column=\"link\")","handlingStrategy":"validation","validationCode":"import pandas as pd\n\ndef validate_csv_columns(csv_path: str, content_column: str, url_column: str) -> None:\n    df = pd.read_csv(csv_path, nrows=0)\n    missing = [c for c in (content_column, url_column) if c not in df.columns]\n    if missing:\n        raise ValueError(f\"Missing columns {missing}; available: {list(df.columns)}\")","typeGuard":"def has_required_columns(df: pd.DataFrame, url_column: str) -> bool:\n    return url_column in df.columns","tryCatchPattern":"try:\n    create_or_update_vector_store(csv_path, content_column=\"text\", url_column=\"url\")\nexcept ValueError as e:\n    if \"URL column\" in str(e):\n        cols = pd.read_csv(csv_path, nrows=0).columns.tolist()\n        # pick the real link-like column or fail with context\n        candidates = [c for c in cols if c.strip().lower() in {\"url\", \"link\", \"source\"}]\n        if not candidates:\n            raise\n        create_or_update_vector_store(csv_path, content_column=\"text\", url_column=candidates[0])\n    else:\n        raise","preventionTips":["Print df.columns.tolist() once when onboarding any new CSV source.","Normalize headers on ingest: df.columns = df.columns.str.strip().str.lower().","Standardize crawl/export scripts to always emit a url column.","Validate required columns with nrows=0 reads before starting an expensive vector-store build."],"tags":["python","pandas","csv","schema-validation","data-ingestion","vector-store"],"backgroundTag":"dataframe-column-not-found","analyzedSha":"fb951af7744dab086e34962e9bc6fe878e145f83","analyzedAt":"2026-08-28T11:56:54.780Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}