{"record":{"id":"291e3b27e272f49c","repo":"unclecode/crawl4ai","slug":"database-missing-columns-missing-columns","errorCode":null,"errorMessage":"Database missing columns: {missing_columns}","messagePattern":"Database missing columns: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crawl4ai/async_database.py","lineNumber":159,"sourceCode":"                            columns = await cursor.fetchall()\n                            column_names = [col[1] for col in columns]\n                            expected_columns = {\n                                \"url\",\n                                \"html\",\n                                \"cleaned_html\",\n                                \"markdown\",\n                                \"extracted_content\",\n                                \"success\",\n                                \"media\",\n                                \"links\",\n                                \"metadata\",\n                                \"screenshot\",\n                                \"response_headers\",\n                                \"downloaded_files\",\n                            }\n                            missing_columns = expected_columns - set(column_names)\n                            if missing_columns:\n                                raise ValueError(\n                                    f\"Database missing columns: {missing_columns}\"\n                                )\n\n                        self.connection_pool[task_id] = conn\n                    except Exception as e:\n                        import sys\n\n                        error_context = get_error_context(sys.exc_info())\n                        error_message = (\n                            f\"Unexpected error in db get_connection at line {error_context['line_no']} \"\n                            f\"in {error_context['function']} ({error_context['filename']}):\\n\"\n                            f\"Error: {str(e)}\\n\\n\"\n                            f\"Code context:\\n{error_context['code_context']}\"\n                        )\n                        self.logger.error(\n                            message=\"{error}\",\n                            tag=\"ERROR\",\n                            params={\"error\": str(error_message)},","sourceCodeStart":141,"sourceCodeEnd":177,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/async_database.py#L141-L177","documentation":"Raised during connection acquisition when the opened 'crawled_data' table is missing one or more expected schema columns (markdown, extracted_content, success, media, links, metadata, screenshot, response_headers, downloaded_files, etc.). This indicates an older database file created by a previous crawl4ai version whose schema predates those columns, and auto-migration did not (or has not yet) run.","triggerScenarios":"Upgrading crawl4ai while keeping an old crawl.db produced by an earlier release; manually created or externally modified tables; a migration step that failed earlier (error 58's path) leaving a half-updated schema; copying a DB file between installations of different versions.","commonSituations":"pip install -U crawl4ai then re-running with the default persistent DB path; shared DB across machines with mixed crawler versions; tests that create a partial schema fixture.","solutions":["Delete or archive the old database file so a fresh schema is created (fastest if the data is disposable).","Trigger the version-manager update path (let initialization detect the version change and run update_db_schema/migrations) instead of reusing a connection created before update.","If the data must be kept, manually ALTER TABLE to add the missing columns listed in the error message.","Pin one crawl4ai version across all machines sharing the DB."],"exampleFix":"// before\n# ~/.crawl4ai/crawl.db created by crawl4ai 0.4.x, now running 0.6.x\n# ValueError: Database missing columns: {'response_headers', 'downloaded_files'}\n\n// after\nimport os\nos.remove(\"~/.crawl4ai/crawl.db\")  # or move aside, then re-run init\nawait AsyncDatabaseManager().initialize()","handlingStrategy":"fallback","validationCode":"import sqlite3\n\nEXPECTED = {\"url\", \"html\", \"markdown\", \"extracted_content\", \"success\", \"media\", \"links\", \"metadata\", \"screenshot\", \"response_headers\", \"downloaded_files\"}\n\ndef schema_current(db_path: str) -> bool:\n    con = sqlite3.connect(db_path)\n    try:\n        cols = {r[1] for r in con.execute(\"PRAGMA table_info(crawled_data)\")}\n        return EXPECTED <= cols\n    finally:\n        con.close()","typeGuard":null,"tryCatchPattern":"try:\n    await db_manager.get_connection()\nexcept ValueError as e:\n    if \"Database missing columns\" in str(e):\n        archive_and_recreate_db(db_path)\n        await db_manager.get_connection()","preventionTips":["Back up and recreate the DB after version upgrades","Pin one crawl4ai version across machines sharing a DB","Run migrations before accepting crawler connections","Never share one DB between old and new crawler versions"],"tags":["database","schema-migration","sqlite","upgrade"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}