{"record":{"id":"8d09c3ccabb9f6b8","repo":"unclecode/crawl4ai","slug":"crawled-data-table-was-not-created","errorCode":null,"errorMessage":"crawled_data table was not created","messagePattern":"crawled_data table was not created","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"critical","filePath":"crawl4ai/async_database.py","lineNumber":62,"sourceCode":"        try:\n            self.logger.info(\"Initializing database\", tag=\"INIT\")\n            # Ensure the database file exists\n            os.makedirs(os.path.dirname(self.db_path), exist_ok=True)\n\n            # Check if version update is needed\n            needs_update = self.version_manager.needs_update()\n\n            # Always ensure base table exists\n            await self.ainit_db()\n\n            # Verify the table exists\n            async with aiosqlite.connect(self.db_path, timeout=30.0) as db:\n                async with db.execute(\n                    \"SELECT name FROM sqlite_master WHERE type='table' AND name='crawled_data'\"\n                ) as cursor:\n                    result = await cursor.fetchone()\n                    if not result:\n                        raise Exception(\"crawled_data table was not created\")\n\n            # If version changed or fresh install, run updates\n            if needs_update:\n                self.logger.info(\"New version detected, running updates\", tag=\"INIT\")\n                await self.update_db_schema()\n                from .migrations import (\n                    run_migration,\n                )  # Import here to avoid circular imports\n\n                await run_migration()\n                self.version_manager.update_version()  # Update stored version after successful migration\n                self.logger.success(\n                    \"Version update completed successfully\", tag=\"COMPLETE\"\n                )\n            else:\n                self.logger.success(\n                    \"Database initialization completed successfully\", tag=\"COMPLETE\"\n                )","sourceCodeStart":44,"sourceCodeEnd":80,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/async_database.py#L44-L80","documentation":"Raised during async database initialization when, after ainit_db() runs, the 'crawled_data' table still cannot be found in sqlite_master. It is a postcondition check: the CREATE TABLE either failed silently, was rolled back, or wrote to a different database file than the one being verified (different db_path resolution between calls).","triggerScenarios":"Two processes racing to initialize the same SQLite file (lock/rollback); a corrupted or pre-existing non-schema database file at the resolved path; db_path pointing to a read-only directory so CREATE TABLE failed; path resolution changing between ainit_db and the verification connect (e.g. relative paths with changed cwd).","commonSituations":"Multiple crawler instances or workers sharing one DB file concurrently; running with insufficient filesystem permissions (read-only volume in Docker); stale/corrupt .db files from older versions; relative db_path resolved differently across async tasks after a cwd change.","solutions":["Delete (or move aside) the corrupt/stale database file and let initialization recreate it.","Check write permissions on the directory containing db_path; in Docker ensure the volume is writable.","Use an absolute db_path and serialize first-time initialization across processes (single initializer, or accept the race is benign and re-run).","Inspect with 'sqlite3 <db> .tables' to see whether the table exists in a different file than expected."],"exampleFix":"// before\nawait AsyncDatabaseManager(database_path=\"crawl.db\").ainit_db()  # relative path, cwd changed\n\n// after\nimport pathlib\nawait AsyncDatabaseManager(database_path=str(pathlib.Path(\"~/.crawl4ai/crawl.db\").expanduser())).ainit_db()","handlingStrategy":"fallback","validationCode":"import sqlite3\n\ndef db_has_table(db_path: str, table: str = \"crawled_data\") -> bool:\n    if not os.path.exists(db_path):\n        return False\n    con = sqlite3.connect(db_path)\n    try:\n        row = con.execute(\n            \"SELECT name FROM sqlite_master WHERE type='table' AND name=?\", (table,)\n        ).fetchone()\n        return row is not None\n    finally:\n        con.close()","typeGuard":null,"tryCatchPattern":"try:\n    await db_manager.initialize()\nexcept Exception as e:\n    if \"crawled_data table was not created\" in str(e):\n        os.remove(db_path)\n        await db_manager.initialize()  # fresh recreate","preventionTips":["Use absolute db_path","Ensure the DB directory is writable","Serialize first-time DB init across processes","Delete corrupt DB files instead of retrying init"],"tags":["database","sqlite","initialization","filesystem"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}