unclecode/crawl4ai · critical · Exception

crawled_data table was not created

Error message

crawled_data table was not created

What it means

Raised during async database initialization when, after ainit_db() runs, the 'crawled_data' table still cannot be found in sqlite_master. It is a postcondition check: the CREATE TABLE either failed silently, was rolled back, or wrote to a different database file than the one being verified (different db_path resolution between calls).

Source

Thrown at crawl4ai/async_database.py:62

        try:
            self.logger.info("Initializing database", tag="INIT")
            # Ensure the database file exists
            os.makedirs(os.path.dirname(self.db_path), exist_ok=True)

            # Check if version update is needed
            needs_update = self.version_manager.needs_update()

            # Always ensure base table exists
            await self.ainit_db()

            # Verify the table exists
            async with aiosqlite.connect(self.db_path, timeout=30.0) as db:
                async with db.execute(
                    "SELECT name FROM sqlite_master WHERE type='table' AND name='crawled_data'"
                ) as cursor:
                    result = await cursor.fetchone()
                    if not result:
                        raise Exception("crawled_data table was not created")

            # If version changed or fresh install, run updates
            if needs_update:
                self.logger.info("New version detected, running updates", tag="INIT")
                await self.update_db_schema()
                from .migrations import (
                    run_migration,
                )  # Import here to avoid circular imports

                await run_migration()
                self.version_manager.update_version()  # Update stored version after successful migration
                self.logger.success(
                    "Version update completed successfully", tag="COMPLETE"
                )
            else:
                self.logger.success(
                    "Database initialization completed successfully", tag="COMPLETE"
                )

View on GitHub (pinned to 7e80152142)

Solutions

1. Delete (or move aside) the corrupt/stale database file and let initialization recreate it.
2. Check write permissions on the directory containing db_path; in Docker ensure the volume is writable.
3. Use an absolute db_path and serialize first-time initialization across processes (single initializer, or accept the race is benign and re-run).
4. Inspect with 'sqlite3 <db> .tables' to see whether the table exists in a different file than expected.

Example fix

// before
await AsyncDatabaseManager(database_path="crawl.db").ainit_db()  # relative path, cwd changed

// after
import pathlib
await AsyncDatabaseManager(database_path=str(pathlib.Path("~/.crawl4ai/crawl.db").expanduser())).ainit_db()

import sqlite3

def db_has_table(db_path: str, table: str = "crawled_data") -> bool:
    if not os.path.exists(db_path):
        return False
    con = sqlite3.connect(db_path)
    try:
        row = con.execute(
            "SELECT name FROM sqlite_master WHERE type='table' AND name=?", (table,)
        ).fetchone()
        return row is not None
    finally:
        con.close()

try:
    await db_manager.initialize()
except Exception as e:
    if "crawled_data table was not created" in str(e):
        os.remove(db_path)
        await db_manager.initialize()  # fresh recreate

When it happens

Trigger: Two processes racing to initialize the same SQLite file (lock/rollback); a corrupted or pre-existing non-schema database file at the resolved path; db_path pointing to a read-only directory so CREATE TABLE failed; path resolution changing between ainit_db and the verification connect (e.g. relative paths with changed cwd).

Common situations: Multiple crawler instances or workers sharing one DB file concurrently; running with insufficient filesystem permissions (read-only volume in Docker); stale/corrupt .db files from older versions; relative db_path resolved differently across async tasks after a cwd change.

Related errors

• Database missing columns: {missing_columns}
• Local file not found: {local_file_path}
• Refusing to write download through symlink: {download_path}
• Local file not found: {path}
• Profile not found: {profile_path}

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/8d09c3ccabb9f6b8. Report an issue: GitHub.