unclecode/crawl4ai · error · ValueError

Database missing columns: {missing_columns}

Error message

Database missing columns: {missing_columns}

What it means

Raised during connection acquisition when the opened 'crawled_data' table is missing one or more expected schema columns (markdown, extracted_content, success, media, links, metadata, screenshot, response_headers, downloaded_files, etc.). This indicates an older database file created by a previous crawl4ai version whose schema predates those columns, and auto-migration did not (or has not yet) run.

Source

Thrown at crawl4ai/async_database.py:159

                            columns = await cursor.fetchall()
                            column_names = [col[1] for col in columns]
                            expected_columns = {
                                "url",
                                "html",
                                "cleaned_html",
                                "markdown",
                                "extracted_content",
                                "success",
                                "media",
                                "links",
                                "metadata",
                                "screenshot",
                                "response_headers",
                                "downloaded_files",
                            }
                            missing_columns = expected_columns - set(column_names)
                            if missing_columns:
                                raise ValueError(
                                    f"Database missing columns: {missing_columns}"
                                )

                        self.connection_pool[task_id] = conn
                    except Exception as e:
                        import sys

                        error_context = get_error_context(sys.exc_info())
                        error_message = (
                            f"Unexpected error in db get_connection at line {error_context['line_no']} "
                            f"in {error_context['function']} ({error_context['filename']}):\n"
                            f"Error: {str(e)}\n\n"
                            f"Code context:\n{error_context['code_context']}"
                        )
                        self.logger.error(
                            message="{error}",
                            tag="ERROR",
                            params={"error": str(error_message)},

View on GitHub (pinned to 7e80152142)

Solutions

1. Delete or archive the old database file so a fresh schema is created (fastest if the data is disposable).
2. Trigger the version-manager update path (let initialization detect the version change and run update_db_schema/migrations) instead of reusing a connection created before update.
3. If the data must be kept, manually ALTER TABLE to add the missing columns listed in the error message.
4. Pin one crawl4ai version across all machines sharing the DB.

Example fix

// before
# ~/.crawl4ai/crawl.db created by crawl4ai 0.4.x, now running 0.6.x
# ValueError: Database missing columns: {'response_headers', 'downloaded_files'}

// after
import os
os.remove("~/.crawl4ai/crawl.db")  # or move aside, then re-run init
await AsyncDatabaseManager().initialize()

import sqlite3

EXPECTED = {"url", "html", "markdown", "extracted_content", "success", "media", "links", "metadata", "screenshot", "response_headers", "downloaded_files"}

def schema_current(db_path: str) -> bool:
    con = sqlite3.connect(db_path)
    try:
        cols = {r[1] for r in con.execute("PRAGMA table_info(crawled_data)")}
        return EXPECTED <= cols
    finally:
        con.close()

try:
    await db_manager.get_connection()
except ValueError as e:
    if "Database missing columns" in str(e):
        archive_and_recreate_db(db_path)
        await db_manager.get_connection()

When it happens

Trigger: Upgrading crawl4ai while keeping an old crawl.db produced by an earlier release; manually created or externally modified tables; a migration step that failed earlier (error 58's path) leaving a half-updated schema; copying a DB file between installations of different versions.

Common situations: pip install -U crawl4ai then re-running with the default persistent DB path; shared DB across machines with mixed crawler versions; tests that create a partial schema fixture.

Related errors

• crawled_data table was not created

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/291e3b27e272f49c. Report an issue: GitHub.