cocoindex-io/cocoindex · error · ValueError

Unexpected column subkey format: {sub_key!r}, expected to st

Error message

Unexpected column subkey format: {sub_key!r}, expected to start with {_COL_SUBKEY_PREFIX!r}

What it means

Target-state column subkeys in the LanceDB connector must be namespaced with _COL_SUBKEY_PREFIX so the engine can distinguish column-level actions from other sub-actions. During diff application, a subkey without that prefix indicates corrupted/unexpected state or a hand-built action map, so a ValueError identifies the bad subkey.

Source

Thrown at python/cocoindex/connectors/lancedb/_target.py:1136

        return pa.schema(fields)

    async def _apply_column_actions(
        self,
        conn: LanceAsyncConnection,
        table_name: str,
        schema: TableSchema[Any],
        column_actions: dict[str, statediff.DiffAction],
    ) -> frozenset[str]:
        """Apply additive column schema changes in place."""
        table = await conn.open_table(table_name)
        existing_cols = set((await table.schema()).names)
        pk_cols = set(schema.primary_key)
        fields_to_add: list[pa.Field] = []
        null_backfilled_columns: set[str] = set()

        for sub_key, action in column_actions.items():
            if not sub_key.startswith(_COL_SUBKEY_PREFIX):
                raise ValueError(
                    f"Unexpected column subkey format: {sub_key!r}, expected to start with {_COL_SUBKEY_PREFIX!r}"
                )

            col_name = sub_key[len(_COL_SUBKEY_PREFIX) :]
            if col_name in pk_cols:
                continue
            if col_name in existing_cols:
                continue

            desired_col = schema.columns.get(col_name)
            if desired_col is None:
                continue

            if action in ("insert", "upsert"):
                fields_to_add.append(
                    # Existing rows are backfilled with null, so additive schema
                    # evolution must materialize the new column as nullable.
                    pa.field(col_name, desired_col.type, nullable=True)

View on GitHub (pinned to e84aa99b32)

Solutions

Clear/reset the target state for this component (drop the LanceDB table state / re-run with fresh state) so keys are regenerated in the current format.
Upgrade or align the cocoindex version between the run that wrote state and the run applying it.
If you construct actions yourself, prefix subkeys with the connector's _COL_SUBKEY_PREFIX format.

Defensive patterns

Strategy: validation

Validate before calling

assert sub_key.startswith("col:"), f"Bad column subkey: {sub_key!r}"  # prefix per connector's _COL_SUBKEY_PREFIX

Try / catch

try:
    await app.update()
except ValueError as e:
    if "Unexpected column subkey format" in str(e):
        ...  # reset stale target state for this component path and re-run
    raise

Prevention

Don't hand-construct target-state action keys; use the connector APIs.
When upgrading cocoindex, verify stored target state compatibility; reset state if formats changed.
Run the same cocoindex version across all writers to a given target.

When it happens

Trigger: _apply_column_actions receiving a column_actions entry whose sub_key does not start with _COL_SUBKEY_PREFIX — e.g. target-state keys constructed manually, a connector-internal format change, or stale persisted state written by an incompatible version.

Common situations: Upgrading CocoIndex across a subkey-format change with pre-existing persisted target state; custom tooling that writes engine state directly; internal bug injecting malformed keys.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

AI-assisted analysis of cocoindex-io/cocoindex@e84aa99b32 (2026-09-08). Data as JSON: /api/errors/f27e73c95c54a3e9. Report an issue: GitHub.