internal error: expect TxnGetResponseGet of get database…

Error message

internal error: expect TxnGetResponseGet of get database meta by db_id, but got {:?}

What it means

In gc_dropped_db_by_id, the code issues a transactional GET on the database meta kv pair and asserts the response is a Get response carrying the DB meta. If the meta-service returns any other response variant (e.g. an empty or unexpected Txn response), this unreachable!() fires with the offending response. It means the transaction API returned a shape the caller never expected for a plain get-by-id — an internal protocol/invariant violation rather than a missing-database error (that is handled earlier via the value being None).

Solutions

Verify all meta nodes run the same databend version; rolling-upgrade mismatches can change txn response shapes
Check the meta store entry for the dropped db id for corruption or missing migration steps
Capture the full log of gc_drop_tables and report as a meta-service bug including the unexpected response variant
Restart/repair the meta cluster from a consistent snapshot if the store is inconsistent

Defensive patterns

Strategy: retry

Type guard

fn is_get_response(resp: &Response) -> bool { matches!(resp, Response::Get(_)) }

Try / catch

match resp.response { Response::Get(g) => process(g), other => { log::error!("unexpected txn response: {:?}", other); retry_with_backoff(); } }

Prevention

Ensure all meta nodes run identical versions
Retry GC after transient meta-store errors
Inspect the dropped db's meta entries if panics repeat

When it happens

Trigger: gc_dropped_db_by_id (invoked from gc_drop_tables) reading back the DB meta by id and receiving a non-Get response from the meta kv transaction — typically after a key-space/version mismatch or a malformed response from a meta node.

Common situations: Mixed-version databend-meta cluster where one node returns a newer/older response shape; a corrupted or partially migrated meta store for the dropped database id; proxying meta traffic through an incompatible node.

Related errors

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/ffe4cb1f3a609f5e. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/api/src/api_impl/garbage_collection_api.rs:462

        let mut new_db_meta = seq_db_meta;
        new_db_meta.gc_in_progress = true;
        let mut txn = TxnRequest::default();
        txn_replace_exact(&mut txn, &dbid, new_db_meta.seq, &new_db_meta.data)?;
        txn.if_then.push(txn_get(&dbid));
        let (success, mut responses) = send_txn(kv_api, txn).await?;
        if !success {
            return Err(KVAppError::AppError(AppError::from(
                MarkDatabaseMetaAsGCInProgressFailed::new(format!(
                    "Failed to mark database {}[{}] as gc_in_progress",
                    db_name, db_id
                )),
            )));
        }

        // Grab the sequence number of new database meta key value pair
        let resp = responses.pop().unwrap();
        let Some(Response::Get(get_resp)) = resp.response else {
            unreachable!(
                "internal error: expect TxnGetResponseGet of get database meta by db_id, but got {:?}",
                resp.response
            )
        };
        db_meta_seq = get_resp
            .value
            .expect("txn op response of get(&dbid) should have value")
            .seq
    };

    // Cleaning up DbIdTableName keys
    {
        let db_id_table_name = DBIdTableName {
            db_id,
            // Going to use 1 level DirName as list prefix, thus the table_name field does not matter
            table_name: "dummy".to_string(),
        };
        let dir_name = DirName::new_with_level(db_id_table_name, 1);

View on GitHub (pinned to 288d84d76e)