risingwavelabs/risingwave · error · MetaError

database {} unavailable {}

Error message

database {} unavailable {}

What it means

mark_blocked_and_notify_failed marks a database's barrier queue as blocked and fails every pending queued start with a MetaError whose text is the database-blocked reason (rendered as `database <id> unavailable <reason>`). It is invoked when aborting barriers after detecting the database must stop making progress (e.g. it was dropped, paused, or its scheduling was aborted).

Source

Thrown at src/meta/src/barrier/schedule.rs:610

    /// Mark command scheduler as blocked and abort all queued scheduled command and notify with
    /// specific reason.
    pub(super) fn abort_and_mark_blocked(
        &self,
        database_id: Option<DatabaseId>,
        reason: impl Into<String>,
    ) {
        let mut queue = self.inner.queue.lock();
        fn database_blocked_reason(database_id: DatabaseId, reason: &String) -> String {
            format!("database {} unavailable {}", database_id, reason)
        }
        fn mark_blocked_and_notify_failed(
            database_id: DatabaseId,
            queue: &mut DatabaseScheduledQueue,
            reason: &String,
        ) {
            let reason = database_blocked_reason(database_id, reason);
            let err: MetaError = anyhow!("{}", reason).into();
            queue.mark_blocked(reason);
            while let Some(ScheduledQueueItem { notifier, .. }) = queue.queue.pop_front() {
                notifier.notify_start_failed(err.clone());
            }
        }
        if let Some(database_id) = database_id {
            let reason = reason.into();
            match queue.queue.entry(database_id) {
                Entry::Occupied(entry) => {
                    let queue = entry.into_mut();
                    if queue.status.is_blocked() {
                        if cfg!(debug_assertions) {
                            panic!("database {} marked as blocked twice", database_id);
                        } else {
                            warn!(?database_id, "database marked as blocked twice");
                        }
                    }
                    info!(?database_id, "database marked as blocked");

View on GitHub (pinned to 6469eb736d)

Solutions

1. Read the `<reason>` suffix in the message — it states why the database was blocked (dropped, paused, aborted, etc.).
2. If the database was dropped intentionally, stop clients targeting it and recreate it if needed.
3. If the block was unintentional, re-enable/unblock the database (admin command or recovery) and retry the operation.
4. During failover, wait for recovery to complete and reconnect before issuing new barrier-dependent queries.

Example fix

// before: retrying blindly after database was blocked
loop { run_query(); }
// after: check database state before retrying
if let Err(e) = run_query().await {
    if is_database_blocked(&e) { unblock_or_recreate_database(db_id).await?; }
}

-- ensure the database exists and is not blocked/paused before issuing work
SELECT id, state FROM rw_databases WHERE id = <database_id>;
-- abort if state indicates dropped/paused

// treat as terminal for this database; do not blind-retry
if is_database_blocked(&err) {
    stop_pending_work_for(db_id);
    // re-enable or recreate the database, then resubmit
}

When it happens

Trigger: abort_and_mark_blocked is called for a database id and the queue has pending items: the database was dropped/paused, an admin command blocked it, or barrier scheduling for that database was aborted; every waiting `notify_start_failed` sender receives this cloned error.

Common situations: A user dropped or paused a database while streaming jobs/barriers were in flight; an operator aborted a stuck barrier schedule; automated tooling issued aborts during failover or migration.

Related errors

• database {database_id} does not exist while handling command
• since_timestamp epoch has not been resolved for snapshot bac
• cannot create batch refresh job while database barrier is pa
• replace sink must not use snapshot backfill
• old sink job {} not found in barrier state

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/33f202d139bfc2f1. Report an issue: GitHub.