risingwavelabs/risingwave · error · Error

failed to access meta store

Error message

failed to access meta store

What it means

hummock::Error::MetaStore wraps any failure accessing the metadata store (SeaORM-backed SQL database, e.g. PostgreSQL or SQLite) that the meta node uses to persist Hummock metadata. The inner anyhow::Error is the `#[source]`, so the real database error (connection failure, lock timeout, constraint violation) is attached. Notably, `Error::retryable()` returns true for this variant, marking it as safe to retry.

Solutions

1. Check the `source` chain for the underlying DbErr and database error code
2. Verify the meta store database is reachable and healthy (network, auth, disk space)
3. Retry the operation — this variant is explicitly marked retryable
4. Check meta store connection pool sizing and DB load
5. Ensure schema migrations for the meta store are applied for your RisingWave version

Example fix

// before
let rows = meta_store.list_sstables().await?; // hummock::Error::MetaStore bubbles up
// after
let rows = retry(backoff::Exponential::default()).try(|| async {
    meta_store.list_sstables().await
}).await?; // MetaStore errors are retryable per Error::retryable()

// Health-check the meta store before Hummock operations
sqlx::query("SELECT 1").fetch_one(&meta_db_pool).await
    .map_err(|e| format!("meta store unhealthy: {e}"))?;

fn is_meta_store_error(e: &hummock::Error) -> Option<&anyhow::Error> {
    match e { hummock::Error::MetaStore(src) => Some(src), _ => None }
}

match hummock_result {
    Err(e @ hummock::Error::MetaStore(_)) if e.retryable() => {
        backoff_retry(op).await? // this variant is retryable by design
    }
    other => other?,
}

When it happens

Trigger: Any Hummock meta operation that reads/writes the metadata database and hits a DbErr converted into this error: writes of SST info, version deltas, pinned snapshots, compaction group config; DB connection loss; constraint violations during concurrent writes.

Common situations: Metadata database (configured meta store) is unreachable, restarted, or saturated; connection pool exhausted; DB migrations not applied after upgrade; disk full on the meta store host.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

• anyhow!(err)
• Barrier read is unavailable for now. Likely the cluster is…
• Catalog error
• Change log retention miss: table
• checkpoint checksum mismatch: expected

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/19ab9e36774ef32b. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/src/hummock/error.rs:30

// See the License for the specific language governing permissions and
// limitations under the License.

use risingwave_common::catalog::TableId;
use risingwave_hummock_sdk::{HummockContextId, HummockSstableObjectId};
use risingwave_object_store::object::ObjectError;
use risingwave_rpc_client::error::ToTonicStatus;
use sea_orm::DbErr;
use thiserror::Error;

use crate::model::MetadataModelError;

pub type Result<T> = std::result::Result<T, Error>;

#[derive(Error, Debug)]
pub enum Error {
    #[error("invalid hummock context {0}")]
    InvalidContext(HummockContextId),
    #[error("failed to access meta store")]
    MetaStore(
        #[source]
        #[backtrace]
        anyhow::Error,
    ),
    #[error(transparent)]
    ObjectStore(
        #[from]
        #[backtrace]
        ObjectError,
    ),
    #[error("compactor {0} is disconnected")]
    CompactorUnreachable(HummockContextId),
    #[error("compaction group error: {0}")]
    CompactionGroup(String),
    #[error("SST {0} is invalid")]
    InvalidSst(HummockSstableObjectId),
    #[error("invalid manual compaction option: {0}")]

View on GitHub (pinned to 6469eb736d)