risingwavelabs/risingwave · error · SchedulerError

Task got killed because compute node running out of memory

Error message

Task got killed because compute node running out of memory

What it means

SchedulerError::TaskRunningOutOfMemory indicates the batch task was killed because the compute node ran out of memory while executing it. The frontend aborts the query with this fixed message. It reflects the batch engine's memory manager deciding to kill a task under memory pressure.

Source

Thrown at src/frontend/src/scheduler/error.rs:39

use crate::error::{ErrorCode, RwError};
use crate::scheduler::plan_fragmenter::QueryId;

#[derive(Error, Debug)]
pub enum SchedulerError {
    #[error("Pin snapshot error: {0} fails to get epoch {1}")]
    PinSnapshot(QueryId, u64),

    #[error(transparent)]
    RpcError(
        #[from]
        #[backtrace]
        RpcError,
    ),

    #[error("{0}")]
    TaskExecutionError(String),

    #[error("Task got killed because compute node running out of memory")]
    TaskRunningOutOfMemory,

    /// Used when receive cancel request for some reason, such as user cancel or timeout.
    #[error("Query cancelled: {0}")]
    QueryCancelled(String),

    #[error(
        "Reject query: the {0} query number reaches the limit: {1}. Use `SHOW PROCESSLIST` to check for hanging queries and cancel them if needed."
    )]
    QueryReachLimit(QueryMode, u64),

    #[error(transparent)]
    BatchError(
        #[from]
        #[backtrace]
        BatchError,
    ),

View on GitHub (pinned to 6469eb736d)

Solutions

1. Add LIMIT or more selective WHERE predicates to reduce the working set of the query.
2. Run the heavy query when the cluster is less loaded, or increase compute node memory / scale out nodes.
3. Raise batch execution memory-related config (e.g. `developer.batch_*` memory limits) if limits are set too low.
4. Split the work into smaller queries or use a streaming (materialized view) computation instead of ad-hoc batch.

Example fix

// before
SELECT * FROM large_mv;
// after
SELECT * FROM large_mv WHERE event_time > now() - interval '1 day' LIMIT 1000;

-- bound the working set before running
SELECT count(*) FROM large_mv WHERE <selective predicate>;

// catch OOM and fall back to a narrower query
catch (e) { if (e.message.includes('running out of memory')) return narrowQuery(); throw e; }

When it happens

Trigger: Executing a batch query whose stage memory footprint exceeds the compute node's task memory limit — large scans, huge sorts/hash aggregations without spill, very wide rows, or many concurrent queries sharing one node.

Common situations: Ad-hoc analytical SELECT over large materialized views; `SELECT *` of huge tables in one query; too many concurrent queries on an under-provisioned cluster; misconfigured batch memory limits.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• {0}
• BatchError
• Not enough memory to run this query, batch memory limit is {
• Pin snapshot error: {0} fails to get epoch {1}
• RpcError

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/edd081ca1f275a45. Report an issue: GitHub.