risingwavelabs/risingwave · error · SchedulerError

RpcError

Error message

RpcError

What it means

SchedulerError::RpcError is a transparent wrapper (via #[from]) over risingwave_rpc_client::RpcError, so the displayed message is just the underlying RPC error. It surfaces when the frontend scheduler's gRPC calls to compute nodes or the meta node fail during query execution — connection failures, deadline exceeded, or transport errors.

Source

Thrown at src/frontend/src/scheduler/error.rs:29

// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

use risingwave_batch::error::BatchError;
use risingwave_common::session_config::QueryMode;
use risingwave_connector::error::ConnectorError;
use risingwave_rpc_client::error::RpcError;
use thiserror::Error;

use crate::error::{ErrorCode, RwError};
use crate::scheduler::plan_fragmenter::QueryId;

#[derive(Error, Debug)]
pub enum SchedulerError {
    #[error("Pin snapshot error: {0} fails to get epoch {1}")]
    PinSnapshot(QueryId, u64),

    #[error(transparent)]
    RpcError(
        #[from]
        #[backtrace]
        RpcError,
    ),

    #[error("{0}")]
    TaskExecutionError(String),

    #[error("Task got killed because compute node running out of memory")]
    TaskRunningOutOfMemory,

    /// Used when receive cancel request for some reason, such as user cancel or timeout.
    #[error("Query cancelled: {0}")]
    QueryCancelled(String),

    #[error(
        "Reject query: the {0} query number reaches the limit: {1}. Use `SHOW PROCESSLIST` to check for hanging queries and cancel them if needed."

View on GitHub (pinned to 6469eb736d)

Solutions

1. Check that all compute nodes are healthy (`SHOW LIVE NODES` / cluster logs) and restart crashed nodes.
2. Retry the query; transient gRPC failures often resolve after the cluster reconverges.
3. Verify network connectivity and configured host/port settings between frontend, meta, and compute nodes.
4. If timeouts recur under load, increase RPC timeout settings or scale out compute resources.

// probe connectivity first
psql -h <frontend> -p 4566 -c 'SHOW LIVE NODES';

// retry on RpcError, surface others
match res { Err(e) if is_rpc_error(&e) => retry(3, backoff), other => other }

When it happens

Trigger: Any gRPC exchange during batch scheduling fails: `SwapOrder`, executing stage requests to compute nodes, or `pin_snapshot` RPCs, when the target node is down, the network partitions, or the request deadline expires.

Common situations: A compute node crashed or was restarted mid-query; misconfigured service ports; network isolation between frontend and compute nodes; timeouts under heavy load.

Related errors

• RpcError
• bigquery insert error: end of resp stream
• Cannot open client to compute node {addr:?}
• Pin snapshot error: {0} fails to get epoch {1}
• {0}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/262248e9366ea6b0. Report an issue: GitHub.