What causes "failed to build data-generation runtime" in risingwavelabs/risingwave?

Raise the container/OS thread limit (ulimit -u, cgroups pids.max) and free memory, then retry.

failed to build data-generation runtime

Error message

failed to build data-generation runtime

What it means

`spawn_data_generation_stream` lazily builds a dedicated multi-thread Tokio runtime (named "rw-datagen") for data-generation streams. If `build()` fails (thread creation failure, resource exhaustion, platform constraints) it panics with this expect message.

Solutions

Raise the container/OS thread limit (ulimit -u, cgroups pids.max) and free memory, then retry.
Reuse an existing runtime instead of building a dedicated one if embedding in your own Tokio app.
Patch to `handle` an existing runtime or fall back to a current-thread runtime builder when multi-thread build fails.

Example fix

// before
.build()
.expect("failed to build data-generation runtime")
// after
.build()
.unwrap_or_else(|e| panic!("failed to build data-generation runtime: {e}")) // with logging, or fallback to current_thread builder

Defensive patterns

Strategy: try-catch

Validate before calling

// Check thread limits before launching datagen workloads:
ulimit -u   # and cgroup pids.max; must allow dozens of threads

Try / catch

std::panic::catch_unwind(|| {
    spawn_data_generation_stream(stream, buffer, gen_fn)
}).unwrap_or_else(|_| {
    tracing::error!("datagen runtime failed to build; check thread/memory limits");
    // fall back to a current-thread runtime or skip datagen
});

Prevention

Raise container thread/pids limits for RisingWave deployments.
Ensure sufficient memory headroom for extra Tokio runtimes.
Test datagen sources in constrained sandboxes before production.

When it happens

Trigger: First call into any datagen source's `into_stream`/`into_data_stream` when the process cannot spawn a new multi-threaded runtime: OS thread/thread-limit exhaustion, low memory, restricted sandbox environments.

Common situations: Containers with very low thread/pid limits (ulimit -u, pids cgroup limit), heavily loaded hosts, or running under test harnesses that restrict threading.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/20b2f8690233612d. Report an issue: GitHub.

Appendix: source

Thrown at src/connector/src/source/data_gen_util.rs:37

use tokio::sync::mpsc;

/// Spawn the data generator to a dedicated runtime, returns a channel receiver
/// for acquiring the generated data. This is used for the [`DatagenSplitReader`]
/// and [`NexmarkSplitReader`] in case that they are CPU intensive
/// and may block the streaming actors.
///
/// [`DatagenSplitReader`]: super::datagen::DatagenSplitReader
/// [`NexmarkSplitReader`]: super::nexmark::source::reader::NexmarkSplitReader
pub fn spawn_data_generation_stream<T: Send + 'static>(
    stream: impl Stream<Item = T> + Send + 'static,
    buffer_size: usize,
) -> impl Stream<Item = T> + Send + 'static {
    static RUNTIME: LazyLock<Runtime> = LazyLock::new(|| {
        tokio::runtime::Builder::new_multi_thread()
            .thread_name("rw-datagen")
            .enable_all()
            .build()
            .expect("failed to build data-generation runtime")
    });

    let (generation_tx, generation_rx) = mpsc::channel(buffer_size);
    RUNTIME.spawn(async move {
        pin_mut!(stream);
        while let Some(result) = stream.next().await {
            if generation_tx.send(result).await.is_err() {
                tracing::warn!("failed to send next event to reader, exit");
                break;
            }
        }
    });

    tokio_stream::wrappers::ReceiverStream::new(generation_rx)
}

View on GitHub (pinned to 6469eb736d)