linera-io/linera-protocol · error

Benchmark process (pid {pid:?}) failed

Error message

Benchmark process (pid {pid:?}) failed

What it means

One of the spawned benchmark worker processes exited with a non-zero status. The orchestrator logs the exact pid and exit status at error level just before raising this, kills all sibling processes, and aborts the whole benchmark run — one dead child means the results are incomplete.

Source

Thrown at linera-service/src/cli/main.rs:1223

                        {
                            join_set.spawn(async move {
                                let pid = child.id();
                                let status = child.wait().await?;
                                stdout_handle.await?;
                                stderr_handle.await?;
                                Ok::<_, anyhow::Error>((pid, status))
                            });
                        }

                        loop {
                            tokio::select! {
                                result = join_set.join_next() => {
                                    match result {
                                        Some(Ok(Ok((pid, status)))) => {
                                            if !status.success() {
                                                error!("Benchmark process (pid {pid:?}) failed with status: {status:?}");
                                                kill_all_processes(&children_pids).await;
                                                return Err(anyhow::anyhow!("Benchmark process (pid {pid:?}) failed"));
                                            }
                                        }
                                        Some(Ok(Err(e))) => {
                                            error!("Benchmark process failed: {e}");
                                            kill_all_processes(&children_pids).await;
                                            return Err(e);
                                        }
                                        Some(Err(e)) => {
                                            error!("Benchmark process panicked: {e}");
                                            kill_all_processes(&children_pids).await;
                                            return Err(e.into());
                                        }
                                        None => {
                                            info!("All benchmark processes have finished");
                                            break;
                                        }
                                    }
                                }

View on GitHub (pinned to 6c226ddcb3)

Solutions

1. Read the `Benchmark process (pid ..) failed with status: ..` error line printed just before this error — it identifies the failing worker and its exit status (signal vs exit code).
2. Check that worker ports, storage paths, and wallet files are unique per process.
3. Kill leftovers from previous runs (pkill / tman kill) before starting.
4. Reproduce with a single process to surface the child's own panic message in its stderr.

Example fix

# before: workers share a storage path -> RocksDB lock panic in one child
--processes 4 --storage rocksdb:bench.db ...

# after: unique storage/port/wallet per worker, leftovers reaped first
pkill -f linera-benchmark || true
# worker i: --storage rocksdb:bench-{i}.db --port {base+i} --wallet bench-wallet-{i}.json

// Pre-flight: claim unique ports/storage per worker before spawning.
for i in 0..processes {
    let port = base_port + i as u16;
    let db = format!("rocksdb:bench-{i}.db");
    ensure!(!TcpListener::bind((host, port)).await?.local_addr().is_unspecified(),
        "port {port} already in use");
}

match join_set.join_next().await {
    Some(Ok(Ok((pid, status)))) if !status.success() => {
        // the orchestrator already logged pid+status and killed siblings;
        // inspect that child's stderr/logs before rerunning
        kill_all_processes(&children_pids).await;
        anyhow::bail!("benchmark child {pid:?} failed: {status:?}");
    }
    other => other.transpose()?.transpose()?,
}

When it happens

Trigger: A child process panics or exits non-zero during the benchmark: port conflicts between workers, two workers opening the same wallet or RocksDB storage path, OOM kills, or an application-level error in the child.

Common situations: Running N processes with non-unique --port/storage/wallet assignments; machine resource exhaustion at high process counts; flaky validators; leftover processes from a previous run still holding ports or DB locks.

Related errors

• Expected balance: {expected_balance}, actual balance: {actua
• Benchmark requires the 'opentelemetry' feature to be enabled
• Child process {self:?} already exited with status: {status}
• Failed to unwrap shared context
• No default chain found for client

AI-assisted analysis of linera-io/linera-protocol@6c226ddcb3 (2026-08-22). Data as JSON: /api/errors/63047a79e6eca59b. Report an issue: GitHub.