{"record":{"id":"63047a79e6eca59b","repo":"linera-io/linera-protocol","slug":"benchmark-process-pid-pid-failed","errorCode":null,"errorMessage":"Benchmark process (pid {pid:?}) failed","messagePattern":"Benchmark process \\(pid (.+?)\\) failed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"linera-service/src/cli/main.rs","lineNumber":1223,"sourceCode":"                        {\n                            join_set.spawn(async move {\n                                let pid = child.id();\n                                let status = child.wait().await?;\n                                stdout_handle.await?;\n                                stderr_handle.await?;\n                                Ok::<_, anyhow::Error>((pid, status))\n                            });\n                        }\n\n                        loop {\n                            tokio::select! {\n                                result = join_set.join_next() => {\n                                    match result {\n                                        Some(Ok(Ok((pid, status)))) => {\n                                            if !status.success() {\n                                                error!(\"Benchmark process (pid {pid:?}) failed with status: {status:?}\");\n                                                kill_all_processes(&children_pids).await;\n                                                return Err(anyhow::anyhow!(\"Benchmark process (pid {pid:?}) failed\"));\n                                            }\n                                        }\n                                        Some(Ok(Err(e))) => {\n                                            error!(\"Benchmark process failed: {e}\");\n                                            kill_all_processes(&children_pids).await;\n                                            return Err(e);\n                                        }\n                                        Some(Err(e)) => {\n                                            error!(\"Benchmark process panicked: {e}\");\n                                            kill_all_processes(&children_pids).await;\n                                            return Err(e.into());\n                                        }\n                                        None => {\n                                            info!(\"All benchmark processes have finished\");\n                                            break;\n                                        }\n                                    }\n                                }","sourceCodeStart":1205,"sourceCodeEnd":1241,"githubUrl":"https://github.com/linera-io/linera-protocol/blob/6c226ddcb332ef55118dc8d0aafbd093d5420899/linera-service/src/cli/main.rs#L1205-L1241","documentation":"One of the spawned benchmark worker processes exited with a non-zero status. The orchestrator logs the exact pid and exit status at error level just before raising this, kills all sibling processes, and aborts the whole benchmark run — one dead child means the results are incomplete.","triggerScenarios":"A child process panics or exits non-zero during the benchmark: port conflicts between workers, two workers opening the same wallet or RocksDB storage path, OOM kills, or an application-level error in the child.","commonSituations":"Running N processes with non-unique --port/storage/wallet assignments; machine resource exhaustion at high process counts; flaky validators; leftover processes from a previous run still holding ports or DB locks.","solutions":["Read the `Benchmark process (pid ..) failed with status: ..` error line printed just before this error — it identifies the failing worker and its exit status (signal vs exit code).","Check that worker ports, storage paths, and wallet files are unique per process.","Kill leftovers from previous runs (pkill / tman kill) before starting.","Reproduce with a single process to surface the child's own panic message in its stderr."],"exampleFix":"# before: workers share a storage path -> RocksDB lock panic in one child\n--processes 4 --storage rocksdb:bench.db ...\n\n# after: unique storage/port/wallet per worker, leftovers reaped first\npkill -f linera-benchmark || true\n# worker i: --storage rocksdb:bench-{i}.db --port {base+i} --wallet bench-wallet-{i}.json","handlingStrategy":"try-catch","validationCode":"// Pre-flight: claim unique ports/storage per worker before spawning.\nfor i in 0..processes {\n    let port = base_port + i as u16;\n    let db = format!(\"rocksdb:bench-{i}.db\");\n    ensure!(!TcpListener::bind((host, port)).await?.local_addr().is_unspecified(),\n        \"port {port} already in use\");\n}","typeGuard":null,"tryCatchPattern":"match join_set.join_next().await {\n    Some(Ok(Ok((pid, status)))) if !status.success() => {\n        // the orchestrator already logged pid+status and killed siblings;\n        // inspect that child's stderr/logs before rerunning\n        kill_all_processes(&children_pids).await;\n        anyhow::bail!(\"benchmark child {pid:?} failed: {status:?}\");\n    }\n    other => other.transpose()?.transpose()?,\n}","preventionTips":["Assign unique port, storage path, and wallet per worker process.","Capture each child's stderr to a per-worker log file for post-mortem.","Reap leftover processes from previous runs before starting.","Reproduce failures with --processes 1 to see the child's own error."],"tags":["benchmark","subprocess","process-management"],"backgroundTag":"subprocess-exit-failure","analyzedSha":"6c226ddcb332ef55118dc8d0aafbd093d5420899","analyzedAt":"2026-08-22T22:49:09.787Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}