{"record":{"id":"a234d6cd29fe7ea9","repo":"netdata/netdata","slug":"startup-sync-install-join-error-e","errorCode":null,"errorMessage":"startup sync: install join error: {e}","messagePattern":"startup sync: install join error: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"src/crates/file-lifecycle/src/recovery/startup.rs","lineNumber":311,"sourceCode":"        Err(_) => {\n            return Err(anyhow::anyhow!(\n                \"startup sync: download {key} timed out after {op_timeout:?}\"\n            ));\n        }\n    };\n\n    if let Err(reason) = validate_catalog(&bytes, parsed, own_machine, signal) {\n        tracing::warn!(key = %key, reason, \"startup sync: catalog failed validation, NOT installing\");\n        return Ok(());\n    }\n\n    // Off-runtime: the atomic install fsyncs the file and its parent dir, so\n    // 8 download workers must not block the runtime on that I/O.\n    let dest = local_catalog_path(catalog_base_dir, parsed);\n    let write_dest = dest.clone();\n    tokio::task::spawn_blocking(move || file_registry::durable::write_atomic(&write_dest, &bytes))\n        .await\n        .map_err(|e| anyhow::anyhow!(\"startup sync: install join error: {e}\"))?\n        .with_context(|| format!(\"startup sync: install catalog {}\", dest.display()))?;\n    tracing::debug!(key = %key, path = %dest.display(), \"startup sync: installed catalog\");\n    // Coarse progress for large restores (a per-catalog line at info would flood).\n    let n = installed.fetch_add(1, Ordering::Relaxed) + 1;\n    if n % 100 == 0 {\n        tracing::info!(\"startup sync: {n}/{total} catalog(s) installed\");\n    }\n    Ok(())\n}\n\n/// Validate a downloaded catalog body against its remote key. Returns\n/// `Err(reason)` (a description for the skip log) on any mismatch. Checks:\n/// container magic/CRC + framing-version, then the JSON envelope's\n/// format-version (via `from_container_bytes`); envelope tenant/date/identity\n/// equal the key's segments + filename fields; the entries fold (max seq,\n/// min/max ts) equals the filename fields; and every entry's `remote_key` is a\n/// well-formed SFST key on this machine AND this tenant AND this signal, whose\n/// embedded `FileId` matches the entry's own `id` AND whose date matches the","sourceCodeStart":293,"sourceCodeEnd":329,"githubUrl":"https://github.com/netdata/netdata/blob/4864de85e26f6734d92cfc27ccbeff5921f49938/src/crates/file-lifecycle/src/recovery/startup.rs#L293-L329","documentation":"The `spawn_blocking` task performing the atomic local install (which fsyncs the file and its parent directory via `file_registry::durable::write_atomic`) did not complete normally — the JoinError means the blocking task panicked or the runtime was shut down while awaiting it. This is not a storage-backend problem; it is a local-filesystem or runtime-lifecycle failure surfaced through the join.","triggerScenarios":"`tokio::task::spawn_blocking(move || write_atomic(&dest, &bytes)).await` returns `Err(join_error)`: the closure panicked (e.g. local disk full, permission denied on the catalog dir, path too long) or the tokio runtime is being torn down mid-startup.","commonSituations":"Local disk exhaustion or read-only filesystem at the catalog base dir; the catalog base dir not created or owned by another user; plugin shutdown racing startup sync.","solutions":["Check local disk space and write permissions on the catalog base directory — panics inside `write_atomic` are usually ENOSPC/EACCES.","Ensure the catalog base dir exists and is writable by the agent user before startup.","If it happened during shutdown, verify whether the runtime was dropped while startup sync was in flight (a restart-loop artifact) rather than a persistent fault.","Re-run startup after remediation; installs are atomic and idempotent."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Before sync: assert the catalog base dir is writable and the disk has room\nfn can_install(base: &std::path::Path) -> bool {\n    let probe = base.join(\".write-probe\");\n    std::fs::write(&probe, b\"x\").is_ok() && std::fs::remove_file(&probe).is_ok()\n}","typeGuard":null,"tryCatchPattern":"// JoinError wraps a panic or shutdown; log which and fail closed\nmatch tokio::task::spawn_blocking(move || write_atomic(&dest, &bytes)).await {\n    Ok(res) => res.with_context(|| format!(\"install {}\", dest.display()))?,\n    Err(join_err) => {\n        if join_err.is_panic() {\n            tracing::error!(path = %dest.display(), \"install task panicked (disk full / permissions?)\");\n        }\n        return Err(anyhow!(\"startup sync: install join error: {join_err}\"));\n    }\n}","preventionTips":["Disk-space and permission checks on the catalog base dir belong in preflight monitoring.","Avoid tearing down the runtime while startup sync is in flight."],"tags":["rust","tokio","spawn-blocking","filesystem","startup"],"backgroundTag":null,"analyzedSha":"4864de85e26f6734d92cfc27ccbeff5921f49938","analyzedAt":"2026-08-15T09:12:38.226Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}