netdata/netdata · error · anyhow::Error

startup sync: install join error: {e}

Error message

startup sync: install join error: {e}

What it means

The `spawn_blocking` task performing the atomic local install (which fsyncs the file and its parent directory via `file_registry::durable::write_atomic`) did not complete normally — the JoinError means the blocking task panicked or the runtime was shut down while awaiting it. This is not a storage-backend problem; it is a local-filesystem or runtime-lifecycle failure surfaced through the join.

Source

Thrown at src/crates/file-lifecycle/src/recovery/startup.rs:311

        Err(_) => {
            return Err(anyhow::anyhow!(
                "startup sync: download {key} timed out after {op_timeout:?}"
            ));
        }
    };

    if let Err(reason) = validate_catalog(&bytes, parsed, own_machine, signal) {
        tracing::warn!(key = %key, reason, "startup sync: catalog failed validation, NOT installing");
        return Ok(());
    }

    // Off-runtime: the atomic install fsyncs the file and its parent dir, so
    // 8 download workers must not block the runtime on that I/O.
    let dest = local_catalog_path(catalog_base_dir, parsed);
    let write_dest = dest.clone();
    tokio::task::spawn_blocking(move || file_registry::durable::write_atomic(&write_dest, &bytes))
        .await
        .map_err(|e| anyhow::anyhow!("startup sync: install join error: {e}"))?
        .with_context(|| format!("startup sync: install catalog {}", dest.display()))?;
    tracing::debug!(key = %key, path = %dest.display(), "startup sync: installed catalog");
    // Coarse progress for large restores (a per-catalog line at info would flood).
    let n = installed.fetch_add(1, Ordering::Relaxed) + 1;
    if n % 100 == 0 {
        tracing::info!("startup sync: {n}/{total} catalog(s) installed");
    }
    Ok(())
}

/// Validate a downloaded catalog body against its remote key. Returns
/// `Err(reason)` (a description for the skip log) on any mismatch. Checks:
/// container magic/CRC + framing-version, then the JSON envelope's
/// format-version (via `from_container_bytes`); envelope tenant/date/identity
/// equal the key's segments + filename fields; the entries fold (max seq,
/// min/max ts) equals the filename fields; and every entry's `remote_key` is a
/// well-formed SFST key on this machine AND this tenant AND this signal, whose
/// embedded `FileId` matches the entry's own `id` AND whose date matches the

View on GitHub (pinned to 4864de85e2)

Solutions

  1. Check local disk space and write permissions on the catalog base directory — panics inside `write_atomic` are usually ENOSPC/EACCES.
  2. Ensure the catalog base dir exists and is writable by the agent user before startup.
  3. If it happened during shutdown, verify whether the runtime was dropped while startup sync was in flight (a restart-loop artifact) rather than a persistent fault.
  4. Re-run startup after remediation; installs are atomic and idempotent.
Defensive patterns

Strategy: try-catch

Validate before calling

// Before sync: assert the catalog base dir is writable and the disk has room
fn can_install(base: &std::path::Path) -> bool {
    let probe = base.join(".write-probe");
    std::fs::write(&probe, b"x").is_ok() && std::fs::remove_file(&probe).is_ok()
}

Try / catch

// JoinError wraps a panic or shutdown; log which and fail closed
match tokio::task::spawn_blocking(move || write_atomic(&dest, &bytes)).await {
    Ok(res) => res.with_context(|| format!("install {}", dest.display()))?,
    Err(join_err) => {
        if join_err.is_panic() {
            tracing::error!(path = %dest.display(), "install task panicked (disk full / permissions?)");
        }
        return Err(anyhow!("startup sync: install join error: {join_err}"));
    }
}

Prevention

When it happens

Trigger: `tokio::task::spawn_blocking(move || write_atomic(&dest, &bytes)).await` returns `Err(join_error)`: the closure panicked (e.g. local disk full, permission denied on the catalog dir, path too long) or the tokio runtime is being torn down mid-startup.

Common situations: Local disk exhaustion or read-only filesystem at the catalog base dir; the catalog base dir not created or owned by another user; plugin shutdown racing startup sync.

Related errors

AI-assisted analysis of netdata/netdata@4864de85e2 (2026-08-15). Data as JSON: /api/errors/a234d6cd29fe7ea9. Report an issue: GitHub.