risingwavelabs/risingwave · error · ConnectorError

serde_json deserialization error (anyhow!(e))

Error message

serde_json deserialization error (anyhow!(e))

What it means

BatchPosixFsSplit::restore_from_json converts a JsonbVal back into a BatchPosixFsSplit using serde_json::from_value. Any failure to deserialize the JSON payload (wrong fields, wrong types, invalid shape) is wrapped with anyhow! and returned as a ConnectorResult error. This happens when the persisted split descriptor does not match the current Rust struct schema.

Source

Thrown at src/connector/src/source/filesystem/opendal_source/batch_posix_fs_source.rs:55

pub struct BatchPosixFsSplit {
    /// For batch posix fs, this is always the root directory. The reader will
    /// scan all files in this directory.
    pub file_path: String,
    /// A unique identifier for the split, typically including a timestamp to force refresh.
    pub split_id: SplitId,
    /// Whether this split has finished reading all data (used for batch sources)
    /// See [`BatchSourceSplit`] for details about recovery.
    #[serde(skip)]
    pub finished: bool,
}

impl SplitMetaData for BatchPosixFsSplit {
    fn id(&self) -> SplitId {
        self.split_id.clone()
    }

    fn restore_from_json(value: JsonbVal) -> ConnectorResult<Self> {
        serde_json::from_value(value.take()).map_err(|e| anyhow!(e).into())
    }

    fn encode_to_json(&self) -> JsonbVal {
        serde_json::to_value(self.clone()).unwrap().into()
    }

    fn update_offset(&mut self, _last_seen_offset: String) -> ConnectorResult<()> {
        // Batch source doesn't use offsets - each file is read completely once
        Ok(())
    }
}

impl BatchSourceSplit for BatchPosixFsSplit {
    fn finished(&self) -> bool {
        self.finished
    }

    fn finish(&mut self) {

View on GitHub (pinned to 6469eb736d)

Solutions

1. Inspect the JSON payload being restored and make sure it has exactly the fields BatchPosixFsSplit expects (split_id, file_path, offset etc.) with correct types.
2. Re-create the source / let it re-list files so splits are re-encoded with the current schema.
3. If upgrading versions, follow the migration path so persisted split state is compatible; otherwise drop and rebuild the MV/source.
4. Add #[serde(default)] or a custom Deserialize to BatchPosixFsSplit for backward compatibility if fields were added.

Example fix

// before
serde_json::from_value(value.take()).map_err(|e| anyhow!(e).into())
// after
serde_json::from_value(value.take())
    .map_err(|e| anyhow!("failed to restore BatchPosixFsSplit from {}: {e}", value.as_str().unwrap_or("json")))
    .into()

// Rust: validate split JSON before restoring
fn split_json_is_valid(v: &serde_json::Value) -> bool {
    v.is_object() && v.get("split_id").map_or(false, |s| s.is_string())
}

fn is_valid_split_value(v: &JsonbVal) -> bool {
    serde_json::from_value::<BatchPosixFsSplit>(v.clone().take()).is_ok()
}

match BatchPosixFsSplit::restore_from_json(value) {
    Ok(split) => split,
    Err(e) => { log::warn!("split restore failed, re-listing: {e}"); re_list_splits() }
}

When it happens

Trigger: SplitManager or source engine calls restore_from_json with a JsonbVal produced by encode_to_json of an older/different struct version, a hand-edited split JSON, or a JSON whose fields don't match BatchPosixFsSplit (missing/renamed fields, wrong types).

Common situations: Upgrading RisingWave across versions where BatchPosixFsSplit fields changed; corrupt or manually edited checkpoint state; splits persisted by a different filesystem connector variant being restored by this one.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• You can't use the macro on this type
• Failed to decode prost: field not found `{}`
• Must have exactly 1 buffer in a jsonb array
• Must have no buffer in a list array
• Must have at least one element in offsets

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/c5223d577f05fd1e. Report an issue: GitHub.