rustfs/rustfs · error · SelectError

JSONParsingError

JSONParsingError

Error message

An error occurred while parsing the JSON file. Check the file and try again.

What it means

SelectError::JsonParsingError is the classification of ArrowError::JsonError (crates/s3select-api/src/lib.rs:228) and of the select object store's JSON line-scan failure (crates/s3select-api/src/object_store.rs:1191). The JSON reader could not parse the object's lines or documents: syntax errors or type-inconsistent records.

Source

Thrown at crates/s3select-api/src/lib.rs:91

}

#[derive(Clone, Debug, Error, PartialEq, Eq)]
pub enum SelectError {
    #[error("The file is not in a supported compression format. Only GZIP and BZIP2 are supported.")]
    InvalidCompressionFormat,

    #[error("The data source type is not valid. Only CSV, JSON, and Parquet are supported.")]
    InvalidDataSource,

    #[error(
        "Object decompression failed. Check that the object is properly compressed using the format specified in the request."
    )]
    TruncatedInput,

    #[error("An error occurred while parsing the CSV file. Check the file and try again.")]
    CsvParsingError,

    #[error("An error occurred while parsing the JSON file. Check the file and try again.")]
    JsonParsingError,

    #[error("An error occurred while parsing the Parquet file. Check the file and try again.")]
    ParquetParsingError,

    #[error("{message}")]
    ParseSelectFailure { message: String },

    #[error("The SQL expression is invalid.")]
    InvalidQuery,

    #[error("The SQL expression contains a data type that is not valid.")]
    InvalidDataType,

    #[error("An incorrect argument type was specified in a function call in the SQL expression.")]
    IncorrectSqlFunctionArgumentType,

    #[error("The data source path in the SQL expression is not supported.")]

View on GitHub (pinned to 35af688cd9)

Solutions

  1. Validate the object client-side before selecting (jq for DOCUMENT; per-line jq -c or python json.loads for LINES)
  2. Make the JSON Type in the request match the actual layout (DOCUMENT for a single pretty-printed value, LINES for newline-delimited records)
  3. Keep types consistent per key across records, or normalize dirty values before upload
  4. Drop or fix malformed lines and re-upload the object
Defensive patterns

Strategy: validation

Validate before calling

// Pre-validate JSON Lines objects line by line before selecting.
for (i, line) in head_bytes.lines().enumerate() {
    let line = line?;
    if line.trim().is_empty() { continue; }
    serde_json::from_str::<serde_json::Value>(line)
        .map_err(|e| err(format!("line {}: {e}", i + 1)))?;
}

Try / catch

match select_object_content(req).await {
    Ok(resp) => { /* records */ }
    Err(e) if matches!(e.select_error(), SelectError::JsonParsingError) => {
        // Report the malformed-object condition; do not retry the same bytes.
    }
    Err(e) => return Err(e),
}

Prevention

When it happens

Trigger: SelectObjectContent with JSON input serialization on a file containing malformed lines (truncated records, trailing commas, single quotes), mixed types for the same key across lines, or invalid UTF-8; JSON Type (DOCUMENT vs LINES) not matching the file layout.

Common situations: JSON Lines files with one truncated line from a partial append; pretty-printed multi-line JSON fed to a line-oriented reader; heterogeneous event logs where a field changes type between records.

Understand the failure class

Background: JSON parse error: "Unexpected token" / "not valid JSON" / "failed to parse" — what JSON parsers are really complaining about — this error's family across 45 libraries.

Related errors

AI-assisted analysis of rustfs/rustfs@35af688cd9 (2026-08-20). Data as JSON: /api/errors/501f807f11f9e619. Report an issue: GitHub.