xai-org/x-algorithm · error · anyhow::Error

not ListArray

Error message

not ListArray

What it means

After locating the 'user_actions' column, the code downcasts it to arrow::array::ListArray. If the column is any other Arrow type (e.g. LargeListArray, StructArray, or a plain non-nested type), the downcast returns None and this error fires. The producer must emit user_actions as a List array.

Source

Thrown at bdsm/rust/abuse-v3-features/src/lib.rs:264

    for b in reader {
        batches.push(b?);
    }
    if batches.is_empty() {
        anyhow::bail!("empty");
    }
    let batch = if batches.len() == 1 {
        batches.into_iter().next().unwrap()
    } else {
        arrow::compute::concat_batches(&batches[0].schema(), &batches)?
    };

    let actions_col = batch
        .column_by_name("user_actions")
        .ok_or_else(|| anyhow::anyhow!("no user_actions"))?;
    let list = actions_col
        .as_any()
        .downcast_ref::<ListArray>()
        .ok_or_else(|| anyhow::anyhow!("not ListArray"))?;
    if list.is_empty() || list.is_null(0) {
        anyhow::bail!("empty list");
    }

    let vals = list.value(0);
    let sa = vals
        .as_any()
        .downcast_ref::<StructArray>()
        .ok_or_else(|| anyhow::anyhow!("not StructArray"))?;
    let total = sa.len();
    if total == 0 {
        anyhow::bail!("zero actions");
    }

    let start = if total > seq_len { total - seq_len } else { 0 };
    let take = total.min(seq_len);
    let sl = sa.slice(start, take);
    let sl = sl.as_any().downcast_ref::<StructArray>().unwrap();

View on GitHub (pinned to 24c60942c5)

Solutions

  1. Print the column's DataType (actions_col.data_type()) to see the actual type
  2. If it is LargeListArray, either change the producer to emit ListArray or downcast to LargeListArray here
  3. Regenerate/align the Arrow schema between producer and consumer (field type must be List(...))
  4. Add an upfront schema check comparing the expected exact DataType before extraction

Example fix

// before
let list = actions_col
    .as_any()
    .downcast_ref::<ListArray>()
    .ok_or_else(|| anyhow::anyhow!("not ListArray"))?;

// after
let list = actions_col
    .as_any()
    .downcast_ref::<ListArray>()
    .ok_or_else(|| anyhow::anyhow!(
        "not ListArray: user_actions is {:?}",
        actions_col.data_type()
    ))?;
Defensive patterns

Strategy: type-guard

Validate before calling

// Check the column type before downcasting
let dt = batch.column_by_name("user_actions").map(|c| c.data_type().clone());
anyhow::ensure!(matches!(dt, Some(DataType::List(_))), "user_actions must be List, got {dt:?}");

Type guard

fn is_list_array(col: &ArrayRef) -> bool {
    matches!(col.data_type(), DataType::List(_))
}
// or handle LargeList too:
fn as_list(col: &ArrayRef) -> Option<&ListArray> {
    col.as_any().downcast_ref::<ListArray>()
}

Try / catch

// branch on the actual type instead of failing on downcast
let list = match actions_col.data_type() {
    DataType::List(_) => actions_col.as_any().downcast_ref::<ListArray>().unwrap(),
    DataType::LargeList(_) => /* convert or handle LargeListArray */,
    other => anyhow::bail!("unsupported user_actions type: {other:?}"),
};

Prevention

When it happens

Trigger: Calling extract_batch where the user_actions column is typed as LargeList, Struct, or any non-List Arrow type — commonly after a producer migration to LargeListArray for >2B row support or a hand-built test batch using the wrong type.

Common situations: Arrow version upgrades where readers/writers default to LargeList, alternate producers writing struct-of-list instead of list-of-struct, or test fixtures constructing batches with mismatched types.

Related errors

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/6a306f05a742af10. Report an issue: GitHub.