stamparm/maltrail · error

unhandled panic in process_packet

Error message

unhandled panic in process_packet (worker {})

What it means

A worker thread's process_packet call panicked; the sensor catches the unwind so a parser bug cannot take down the whole sensor, increments panics_recovered, and logs this error naming the worker. The offending packet is dropped but processing continues on the next packet.

Solutions

  1. Capture the offending packet: enable packet dump or note the timestamp (sec, usec) so the triggering bytes can be reproduced in a test.
  2. Report/reproduce via a minimized pcap — panics in process_packet are parser bugs and should be filed with the triggering packet bytes and offset.
  3. Verify panics_recovered in metrics to scope the impact (how many packets are being dropped).
  4. Update the sensor to a version with the parser fix; as an interim, exclude the offending traffic source or interface if the panic rate is high.
Defensive patterns

Strategy: try-catch

Try / catch

// the sensor already isolates the panic per-packet; operators should
// watch the recovery counter and file a bug with the triggering packet
if st.metrics.panics_recovered_delta > 0 {
    dump_ring_buffer_packet(sec, usec, offset); // capture bytes for reproduction
    file_parser_bug(packet_bytes, offset);
}

Prevention

When it happens

Trigger: run_all's per-packet catch_unwind(AssertUnwindSafe(|| process::process_packet(&mut st, data, sec, usec, offset))) returns Err — process_packet panicked on a particular packet's bytes, timestamp, or offset value.

Common situations: Crafted or malformed packet data triggering an unwrap/overflow/index bug in a protocol parser; offset from the DLT heuristic (see the unexpected-datalink path) pointing mid-header; a new protocol variant in live traffic hitting an untested parser branch.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of stamparm/maltrail@77cfb06d76 (2026-09-13). Data as JSON: /api/errors/4c373cb1dcd201b4. Report an issue: GitHub.

Appendix: source

Thrown at sensor/src/worker.rs:183

                    // distorts the thing it reports. Sampling costs <1 ns/packet and the estimate
                    // is within a couple of percent on any realistic packet mix.
                    timing_countdown -= 1;
                    let started = if timing_countdown == 0 {
                        timing_countdown = TIMING_SAMPLE_STRIDE;
                        Some(Instant::now())
                    } else {
                        None
                    };
                    let ip_offset = st.dlt.resolve(datalink, data);
                    if let Some(offset) = ip_offset {
                        // Mirrors sensor.py's blanket `except Exception` around
                        // _process_packet: a parser bug must never take the sensor down.
                        let result = catch_unwind(AssertUnwindSafe(|| {
                            process::process_packet(&mut st, data, sec, usec, offset);
                        }));
                        if result.is_err() {
                            st.metrics.panics_recovered += 1;
                            crate::output::log_error(
                                &format!("unhandled panic in process_packet (worker {})", ctx.id),
                                true,
                            );
                        }
                    } else {
                        st.metrics.packets_ignored += 1;
                    }
                    if let Some(started) = started {
                        st.metrics.processing_nanos += started.elapsed().as_nanos() as u64;
                        st.metrics.processing_samples += 1;
                    }

                    if drained >= HOUSEKEEPING_INTERVAL as u32 {
                        break;
                    }
                }
                // offline: end of file. live: nothing available right now.
                Ok(None) => {

View on GitHub (pinned to 77cfb06d76)