xai-org/x-algorithm · critical

Failed to create consumer for thread {}: {:#}

Error message

Failed to create consumer for thread {}: {:#}

What it means

This panic occurs when one of the tweet-event processing worker threads fails to create its rd_kafka consumer during startup of the v2 tweet events listener. Consumer creation in rdkafka can fail for reasons such as an invalid broker address, bad consumer group config, authentication failure, or DNS resolution problems. Because the feeder cannot function without tweet event processing, the code treats this as a fatal, non-recoverable condition and panics inside the spawned thread.

Source

Thrown at thunder/kafka/tweet_events_listener_v2.rs:103

                    );

                    if let Err(e) = process_tweet_events_v2(
                        consumer,
                        post_store_clone,
                        batch_size,
                        tx_clone,
                        semaphore_clone,
                    )
                    .await
                    {
                        panic!(
                            "Tweet events processing thread {} exited unexpectedly: {:#}. This is a critical failure - the feeder cannot function without tweet event processing.",
                            thread_id, e
                        );
                    }
                }
                Err(e) => {
                    panic!(
                        "Failed to create consumer for thread {}: {:#}",
                        thread_id, e
                    );
                }
            }
        });
    }
}

fn deserialize_batch(
    messages: Vec<KafkaMessage>,
) -> Result<(Vec<LightPost>, Vec<TweetDeleteEvent>)> {
    let start_time = Instant::now();
    let num_messages = messages.len();
    let results = deserialize_kafka_messages(messages, deserialize_tweet_event_v2)?;
    let deser_elapsed = start_time.elapsed();
    if DESER_LOG_COUNTER
        .fetch_add(1, Ordering::Relaxed)

View on GitHub (pinned to 24c60942c5)

Solutions

1. Verify the Kafka bootstrap servers address and connectivity (kcat -L -b $BROKER or telnet) from the host running the feeder.
2. Check the consumer config passed to the thread (group.id, SASL/SSL settings) against the cluster's requirements.
3. Ensure Kafka is reachable at startup; add retry/backoff around consumer creation or delay service start until the broker is available.
4. Inspect the {:#} formatted rdkafka error in the panic output — it names the exact cause (e.g. 'InvalidData', 'Broker not available').
5. If brokers flake at boot, consider catching the error in spawn and retrying thread creation instead of panicking.

Example fix

// before
Err(e) => {
    panic!("Failed to create consumer for thread {}: {:#}", thread_id, e);
}

// after (retry with backoff instead of aborting)
Err(e) => {
    error!("Consumer creation failed for thread {}: {:#}; retrying", thread_id, e);
    continue; // or implement bounded retry before panicking
}

// Before starting processing, verify broker reachability
use std::net::TcpStream;
fn broker_reachable(addr: &str) -> bool {
    TcpStream::connect(addr).is_ok()
}
assert!(broker_reachable(kafka_broker_addr), "Kafka broker unreachable");
start_tweet_event_processing_v2(...);

// Rust: catch_unwind around thread spawn to log and retry instead of crashing the process
let result = std::panic::catch_unwind(|| spawn_processing_threads_v2(cfg));
if result.is_err() { /* re-init with backoff, alert on-call */ }

When it happens

Trigger: Calling start_tweet_event_processing_v2 with a Kafka bootstrap broker that is unreachable/misconfigured, an invalid group.id or auth credentials, or a broker version incompatibility. Each spawned thread calls its consumer constructor; the Err branch of that constructor panics with 'Failed to create consumer for thread {thread_id}'.

Common situations: Misconfigured KAFKA_BOOTSTRAP_SERVERS env var, typos in broker hostnames, missing SASL/SSL config, Kafka cluster not yet up when the service starts, or a network/firewall block between the service and the broker.

Related errors

• {ENV_CACHE_WARM_SAMPLE_PCT}: {error}
• {misconfiguration}
• Kafka mTLS requires CA at {ca_file!r} (mount internal-ca Con
• `clusters` must contain at least one region
• Region {region!r} must list at least one broker

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/de09135d5f4fe0fd. Report an issue: GitHub.