quickwit-oss/quickwit · error · anyhow::Error

Quickwit currently supports multiple pipelines only for GCP

Error message

Quickwit currently supports multiple pipelines only for GCP PubSub or Kafka sources. open an issue https://github.com/quickwit-oss/quickwit/issues if you need the feature for other source types

What it means

`num_pipelines` > 1 is only supported for source types that can partition work across pipelines (GCP PubSub, Kafka, and file-with-notifications). For any other source type, `validate_and_build` bails when `num_pipelines` exceeds 1, because there is no safe way to shard that source across multiple indexing pipelines.

Source

Thrown at quickwit/quickwit-config/src/source_config/serialize.rs:132

            | SourceParams::Kafka(_)
            | SourceParams::Kinesis(_)
            | SourceParams::Pulsar(_) => {
                // TODO consider any validation opportunity
            }
            SourceParams::PubSub(_)
            | SourceParams::Ingest
            | SourceParams::IngestApi
            | SourceParams::IngestCli
            | SourceParams::Vec(_)
            | SourceParams::Void(_) => {}
        }
        match &self.source_params {
            SourceParams::PubSub(_)
            | SourceParams::Kafka(_)
            | SourceParams::File(FileSourceParams::Notifications(_)) => {}
            _ => {
                if self.num_pipelines > 1 {
                    bail!(
                        "Quickwit currently supports multiple pipelines only for GCP PubSub or Kafka sources. open an issue https://github.com/quickwit-oss/quickwit/issues if you need the feature for other source types"
                    );
                }
            }
        }

        if let Some(transform_config) = &self.transform {
            if matches!(
                self.input_format,
                SourceInputFormat::OtlpLogsJson
                    | SourceInputFormat::OtlpLogsProtobuf
                    | SourceInputFormat::OtlpTracesJson
                    | SourceInputFormat::OtlpTracesProtobuf
            ) {
                bail!("VRL transforms are not supported for OTLP input formats");
            }
            transform_config.validate_vrl_script()?;
        }

View on GitHub (pinned to a39730c5cd)

Solutions

Set `num_pipelines` to 1 for the unsupported source type.
If higher throughput is needed, switch to Kafka or GCP PubSub (or a file source with notifications), which support multiple pipelines.
Open an issue at https://github.com/quickwit-oss/quickwit/issues to request multi-pipeline support for your source type.

Example fix

# before
source_id: kinesis-src
source_type: kinesis
num_pipelines: 4

# after
source_id: kinesis-src
source_type: kinesis
num_pipelines: 1

Defensive patterns

Strategy: validation

Validate before calling

fn supports_multiple_pipelines(params: &SourceParams) -> bool {
    matches!(
        params,
        SourceParams::PubSub(_)
            | SourceParams::Kafka(_)
            | SourceParams::File(FileSourceParams::Notifications(_))
    )
}

if source_config.num_pipelines > 1
    && !supports_multiple_pipelines(&source_config.source_params)
{
    return Err(anyhow!(
        "num_pipelines > 1 only supported for PubSub/Kafka/notification file sources"
    ));
}

Type guard

fn multipipeline_capable(params: &SourceParams) -> bool {
    matches!(
        params,
        SourceParams::PubSub(_)
            | SourceParams::Kafka(_)
            | SourceParams::File(FileSourceParams::Notifications(_))
    )
}

Try / catch

match load_source_config_from_user_config(&user_config) {
    Ok(sc) => apply(sc),
    Err(e) if e.to_string().contains("multiple pipelines") => {
        eprintln!("set num_pipelines=1 or switch to Kafka/PubSub: {}", e);
    }
    Err(e) => return Err(e),
}

Prevention

Only set num_pipelines > 1 on Kafka or GCP PubSub sources (or file sources with notifications).
Default num_pipelines to 1 and increase only after confirming source-type support.
Clamp desired_num_pipelines to 1 in tooling for unsupported source types.

When it happens

Trigger: Setting `num_pipelines` > 1 on a source config whose `source_params` is not PubSub, Kafka, or `File(FileSourceParams::Notifications)` — e.g. a plain file, kinesis, or pulsar source — when loaded via `load_source_config_from_user_config`/`load_source_config_update`.

Common situations: Scaling ingestion throughput by raising `desired_num_pipelines` on a Kinesis or file source; copy-pasting Kafka source tuning (`num_pipelines`) onto other source types.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/c1e845f36fc766fb. Report an issue: GitHub.