{"record":{"id":"c1e845f36fc766fb","repo":"quickwit-oss/quickwit","slug":"quickwit-currently-supports-multiple-pipelines-onl","errorCode":null,"errorMessage":"Quickwit currently supports multiple pipelines only for GCP PubSub or Kafka sources. open an issue https://github.com/quickwit-oss/quickwit/issues if you need the feature for other source types","messagePattern":"Quickwit currently supports multiple pipelines only for GCP PubSub or Kafka sources\\. open an issue https://github\\.com/quickwit-oss/quickwit/issues if you need the feature for other source types","errorType":"validation","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-config/src/source_config/serialize.rs","lineNumber":132,"sourceCode":"            | SourceParams::Kafka(_)\n            | SourceParams::Kinesis(_)\n            | SourceParams::Pulsar(_) => {\n                // TODO consider any validation opportunity\n            }\n            SourceParams::PubSub(_)\n            | SourceParams::Ingest\n            | SourceParams::IngestApi\n            | SourceParams::IngestCli\n            | SourceParams::Vec(_)\n            | SourceParams::Void(_) => {}\n        }\n        match &self.source_params {\n            SourceParams::PubSub(_)\n            | SourceParams::Kafka(_)\n            | SourceParams::File(FileSourceParams::Notifications(_)) => {}\n            _ => {\n                if self.num_pipelines > 1 {\n                    bail!(\n                        \"Quickwit currently supports multiple pipelines only for GCP PubSub or Kafka sources. open an issue https://github.com/quickwit-oss/quickwit/issues if you need the feature for other source types\"\n                    );\n                }\n            }\n        }\n\n        if let Some(transform_config) = &self.transform {\n            if matches!(\n                self.input_format,\n                SourceInputFormat::OtlpLogsJson\n                    | SourceInputFormat::OtlpLogsProtobuf\n                    | SourceInputFormat::OtlpTracesJson\n                    | SourceInputFormat::OtlpTracesProtobuf\n            ) {\n                bail!(\"VRL transforms are not supported for OTLP input formats\");\n            }\n            transform_config.validate_vrl_script()?;\n        }","sourceCodeStart":114,"sourceCodeEnd":150,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-config/src/source_config/serialize.rs#L114-L150","documentation":"`num_pipelines` > 1 is only supported for source types that can partition work across pipelines (GCP PubSub, Kafka, and file-with-notifications). For any other source type, `validate_and_build` bails when `num_pipelines` exceeds 1, because there is no safe way to shard that source across multiple indexing pipelines.","triggerScenarios":"Setting `num_pipelines` > 1 on a source config whose `source_params` is not PubSub, Kafka, or `File(FileSourceParams::Notifications)` — e.g. a plain file, kinesis, or pulsar source — when loaded via `load_source_config_from_user_config`/`load_source_config_update`.","commonSituations":"Scaling ingestion throughput by raising `desired_num_pipelines` on a Kinesis or file source; copy-pasting Kafka source tuning (`num_pipelines`) onto other source types.","solutions":["Set `num_pipelines` to 1 for the unsupported source type.","If higher throughput is needed, switch to Kafka or GCP PubSub (or a file source with notifications), which support multiple pipelines.","Open an issue at https://github.com/quickwit-oss/quickwit/issues to request multi-pipeline support for your source type."],"exampleFix":"# before\nsource_id: kinesis-src\nsource_type: kinesis\nnum_pipelines: 4\n\n# after\nsource_id: kinesis-src\nsource_type: kinesis\nnum_pipelines: 1","handlingStrategy":"validation","validationCode":"fn supports_multiple_pipelines(params: &SourceParams) -> bool {\n    matches!(\n        params,\n        SourceParams::PubSub(_)\n            | SourceParams::Kafka(_)\n            | SourceParams::File(FileSourceParams::Notifications(_))\n    )\n}\n\nif source_config.num_pipelines > 1\n    && !supports_multiple_pipelines(&source_config.source_params)\n{\n    return Err(anyhow!(\n        \"num_pipelines > 1 only supported for PubSub/Kafka/notification file sources\"\n    ));\n}","typeGuard":"fn multipipeline_capable(params: &SourceParams) -> bool {\n    matches!(\n        params,\n        SourceParams::PubSub(_)\n            | SourceParams::Kafka(_)\n            | SourceParams::File(FileSourceParams::Notifications(_))\n    )\n}","tryCatchPattern":"match load_source_config_from_user_config(&user_config) {\n    Ok(sc) => apply(sc),\n    Err(e) if e.to_string().contains(\"multiple pipelines\") => {\n        eprintln!(\"set num_pipelines=1 or switch to Kafka/PubSub: {}\", e);\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Only set num_pipelines > 1 on Kafka or GCP PubSub sources (or file sources with notifications).","Default num_pipelines to 1 and increase only after confirming source-type support.","Clamp desired_num_pipelines to 1 in tooling for unsupported source types."],"tags":["rust","config","quickwit"],"backgroundTag":"unsupported-config-value","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}