quickwit-oss/quickwit · error

shard_throughput_limit

Error message

shard_throughput_limit ({}) must be within 1mb and 20mb

What it means

The ingestion shard throughput limit must be between 1 MiB and 20 MiB per second; values outside this range are rejected during ingest config validation. This bounds per-shard consumption so a single shard can neither starve (too low) nor overwhelm the node (too high).

Solutions

1. Set `shard_throughput_limit` within [1mib, 20mib], e.g. `10mib`
2. Remove the setting to use the default if custom tuning isn't required
3. To raise total ingestion throughput, add shards rather than exceeding the per-shard cap

Example fix

// before
shard_throughput_limit: 50mb
// after
shard_throughput_limit: 20mb

fn ensure_shard_throughput(v: ByteSize) -> anyhow::Result<()> {
    if !(ByteSize::mib(1)..=ByteSize::mib(20)).contains(&v) {
        anyhow::bail!("shard_throughput_limit ({}) must be within 1mb and 20mb", v.display().si());
    }
    Ok(())
}

if let Err(e) = ingest_config.validate() {
    if e.to_string().contains("shard_throughput_limit") {
        eprintln!("Set shard_throughput_limit between 1 MiB and 20 MiB.");
    }
}

When it happens

Trigger: Setting `shard_throughput_limit` in the node config below 1 MiB or above 20 MiB and starting/validating the node.

Common situations: Aggressive tuning for high-throughput ingestion (setting 50mb); setting a near-zero limit by mistake; copying a value in bytes rather than a byte-size string and ending up far outside the range.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• max_queue_disk_usage must be at least 256 MiB, got
• max_queue_disk_usage
• concatenate field uses an unknown field
• contains an empty label name
• contains an empty label value

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/78070fa235dfa460. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-config/src/node_config/mod.rs:822

    fn validate(&self) -> anyhow::Result<()> {
        self.warn_if_replication_factor_is_set();
        ensure!(
            self.max_queue_disk_usage > ByteSize::mib(256),
            "max_queue_disk_usage must be at least 256 MiB, got `{}`",
            self.max_queue_disk_usage.display().si()
        );
        ensure!(
            self.max_queue_disk_usage >= self.max_queue_memory_usage,
            "max_queue_disk_usage ({}) must be at least max_queue_memory_usage ({})",
            self.max_queue_disk_usage.display().si(),
            self.max_queue_memory_usage.display().si()
        );
        info!(
            "ingestion shard throughput limit: {}",
            self.shard_throughput_limit.display().si()
        );
        ensure!(
            self.shard_throughput_limit >= ByteSize::mib(1)
                && self.shard_throughput_limit <= ByteSize::mib(20),
            "shard_throughput_limit ({}) must be within 1mb and 20mb",
            self.shard_throughput_limit.display().si()
        );
        // The newline delimited format is persisted as something a bit larger
        // (lines prefixed with their length)
        let estimated_persist_size = ByteSize::b(3 * self.content_length_limit.as_u64() / 2);
        ensure!(
            self.shard_burst_limit >= estimated_persist_size,
            "shard_burst_limit ({}) must be at least 1.5*content_length_limit ({})",
            self.shard_burst_limit,
            estimated_persist_size,
        );
        ensure!(
            self.shard_scale_up_factor > 1.0,
            "shard_scale_up_factor ({}) must be greater than 1",
            self.shard_scale_up_factor,

View on GitHub (pinned to a39730c5cd)