BurntSushi/ripgrep · error

configured allocation limit ({}) exceeded

Error message

configured allocation limit ({}) exceeded

What it means

This is an io::Error produced by grep-searcher's LineBuffer (the streaming line reader that backs the Searcher). It is returned from LineBufferReader::fill / LineBuffer::fill via ensure_capacity (crates/searcher/src/line_buffer.rs:499) only when the buffer was built with BufferAllocation::Error(limit) instead of the default BufferAllocation::Eager. The value printed in the message is capacity+limit (line 512), i.e. the total heap budget the searcher is permitted to use; once a single logical line needs more room than that budget allows, the buffer refuses to grow and surfaces this error rather than OOM-ing the process. The public API that turns it on is SearcherBuilder::heap_limit (crates/searcher/src/searcher/mod.rs:457), which maps the user-supplied byte count into capacity + BufferAllocation::Error(additional) at searcher/mod.rs:224-233.

Source

Thrown at crates/searcher/src/line_buffer.rs:39

    /// This is the default.
    Eager,
    /// Limit the amount of additional memory allocated to the given size. If
    /// a line is found that requires more memory than is allowed here, then
    /// stop reading and return an error.
    Error(usize),
}

impl Default for BufferAllocation {
    fn default() -> BufferAllocation {
        BufferAllocation::Eager
    }
}

/// Create a new error to be used when a configured allocation limit has been
/// reached.
pub(crate) fn alloc_error(limit: usize) -> io::Error {
    let msg = format!("configured allocation limit ({}) exceeded", limit);
    io::Error::new(io::ErrorKind::Other, msg)
}

/// The behavior of binary detection in the line buffer.
///
/// Binary detection is the process of _heuristically_ identifying whether a
/// given chunk of data is binary or not, and then taking an action based on
/// the result of that heuristic. The motivation behind detecting binary data
/// is that binary data often indicates data that is undesirable to search
/// using textual patterns. Of course, there are many cases in which this isn't
/// true, which is why binary detection is disabled by default.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(crate) enum BinaryDetection {
    /// No binary detection is performed. Data reported by the line buffer may
    /// contain arbitrary bytes.
    None,
    /// The given byte is searched in all contents read by the line buffer. If
    /// it occurs, then the data is considered binary and the line buffer acts
    /// as if it reached EOF. The line buffer guarantees that this byte will

View on GitHub (pinned to 3fce3b5bb0)

Solutions

1. If you can afford the memory, raise or remove heap_limit: pass None (or a larger N) to SearcherBuilder::heap_limit. The default Eager strategy will grow the buffer up to available memory and never produce this error.
2. If the offending file genuinely has pathologically long lines, pre-process or skip it (e.g. enable BinaryDetection::Quit, filter by file size, or use --no-line-buffer style streaming) so the buffer never has to hold the whole line.
3. Reduce the buffer pressure by lowering before_context/after_context (Config::before_context / after_context) so the required buffer window is smaller relative to your heap_limit.
4. Enable memory-map search (MmapChoice::auto or ::when appropriate) via SearcherBuilder::memory_map; mmap reads do not consume the line buffer's heap budget, sidestepping the limit entirely.
5. If the cap must stay tiny, set heap_limit to at least a few times the size of your longest expected line plus context; do not set it to 0 or to a value <= the longest line length.
6. Handle the error at the call site: match on io::ErrorKind::Other and the 'configured allocation limit' message, then skip the file or fall back to a streaming/partial search rather than aborting the whole search.

Example fix

// before
use grep_searcher::{Searcher, SearcherBuilder};
let mut searcher = SearcherBuilder::new()
    .heap_limit(Some(64 * 1024)) // 64 KiB cap
    .build();
let result = searcher.search_reader(&matcher, &mut file, &mut sink);
// -> io error: "configured allocation limit (65536) exceeded" on long-line files

// after — raise the cap to fit the worst-case line + context, or remove it
use grep_searcher::{Searcher, SearcherBuilder};
let mut searcher = SearcherBuilder::new()
    .heap_limit(Some(64 * 1024 * 1024)) // 64 MiB, well above any single line
    .build();
// or, if memory is not a concern, omit heap_limit entirely to use Eager:
// let mut searcher = SearcherBuilder::new().build();

// Before searching, size heap_limit against the file's longest line + context window.
use std::fs::File;
use std::io::{self, Read, BufReader};

/// Returns the length (in bytes) of the longest line in `path`, or an io::Error.
/// Use this to pick a heap_limit that is guaranteed not to trip the alloc_error.
fn longest_line_len(path: &std::path::Path) -> io::Result<usize> {
    let f = File::open(path)?;
    let mut r = BufReader::new(f);
    let mut buf = [0u8; 64 * 1024];
    let mut max = 0usize;
    let mut cur = 0usize;
    loop {
        let n = r.read(&mut buf)?;
        if n == 0 { break; }
        for &b in &buf[..n] {
            if b == b'\n' {
                if cur > max { max = cur; }
                cur = 0;
            } else {
                cur += 1;
            }
        }
    }
    if cur > max { max = cur; }
    Ok(max)
}

// Then choose heap_limit so that capacity (64 KiB) + additional > longest_line + context.
// Keep a generous multiplier; the line buffer must hold the line AND one more byte to detect EOF.
let longest = longest_line_len(&path)?;
let context = searcher_before_ctx + searcher_after_ctx;
let needed = longest.saturating_add(context).saturating_add(1);
let heap_limit = if needed < 64 * 1024 { 64 * 1024 } else { needed };
let mut searcher = SearcherBuilder::new().heap_limit(Some(heap_limit)).build();

// Rust: there is no dedicated error type; narrow by kind + message substring.
fn is_allocation_limit_error(e: &std::io::Error) -> bool {
    use std::io::ErrorKind;
    e.kind() == ErrorKind::Other
        && e.to_string().contains("configured allocation limit")
}

// Usage in a sink/search call:
match searcher.search_reader(&matcher, &mut file, &mut sink) {
    Ok(()) => {},
    Err(ref e) if is_allocation_limit_error(e) => {
        // line too long for the configured heap_limit; handle gracefully
    }
    Err(other) => return Err(other.into()),
}

// Recommended pattern when iterating many files with a heap cap:
for path in paths {
    let result = searcher.search_path(&matcher, path, &mut sink);
    match result {
        Ok(()) => {},
        Err(ref e) if is_allocation_limit_error(&e.into_io().unwrap_or_else(|_| io::Error::new(io::ErrorKind::Other, e.to_string()))) => {
            // Log and skip; do NOT abort the whole search over one oversized line.
            tracing::warn!(?path, "skipped: line exceeds heap_limit");
            continue;
        }
        Err(e) => return Err(e),
    }
}

When it happens

Trigger: Calling SearcherBuilder::new().heap_limit(Some(N)).build() and then searching a file whose longest line (including any before/after context window, since the buffer must hold the whole window) exceeds the configured N bytes. Concretely: ensure_capacity hits BufferAllocation::Error(limit), computes used = buf.len() - capacity, and when min(len*2, limit - used) == 0 it returns Err(alloc_error(capacity+limit)) (line_buffer.rs:508-513). It is also returned up front when heap_limit is set so low that no buffer can be allocated at all (searcher/mod.rs:996 and :1021, when additional == 0). Streaming search (Searcher::search_reader) with a fixed heap_limit over a file with a very long single line is the canonical trigger; multi-line mode with context enabled makes it more likely because the buffer must hold the context window plus the matched line.

Common situations: Operators running ripgrep-like tools with a memory cap (e.g. a --max-columns or memory-limit flag, or an embedded grep-searcher inside a constrained service) on minified JS, concatenated CSV/TSV without newlines, log files with one giant line, SQL dumps, or base64 blobs. Setting heap_limit to a small value while also enabling before_context/after_context (which expand the required buffer) is a frequent misconfiguration. Migrating from the default (no limit, Eager allocation) to a capped heap_limit without raising it for files known to have long lines is the typical regression. Enabling memory-map search disabled (MmapChoice::never) plus heap_limit on a huge-line file guarantees the error because there is no fallback strategy.

AI-assisted analysis of BurntSushi/ripgrep@3fce3b5bb0 (2026-08-06). Data as JSON: /data/errors/c40ced9a425237f4.json. Report an issue: GitHub.