zed-industries/zed · error

sweep prompt {field_name} contains reserved tokens

Error message

sweep prompt {field_name} contains reserved tokens

What it means

Thrown by validate_prompt_field when building a sweep prompt in Zed's edit_prediction crate. The prompt reserves two sentinel tokens, `<|file_sep|>` and `</s>` (RESERVED_SWEEP_TOKENS, sweep_prompt.rs:22), as structural separators between file sections. If any user-supplied prompt field (e.g. related file content, the current buffer text) literally contains one of these tokens, the model would mis-parse the prompt boundaries, so construction fails fast instead of shipping a corrupted prompt.

Source

Thrown at crates/edit_prediction/src/sweep_prompt.rs:277

    }

    for related_file in &input.related_files {
        validate_prompt_field(
            "related file path",
            &related_file.file_path.display().to_string(),
        )?;
        validate_prompt_field("related file content", &related_file.content)?;
    }

    Ok(())
}

fn validate_prompt_field(field_name: &str, value: &str) -> Result<()> {
    if RESERVED_SWEEP_TOKENS
        .iter()
        .any(|reserved_token| value.contains(reserved_token))
    {
        anyhow::bail!("sweep prompt {field_name} contains reserved tokens");
    }

    Ok(())
}

pub(crate) fn original_window_for_current_window(
    current_window: Range<Point>,
    latest_event: Option<&StoredEvent>,
    current_snapshot: &BufferSnapshot,
) -> Option<String> {
    let latest_event = latest_event?;
    if latest_event.old_snapshot.remote_id() != current_snapshot.remote_id() {
        return None;
    }

    let old_range = current_snapshot.range_to_version(
        current_window.to_offset(current_snapshot),
        latest_event.old_snapshot.version(),

View on GitHub (pinned to f4178619ac)

Solutions

1. Move or remove the literal `<|file_sep|>` / `</s>` text from the file being edited or from the related file that got pulled into the prompt
2. Close/skip the related file whose content contains the token so it is not attached to the prompt
3. If you control the data, escape or mangle the tokens (e.g. insert a zero-width space) before the buffer is used for sweep prediction

Example fix

// before: buffer literally contains
// let sep = "<|file_sep|>";
// which ends up in related_file.content -> bail!

// after: avoid the literal token in content fed to the prompt
// let sep = "<|file" + "_sep|>";  // constructed, not literal

const RESERVED_SWEEP_TOKENS: [&str; 2] = ["<|file_sep|>", "</s>"];

fn prompt_field_is_clean(field_name: &str, value: &str) -> Result<(), anyhow::Error> {
    if let Some(token) = RESERVED_SWEEP_TOKENS.iter().find(|t| value.contains(*t)) {
        anyhow::bail!("field {field_name} contains reserved token {token}; strip it before building the sweep prompt");
    }
    Ok(())
}

match build_sweep_prompt(&inputs) {
    Err(e) if e.to_string().contains("contains reserved tokens") => {
        // log and skip this example; content would corrupt the prompt structure
        log::warn!("skipping example: {e:#}");
        continue;
    }
    rest => rest?,
}

When it happens

Trigger: Calling the sweep prompt builder (the function that ends with validate_prompt_field on 'related file content') where the edited buffer or a related file contains the literal string `<|file_sep|>` or `</s>`. This typically happens when the user is editing source that itself defines or documents these tokens (e.g. prompt templates, tokenizer configs, test fixtures for the model).

Common situations: Editing the edit_prediction crate's own test fixtures or prompt templates inside Zed; opening a tokenizer/vocab file that contains `</s>` (a standard EOS token in SentencePiece models); pasting sample prompts into a buffer while dogfooding the sweep feature.

Related errors

• Cannot list directory because its path matches the user's wo
• Cannot list directory because its path matches the user's wo
• Path not found: {}
• {} is not a directory.
• Cannot read file because its path matches the global `file_s

AI-assisted analysis of zed-industries/zed@f4178619ac (2026-08-20). Data as JSON: /api/errors/231d431145e41826. Report an issue: GitHub.