{"record":{"id":"231d431145e41826","repo":"zed-industries/zed","slug":"sweep-prompt-field-name-contains-reserved-tokens","errorCode":null,"errorMessage":"sweep prompt {field_name} contains reserved tokens","messagePattern":"sweep prompt (.+?) contains reserved tokens","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/edit_prediction/src/sweep_prompt.rs","lineNumber":277,"sourceCode":"    }\n\n    for related_file in &input.related_files {\n        validate_prompt_field(\n            \"related file path\",\n            &related_file.file_path.display().to_string(),\n        )?;\n        validate_prompt_field(\"related file content\", &related_file.content)?;\n    }\n\n    Ok(())\n}\n\nfn validate_prompt_field(field_name: &str, value: &str) -> Result<()> {\n    if RESERVED_SWEEP_TOKENS\n        .iter()\n        .any(|reserved_token| value.contains(reserved_token))\n    {\n        anyhow::bail!(\"sweep prompt {field_name} contains reserved tokens\");\n    }\n\n    Ok(())\n}\n\npub(crate) fn original_window_for_current_window(\n    current_window: Range<Point>,\n    latest_event: Option<&StoredEvent>,\n    current_snapshot: &BufferSnapshot,\n) -> Option<String> {\n    let latest_event = latest_event?;\n    if latest_event.old_snapshot.remote_id() != current_snapshot.remote_id() {\n        return None;\n    }\n\n    let old_range = current_snapshot.range_to_version(\n        current_window.to_offset(current_snapshot),\n        latest_event.old_snapshot.version(),","sourceCodeStart":259,"sourceCodeEnd":295,"githubUrl":"https://github.com/zed-industries/zed/blob/f4178619acd0d47ea1f76a2025c42962c6d6638c/crates/edit_prediction/src/sweep_prompt.rs#L259-L295","documentation":"Thrown by validate_prompt_field when building a sweep prompt in Zed's edit_prediction crate. The prompt reserves two sentinel tokens, `<|file_sep|>` and `</s>` (RESERVED_SWEEP_TOKENS, sweep_prompt.rs:22), as structural separators between file sections. If any user-supplied prompt field (e.g. related file content, the current buffer text) literally contains one of these tokens, the model would mis-parse the prompt boundaries, so construction fails fast instead of shipping a corrupted prompt.","triggerScenarios":"Calling the sweep prompt builder (the function that ends with validate_prompt_field on 'related file content') where the edited buffer or a related file contains the literal string `<|file_sep|>` or `</s>`. This typically happens when the user is editing source that itself defines or documents these tokens (e.g. prompt templates, tokenizer configs, test fixtures for the model).","commonSituations":"Editing the edit_prediction crate's own test fixtures or prompt templates inside Zed; opening a tokenizer/vocab file that contains `</s>` (a standard EOS token in SentencePiece models); pasting sample prompts into a buffer while dogfooding the sweep feature.","solutions":["Move or remove the literal `<|file_sep|>` / `</s>` text from the file being edited or from the related file that got pulled into the prompt","Close/skip the related file whose content contains the token so it is not attached to the prompt","If you control the data, escape or mangle the tokens (e.g. insert a zero-width space) before the buffer is used for sweep prediction"],"exampleFix":"// before: buffer literally contains\n// let sep = \"<|file_sep|>\";\n// which ends up in related_file.content -> bail!\n\n// after: avoid the literal token in content fed to the prompt\n// let sep = \"<|file\" + \"_sep|>\";  // constructed, not literal","handlingStrategy":"validation","validationCode":"const RESERVED_SWEEP_TOKENS: [&str; 2] = [\"<|file_sep|>\", \"</s>\"];\n\nfn prompt_field_is_clean(field_name: &str, value: &str) -> Result<(), anyhow::Error> {\n    if let Some(token) = RESERVED_SWEEP_TOKENS.iter().find(|t| value.contains(*t)) {\n        anyhow::bail!(\"field {field_name} contains reserved token {token}; strip it before building the sweep prompt\");\n    }\n    Ok(())\n}","typeGuard":null,"tryCatchPattern":"match build_sweep_prompt(&inputs) {\n    Err(e) if e.to_string().contains(\"contains reserved tokens\") => {\n        // log and skip this example; content would corrupt the prompt structure\n        log::warn!(\"skipping example: {e:#}\");\n        continue;\n    }\n    rest => rest?,\n}","preventionTips":["Pre-scan buffers and related-file contents for '<|file_sep|>' and '</s>' before building sweep prompts","Never paste raw prompt templates or tokenizer vocab entries into buffers used for edit-prediction testing","Treat this bail as a data-hygiene signal: the offending content, not the code, must change"],"tags":["rust","zed","edit-prediction","prompt-validation","reserved-tokens"],"backgroundTag":"input-validation-failed","analyzedSha":"f4178619acd0d47ea1f76a2025c42962c6d6638c","analyzedAt":"2026-08-20T19:29:52.058Z","contentChangedAt":"2026-08-20T19:29:52.058Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}