What causes "missing tokens for node: {:?}" in rust-lang/rust?

If you hit this in normal code on nightly, file an ICE report at the rust-lang/rust issue tracker with the reproduction (include the --version hash).

rust-lang/rust · critical

missing tokens for node: {:?}

Error message

missing tokens for node: {:?}

What it means

Panic raised by TokenStream::from_ast when an AST node returns None from its tokens() accessor. The compiler expects every node that implements HasTokens to either carry captured tokens or to have been assigned them during parsing/macro expansion; reaching this panic means an internal invariant was violated while reconstructing a token stream from an AST node for proc-macro/span recovery purposes. It is an ICE-only path: well-formed user input never triggers it directly.

Source

Thrown at compiler/rustc_ast/src/tokenstream.rs:657

    pub fn get(&self, index: usize) -> Option<&TokenTree> {
        self.0.get(index)
    }

    pub fn iter(&self) -> TokenStreamIter<'_> {
        TokenStreamIter::new(self)
    }

    /// Create a token stream containing a single token with alone spacing. The
    /// spacing used for the final token in a constructed stream doesn't matter
    /// because it's never used. In practice we arbitrarily use
    /// `Spacing::Alone`.
    pub fn token_alone(kind: TokenKind, span: Span) -> TokenStream {
        TokenStream::new(vec![TokenTree::token_alone(kind, span)])
    }

    pub fn from_ast(node: &(impl HasTokens + fmt::Debug)) -> TokenStream {
        let tokens = node.tokens().unwrap_or_else(|| panic!("missing tokens for node: {:?}", node));
        let mut tts = vec![];
        attrs_and_tokens_to_token_trees(node.attrs(), tokens, &mut tts);
        TokenStream::new(tts)
    }

    // If `vec` is not empty, try to glue `tt` onto its last token. The return
    // value indicates if gluing took place.
    fn try_glue_to_last(vec: &mut [TokenTree], tt: &TokenTree) -> bool {
        if let Some(TokenTree::Token(last_tok, Spacing::Joint | Spacing::JointHidden)) = vec.last()
            && let TokenTree::Token(tok, spacing) = tt
            && let Some(glued_tok) = last_tok.glue(tok)
        {
            // ...then overwrite the last token tree in `vec` with the glued token.
            *vec.last_mut().unwrap() = TokenTree::Token(glued_tok, *spacing);
            true
        } else {
            false
        }

View on GitHub (pinned to 22057b88b0)

Solutions

If you hit this in normal code on nightly, file an ICE report at the rust-lang/rust issue tracker with the reproduction (include the --version hash).
Bisect between nightly rustc versions to find the regression commit, then pin to the last working nightly.
If modifying rustc itself, ensure the node type's parser/expansion path populates tokens (set via node.tokens = Some(...)) before from_ast is invoked.
Reduce the failing crate to a minimal reproduction with `cargo build` then `rustc +nightly` to isolate the offending AST node printed by the {:?} debug output.

Defensive patterns

Strategy: validation

Validate before calling

// TokenStream::from_ast panics when node.tokens() is None.
// Validate the node carries tokens before reconstructing a stream.
fn safe_from_ast(node: &(impl HasTokens + fmt::Debug)) -> Option<TokenStream> {
    node.tokens().map(|tokens| {
        let mut tts = vec![];
        attrs_and_tokens_to_token_trees(node.attrs(), tokens, &mut tts);
        TokenStream::new(tts)
    })
}
// Usage: match safe_from_ast(&node) { Some(ts) => ..., None => return Err(format!("node {:?} has no cached tokens", node)) }

Type guard

// Narrow to nodes guaranteed to have tokens after expansion.
fn has_tokens(node: &(impl HasTokens + fmt::Debug)) -> bool {
    node.tokens().is_some()
}

Try / catch

// Only for tooling that must survive an ICE here (proc-macro servers, rust-analyzer-like tools):
let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
    TokenStream::from_ast(&node)
}));
match result {
    Ok(ts) => ts,
    Err(payload) => return Err(format!("from_ast panicked: missing tokens for node {:?}", node)),
}

Prevention

Never call TokenStream::from_ast on a node that was not produced by (or recovered after) macro expansion; only nodes whose tokens() is Some are reconstructable.
If you build AST nodes manually (e.g. in a proc-macro or codegen tool), call node.store_tokens(...) / ensure attrs_and_tokens are populated so tokens() returns Some.
When walking a partially-expanded AST, gate every from_ast call behind a tokens().is_some() check rather than assuming expansion populated them.
Treat a None from tokens() as a hard error in your tool, not as a recoverable default — silently skipping it produces a malformed stream downstream.

When it happens

Trigger: Internally triggered when from_ast is called on a node (item, expr, stmt, etc.) whose tokens field is None — typically during macro token replay, derive expansion, or rustfmt-style regeneration. Reproducible by feeding proc-macro/derive code that constructs AST nodes without setting tokens, or by a compiler refactor that stops populating tokens for some node kind.

Common situations: Nightly rustc regressions after a refactor to AST node structures; mismatched versions between rustc and a custom build of rustc_ast; proc-macro code that manipulates AST nodes via unstable internal APIs. Almost never seen by end users writing ordinary Rust.

Related errors

AI-assisted analysis of rust-lang/rust@22057b88b0 (2026-08-03). Data as JSON: /data/errors/984fefdc1498b1ae.json. Report an issue: GitHub.