neondatabase/neon · error

unexpected in-memory layer

Error message

unexpected in-memory layer

What it means

The index_part ctl command reconstructs a layer map from the tenant's index_part.json. That index only ever references persistent (delta/image) layers on remote storage, so a search result that is an in-memory layer violates the format's invariant. In practice this means the ctl binary's layer-map construction is out of sync with the index content (version skew) or the index_part.json is corrupt or written by an incompatible format version.

Solutions

1. Rebuild or reinstall the ctl binary from the same commit as the pageserver that owns the tenant.
2. Inspect the index_part.json layer list directly (jq or the ctl's own printing) for unexpected entries.
3. If the index is genuinely corrupt, re-attach the tenant in the pageserver so it rewrites index_part.json, or reconstruct from the layer files on remote storage.

match layer_map.search(key, end_lsn) {
    Some(SearchResult { layer: ReadableLayerWeak::PersistentLayer(layer), lsn_floor }) => {
        // proceed with the on-disk layer
    }
    Some(SearchResult { layer: ReadableLayerWeak::InMemoryLayer(_), .. }) => {
        // index_part never contains in-memory layers: treat as version/corruption issue
        anyhow::bail!("index_part contains an unexpected in-memory layer; rebuild ctl from the pageserver's commit or re-attach the tenant");
    }
    None => { /* handle no covering layer */ }
}

When it happens

Trigger: layer_map.search(key, end_lsn) returning ReadableLayerWeak::InMemoryLayer while walking a layer map derived purely from index_part.json.

Common situations: Running a ctl built from a different branch or commit than the pageserver that wrote the index; inspecting an index produced by a newer or older on-disk format; a hand-edited or partially written index_part.json.

Related errors

• failed to parse layer name
• cannot find 'blkno='
• cannot find 'rel='
• failed to parse as hex or span attributes
• found no blocknum in arguments

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/b4a30583fd7b2a99. Report an issue: GitHub.

Appendix: source

Thrown at pageserver/ctl/src/index_part.rs:91

    let tenant_shard_id = TenantShardId::unsharded(tenant_id);
    let timeline_id = TimelineId::from_str(timeline_id).unwrap();
    let index_json = {
        let bytes = tokio::fs::read(path).await?;
        IndexPart::from_json_bytes(&bytes).unwrap()
    };
    let layer_map = create_layer_map_from_index_part(&index_json, tenant_shard_id, timeline_id);
    let key = Key::from_hex(key)?;

    let lsn = Lsn::from_str(lsn).unwrap();
    let mut end_lsn = lsn;
    loop {
        let result = layer_map.search(key, end_lsn);
        match result {
            Some(SearchResult { layer, lsn_floor }) => {
                let disk_layer = match layer {
                    ReadableLayerWeak::PersistentLayer(layer) => layer,
                    ReadableLayerWeak::InMemoryLayer(_) => {
                        anyhow::bail!("unexpected in-memory layer")
                    }
                };

                let metadata = index_json
                    .layer_metadata
                    .get(&disk_layer.layer_name())
                    .unwrap();
                println!(
                    "{}",
                    remote_layer_path(
                        &tenant_id,
                        &timeline_id,
                        metadata.shard,
                        &disk_layer.layer_name(),
                        metadata.generation
                    )
                );
                end_lsn = lsn_floor;

View on GitHub (pinned to 8f60b04da4)