neondatabase/neon · error

unexpected in-memory layer

Error message

unexpected in-memory layer

What it means

The index_part ctl command reconstructs a layer map from the tenant's index_part.json. That index only ever references persistent (delta/image) layers on remote storage, so a search result that is an in-memory layer violates the format's invariant. In practice this means the ctl binary's layer-map construction is out of sync with the index content (version skew) or the index_part.json is corrupt or written by an incompatible format version.

Source

Thrown at pageserver/ctl/src/index_part.rs:91

    let tenant_shard_id = TenantShardId::unsharded(tenant_id);
    let timeline_id = TimelineId::from_str(timeline_id).unwrap();
    let index_json = {
        let bytes = tokio::fs::read(path).await?;
        IndexPart::from_json_bytes(&bytes).unwrap()
    };
    let layer_map = create_layer_map_from_index_part(&index_json, tenant_shard_id, timeline_id);
    let key = Key::from_hex(key)?;

    let lsn = Lsn::from_str(lsn).unwrap();
    let mut end_lsn = lsn;
    loop {
        let result = layer_map.search(key, end_lsn);
        match result {
            Some(SearchResult { layer, lsn_floor }) => {
                let disk_layer = match layer {
                    ReadableLayerWeak::PersistentLayer(layer) => layer,
                    ReadableLayerWeak::InMemoryLayer(_) => {
                        anyhow::bail!("unexpected in-memory layer")
                    }
                };

                let metadata = index_json
                    .layer_metadata
                    .get(&disk_layer.layer_name())
                    .unwrap();
                println!(
                    "{}",
                    remote_layer_path(
                        &tenant_id,
                        &timeline_id,
                        metadata.shard,
                        &disk_layer.layer_name(),
                        metadata.generation
                    )
                );
                end_lsn = lsn_floor;

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Rebuild or reinstall the ctl binary from the same commit as the pageserver that owns the tenant.
2. Inspect the index_part.json layer list directly (jq or the ctl's own printing) for unexpected entries.
3. If the index is genuinely corrupt, re-attach the tenant in the pageserver so it rewrites index_part.json, or reconstruct from the layer files on remote storage.

match layer_map.search(key, end_lsn) {
    Some(SearchResult { layer: ReadableLayerWeak::PersistentLayer(layer), lsn_floor }) => {
        // proceed with the on-disk layer
    }
    Some(SearchResult { layer: ReadableLayerWeak::InMemoryLayer(_), .. }) => {
        // index_part never contains in-memory layers: treat as version/corruption issue
        anyhow::bail!("index_part contains an unexpected in-memory layer; rebuild ctl from the pageserver's commit or re-attach the tenant");
    }
    None => { /* handle no covering layer */ }
}

When it happens

Trigger: layer_map.search(key, end_lsn) returning ReadableLayerWeak::InMemoryLayer while walking a layer map derived purely from index_part.json.

Common situations: Running a ctl built from a different branch or commit than the pageserver that wrote the index; inspecting an index produced by a newer or older on-disk format; a hand-edited or partially written index_part.json.

Related errors

• failed to parse layer name: {e}
• 'REMOTE_STORAGE_CONFIG' environment variable must be set to
• unsupported linekind: {s}
• need 1..N positional arguments describing the key, try hex o
• failed to parse {one:?} as hex or span attributes: - {e1:#}

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/b4a30583fd7b2a99. Report an issue: GitHub.