{"record":{"id":"b4a30583fd7b2a99","repo":"neondatabase/neon","slug":"unexpected-in-memory-layer","errorCode":null,"errorMessage":"unexpected in-memory layer","messagePattern":"unexpected in-memory layer","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pageserver/ctl/src/index_part.rs","lineNumber":91,"sourceCode":"    let tenant_shard_id = TenantShardId::unsharded(tenant_id);\n    let timeline_id = TimelineId::from_str(timeline_id).unwrap();\n    let index_json = {\n        let bytes = tokio::fs::read(path).await?;\n        IndexPart::from_json_bytes(&bytes).unwrap()\n    };\n    let layer_map = create_layer_map_from_index_part(&index_json, tenant_shard_id, timeline_id);\n    let key = Key::from_hex(key)?;\n\n    let lsn = Lsn::from_str(lsn).unwrap();\n    let mut end_lsn = lsn;\n    loop {\n        let result = layer_map.search(key, end_lsn);\n        match result {\n            Some(SearchResult { layer, lsn_floor }) => {\n                let disk_layer = match layer {\n                    ReadableLayerWeak::PersistentLayer(layer) => layer,\n                    ReadableLayerWeak::InMemoryLayer(_) => {\n                        anyhow::bail!(\"unexpected in-memory layer\")\n                    }\n                };\n\n                let metadata = index_json\n                    .layer_metadata\n                    .get(&disk_layer.layer_name())\n                    .unwrap();\n                println!(\n                    \"{}\",\n                    remote_layer_path(\n                        &tenant_id,\n                        &timeline_id,\n                        metadata.shard,\n                        &disk_layer.layer_name(),\n                        metadata.generation\n                    )\n                );\n                end_lsn = lsn_floor;","sourceCodeStart":73,"sourceCodeEnd":109,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/pageserver/ctl/src/index_part.rs#L73-L109","documentation":"The index_part ctl command reconstructs a layer map from the tenant's index_part.json. That index only ever references persistent (delta/image) layers on remote storage, so a search result that is an in-memory layer violates the format's invariant. In practice this means the ctl binary's layer-map construction is out of sync with the index content (version skew) or the index_part.json is corrupt or written by an incompatible format version.","triggerScenarios":"layer_map.search(key, end_lsn) returning ReadableLayerWeak::InMemoryLayer while walking a layer map derived purely from index_part.json.","commonSituations":"Running a ctl built from a different branch or commit than the pageserver that wrote the index; inspecting an index produced by a newer or older on-disk format; a hand-edited or partially written index_part.json.","solutions":["Rebuild or reinstall the ctl binary from the same commit as the pageserver that owns the tenant.","Inspect the index_part.json layer list directly (jq or the ctl's own printing) for unexpected entries.","If the index is genuinely corrupt, re-attach the tenant in the pageserver so it rewrites index_part.json, or reconstruct from the layer files on remote storage."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"match layer_map.search(key, end_lsn) {\n    Some(SearchResult { layer: ReadableLayerWeak::PersistentLayer(layer), lsn_floor }) => {\n        // proceed with the on-disk layer\n    }\n    Some(SearchResult { layer: ReadableLayerWeak::InMemoryLayer(_), .. }) => {\n        // index_part never contains in-memory layers: treat as version/corruption issue\n        anyhow::bail!(\"index_part contains an unexpected in-memory layer; rebuild ctl from the pageserver's commit or re-attach the tenant\");\n    }\n    None => { /* handle no covering layer */ }\n}","preventionTips":["Build the ctl binary from the same commit as the pageserver that wrote the index.","Snapshot index_part.json before running investigative commands so corruption can be diffed later.","Prefer re-attaching the tenant in the pageserver to regenerate the index rather than hand-editing it."],"tags":["rust","neon","ctl","layer-map","index-part"],"backgroundTag":"version-mismatch","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}