quickwit-oss/quickwit · error

source `{}` is defined more than once

Error message

source `{}` is defined more than once

What it means

Thrown when deserializing a v0.8 index metadata document that contains two sources with the same source_id. The migration code converts the source list to a map keyed by source_id and treats duplicates as corrupted input rather than silently dropping one.

Source

Thrown at quickwit/quickwit-metastore/src/metastore/index_metadata/serialize.rs:87

    pub index_uid: IndexUid,
    #[schema(value_type = VersionedIndexConfig)]
    pub index_config: IndexConfig,
    #[schema(value_type = Object)]
    pub checkpoint: IndexCheckpoint,
    #[serde(default = "utc_now_timestamp")]
    pub create_timestamp: i64,
    #[schema(value_type = Vec<VersionedSourceConfig>)]
    pub sources: Vec<SourceConfig>,
}

impl TryFrom<IndexMetadataV0_8> for IndexMetadata {
    type Error = anyhow::Error;

    fn try_from(v0_8: IndexMetadataV0_8) -> anyhow::Result<Self> {
        let mut sources: HashMap<String, SourceConfig> = Default::default();
        for source in v0_8.sources {
            if sources.contains_key(&source.source_id) {
                anyhow::bail!("source `{}` is defined more than once", source.source_id);
            }
            sources.insert(source.source_id.clone(), source);
        }
        Ok(Self {
            index_uid: v0_8.index_uid,
            index_config: v0_8.index_config,
            checkpoint: v0_8.checkpoint,
            create_timestamp: v0_8.create_timestamp,
            sources,
        })
    }
}

View on GitHub (pinned to a39730c5cd)

Solutions

  1. Inspect the metastore index JSON file and remove/merge the duplicate source entries with the same source_id
  2. Fix whatever tool or manual edit produced the duplicate and re-export the metadata
  3. Restore the index metadata file from a backup taken before the corruption

Example fix

// before
"sources": [{"source_id":"kafka"},{"source_id":"kafka"}]
// after
"sources": [{"source_id":"kafka"}]
Defensive patterns

Strategy: validation

Validate before calling

let ids: HashSet<&str> = v08.sources.iter().map(|s| s.source_id.as_str()).collect();
if ids.len() != v08.sources.len() {
    panic!("duplicate source_id in index metadata");
}

Type guard

fn has_unique_sources(sources: &[SourceConfig]) -> bool {
    let ids: HashSet<_> = sources.iter().map(|s| s.source_id.as_str()).collect();
    ids.len() == sources.len()
}

Try / catch

match IndexMetadata::try_from(v0_8) {
    Err(e) if e.to_string().contains("defined more than once") => {
        eprintln!("corrupted metadata: {e}; restore from backup");
    }
    other => other?,
}

Prevention

When it happens

Trigger: Loading/migrating an old IndexMetadataV0_8 whose `sources` array contains two entries with identical source_id, e.g. a hand-edited or corrupted metastore JSON file.

Common situations: Upgrading Quickwit from an old version with metastore files edited manually or written by a buggy tool; JSON merge conflicts producing duplicated source entries.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/da4b8c98b3941598. Report an issue: GitHub.