xai-org/x-algorithm · error · IllegalArgumentException

Job config specified EntityType.SemanticCore, but non-semant

Error message

Job config specified EntityType.SemanticCore, but non-semantic core entity was found.

What it means

While building the user-entity matrix, the job expects every entity to be an Entity.SemanticCore when entityType == SemanticCore. Encountering any other Entity variant (Hashtag, TweetId, UserId) hits the catch-all and throws, protecting downstream InternalId.SemanticCore conversion from ClassCastException-like failures.

Source

Thrown at simclusters/simclusters_v2/scalding/embedding/LocaleEntitySimClustersEmbeddingsJob.scala:330

    semanticCoreEntityIdsToKeep: Option[TypedPipe[Long]],
    applyLogTransform: Boolean = false
  ): TypedPipe[(UserId, (Entity, Double))] =
    jobConfig.entityType match {
      case EntityType.SemanticCore =>
        semanticCoreEntityIdsToKeep match {
          case Some(entityIdsToKeep) =>
            getEntityUserMatrix(entityRealGraphSource, jobConfig.halfLife, EntityType.SemanticCore)
              .map {
                case (entity, (userId, score)) =>
                  entity match {
                    case Entity.SemanticCore(SemanticCoreEntity(entityId, _)) =>
                      if (applyLogTransform) {
                        (entityId, (userId, (entity, Math.log(score + 1))))
                      } else {
                        (entityId, (userId, (entity, score)))
                      }
                    case _ =>
                      throw new IllegalArgumentException(
                        "Job config specified EntityType.SemanticCore, but non-semantic core entity was found.")
                  }
              }.hashJoin(entityIdsToKeep.asKeys).values.map {
                case ((userId, (entity, score)), _) => (userId, (entity, score))
              }
          case _ =>
            getEntityUserMatrix(entityRealGraphSource, jobConfig.halfLife, EntityType.SemanticCore)
              .map { case (entity, (userId, score)) => (userId, (entity, score)) }
        }
      case EntityType.Hashtag =>
        getEntityUserMatrix(entityRealGraphSource, jobConfig.halfLife, EntityType.Hashtag)
          .map { case (entity, (userId, score)) => (userId, (entity, score)) }
      case _ =>
        throw new IllegalArgumentException(
          s"Argument [--entity-type] must be provided. Supported options [${EntityType.SemanticCore.name}, ${EntityType.Hashtag.name}]")
    }

  def toSimClustersEmbeddingId(

View on GitHub (pinned to 24c60942c5)

Solutions

Filter the input to only SemanticCore entities before this stage
Verify --entity-type matches the actual entity source being read
Add a reprocessing/validation job upstream to separate entity types

Example fix

// before
userEntityMatrix.flatMap {
  case (userId, (entity: Entity.SemanticCore, score)) => Some(...)
  case _ => throw new IllegalArgumentException("Job config specified EntityType.SemanticCore, but non-semantic core entity was found.")
}

// after
userEntityMatrix.flatMap {
  case (userId, (entity: Entity.SemanticCore, score)) => Some(...)
  case _ =>
    println(s"Skipping non-semantic-core entity")
    None // or log & count instead of failing the whole job
}

Defensive patterns

Strategy: type-guard

Validate before calling

val semanticCoreOnly = input.filter { case (_, (e: Entity, _)) => e.isInstanceOf[Entity.SemanticCore] }

Type guard

def isSemanticCore(e: Entity): Boolean = e match {
  case _: Entity.SemanticCore => true
  case _ => false
}

Try / catch

catch { case e: IllegalArgumentException if e.getMessage.contains("SemanticCore") => log.error("Non-semantic-core entity in input; check upstream source", e); sys.exit(2) }

Prevention

Filter the input stream by entity type before joining
Run a data-quality job to assert entity-type homogeneity before this pipeline

When it happens

Trigger: Running with --entity-type SemanticCore while the input entity stream contains Entity.Hashtag or other non-SemanticCore entities (dirty upstream data or wrong source selection).

Common situations: Mixing entity sources in the RealGraph/input data; upstream schema change emitting hashtags into the semantic-core stream; running the hashtag pipeline with the semantic-core flag.

Related errors

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/d2d069d9a6a54c38. Report an issue: GitHub.