Semantic Model Aggregates and Cuboid Replication

Semantic Model Aggregates and Cuboid Replication

1. Overview

This section covers semantic model processing and cuboid management, detailing processing stages, aggregate generation, and cuboid replication.

The objective is to provide an understanding of how Kyvos manages semantic model processing, optimizes query execution through aggregates, and distributes cuboids across Query Engines to maintain performance and stability.

2. Semantic Model Processing Lifecycle

A semantic model processing involves multiple processing activities, starting with the identification of the data and model changes and ending with the availability of processed cuboids for querying.

At a high level, the processing lifecycle includes the following stages:

2.1 Initialization

  • A processing request is submitted for the semantic model.

  • Kyvos initializes the processing job and validates the model configuration.

  • The processing scope is determined based on the selected processing type (Full or Incremental).

  • Required metadata, data sources, partitions, and aggregation configurations are identified.

2.2 Data Processing and Transformation

  • Kyvos reads the underlying source data.

  • Data is processed according to the semantic model definition, including relationships, dimensions, measures, hierarchies, and calculated expressions.

  • For incremental processing, only the impacted partitions are identified for processing.

  • Required intermediate data structures are generated for subsequent aggregate creation.

2.3 Aggregate and Cuboid Generation

  • Kyvos generates the required aggregates based on the aggregation strategy configured for the semantic model.

  • The aggregates represent precomputed summaries of data at different dimensional granularities.

  • These summaries are stored as cuboids.

  • Cuboids are generated and progressively replicated to the Query Engines during processing.

2.4 Cuboid Replication

  • Newly generated cuboids are progressively copied to the Query Engines while processing is underway.

  • This allows Query Engines to receive the new cuboids without waiting for the entire processing job to finish.

  • During post-processing, Kyvos validates whether all required cuboids have been successfully copied.

  • Any cuboids that were not replicated during the model processing are copied during post-processing.

2.5 Post-Processing and Validation

  • Kyvos performs the required post-processing activities.

  • The system validates the successful completion of processing and the availability of required cuboids.

  • Once the model processing completes successfully, the new cuboids become the active version for querying.

  • Older cuboids are marked for purging.

2.6 Purge and Cleanup

  • Purging is performed asynchronously.

  • A purge activity is triggered on the Query Engines after processing completes.

  • The purge cycle starts 10 minutes after processing completion.

  • Each Query Engine cleans up the cuboids available on its respective node.

Important: The above represents the logical processing lifecycle. Individual job names, internal execution stages, and their sequencing may vary depending on the Kyvos version and processing architecture.

3. Aggregate Generation

3.1 What Are Aggregates?

An aggregate is a precomputed summary of measure data at a particular dimensional granularity.

For example, consider a sales semantic model containing the following dimensions:

  • Geography

  • Product

  • Customer

  • Time

A query requesting total sales by country does not necessarily need to scan the underlying transaction-level data. Kyvos can use a previously generated aggregate containing sales summarized at the country level.

Aggregates help reduce the amount of data that needs to be scanned and processed during query execution, thereby improving query response times.

3.2 Base Aggregates vs. Smart Aggregates

Kyvos supports different aggregation strategies, including Expert (user-driven) and Smart (system-driven) aggregation.

Parameter

Base / Expert Aggregation

Smart Aggregation

Approach

Configuration-driven

System-driven

Selection

Based on the aggregation configuration defined for the semantic model

Based on query patterns, data profiles, and system recommendations

Inputs

Model dimensions, hierarchies, materialization settings, and precompute thresholds

Historical query information, data profiles, and processing/test results

Control

Provides greater control over the aggregates to be materialized

Kyvos recommends aggregates based on observed workload patterns

Objective

Materialize the required dimensional combinations based on the defined strategy

Optimize query performance while controlling aggregate storage and size

Maintenance

Updated through changes to the aggregation strategy and subsequent processing

Recommendations evolve as the system learns from query and data patterns

How Smart Aggregates are generated

Smart Aggregation uses information such as:

  1. Historical query patterns.

  2. Data profiles.

  3. Processing and test results.

  4. Dimensional and hierarchy usage patterns.

Based on this information, Kyvos recommends an aggregation strategy that aims to optimize frequently executed queries while keeping the overall aggregate footprint under control.

As users continue to execute queries, Kyvos learns from the workload and can recommend additional or modified aggregates.

From Kyvos 2026.6 onwards, there is an additional optimization in Smart Aggregate generation: new Smart Aggregates can be created using the closest previously materialized Smart Aggregates rather than always using the Base Aggregate. This reduces the amount of data that needs to be processed during Smart Aggregate generation.

3.3 Algorithm Used for Base Aggregate Generation

The Expert aggregation strategy is configuration-driven.

The aggregation configuration determines which dimensional and hierarchical combinations should be materialized. The materialization level and precompute threshold influence the resulting aggregation strategy.

The general process is:

  1. Identify the dimensions, hierarchies, and levels defined in the semantic model.

  2. Apply the configured materialization settings.

  3. Evaluate the combinations that qualify for materialization based on the configured thresholds.

  4. Generate the corresponding aggregates from the underlying processed data.

  5. Store the resulting aggregates as cuboids.

The precise internal algorithm used to enumerate, select, and optimize Base Aggregate combinations, including the detailed cost and cardinality calculations, is an implementation-level detail and should be confirmed with Kyvos Engineering before sharing a more granular algorithmic explanation.

3.4 How Aggregates Impact Query Performance

Aggregates improve query performance by reducing the volume of data that must be scanned and processed.

For example:

  • Without an appropriate aggregate, a query requesting sales by country may need to process a larger volume of detailed transaction data.

  • With a suitable aggregate, Kyvos can use precomputed country-level summaries.

  • This reduces data scanning, processing overhead, and potentially the amount of data transferred between processing components.

The actual performance improvement depends on:

  • Query patterns and dimensional granularity.

  • Aggregate availability and coverage.

  • Data cardinality.

  • Aggregate size.

  • Cache availability.

  • Query complexity and concurrency.

An excessive number of aggregates can increase processing time, storage consumption, and replication overhead. Therefore, the objective is not to generate every possible aggregate but to identify an optimal set that balances query performance with storage and processing costs.

3.5 Lifecycle of an Aggregate

The aggregate lifecycle can be understood through the following stages:

  1. Recommendation / Definition: Aggregates are defined manually or recommended by Smart Aggregation.

  2. Generation: Aggregates are generated during semantic model processing or an Update Aggregates operation.

  3. Replication: Generated cuboids are progressively copied to Query Engines.

  4. Validation: Post-processing verifies that all required cuboids have been replicated.

  5. Query Execution: Query Engines use the available cuboids to serve queries.

  6. Replacement: During subsequent processing, new cuboids are generated to reflect the latest processed data or aggregation strategy.

  7. Purge: Superseded cuboids are marked for cleanup and removed through the asynchronous purge process.

4. Full vs. Incremental Processing and Cuboid Cleanup

4.1 Full Processing

When a full model processing is triggered, the entire semantic model is reprocessed from the current state of the ingested data.

How it works:

  1. A full model processing starts, and new cuboids are created for all aggregates.

  2. As processing runs, new cuboids are copied to the Query Engines progressively, not just at the end.

  3. Once the full model processing completes successfully, the older cuboids are marked for purging.

  4. Purge is executed asynchronously. A purge activity is triggered on the Query Engines after the model processing, so queries are never blocked waiting for cleanup.

  5. The purge cycle starts 10 minutes after processing is completed.

Key point: New cuboids are made available for querying as soon as processing completes successfully. Old cuboids are purged during the subsequent purge event.

4.2 Incremental Processing

Incremental processing applies only the newly ingested data delta to the already-processed semantic model.

How it works:

  1. Only the impacted partitions are identified and rebuilt, not the entire model.

  2. When run with Replace Partition = Auto, new cuboids are built specifically for those affected partitions and any additional partitions added to the model.

  3. New cuboids are copied to the Query Engines during processing, following the same approach as full processing.

  4. Once new cuboids are in place, the old partition cuboids are marked for purging.

  5. The purge cycle starts 10 minutes after processing is completed.

Key point: Only the changed partitions go through the model processing-and-purge cycle, making incremental processing significantly lighter than full processing while following the same asynchronous purge pattern.

5. Cuboid Replication and Purging

5.1 What Happens to Cuboids Generated During a Failed, Canceled, or Interrupted?

Cuboids generated during failed, canceled, or interrupted model processings are handled through subsequent cleanup processing.

The cleanup mechanism depends on the state of the semantic model:

  • If the cube is in a resumable state: Cleanup is performed after the next successful Semantic Model processing.

  • If the cube is not in a resumable state: The cuboids are picked up for cleanup during the next iteration of the purge process.

Additionally, cuboids generated during failed, canceled, or interrupted model processing are discarded and are not copied locally.

5.2 What Happens if a Query Engine Is Unavailable or Restarted When Purge Is Triggered?

The purge thread starts during Query Engine boot-up and runs at a configurable frequency.

  • Default purge frequency: 30 minutes.

  • Configuration property: CUBOID_PURGE_FREQUENCY.

If a Query Engine is unavailable or restarted when a purge operation is triggered, the purge operation is retried during the next scheduled run.

This ensures that a temporary Query Engine outage does not permanently prevent cuboid cleanup.

5.3 What Happens if Another Processing Run Starts Within the 10-Minute Purge Window?

Cuboids are unique to each processing run, and every processing run creates a new set of cuboids.

Therefore, if another processing run starts within the 10-minute window:

  • The previous processing run's cuboids remain identifiable.

  • The purge flow for the previous run remains unchanged.

  • The purge is neither deferred nor skipped because of the subsequent processing run.

5.4 How Does Purging Work When the Number of Query Engines Changes?

The purge behavior remains consistent across all Query Engines.

Each Query Engine purges the cuboids available on its respective node.

Therefore, changes in the number of Query Engines, such as scaling down or scaling up, do not change the fundamental purge mechanism.

5.5 What Exactly Is Replicated During Processing and Post-Processing?

Cuboids are progressively copied to the Query Engines while the model processing is in progress.

During post-processing:

  1. Kyvos validates whether all required cuboids have been successfully copied to the Query Engines.

  2. Kyvos identifies any cuboids not copied during model processing.

  3. The remaining cuboids are replicated during post-processing.

This approach allows replication to happen in parallel with processing while retaining a final validation step to ensure completeness.

 

Copyright Kyvos, Inc. 2026. All rights reserved.