Kyvos Conversational Analytics uses Large Language Models (LLMs) to understand natural-language questions, generate analytical queries, and, when enabled, summarize analytical results. Kyvos incorporates multiple security and safety controls to help protect customer data and ensure that LLM interactions remain focused on their intended analytical purpose.

Security for LLM-based Conversational Analytics is implemented through multiple layers of controls, including:

Kyvos leverages an Enterprise Subscription of your LLM service (like Azure OpenAI, Vertex AI, AWS Bedrock) within the customer environment, based on the applicable Kyvos deployment architecture. This helps keep customer metadata and analytical data within the customer's controlled environment.

Conversational Analytics interacts with the LLM primarily for two purposes:

1. Natural Language to SQL Generation
   When a user asks a question in natural language, Kyvos uses prompt engineering and Retrieval-Augmented Generation (RAG) techniques to provide the LLM with the relevant semantic model context needed to generate an SQL query.

The context provided for SQL generation can include semantic model metadata such as:

1. Column names

2. Column descriptions

3. Tags

The generated SQL is then executed by the Kyvos analytical engine. Data access continues to be governed by the security controls configured in Kyvos.

2. Generating Natural Language Summary of Results

Kyvos can optionally use an LLM to generate a natural-language summary of analytical query results. This capability is configurable. When enabled, relevant query-response data is provided to the LLM to generate a summary of key insights. If the customer does not want query-response data to be sent to the LLM for summarization, this capability can be disabled.

Kyvos applies prompt-level guardrails to keep LLM interactions focused on the intended Conversational Analytics tasks.

The LLM is instructed to perform supported tasks such as:

• Generating SQL based on the available semantic-model context.

• Summarizing analytical query results.

It is not intended to function as a general-purpose chatbot within the Conversational Analytics workflow.

For example, if a semantic model contains retail-order data, questions such as:

• "Which movies were released last week?"

• "How is the weather outside?"

• "How do I make a bomb?"

The system would respond with a message such as:

Sorry! Unable to answer your query. The provided columns do not include information specific to movies or weather. The database and columns are related to retail orders, so a query about movies or weather cannot be generated.

These controls help prevent the Conversational Analytics LLM from being used for unrelated content generation.

Kyvos uses prompt engineering and contextual controls to restrict the LLM to its intended analytical tasks.

The context provided to the LLM is limited to information required for the specific operation. This helps reduce the risk of unintended model behavior and prevents unrelated instructions from changing the intended purpose of the interaction.

The generated SQL is based on the semantic-model context provided by Kyvos and is executed through the Kyvos analytical engine.

Conversational Analytics respects the security controls configured in Kyvos when accessing analytical data.

Depending on the deployment and configuration, this includes applicable:

• Authentication and authorization controls

• Role-based access controls

• Row-level security

• Semantic-model security

• Other data-access restrictions configured in Kyvos

Conversational Analytics does not provide a mechanism to bypass the security controls applied to the underlying Kyvos data.

Kyvos provides capabilities that help users understand and validate AI-generated responses.

Kyvos logs LLM interactions to support troubleshooting, monitoring, and auditing. Depending on the configuration, logged information can include:

• User question

• Relevant prompt or context sent to the LLM

• LLM-generated response

• Generated SQL

• Generated analytical summary

Access to this information is controlled by the applicable Kyvos and customer environment configuration.

Where applicable, Kyvos provides visibility into the context used to generate an answer.

Users may be able to view:

• Generated SQL

• AI-generated summary

• Relevant semantic-model context

• Columns used to generate the response

This provides greater transparency into how an AI-generated analytical response was produced.

AI-generated responses are identified as AI-generated where applicable.

Users should validate AI-generated SQL, summaries, and insights before relying on them for important business decisions.

When Azure OpenAI is used, Azure AI safety capabilities provide an additional layer of protection for LLM interactions.

Depending on the Azure configuration, these capabilities can include:

Azure content filters can be configured to detect or restrict prompts and generated responses associated with potentially harmful content categories, including:

• Hate

• Violence

• Sexual content

• Self-harm

• Other supported safety categories

Additional models or safety mechanisms can be used to identify potentially unsafe or inappropriate content before or after interaction with the primary LLM.

Blocklists can be configured to prevent specific words, phrases, or statements from being submitted to or generated by the LLM.

These controls provide an additional layer of defense alongside the guardrails implemented by Kyvos.

Kyvos follows a data-minimization approach when interacting with LLMs.

For SQL generation, only the metadata and contextual information required to generate the SQL query is provided to the LLM. Underlying analytical data is not required for this operation.

For result summarization, query-response data is provided only when the optional result-summary capability is enabled.

Kyvos can support user feedback mechanisms for AI-generated responses. Feedback can help identify inaccurate or unexpected responses and improve the quality of Conversational Analytics.