Common Prerequisites for Spark and No-Spark

This section describes the common prerequisites and required configurations that must be completed before deploying Kyvos using either Spark or No-Spark on Google Cloud Platform.

• To continue with Postgres 16 as an installed service, download the PostgreSQL 16 script.

• Configure Kyvos CPU and memory allocation based on the capacity of the selected VM instance to ensure optimal resource utilization. For example, if a node is provisioned with a 64 GB RAM VM, configure the Kyvos processes to utilize the available 64 GB memory (while reserving resources for the operating system and other required services) and the corresponding CPU cores.

• Kyvos now supports N4 series machine types for Kyvos components (Analytical Server, Query Engine and Kyvos Web Portal) and dedicated compute cluster. Ensure that you change the disk type to hyperdisk-balanced when selecting N4 instance type.

• gcloud must be installed on the Virtual Machine instances required for upgrade and patch.

• You can download a single script for wizard-based deployments by automatically installing all required prerequisites across all nodes.

• You need a valid Google Cloud Platform account. This account will be used to authenticate Terraform to interact with GCP resources.

• The following permissions must be given to the logged-in user account:

  • Editor

  • Secret Manager Admin

  • Storage Object Admin

  • Create a custom role and assign the below permission to the role. Ensure that custom role must be attached to logged-in user account.

    • iam.roles.create  

    • iam.serviceAccounts.setIamPolicy

    • resourcemanager.projects.setIamPolicy  

• Kyvos needs a service account to launch the Kyvos instance. Refer to the steps given in the Service Account section to create it.

• Ensure that the following ports are opened/allowed in the Firewall inbound rules for all internal communication between Kyvos instances.
  2121, 2181, 2888, 3888, 4000, 6602, 6903, 6703, 45450, 45460, 45461, 45462, 45463, 45464, 45465, 6603, 6702, 6803, 7003, 45440, 6605, 45421, 45564, 4000, 8080, 8081, 8005, 8009, 8443, 8444, 9443, 22 and 9444.

• Ports 22, 8080, and 8081 should be accessible from outside of the cluster from where you want to access the Web application.

• Kyvos will need the Storage Legacy Bucket Owner role on the storage bucket to store data (semantic models). Grant this role if using existing bucket in Kyvos deployment

• To access the storage bucket from the Kyvos instances, a NAT Gateway in VPC or Endpoint between storage and VPC should be available.  

• To send requests to your VPC network and receive the corresponding responses without using the public internet, you must use the Serverless VPC Access connector.
  Serverless VPC Access uses the Serverless VPC Access Service Agent service account. This service account's email address has the following form:
  mailto:service-PROJECT_NUMBER@gcp-sa-vpcaccess.iam.gserviceaccount.com

  Permissions required

  1. By default, the above service account has the Serverless VPC Access Service Agent role (roles/vpcaccess.serviceAgent). Serverless VPC Access operations may fail if you change this account's permissions.

  2. If using a Shared Network, grant the above service account the Serverless VPC Access Service Agent predefined role to the project where the network originally resides.

     NOTE: You can refer to the GCP documentation to create a Serverless VPC Access connector. 

• If your bucket is in another project, then for cross-project bucket access, you must provide the following permissions on your bucket.

  • storage.object.list

  • storage.object.get

• If using shared VPC, the VPC must be shared with the project that you want to access.

  1. Navigate to the VPC network.

  2. Click the Shared VPC. This should be performed from the project where the shared VPC network originally resides.

  3. Go to the ATTACHED PROJECTS tab and attach the project.

     Open

     

• When using an existing VPC for Kyvos, the subnet must have a minimum mask range of /22

• Click Roles > Create new role. Provide a name like Kyvos-role for storage service and assign the following permissions. This role should be attached to Kyvos service account.

• From Kyvos 2026.6 onwards, existing service accounts used by the Kyvos cluster must be assigned the Secret Manager Secret Version Manager predefined IAM roles to support automatic retention management.

• Add the following predefined roles to the service account used by the Kyvos cluster.

  • BigQuery data viewer

  • BigQuery user

  • Dataproc Worker

  • Cloud Functions Admin

  • Cloud Scheduler Admin

  • Cloud Scheduler Service Agent

  • Service Account User

  • Logs Writer

• If an existing service account is used, add the required invoker permission by running the following command: 
  gcloud functions add-invoker-policy-binding functionName  --project=projectId   --region=region   --member="allUsers"

• Prerequisites for setup between Cross-Project BigQuery and Kyvos VMs:

  • Grant the following roles to the applicable service account on the BigQuery Project based on the authentication type (mentioned below).

    • BigQuery Data Viewer

    • BigQuery User

• For accessing BigQuery Views, add the following permissions to the Kyvos custom role (created above).

  • bigquery.tables.create

  • bigquery.tables.delete

  • bigquery.tables.update

  • bigquery.tables.updateData

• To create secret for Google Cloud Platform, refer to Creating Secret for Google Cloud Platform Cluster for secret manager permissions.

• The secretmanager.secrets.list permission must be attached to the existing service account. This permission is required to list the Secret Manager on the replication configuration page in Kyvos Manager.

• In the API and identity management section of VM, for Cloud API access scopes, the Allow full access to all Cloud APIs must be set.

  Open

  

Apart from the existing permissions mentioned in the Creating a service account from Google Cloud Console section, you need the following permissions for GCP Enterprise:

Permissions required in GCP

• compute.instanceGroups.get

• compute.instances.create

• compute.disks.create

• compute.disks.use

• compute.subnetworks.use

• compute.instances.setServiceAccount

• compute.instances.delete

• compute.instanceGroups.update

• compute.instances.use

• compute.instances.detachDisk

• compute.disks.delete

• compute.instances.attachDisk

• compute.healthChecks.get

• compute.regionBackendServices.get

• compute.regionBackendServices.list

• compute.regionHealthChecks.get

• compute.regionHealthChecks.update

Conditional permission needed if using Shared Network

• compute.subnetworks.use (on the Kyvos service account in the project where your network resides)