Deploying Managed Efficiency: Aiven BYOC On GCP

In the modern data landscape, the choice between managed services and infrastructure control is often presented as a trade-off. However, with Aiven Bring Your Own Cloud (BYOC), organizations no longer have to choose between the ease of a managed service, the security of their own VPC, and the sovereignty of their own data. Additionally, BYOC offers the potential financial benefit of using the "Committed Used Discounts" and avoiding hidden data egress transfer fees if they have to send the data to non-BYOC solutions such as Confluent Cloud.

Understanding the shared responsibility model

Under the BYOC model Aiven manages the OS and Kafka software lifecycle within your existing GCP project while you maintain ownership of the underlying GCP infrastructure.

Aiven operates within your environment by impersonating a Privilege-Limited Service Account. This account is strictly limited to the compute, networking, and storage resources required to run your managed services. It possesses no long-lived credentials and cannot access billing, IAM, or other unrelated projects, providing a clean "least-privilege" security model.

Below are the steps when executed by both sides in the shared responsibility view.

Table 1. Shared Responsibility Table

Architecting the security foundation for BYOC on GCP

The BYOC security foundation is under the bigger system architecture shown below.

Diagram 1. Aiven BYOC System Architecture

Your GCP project must be prepared to host the Aiven ecosystem. This involves the design and preparation of two identities.

1. The Bootstrap Identity, used by your cloud admin or GCP project owner, with greater privilege to create resources such as VPC and Compute nodes, create another service account such as the below less privileged service account, and manage networks such as subnets and firewalls.
2. The less privileged identity, also called Privilege-Limited Service Account, created by the Bootstrap Identity, intended to be used by Aiven to manage Aiven services in the BYOC. This Privilege-Limited Service Account does not use any long-lived credentials and the privilege can be revoked at any time. Further, every activity can be logged via GCP Cloud Audit Logs.

The Bootstrap Identity, typically using either a user account with owner rights to the target project, or a temporary service account that can be discarded after creation for security reasons.

Before deploying, ensure your Bootstrap Identity is equipped with the necessary roles, APIs and credentials. This includes 5 roles, 4 APIs, and 1 access token for GCP CLI.

The roles that Bootstrap Identity needs temporarily in order to set up BYOC in the bootstrap process are for these purpose:

• Compute and networking: roles/compute.admin is essential to provision the VPCs, subnets, firewall rules that house your Kafka brokers.
• Identity management: roles/iam.serviceAccountAdmin and roles/iam.securityAdmin allow for the creation and secure binding of the privilege-limited service account intended for Aiven to use.
• Access management: roles/resourcemanager.projectIamAdmin is required to attach IAM policies at project scope.
• Service enablement: roles/serviceusage.serviceUsageAdmin is necessary to enable service APIs such as Compute, IAM, etc.

The 4 APIs that the creation process needs are below. They must be enabled using the gcloud services enable command to allow automated orchestration.

1. compute.googleapis.com
2. iam.googleapis.com
3. cloudresourcemanager.googleapis.com
4. serviceusage.googleapis.com

The access token is intended to be used to run the TerraForm script. This reference implementation assumes Application Default Credentials using the active gcloud user by issuing: gcloud auth application-default login command. The terraform script is created by Aiven after you provide configuration through Aiven console and downloaded and applied by you defined by the Shared Responsibility Table step 5 and step 6.

Step by step BYOC creation

Below are the example screenshots to configure BYOC on the Aiven management console. However, it is also possible to do the same using the two command lines:

Loading code...

Loading code...

1. Pick cloud, region, and custom cloud name (your BYOC name)

2. Provide custom cloud's CIDR range

3. Download Aiven generated template, run the template, and enter generated Privilege-Limited Service Account

4. Assign BYOC to projects to be used

Streamlining infrastructure with Terraform

Aiven simplifies the deployment process by providing a guided setup as shown above within the Aiven Console to generate Terraform scripts. This automation ensures that your environment is built according to best practices without manual configuration errors.

The generated scripts perform several key functions:

• Network isolation: They create a dedicated VPC with two subnets, providing a clean separation for your data traffic.
• Traffic control: They automatically configure routing and firewall rules to support both private and public BYOC models.
• Handshake completion: Once the scripts are applied, you provide the generated Service Account ID back to Aiven to activate the custom cloud and assign your projects.

Limitation of the Privilege-Limited Service Account for Aiven

The Privilege-Limited Service Account created by you in this process can create and manage resources within the BYOC VPC but have intentionally limited other privileges for security purposes based on the least privilege principle.

While the customer uses high-level roles (like roles/owner or roles/resourcemanager.projectIamAdmin) to set up the infrastructure via Terraform, the Privileged-Limited Service Account itself is typically assigned more granular roles:

• roles/compute.instanceAdmin.v1: For managing the lifecycle of service nodes and networks.
• roles/compute.securityAdmin: For managing firewall rules and network security of the bastion node.
• roles/iam.serviceAccountUser: Often granted to the Aiven platform to allow it to "act as" or impersonate this account.

The Privilege-Limited Service Account is designed with the following functional capabilities (can do):

• Compute and instance management: Required to create and manage the virtual machines (VMs) that serve as Kafka nodes.
• Networking infrastructure: Permissions to create and manage dedicated VPCs, configure subnets, and establish routing.
• Security and firewalls: Capabilities to define and manage firewall rules and security groups specifically for the Aiven service nodes.
• Storage access: Necessary permissions to manage the local or tiered storage used by some services.

Aiven's access is strictly scoped to prevent unauthorized changes to your broader GCP environment (can't do):

• No long-lived credentials: Aiven does not possess static keys; it uses short-lived tokens through impersonation.
• Restricted project access: The account cannot modify IAM policies for other users, access billing information, or interact with other projects not assigned to the custom cloud.
• VPC isolation: Aiven typically accesses these resources through a bastion proxy or a dedicated network bridge, ensuring workload nodes are not directly exposed.

Conclusion

Aiven BYOC on GCP offers a sophisticated solution for enterprises that require the security of their own cloud environment without the overhead of managing complex services. It provides a secure, automated, and high-performance path to modern data products.

Ready to build? Sign up for an Aiven account and deploy your first BYOC cluster on Google Cloud today. For more information, check out the Aiven for BYOC on GCP Documentation.