How to Manage Google Cloud Compute Engine with Python

Compute Engine is an IaaS offering by Google Cloud. It is a powerful platform that can be used to create and manage virtual machines. However, managing Compute Engine can be a complex task. In this blog post, I will show you how to use Python to automate the management of Compute Engine instances.

Setting up your environment

Make sure you have python and pip installed.

Installation

Mac/Linux

pip install virtualenv
virtualenv <your-env>
source <your-env>/bin/activate
<your-env>/bin/pip install google-cloud-compute

Windows

pip install virtualenv
virtualenv <your-env>
<your-env>\Scripts\activate
<your-env>\Scripts\pip.exe install google-cloud-compute

Authentication

For authentication let's create a service account.

IAM & Admin --> Service Accounts
Click on CREATE SERVICE ACCOUNT.
Add a Service account name and Service account ID.
Click on CREATE AND CONTINUE.
Add Role of Compute Admin.
Click on DONE.

Now let's generate a JSON key which we can use in our local environment.

Click on the service account name you just created.
Click on the KEYS option.
ADD KEY --> Create new key
Select JSON and click on CREATE.
Download and Save the key.

Now that we have authentications sorted let's look at some code!

Creating a new VM instance

from google.cloud import compute_v1
import os

# Using os we are storing the service account json key in the variable
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]= "key.json"

INSTANCE_NAME = 'instance-name'
MACHINE_TYPE = 'projects/project-id/zones/us-west4-b/machineTypes/e2-medium' 
SUBNETWORK = 'projects/project-id/regions/us-west4/subnetworks/default'
SOURCE_IMAGE = 'projects/project-id/global/images/ubuntu-2004-focal-v20230831'
NETWORK_INTERFACE = {
    'subnetwork':SUBNETWORK,
    'access_configs': [
        {
            'name':'External NAT'
        }
    ]
}

compute_client = compute_v1.InstancesClient()

config = {
    'name' : INSTANCE_NAME,
    'machine_type' : MACHINE_TYPE,
    'disks': [
        {
            'boot': True,
            'auto_delete': True,
            'initialize_params': {
                'source_image': SOURCE_IMAGE,
            }
        }
    ],

    'network_interfaces' : [NETWORK_INTERFACE]
}

print("Creating instace.....")
operation = compute_client.insert(
    project='project-id',
    zone='us-west4-b',
    instance_resource=config
)

operation.result()

print(f'Created VM Instance:{INSTANCE_NAME}')

Imported Modules :

google.cloud.compute_v1: This module provides the client library for the Google Cloud Compute Engine API.
os: This module provides access to operating system functionality.

This is how the code works:

Sets the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of the JSON file that contains the Google Cloud service account credentials.
Defines the following variables:
- INSTANCE_NAME: The name of the instance to create.
- MACHINE_TYPE: The machine type of the instance to create.
- SUBNETWORK: The subnetwork of the instance to create.
- SOURCE_IMAGE: The image to use for the instance.
- NETWORK_INTERFACE: The network interface of the instance to create.
Creates an instance of the InstancesClient class. This class provides methods for managing Google Compute Engine instances.
Creates a dictionary called config. This dictionary contains the configuration for the instance to create.
Calls the insert() method of the InstancesClient class. This method creates the instance.
Waits for the operation to complete.
Prints the name of the instance that was created.

If you are not sure about arguments like MACHINE_TYPE or SOURCE_IMAGE then you can refer to the Google Cloud Console.

Compute Engine --> VM instances
Click on CREATE INSTANCE.
Choose the arguments according to you.

Once done click on the EQUIVALENT CODE option in the upper right corner and choose REST

Here you can see all the available arguments and you can choose as per your requirements.

List all the VM instances

from google.cloud import compute_v1
from collections import defaultdict
import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]= "key.json"

instace_client = compute_v1.InstancesClient()
request = compute_v1.AggregatedListInstancesRequest()
request.project = 'project-id'

agg_list = instace_client.aggregated_list(request=request)

all_instances = defaultdict(list)
print("Instances found:")
for zone, response in agg_list:
        if response.instances:
            all_instances[zone].extend(response.instances)
            print(f" {zone}:")
            for instance in response.instances:
                print(f" - {instance.name} ({instance.machine_type}) {instance.status}")

Imported Modules :

collections.defaultdict: This module provides a defaultdict data structure, which is a dictionary where the default value is set to a specified value.

This is how the code works:

Sets the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of the JSON file that contains the Google Cloud service account credentials.
Creates an instance of the InstancesClient class. This class provides methods for managing Google Compute Engine instances.
Creates an instance of the AggregatedListInstancesRequest class. This class represents a request to list Google Compute Engine instances.
Sets the project ID of the project that contains the instances to list.
Calls the aggregated_list() method of the InstancesClient class. This method lists the instances in the specified project.
Creates a dictionary called all_instances. This dictionary will be used to store the instances that are found.
Iterates over the response from the aggregated_list() method.
For each zone in the response, adds the instances in that zone to the all_instances dictionary.
Prints the name, machine type, and status of each instance in the all_instances dictionary.

This how the output looks like. I have printed the instance name, machine type and status but you can print every attribute of the instance from the all_instances dictionary.

Start and stop a VM instance

from google.cloud import compute_v1
import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]= "key.json"

compute_client = compute_v1.InstancesClient()

operation = compute_client.stop(
    project='project-id',
    zone='us-west4-b',
    instance='python-instance'
)

operation.result()
print("Instace Stopped!")

This is straightforward. All you have to provide is project-id, zone and instance name.

Creates an instance of the InstancesClient class. This class provides methods for managing Google Compute Engine instances.
Calls the stop() method of the InstancesClient class. This method stops the instance.

The same applies for starting an instance. Rather than stop you have to use the start method rest is the same.

from google.cloud import compute_v1
import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]= "key.json"

compute_client = compute_v1.InstancesClient()

operation = compute_client.start(
    project='project-id',
    zone='us-west4-b',
    instance='python-instance'
)

operation.result()
print("Instace Started!")

Delete a VM Instance

from google.cloud import compute_v1
import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]= "key.json"

compute_client = compute_v1.InstancesClient()

operation = compute_client.delete(
    project='project-id',
    zone='us-west4-b',
    instance='instance-name'
)

operation.result()
print("Instace deleted successfully!")

Deleting also has the same logic as starting or stopping.

Creates an instance of the InstancesClient class. This class provides methods for managing Google Compute Engine instances.
Calls the delete() method of the InstancesClient class. This method deletes the instance.

All the code mentioned in this blog is available on GitHub

https://github.com/pratikkalein/GCP-Python

That's all about how you can manage VM instances using python. Let me know your thoughts or if I missed something in the comments.

Thank you for reading!

Connect With Me :

https://pratik.linkb.org/