I recently completed the Certified Kubernetes Administrator (CKA) certification. After learning about Kubernetes I wanted to try using it in my homelab. My homlab runs on Proxmox so I wanted to deploy a Kubernetes cluster running on a few Proxmox virtual machines. While it would be fairly easy to set up the virtual machines manually, I wanted to experiment with infrastructure as code, so I decided to use Terraform to do it instead.
Terraform is a tool for building and managing infrastructure declaratively. This means that the infrastructure is declared in configuration files that can be version controlled, reused, and shared. Terraform can manage low-level components like compute, storage, and networking resources, as well as high-level components like DNS entries and SaaS features.
In this article I will show how to use Terraform to deploy two virtual machines on Proxmox and provision them with cloud-init. Later, in Part 2 I will show how to integrate Ansible with Terraform and use Ansible to install a Kubernetes cluster on the virtual machines.
Using Terraform with Proxmox
Terraform integrates with third-party services (such as Proxmox) through plugins called providers. There are several providers available for Proxmox. At the time of writing (late 2022) I found the bpg proxmox provider to be the most feature rich and stable.
To get started with the bpg proxmox provider, create a new directory and add a file called main.tf containing the following:
Then run the following command in the same directory to download and install the provider:
$ terraform init
To use the provider, we need to give it a username and password to access the Proxmox API. There are several ways to do this, but I prefer using environment variables. I use a tool called direnv to set the environment variables. With this tool, the environment variables are automatically set when I enter the terraform directory and removed when I leave. This ensures that the username and password are not exposed to other applications on my system. To use direnv, create a file called .envrc and add the following:
Note: Since this file contains sensitive information you should avoid pushing it to a shared git repository. I recommend either adding it to .gitignore or using a tool like git-crypt to encrypt it.
Creating Proxmox VMs
To keep things simple we will create a Kubernetes cluster with one control plane node and one worker node. We therefore needed to create two Proxmox virtual machines. We will be using Debian as the operating system because the Debian maintainers provide pre-made cloud images that can be used with Proxmox. To use one of the images, we first need to upload it to the Proxmox server. This can be done with Terraform by creating a file called files.tf with the following content:
This will download the latest (at the time of writing) image and add it to the Proxmox server. Note that we have to change the file extension to .img because Proxmox only allows files with certain whitelisted extensions to be uploaded.
The next step is to create the two virtual machines. To do this, create a new file called virtual_machines.tf and add the following content:
In this example I have given the control plane node 2 cores and 6GB of RAM and the worker node 1 core and 2GB of RAM. This can be tweaked to match your requirements, but I would recommend at least 2GB RAM for each node.
Provisioning the VMs with cloud-init
Now that we have defined the virtual machines we need some way to access them. The Debian cloud images support cloud-init, so we will be using it to create a user and enable SSH so that we can log in. We wil also use cloud-init to install the QEMU guest agent so that Proxmox can detect the IP-address of the VMs. This will help Terraform know that the VMs have been created successfully.
To use cloud-init, create a new directory called cloud-init and add a file called user-data.yml. Add the following content and replace the placeholders with your username, password and SSH public key:
After creating this file, your directory structure should look like this:
.
├── cloud-init
│ └── user-data.yml
├── files.tf
├── main.tf
└── virtual_machines.tf
To use the cloud-init file, we need to upload it to Proxmox. To do this, modify files.tf and add the following lines:
Finally, we need to create a cloud-init drive and mount it to the virtual machines. Terraform can do this for us with only a few lines of code. Add the following to virtual_machines.tf:
In part 2 we will use Ansible to install Kubernetes and set up a cluster on the two virtual machines.