Proxmox, Terraform, Ansible & Kubernetes

See how I plan to have private cloud via Proxmox, Terraform, Ansible and Kubernetes cluster on my Homelab machine!

November 3, 2022 - 3 minute read -
Homelab Proxmox Terraform Ansible Kubernetes

Equilibrium Engine is out!

After a year of development I am happy to publish my open source Game Engine on GitHub under permissive license.

It focuses mainly on modularity and the ability to reload your scripts & shaders on the fly. It can also be used to familiarize programmers with C language and Data Oriented Design applying ECS (Entity Component System) pattern. Get it here on GitHub

What’s next?

Currently I am making a private cloud at home with Proxmox, Terraform, Ansible and Kubernetes! It is currently deployed on a single physical machine with 64 threads, 128GB of RAM and 4TB of RAID 10 storage.

Grafana dashboard

Grafana dashboard

I use Proxmox as a Hypervisor and provision it via Terraform IAC (Infrastructure as Code) and Ansible that allows me to deploy Kubernetes in HA (high availability mode) on all the Virtual Machines.

Terraform & Ansible

With Terraform & Ansible I am able to automate my provisioning and deployment. By provisioning I mean everything from A to Z , from creating and setting up VM resources, OS installation via cloudinit, and Kubernetes installation combined with joining to an existing Kubernetes cluster.

Terraform allows me to locally provision my VMs but also provision Amazon or Google VMs. So if I ever decide to go to a public cloud I will be all set with couple extra lines of code. Ansible playbook is another useful automation tool that allows me to simply connect to my VMs via SSH and deploy Kubernetes on them. I strongly recommend check out Techno Tim repository and this article

Why Kubernetes?

Gearing towards micro service architecture I realized that having a VM per service leads to poor resource utilization - some VMs can be idling while others being super busy. I decided to dive into Kubernetes as it’s industry proven Container Orchestrator that helps with automatic deployments, scaling and management of containerized applications.

I chose k3s implementation of Kubernetes as it is lightweight and distributed as single binary.

What is my setup?

Proxmox dashboard

Proxmox dashboard

All 16 VMs are running on Ubuntu Focal Server 20.04, I have also tried Debian 11 but decided to stick with Ubuntu for better compatibility with Rancher.

Currently I have 16 VMs each of them have 4 vCPU cores (threads) and 8GB of RAM with 107GB boot disk.

The setup of k3s cluster is the following:

  • 3 master nodes in HA mode (if 1 master fails the system continues to work)
  • 13 worker nodes

This gives me 1.3TB of usable storage for my services. Also I use Rancher UI for Cluster management and app deployment with Helm Charts. I use metallb as bare metal load balancer, kube vip for service balance and traefik as ingress controller. Proxmox storage is a zfs RAID 10 so all my VM data is safe even if 1 or 2 disks fail (unless they are from the same pair).

Rancher UI

Rancher UI

What do I deploy on my Cluster?

Currently I have deployed Minio (S3 compatible storage) for my project, I deployed it with Operator and 1 Tennant that has 4 drives.

Minio Console

Minio Console

I also host wiki Outline as an alternative to Notion where I document all my work and sketch up system designs. Next steps would be to use ArgoCD to setup build system so my project can be built for Mac, Windows & Linux.

Outline

Outline

I plan to use Zenoh for my next big project, stay tuned!