Skip to content

Home Cluster

This repository is my home Kubernetes cluster in a declarative state. Flux watches my cluster folder and makes the changes to my cluster based on the YAML manifests.

Feel free to open a Github issue or join the k8s@home Discord if you have any questions.

This repository is built off the k8s-at-home/template-cluster-k3s repository.

Cluster setup

My cluster is k3s provisioned overtop Ubuntu 21.04 using the Ansible galaxy role ansible-role-k3s. This is a semi hyper-converged cluster, workloads and block storage are sharing the same available resources on my nodes while I have a separate server for (NFS) file storage.

See my ansible directory for my playbooks and roles.

Cluster components

  • calico: For internal cluster networking using BGP configured on Opnsense.
  • rook-ceph: Provides persistent volumes, allowing any application to consume RBD block storage.
  • Mozilla SOPS: Encrypts secrets which is safe to store - even to a public repository.
  • external-dns: Creates DNS entries in a separate coredns deployment which is backed by my clusters etcd deployment.
  • cert-manager: Configured to create TLS certs for all ingress services automatically using LetsEncrypt.
  • kube-vip: HA solution for Kubernetes control plane
  • Kasten: Data backup and recovery

Repository structure

The Git repository contains the following directories under cluster and are ordered below by how Flux will apply them.

  • base directory is the entrypoint to Flux
  • crds directory contains custom resource definitions (CRDs) that need to exist globally in your cluster before anything else exists
  • core directory (depends on crds) are important infrastructure applications (grouped by namespace) that should never be pruned by Flux
  • apps directory (depends on core) is where your common applications (grouped by namespace) could be placed, Flux will prune resources here if they are not tracked by Git anymore
./cluster
├── ./apps
├── ./base
├── ./core
└── ./crds

Automate all the things!

Hardware

Device Count OS Disk Size Data Disk Size Ram Purpose
Intel NUC8i3BEK 3 256GB NVMe N/A 16GB k3s Masters (embedded etcd)
Intel NUC8i5BEH 1 240GB SSD 1TB NVMe (rook-ceph) 32GB k3s Workers
Intel NUC8i7BEH 2 240GB SSD 1TB NVMe (rook-ceph) 32GB k3s Workers
PowerEdge T340 1 120GB SSD 8x12TB RAIDz2 32GB Shared file storage

Tools

Tool Purpose
direnv Sets environment variable based on present working directory
go-task Alternative to makefiles, who honestly likes that?
pre-commit Enforce code consistency and verifies no secrets are pushed
stern Tail logs in Kubernetes

Thanks

A lot of inspiration for my cluster came from the people that have shared their clusters over at awesome-home-kubernetes