First version of my proxmox k8s notes
parents
LICENSE
0 → 100644
README.md
0 → 100644
# Installing kubernetes master on proxmox behind NAT using wireshark | ||
**NOTE: These are my personal notes on what I needed to do to get kubernetes working in proxmox using LXC and a container, they may not be complete/work on your machine, PR's are welcome** | ||
``` | ||
# <- execute inside the (proxmox) host | ||
$ <- execite inside the container | ||
``` | ||
## Step 1: Prepare the proxmox host | ||
Ensure the following modules are loaded: | ||
``` | ||
# cat /proc/sys/net/bridge/bridge-nf-call-iptables | ||
``` | ||
Now make sure swapiness is on 0, so that swap will not be used, otherwise kubernetes will not start: | ||
``` | ||
# cat /proc/sys/vm/swappiness | ||
[should be 0] | ||
``` | ||
Define the new one | ||
``` | ||
# sysctl vm.swappiness=0 | ||
``` | ||
Disable SWAP, it'll take some times to clean the SWAP area | ||
``` | ||
#s wapoff -a | ||
``` | ||
Now wait for swap to be empty. | ||
## Step 1: Creating the kubernetes container | ||
1) Create a new container in proxmox, making sure to give it 0 swap, and make it a privileged container | ||
2) Edit the config file `/etc/pve/lxc/$ID.conf` and add the following part: | ||
``` | ||
lxc.apparmor.profile: unconfined | ||
lxc.cgroup.devices.allow: a | ||
lxc.cap.drop: | ||
lxc.mount.auto: "proc:rw sys:rw" | ||
``` | ||
If you are using zfs on proxmos, make sure to create a ext4 volume, as zfs is not supported with kubeadm | ||
See: https://github.com/corneliusweig/kubernetes-lxd | ||
``` | ||
zfs create -V 50G mypool/my-dockervol | ||
zfs create -V 5G mypool/my-kubeletvol | ||
mkfs.ext4 /dev/zvol/mypool/my-dockervol | ||
mkfs.ext4 /dev/zvol/mypool/my-kubeletvol | ||
``` | ||
Then make sure to mount it inside of the container: | ||
``` | ||
mp0: /dev/zvol/mypool/my-dockervol,mp=/var/lib/docker,backup=0 | ||
mp1: /dev/zvol/mypool/my-kubeletvol,mp=/var/lib/kubelet,backup=0 | ||
``` | ||
Next make sure `conntrack` is working in the container | ||
``` | ||
$ sudo conntrack -L | ||
``` | ||
Now we can setup the VPN we need, see documentation for wireguard to install | ||
``` | ||
$ sudo add-apt-repository ppa:wireguard/wireguard | ||
$ sudo apt-get update | ||
$ sudo apt-get install wireguard | ||
``` | ||
Create the config: | ||
``` | ||
$ cat > /etc/wireguard/wg0.conf | ||
[Interface] | ||
Address = 10.0.0.1/32 | ||
ListenPort = 55555 | ||
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE | ||
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE | ||
PrivateKey = WNeaIBT40mN/asu9zXrPeSYA+4pFmZA9lUBvHTx+TG8= | ||
[Peer] | ||
# server2 | ||
PublicKey = NSWzZOIUHPqRxOxUmB/A7+Gs6oECYGojREvGs/ZEi2o= | ||
AllowedIPs = 10.0.0.2/32 | ||
[Peer] | ||
# server3 | ||
PublicKey = JhT41so2SiITMe2uqPoNB40kkwxRqyklWiILyhT1uVY= | ||
AllowedIPs = 10.0.0.3/32 | ||
``` | ||
And start the vpn: | ||
``` | ||
$ wg-quick up wg0 | ||
$ wg show | ||
``` | ||
To make sure we start the vpn on boot, and to fix some other small issues create the following rc.local file: | ||
``` | ||
$ cat > /etc/rc.local | ||
#!/bin/sh -e | ||
# Kubeadm 1.15 needs /dev/kmsg to be there, but it's not in lxc, but we can just use /dev/console instead | ||
# see: https://github.com/kubernetes-sigs/kind/issues/662 | ||
if [ ! -e /dev/kmsg ]; then | ||
ln -s /dev/console /dev/kmsg | ||
fi | ||
# Make sure our VPN is setup so we can connect to the other nodes | ||
wg-quick up wg0 | ||
# https://medium.com/@kvaps/run-kubernetes-in-lxc-container-f04aa94b6c9c | ||
mount --make-rshared /' > /etc/rc.local | ||
exit 0 | ||
``` | ||
Set the permissions and reboot | ||
``` | ||
$ chmod +x /etc/rc.local | ||
# sudo reboot | ||
``` | ||
Now make sure we have booted properly: | ||
``` | ||
$ ls -l /dev/kmsg | ||
[this should exist] | ||
$ wg show | ||
[this should show wireguard is started] | ||
``` | ||
Now we can start installing kubernetes: | ||
``` | ||
$ apt-get update && apt-get install -y apt-transport-https curl | ||
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - | ||
$ cat <<EOF >/etc/apt/sources.list.d/kubernetes.list | ||
deb https://apt.kubernetes.io/ kubernetes-xenial main | ||
EOF | ||
$ apt-get update | ||
$ apt-get install -y kubelet kubeadm kubectl | ||
$ apt-mark hold kubelet kubeadm kubectl | ||
``` | ||
To make sure kubernets connects using the correct ip, and will use the VPN to connect though, we must tell kubelet to use the vpn server ip: | ||
``` | ||
$ echo "KUBELET_EXTRA_ARGS=--node-ip=10.0.0.1" >> /etc/default/kubelet | ||
``` | ||
Now we can setup kubeadm: | ||
Make sure to specify the pod-network-cidr and service-cidr if they are overlapping with your normal LAN network, which it is in my case, as I run 192.168.x.x on my proxmox LAN. Note that I add an extra apiserver-cert-extra-sans, so I can just connect to the api server from it's LAN ip too. | ||
To make sure it will work we need to ignore all preflight errors, but it should work just fine. | ||
``` | ||
$ kubeadm init --pod-network-cidr=10.250.0.0/16 --service-cidr=172.31.0.0/16 --apiserver-advertise-address 10.0.0.1 --apiserver-cert-extra-sans k8s.mydomain.com --apiserver-cert-extra-sans 192.168.1.13 --apiserver-cert-extra-sans 10.0.0.1 --ignore-preflight-errors=all | ||
``` | ||
Next copy the kubectl config: | ||
``` | ||
$ mkdir -p $HOME/.kube | ||
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config | ||
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config | ||
``` | ||
And apply calico config | ||
``` | ||
$ curl https://docs.projectcalico.org/v3.8/manifests/calico.yaml -O | ||
$ vim calico.yaml | ||
Change: | ||
IP_AUTODETECTION_METHOD: "interface=wg*" | ||
CALICO_IPV4POOL_CIDR: "10.250.0.0/16" | ||
``` | ||
The `IP_AUTODETECTION_METHOD` needs to be wg0 so that all nodes will connect through the VPN, that will make sure it's working fine behind NAT, and all will be secure. | ||
The `CALICO_IPV4POOL_CIDR` is needed to make sure all pods are created in the correct network. | ||
When this is correct we can apply the network config: | ||
``` | ||
$ kubectl apply -f calico.yaml | ||
``` | ||
Keep note of the join command, so we can add another node. | ||
# Adding another node | ||
1) Prepare the node, in this example 'server2' with wireguard | ||
VPN on server2: | ||
``` | ||
[Interface] | ||
PrivateKey = wPMsKBkqbdz1WBx8MhYM7/GwzYd6U7DWuef1FoeUdkg= | ||
Address = 10.0.0.2/32 | ||
[Peer] | ||
PublicKey = 8q+JKbrXDs86lnBvAl4lx6QiCzgoOOaAc7jtjz/lFBM= | ||
AllowedIPs = 10.0.0.0/24 | ||
Endpoint = 123.123.123.123:55555 | ||
PersistentKeepalive = 15 | ||
``` | ||
Start the vpn, and make sure master and server2 can connect to each other: | ||
``` | ||
master > ping 10.0.0.2 | ||
server2 > ping 10.0.0.1 | ||
``` | ||
Now install `kubeadm` like before, and fix the `--node-ip` like we did before. | ||
We can now join the node: | ||
``` | ||
kubeadm join 10.0.0.1:6443 --token j9dg9i.u023uf023pr902u4 \ | ||
--discovery-token-ca-cert-hash sha256:bb69ce798968473041754992927ce3b8154b526485055a0bf9fdd34c2aa34944 | ||
``` | ||
# Making sure everything is ok | ||
Couple of things we can do to make sure it's all ok, first check all nodes are ready and using the vpn ip: | ||
``` | ||
$ kubectl get node -o wide | ||
[email protected]:~# kubectl get node -o wide | ||
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME | ||
master Ready master 9h v1.15.2 10.0.0.1 <none> Ubuntu 18.04 LTS 4.15.18-13-pve docker://19.3.1 | ||
server2 Ready <none> 9h v1.15.3 10.0.0.2 <none> CentOS Linux 7 (Core) 3.10.0-957.27.2.el7.x86_64 docker://1.13.1 | ||
``` | ||
If the internal ip's are wrong, check your calico settings, beware that you might need to recreat the cluster using `kubeadm reset` first, as calico mentions changing the config later might not have effect. | ||
Also make sure calico is fine: | ||
``` | ||
$ curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.8.2/calicoctl | ||
$ chmod +x calicoctl | ||
$ DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config ./calicoctl get node -o wide | ||
NAME ASN IPV4 IPV6 | ||
k8s (64512) 10.0.0.1/32 | ||
server2 (64512) 10.0.0.2/32 | ||
$ DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config ./calicoctl node status | ||
Calico process is running. | ||
IPv4 BGP status | ||
+--------------+-------------------+-------+----------+-------------+ | ||
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | | ||
+--------------+-------------------+-------+----------+-------------+ | ||
| 10.0.0.2 | node-to-node mesh | up | 07:22:50 | Established | | ||
+--------------+-------------------+-------+----------+-------------+ | ||
``` | ||
If it's not make sure calico settings are correct and firewall is ok: | ||
``` | ||
master > $ nc -v 10.0.0.3 179 | ||
Connection to 10.0.0.3 179 port [tcp/bgp] succeeded! | ||
``` | ||
After this you should have a k8s cluster running, from within proxmox. | ||
References: | ||
[1] https://medium.com/@kvaps/run-kubernetes-in-lxc-container-f04aa94b6c9c | ||
[2] https://gist.github.com/kvaps/25f730e0ec39dd2e5749fb6b020e71fc | ||
[3] https://stackoverflow.com/questions/55813994/install-and-create-a-kubernetes-cluster-on-lxc-proxmox | ||
[4] https://github.com/corneliusweig/kubernetes-lxd | ||
[5] https://www.wireguard.com/install/#ubuntu-module-tools | ||
[6] https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ | ||
[7] https://blog.lbdg.me/proxmox-best-performance-disable-swappiness/ |
Please register or sign in to comment