Unverified Commit 5eb167d5 authored by Marijn Koesen's avatar Marijn Koesen
Browse files

First version of my proxmox k8s notes

parents
MIT License
Copyright (c) 2019 Marijn Koesen
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
# Installing kubernetes master on proxmox behind NAT using wireshark
**NOTE: These are my personal notes on what I needed to do to get kubernetes working in proxmox using LXC and a container, they may not be complete/work on your machine, PR's are welcome**
```
# <- execute inside the (proxmox) host
$ <- execite inside the container
```
## Step 1: Prepare the proxmox host
Ensure the following modules are loaded:
```
# cat /proc/sys/net/bridge/bridge-nf-call-iptables
```
Now make sure swapiness is on 0, so that swap will not be used, otherwise kubernetes will not start:
```
# cat /proc/sys/vm/swappiness
[should be 0]
```
Define the new one
```
# sysctl vm.swappiness=0
```
Disable SWAP, it'll take some times to clean the SWAP area
```
#s wapoff -a
```
Now wait for swap to be empty.
## Step 1: Creating the kubernetes container
1) Create a new container in proxmox, making sure to give it 0 swap, and make it a privileged container
2) Edit the config file `/etc/pve/lxc/$ID.conf` and add the following part:
```
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
lxc.mount.auto: "proc:rw sys:rw"
```
If you are using zfs on proxmos, make sure to create a ext4 volume, as zfs is not supported with kubeadm
See: https://github.com/corneliusweig/kubernetes-lxd
```
zfs create -V 50G mypool/my-dockervol
zfs create -V 5G mypool/my-kubeletvol
mkfs.ext4 /dev/zvol/mypool/my-dockervol
mkfs.ext4 /dev/zvol/mypool/my-kubeletvol
```
Then make sure to mount it inside of the container:
```
mp0: /dev/zvol/mypool/my-dockervol,mp=/var/lib/docker,backup=0
mp1: /dev/zvol/mypool/my-kubeletvol,mp=/var/lib/kubelet,backup=0
```
Next make sure `conntrack` is working in the container
```
$ sudo conntrack -L
```
Now we can setup the VPN we need, see documentation for wireguard to install
```
$ sudo add-apt-repository ppa:wireguard/wireguard
$ sudo apt-get update
$ sudo apt-get install wireguard
```
Create the config:
```
$ cat > /etc/wireguard/wg0.conf
[Interface]
Address = 10.0.0.1/32
ListenPort = 55555
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
PrivateKey = WNeaIBT40mN/asu9zXrPeSYA+4pFmZA9lUBvHTx+TG8=
[Peer]
# server2
PublicKey = NSWzZOIUHPqRxOxUmB/A7+Gs6oECYGojREvGs/ZEi2o=
AllowedIPs = 10.0.0.2/32
[Peer]
# server3
PublicKey = JhT41so2SiITMe2uqPoNB40kkwxRqyklWiILyhT1uVY=
AllowedIPs = 10.0.0.3/32
```
And start the vpn:
```
$ wg-quick up wg0
$ wg show
```
To make sure we start the vpn on boot, and to fix some other small issues create the following rc.local file:
```
$ cat > /etc/rc.local
#!/bin/sh -e
# Kubeadm 1.15 needs /dev/kmsg to be there, but it's not in lxc, but we can just use /dev/console instead
# see: https://github.com/kubernetes-sigs/kind/issues/662
if [ ! -e /dev/kmsg ]; then
ln -s /dev/console /dev/kmsg
fi
# Make sure our VPN is setup so we can connect to the other nodes
wg-quick up wg0
# https://medium.com/@kvaps/run-kubernetes-in-lxc-container-f04aa94b6c9c
mount --make-rshared /' > /etc/rc.local
exit 0
```
Set the permissions and reboot
```
$ chmod +x /etc/rc.local
# sudo reboot
```
Now make sure we have booted properly:
```
$ ls -l /dev/kmsg
[this should exist]
$ wg show
[this should show wireguard is started]
```
Now we can start installing kubernetes:
```
$ apt-get update && apt-get install -y apt-transport-https curl
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
$ cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
$ apt-get update
$ apt-get install -y kubelet kubeadm kubectl
$ apt-mark hold kubelet kubeadm kubectl
```
To make sure kubernets connects using the correct ip, and will use the VPN to connect though, we must tell kubelet to use the vpn server ip:
```
$ echo "KUBELET_EXTRA_ARGS=--node-ip=10.0.0.1" >> /etc/default/kubelet
```
Now we can setup kubeadm:
Make sure to specify the pod-network-cidr and service-cidr if they are overlapping with your normal LAN network, which it is in my case, as I run 192.168.x.x on my proxmox LAN. Note that I add an extra apiserver-cert-extra-sans, so I can just connect to the api server from it's LAN ip too.
To make sure it will work we need to ignore all preflight errors, but it should work just fine.
```
$ kubeadm init --pod-network-cidr=10.250.0.0/16 --service-cidr=172.31.0.0/16 --apiserver-advertise-address 10.0.0.1 --apiserver-cert-extra-sans k8s.mydomain.com --apiserver-cert-extra-sans 192.168.1.13 --apiserver-cert-extra-sans 10.0.0.1 --ignore-preflight-errors=all
```
Next copy the kubectl config:
```
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
```
And apply calico config
```
$ curl https://docs.projectcalico.org/v3.8/manifests/calico.yaml -O
$ vim calico.yaml
Change:
IP_AUTODETECTION_METHOD: "interface=wg*"
CALICO_IPV4POOL_CIDR: "10.250.0.0/16"
```
The `IP_AUTODETECTION_METHOD` needs to be wg0 so that all nodes will connect through the VPN, that will make sure it's working fine behind NAT, and all will be secure.
The `CALICO_IPV4POOL_CIDR` is needed to make sure all pods are created in the correct network.
When this is correct we can apply the network config:
```
$ kubectl apply -f calico.yaml
```
Keep note of the join command, so we can add another node.
# Adding another node
1) Prepare the node, in this example 'server2' with wireguard
VPN on server2:
```
[Interface]
PrivateKey = wPMsKBkqbdz1WBx8MhYM7/GwzYd6U7DWuef1FoeUdkg=
Address = 10.0.0.2/32
[Peer]
PublicKey = 8q+JKbrXDs86lnBvAl4lx6QiCzgoOOaAc7jtjz/lFBM=
AllowedIPs = 10.0.0.0/24
Endpoint = 123.123.123.123:55555
PersistentKeepalive = 15
```
Start the vpn, and make sure master and server2 can connect to each other:
```
master > ping 10.0.0.2
server2 > ping 10.0.0.1
```
Now install `kubeadm` like before, and fix the `--node-ip` like we did before.
We can now join the node:
```
kubeadm join 10.0.0.1:6443 --token j9dg9i.u023uf023pr902u4 \
--discovery-token-ca-cert-hash sha256:bb69ce798968473041754992927ce3b8154b526485055a0bf9fdd34c2aa34944
```
# Making sure everything is ok
Couple of things we can do to make sure it's all ok, first check all nodes are ready and using the vpn ip:
```
$ kubectl get node -o wide
[email protected]:~# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master Ready master 9h v1.15.2 10.0.0.1 <none> Ubuntu 18.04 LTS 4.15.18-13-pve docker://19.3.1
server2 Ready <none> 9h v1.15.3 10.0.0.2 <none> CentOS Linux 7 (Core) 3.10.0-957.27.2.el7.x86_64 docker://1.13.1
```
If the internal ip's are wrong, check your calico settings, beware that you might need to recreat the cluster using `kubeadm reset` first, as calico mentions changing the config later might not have effect.
Also make sure calico is fine:
```
$ curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.8.2/calicoctl
$ chmod +x calicoctl
$ DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config ./calicoctl get node -o wide
NAME ASN IPV4 IPV6
k8s (64512) 10.0.0.1/32
server2 (64512) 10.0.0.2/32
$ DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config ./calicoctl node status
Calico process is running.
IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+-------------------+-------+----------+-------------+
| 10.0.0.2 | node-to-node mesh | up | 07:22:50 | Established |
+--------------+-------------------+-------+----------+-------------+
```
If it's not make sure calico settings are correct and firewall is ok:
```
master > $ nc -v 10.0.0.3 179
Connection to 10.0.0.3 179 port [tcp/bgp] succeeded!
```
After this you should have a k8s cluster running, from within proxmox.
References:
[1] https://medium.com/@kvaps/run-kubernetes-in-lxc-container-f04aa94b6c9c
[2] https://gist.github.com/kvaps/25f730e0ec39dd2e5749fb6b020e71fc
[3] https://stackoverflow.com/questions/55813994/install-and-create-a-kubernetes-cluster-on-lxc-proxmox
[4] https://github.com/corneliusweig/kubernetes-lxd
[5] https://www.wireguard.com/install/#ubuntu-module-tools
[6] https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
[7] https://blog.lbdg.me/proxmox-best-performance-disable-swappiness/
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment