Guide: Storage, Ingress and Web UIs for Kubernetes Cluster

Kubernetes Sep 29, 2020

Hello everybody, tansanrao here! This is a follow up on the previous guide about how to setup an HA Kubernetes Cluster. We will be dealing with Distributed Block Storage using Longhorn, Ingress Controller using Traefik, A Few useful Middleware Examples for Traefik and deploying the Kubernetes Dashboard.

Storage: Longhorn distributed block storage for Kubernetes

What is Longhorn?

Longhorn is a lightweight, reliable and easy-to-use distributed block storage system for Kubernetes.

Longhorn is free, open source software. Originally developed by Rancher Labs, it is now being developed as a sandbox project of the Cloud Native Computing Foundation.

Installing Longhorn

Install Longhorn by applying the following manifest with kubectl

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml

To monitor the progress of the installation, you can use this

kubectl get pods \
--namespace longhorn-system \
--watch

To check for successful deployment, see that all the pods show a ready and running state like so

NAME                                        READY     STATUS    RESTARTS   AGE
csi-attacher-6fdc77c485-8wlpg               1/1       Running   0          9d
csi-attacher-6fdc77c485-psqlr               1/1       Running   0          9d
csi-attacher-6fdc77c485-wkn69               1/1       Running   0          9d
csi-provisioner-78f7db7d6d-rj9pr            1/1       Running   0          9d
csi-provisioner-78f7db7d6d-sgm6w            1/1       Running   0          9d
csi-provisioner-78f7db7d6d-vnjww            1/1       Running   0          9d
engine-image-ei-6e2b0e32-2p9nk              1/1       Running   0          9d
engine-image-ei-6e2b0e32-s8ggt              1/1       Running   0          9d
engine-image-ei-6e2b0e32-wgkj5              1/1       Running   0          9d
longhorn-csi-plugin-g8r4b                   2/2       Running   0          9d
longhorn-csi-plugin-kbxrl                   2/2       Running   0          9d
longhorn-csi-plugin-wv6sb                   2/2       Running   0          9d
longhorn-driver-deployer-788984b49c-zzk7b   1/1       Running   0          9d
longhorn-manager-nr5rs                      1/1       Running   0          9d
longhorn-manager-rd4k5                      1/1       Running   0          9d
longhorn-manager-snb9t                      1/1       Running   0          9d
longhorn-ui-67b9b6887f-n7x9q                1/1       Running   0          9d

Dashboard: Kubernetes Web UI

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml

To protect your cluster data, Dashboard deploys with a minimal RBAC configuration by default. Currently, Dashboard only supports logging in with a Bearer Token. To create a token for this demo, create a ServiceAccount called admin-user.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
EOF

Next we create a ClusterRoleBinding between the ClusterRole cluster-admin and the ServiceAccount admin-user

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
EOF

Next we need to fetch a Bearer Token,

For Bash, do this:

kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')

For PowerShell, do this:

kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | sls admin-user | ForEach-Object { $_ -Split '\s+' } | Select -First 1)

It should output something like this

Name:         admin-user-token-v57nw
Namespace:    kubernetes-dashboard
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: admin-user
              kubernetes.io/service-account.uid: 0303243c-4040-4a58-8a47-849ee9ba79c1

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1066 bytes
namespace:  20 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXY1N253Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIwMzAzMjQzYy00MDQwLTRhNTgtOGE0Ny04NDllZTliYTc5YzEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.Z2JrQlitASVwWbc-s6deLRFVk5DWD3P_vjUFXsqVSY10pbjFLG4njoZwh8p3tLxnX_VBsr7_6bwxhWSYChp9hwxznemD5x5HLtjb16kI9Z7yFWLtohzkTwuFbqmQaMoget_nYcQBUC5fDmBHRfFvNKePh_vSSb2h_aYXa8GV5AcfPQpY7r461itme1EXHQJqv-SN-zUnguDguCTjD80pFZ_CmnSE1z9QdMHPB8hoB4V68gtswR1VLa6mSYdgPwCHauuOobojALSaMc3RH7MmFUumAgguhqAkX3Omqd3rJbYOMRuMjhANqd08piDC3aIabINX6gP5-Tuuw2svnV6NYQ

Now copy the token and store it somewhere safe, we will need it in a minute.

Accessing the Web UI

We create a proxy to our cluster using kubectl

kubectl proxy

kubectl will make Dashboard available at http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/.

The UI can only be accessed from the machine where the command is executed.

Web UI Login
Web UI Login

Enter the token from the previous step here to login.

Ingress: Traefik Proxy as Ingress Controller

We will be setting up Traefik with Let’s Encrypt with optional Cloudflare DNS Challenge Configuration

Installing Traefik via Helm Chart

Make sure you have Helm installed, then add the Traefik repo

helm repo add traefik https://helm.traefik.io/traefik

Update Helm Repos

helm repo update

Create a traefik-values.yaml file

nano traefik-values.yaml

Paste the following into the file

additionalArguments:
  - "[email protected]"
  - "--certificatesresolvers.http-le.acme.storage=/data/acme.json"
  - "--certificatesresolvers.http-le.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
  - "--certificatesResolvers.http-le.acme.httpchallenge=true"
  - "--certificatesResolvers.http-le.acme.httpchallenge.entrypoint=web"
  - "--api.insecure=true"
  - "--accesslog=true"
  - "--log.level=WARN"
  - "--serversTransport.insecureSkipVerify=true"

Change the email to match your valid email address for Let’s Encrypt. That will setup an ACME certResolver named http-le which will use the web entry point to perform an HTTP-01 challenge and issue certs.

To use Cloudflare DNS via certResolver named dns-le to perform a DNS-01 challenge you can use the values below.

additionalArguments:
  - "[email protected]"
  - "--certificatesresolvers.dns-le.acme.storage=/data/acme.json"
  - "--certificatesresolvers.dns-le.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
  - "--certificatesResolvers.dns-le.acme.dnschallenge=true"
  - "--certificatesResolvers.dns-le.acme.dnschallenge.provider=cloudflare"
  - "--api.insecure=true"
  - "--accesslog=true"
  - "--log.level=WARN"
  - "--serversTransport.insecureSkipVerify=true"
env:
  - name: CF_DNS_API_TOKEN
    valueFrom:
      secretKeyRef:
        name: cloudflare
        key: dns-token

In the Cloudflare version of the values file, we are loading the Cloudflare API token from a secret, let’s go ahead and create one.

In your Cloudflare Dashboard, navigate to your domain page. On the right, you’ll see a section called API. Click on get API Token

Get API Token
Get API Token

Then on the API Tokens Tab, click on Create Token

Api Tokens
Api Tokens

Use the Edit Zone DNS Template and add Edit permissions for Zone.Zone and Zone.DNS, then under Zone Resources, Include the zone for the domain you plan on using, in my case, I’ve chosen to grant access to all zones.

Token Config
Token Config

Save the Token as it’ll only be shown once.

Next we create a secret for the same using kubectl

kubectl create secret generic cloudflare \
  --from-literal=dns-token='<cloudflare_token_here>'

Now we are ready to install Traefik using Helm

helm install traefik traefik/traefik --values=traefik-values.yaml

Check for successful installation by running

kubectl get pods

Traefik should be visible with Ready Pods and a status of Running. The dashboard can be accessed by port forwarding through kubectl

kubectl port-forward $(kubectl get pods --selector "app.kubernetes.io/name=traefik" --output=name) 9000:9000

The dashboard will be available at http://localhost:9000/dashboard.

Basic Auth Middleware for Traefik

Create a file named auth-middleware.yaml and paste the following into it

# Declaring the user list
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: admin-auth
  namespace: default
spec:
  basicAuth:
    secret: authsecret

---
# Note: in a kubernetes secret the string (e.g. generated by htpasswd) must be base64-encoded first.
# To create an encoded user:password pair, the following command can be used:
# htpasswd -nb user password | openssl base64

apiVersion: v1
kind: Secret
metadata:
  name: authsecret
  namespace: default

data:
  users: 
    <base64_encoded_string>

Replace the line at the end with a base64-encoded string from htpasswd generated using this command

htpasswd -nb user password | openssl base64

Replace user and password with the username and password that you would like to use.

Apply the file using kubectl

kubectl apply -f auth-middleware.yaml

Middleware to add CORS and svc-wss-headers

CORS Middleware

Create a file named cors-middleware.yaml and paste the following

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: cors-allow-all
  namespace: default
spec:
  headers:
    accessControlAllowCredentials: true
    accessControlAllowMethods:
      - "*"
    accessControlAllowOriginList:
      - "*"
    accessControlMaxAge: 100
    addVaryHeader: true

It grants all origins permission to use all methods and credentials. Apply it with kubectl

kubectl apply -f cors-middleware.yaml

SVC-WSS Middleware

Create a file named wss-middleware.yaml and paste the following

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: svc-wss-headers
  namespace: default
spec:
  headers:
    customRequestHeaders:
      X-Forwarded-Proto: "https"

It adds https in the Forwarded-Photo header to force https WebSocketSecure connections. It is needed only sometimes to get Longhorn UI to play nicely with https in Traefik.

Apply it with kubectl

kubectl apply -f wss-middleware.yaml

Longhorn UI Router using IngressRoutes

Create a file named ingress-longhorn.yaml and add the following to it

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: longhorn-ui-ingress
  namespace: longhorn-system
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`longhorn.k8s.example.com`)
      kind: Rule
      services:
        - name: longhorn-frontend
          port: 80
          namespace: longhorn-system
      middlewares:
        - name: cors-allow-all
          namespace: default
        - name: admin-auth
          namespace: default
        - name: svc-wss-headers
          namespace: default
  tls:
    certResolver: http-le

Here we are creating an IngressRoute that matches the Hostname longhorn.k8s.example.com substitute it for whatever you are going to use. We are also chaining the 3 middleware we defined above to add core, was and basic auth to the route.

Apply it with kubectl

kubectl apply -f ingress-longhorn.yaml

Done, the UI should be up on the host url specified in the ingress route.

For any doubts, suggestions or issues, leave a comment below and subscribe to the newsletter for the latest posts delivered straight to your inbox! Follow me on Twitter & Instagram for behind the scenes and updates.

Tags

Tanuj Ravi Rao

99% of the time my brain is thinking blah, meh, why, huh, WTF, food, and computers. The other 1% I’m usually asleep.

Great! You've successfully subscribed.
Great! Next, complete checkout for full access.
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.