Home Lab: Nextcloud on Kubernetes with S3 as primary storage — Part 2

Infrastructure May 7, 2021

This guide is going to walk you through setting up nextcloud with an S3 backend on Kubernetes. In this example, I will be using MicroK8s and Minio for self-hosted S3. If you do not already have a Kubernetes Cluster setup, get one from a cloud provider. If you would like to set up your own bare-metal cluster, you can refer my single node MicroK8s guide here, or the multi-node HA Cluster Guide using kubeadm here.

Configure DNS

We begin by adding an A record for a subdomain pointing back to our ingress IP address. This would be your LoadBalancer Public IP if you are hosted in the cloud, or if you are hosted at home behind NAT, you would point the sub-domain to your public IP. The exact steps for This would vary between providers, so I would suggest checking out the documentation or guides provided by your DNS provider.

For this guide, I will be using drive.example.com

Deployment in Kubernetes

On your client machine, make sure you have helm installed and are connected to your Kubernetes cluster.

Add the Helm repository

Add the nextcloud helm repo and update your charts.

helm repo add nextcloud https://nextcloud.github.io/helm/
helm repo update

Prepare values.yaml

Create a values.yaml file with nano values.yaml

Paste the below code sections in the values.yaml file one below the other. Change values as needed.

Cronjob

This is the configuration for the Kubernetes cronjob, this is used to curl the cron endpoint every 5 minutes to run background tasks. I would suggest running this every minute if your nextcloud server is going to have a lot of load.

cronjob:
  annotations: {}
  curlInsecure: false
  enabled: true
  failedJobsHistoryLimit: 5
  image: {}
  schedule: '*/5 * * * *'
  successfulJobsHistoryLimit: 2

Disable HPA

Since HorizontalPodAutoscaler requires a ReadWriteMany persistent volume, and I do not have it available on my cluster, I will be disabling HPA.

If your usage is under 10 concurrent users, you should be fine with 1 pod and all the other defaults listed in this guide.

hpa:
  enabled: false

Image

The official chart as of this post still uses nextcloud-19, we will be updating the image to the latest stable tag available.

image:
  repository: nextcloud
  tag: 20.0.9-apache
  pullPolicy: IfNotPresent

Database

We have 3 options for the database, an internal db powered by SQLite, MariaDB, and PostgreSQL. I prefer MariaDB, so I will be enabling that and disabling the other two.

internalDatabase:
  enabled: false
mariadb:
  db:
    name: nextcloud
    password: db-password
    user: nextcloud
  enabled: true
  master:
    persistence:
      accessMode: ReadWriteOnce
      enabled: true
      size: 8Gi
  replication:
    enabled: false
  rootUser:
    password: root-db-password
    forcePassword: true
postgresql:
  enabled: false

Metrics

This is optional, only enable this if you want to be able to monitor metrics with Prometheus.

metrics:
  enabled: true

Nextcloud config files

We can add custom config.php files to the nextcloud install here, and these will be merged and used.

nextcloud:
  configs:
    custom.config.php: |-
      <?php
      $CONFIG = array (
        'overwriteprotocol' => 'https',
        'overwrite.cli.url' => 'https://drive.example.com',
        'filelocking.enabled' => 'true',
        'loglevel' => '2',
        'enable_previews' => true,
        'trusted_domains' =>
           [
            'nextcloud',
            'drive.example.com'
           ]
      );

In this config file, I will be setting the overwriteprotocol and overwrite.cli.url keys to https and the DNS name drive.example.com.

filelocking.enabled enables file locking to prevent issues when multiple clients are accessing or updating the same file.

Redis config

This config file is for Redis caching, this improves performance by a lot and I would highly recommend leaving this section as it is.

    redis.config.php: |-
      <?php
      $CONFIG = array (
        'memcache.local' => '\\OC\\Memcache\\Redis',
        'memcache.distributed' => '\OC\Memcache\Redis',
        'memcache.locking' => '\OC\Memcache\Redis',
        'redis' => array(
          'host' => getenv('REDIS_HOST'),
          'port' => getenv('REDIS_HOST_PORT') ?: 6379,
          'password' => getenv('REDIS_HOST_PASSWORD')
        )
      );

This overrides the default config so that it takes the Redis password into account. The Helm chart installs Redis without a password by default, and that has caused authentication problems in the past.

S3 config

The last custom config file is for the S3 configuration.

    s3.config.php: |-
      <?php
      $CONFIG = array (
        'objectstore' => array(
          'class' => '\\OC\\Files\\ObjectStore\\S3',
          'arguments' => array(
            'bucket'     => 'bucket-name',
            'autocreate' => true,
            'key'        => 's3-access-key',
            'secret'     => 's3-secret-key',
            'region'     => 's3-region',
            'hostname'   => 's3-endpoint',
            'use_ssl'    => true,
            'use_path_style' => true
          )
        )
      );

Minio uses Path style accessing of buckets, so I have set usepathstyle to true. If you are using AWS S3, Wasabi or something else that allows you to access a bucket by the DNS name, set this to false. Fill in all the other details as per your S3 Provider.

Default configs

Here we disable the Redis config since we are overriding it above.

  defaultConfigs:
    .htaccess: true
    apache-pretty-urls.config.php: true
    apcu.config.php: true
    apps.config.php: true
    autoconfig.php: false
    redis.config.php: false
    smtp.config.php: true

Hostname and admin user

Here we specify the hostname and username/password for the admin user.

  host: drive.example.com
  password: admin-password
  username: admin-user

Email settings

Here you can configure any SMTP service to send email. This is useful as it allows nextcloud to send notifications and password reset functionality via email.

  mail:
    domain: domain.com
    enabled: false
    fromAddress: user
    smtp:
      authtype: LOGIN
      host: domain.com
      name: user
      password: pass
      port: 465
      secure: ssl

Persistence

This persistence config is for nextcloud itself, the user data is stored in S3, but Nextcloud still stores other stuff here.

persistence:
  accessMode: ReadWriteOnce
  annotations: {}
  enabled: true
  size: 8Gi

Redis deployment

Here we configure the helm chart to set up Redis with a password.

redis:
  enabled: true
  password: 'redis-password'
  usePassword: true

Replica count

And if you have decided to leave HPA off, add this line to make sure the deployment is configured with 1 replica only.

replicaCount: 1

Installation

Finally, to install Nextcloud, MariaDB and Redis, create a namespace with kubectl and use helm to create a release.

kubectl create ns nextcloud

helm upgrade --install --namespace nextcloud -f your-values.yaml nextcloud nextcloud/nextcloud

You will see an output like below

Release "nextcloud" has been upgraded. Happy Helming!
NAME: nextcloud
LAST DEPLOYED: Fri Apr 23 15:08:35 2021
NAMESPACE: nextcloud
STATUS: deployed

Traefik IngressRoute

Since my cluster is configured to use Traefik Proxy for ingress, which I highly recommend you do so. I will be creating an IngressRoute for Nextcloud.

For more information on how to set up Traefik, refer my previous post here.

Create a yaml file using nano nextcloud-ingress.yaml

Paste the following in it.

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: nextcloud-web
spec:
  entryPoints:
    - web
  routes:
  - kind: Rule
    match: Host(`drive.example.com`) && PathPrefix(`/`)
    services:
    - name: nextcloud
      namespace: nextcloud
      port: 8080
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: nextcloud-websecure
spec:
  entryPoints:
    - websecure
  routes:
  - kind: Rule
    match: Host(`drive.example.com`) && PathPrefix(`/`)
    services:
    - name: nextcloud
      namespace: nextcloud
      port: 8080
  tls:
    certResolver: dns-le

Apply the IngressRoute using

kubectl apply -f nextcloud-ingress.yaml -n nextcloud

Tags

Tanuj Ravi Rao

99% of the time my brain is thinking blah, meh, why, huh, WTF, food, and computers. The other 1% I’m usually asleep.

Great! You've successfully subscribed.
Great! Next, complete checkout for full access.
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.