stacks_adding_alloy/template/stacks/local-backup

Local Backup with Velero and Minio

This is example is adapted from the original icpbuilder stack.

The two significant changes from the original were made:

• disabled hostPath mount to persist backups within kind, since backups do not work sufficiently in this example due to PVC issues, see below.
• renamed minio namespace to minio-backup so it does not collide with other minio examples.

Within kind, it can only backup kubernetes objects. Data from PVC's is skipped, see below why.

Velero requires some compatible storage providers as its backup target. This local installation uses MinIO as an example. However, MinIO is not officially supported by Velero but works due to S3 compatibility.

The current setup does NOT persist backups but stores them in MinIO's PVCs. Proper backups should configure external storage, see Supported Providers.

Installation

The stack is installed as part of the ./example.sh run.

In order to persist a local backup you have to mount a local directory within main.go:

nodes:
  - role: control-plane
    extraMounts:
      - hostPath: /some/path/backup # replace with your own path
        containerPath: /backup

Kind creates the directory on the host but you might have to adjust the permissions, otherwise the minio pod fails to start.

Using it

After the installation velero and minio should be visible in ArgoCD.

During the installation credentials for minio are generated and shared with velero. You can access them manually:

kubectl -n minio-backup get secret root-creds -o go-template='{{ range $key, $value := .data }}{{ printf "%s: %s\n" $key ($value | base64decode) }}{{ end }}'
# example output
# rootPassword: aKKZzLnyry6OYZts17vMTf32H5ghFL4WYgu6bHujm
# rootUser: ge8019yksArb7BICt3MLY9

A bucket in minio was created and velero uses it for its backups by default, see helm values.yaml files.

Backup and Restore

Backups and subsequent restores can be scheduled by either using the velero cli or by creating CRD objects.

Check the ./demo directory for equivalent CRD manifests.

Create a backup of the backstage namespace, see schedule task for more permanent setups:

velero backup create backstage-backup --include-namespaces backstage

There are more options to create a fine granular backup and to set the backup storage. See velero's docs for details.

Check the backup with:

velero backup get

To get more details on the backup you need to be able to connect to velero's backup storage, i.e. minio. Using kubefwd here helps a lot (this is not necessary for restore).

kubefwd services -n minio-backup

More details with describe and logs:

velero backup describe backstage-backup --details
velero backup logs backstage-backup

Restore the backup into the original namespace, you might want to delete the existing namespace beforehand:

kubectl delete namespace backstage
velero restore create --from-backup backstage-backup

When restoring, velero does not replace existing objects in the backup target.

ArgoCD does pickup on the changes and also validates that the backup is in sync.

Issues with Persistent Volumes

Velero has no issue to backup kubernetes objects like Deployments, ConfigMaps, etc. since they are just yaml/json definitions. Volumes containing data are, however, more complex. The preferred type of backup are kubernetes' VolumeSnapshots as they consistently store the state of a volume at a given point in time in an atomic action. Those snapshots live within the cluster and are subsequently downloaded into one of velero's storage backends for safekeeping.

However, VolumeSnapshots are only possible on storage backends that support them via CSI drivers. Backends like nfs or hostPath do NOT support them. Here, velero uses an alternative method called File System Backups. In essence, this a simple copy operation based on the file system. Even though this uses more sophisticated tooling under the hood, i.e. kopia, it is not possible to create a backup in an atomic transaction. Thus, the resulting backup might be inconsistent.

Furthermore, for file system backups to work velero installs a node-agent as a DaemonSet on each Kubernetes node. The agent is aware of the node's internal storage and accesses the directories on the host directly to copy the files. This is not supported for hostPath volumes as they mount an arbitrary path on the host. In theory, a backup is possible but due extra config and security considerations intentionally skipped. Kind's local-path provider storage uses a hostPath and is thus not supported for any kind of backup.

TODOs

• The MinIO -backup installation is only intended as an example and must either be configured properly or replaced.

• The current example does not automatically schedule backups.

• velero chart must be properly parameterized