stacks_adding_alloy/template/stacks/local-backup/README.md

# Local Backup with Velero and Minio

This is example is adapted from the original icpbuilder stack.

The two significant changes from the original were made:

* disabled `hostPath` mount to persist backups within kind, since backups do not work sufficiently in this example due to PVC issues, see below.
* renamed `minio` namespace to `minio-backup` so it does not collide with other minio examples.

Within kind, it can only backup kubernetes objects. Data from PVC's is skipped, see below why.

[Velero](https://velero.io/) requires some compatible storage providers as its backup target. This local installation uses [MinIO](https://min.io/) as an example.
However, MinIO is not officially supported by Velero but works due to S3 compatibility.

The current setup does NOT persist backups but stores them in MinIO's PVCs. Proper backups should configure external storage, see [Supported Providers](https://velero.io/docs/main/supported-providers/).

## Installation

The stack is installed as part of the `./example.sh` run.

In order to persist a local backup you have to mount a local directory within `main.go`:

```yaml
nodes:
  - role: control-plane
    extraMounts:
      - hostPath: /some/path/backup # replace with your own path
        containerPath: /backup
```

Kind creates the directory on the host but you might have to adjust the permissions, otherwise the minio pod fails to start.

## Using it

After the installation velero and minio should be visible in ArgoCD.

During the installation credentials for minio are generated and shared with velero. You can access them manually:

```bash
kubectl -n minio-backup get secret root-creds -o go-template='{{ range $key, $value := .data }}{{ printf "%s: %s\n" $key ($value | base64decode) }}{{ end }}'
# example output
# rootPassword: aKKZzLnyry6OYZts17vMTf32H5ghFL4WYgu6bHujm
# rootUser: ge8019yksArb7BICt3MLY9
```

A bucket in minio was created and velero uses it for its backups by default, see helm `values.yaml` files.


### Backup and Restore

Backups and subsequent restores can be scheduled by either using the velero cli or by creating CRD objects.

Check the `./demo` directory for equivalent CRD manifests.

Create a backup of the backstage namespace, see `schedule` task for more permanent setups:

```shell
velero backup create backstage-backup --include-namespaces backstage
```

There are more options to create a fine granular backup and to set the backup storage.
See velero's docs for details.

Check the backup with:
```shell
velero backup get
```

To get more details on the backup you need to be able to connect to velero's backup storage, i.e. minio.
Using `kubefwd` here helps a lot (this is not necessary for restore).

```shell
kubefwd services -n minio-backup
```

More details with `describe` and `logs`:

```shell
velero backup describe backstage-backup --details
velero backup logs backstage-backup
```

Restore the backup into the original namespace, you might want to delete the existing namespace beforehand:

```shell
kubectl delete namespace backstage
velero restore create --from-backup backstage-backup
```

When restoring, velero does not replace existing objects in the backup target.

ArgoCD does pickup on the changes and also validates that the backup is in sync.


## Issues with Persistent Volumes

Velero has no issue to backup kubernetes objects like Deployments, ConfigMaps, etc. since they are just yaml/json definitions.
Volumes containing data are, however, more complex. The preferred type of backup are kubernetes' VolumeSnapshots as they consistently store the state
of a volume at a given point in time in an atomic action. Those snapshots live within the cluster and are subsequently downloaded into one of velero's
storage backends for safekeeping.

However, VolumeSnapshots are only possible on storage backends that support them via CSI drivers.
Backends like `nfs` or `hostPath` do NOT support them. Here, velero uses an alternative method
called [File System Backups](https://velero.io/docs/main/file-system-backup/).
In essence, this a simple copy operation based on the file system. Even though
this uses more sophisticated tooling under the hood, i.e. kopia, it is not
possible to create a backup in an atomic transaction. Thus, the resulting backup
might be inconsistent.

Furthermore, for file system backups to work velero installs a node-agent as a
DaemonSet on each Kubernetes node. The agent is aware of the node's internal
storage and accesses the directories on the host directly to copy the files.
This is not supported for hostPath volumes as they mount an arbitrary path
on the host. In theory, a backup is possible but due extra config and security
considerations intentionally skipped. Kind's local-path provider storage uses
a hostPath and is thus not supported for any kind of backup.

## TODOs

* The MinIO -backup installation is only intended as an example and must either
be configured properly or replaced.

* The current example does not automatically schedule backups.

* velero chart must be properly parameterized