edp-doc/docs/technical-documentation/solution/scenarios/local-development/registry-mirror-and-ache/3-registry-mirror-and-cache-proxy-hacks.md

# Hacks

This documentation describes how docker/OCI image pulls on a local linux box can be configured to connect to mirrors or pull through cache proxies.

The audience is developers who want to have faster or more reliable pulls, and want to avoid rate limits from external registries.

This part is called 'hacks' and describes some more hands-on components and investigations on the command line.


## Create an own registry mirror to test a kind mirror setting

May be you don't have or need a mirror, but you would like to run all sceanrios of part 2 and thus need a local mirror.
Or you would like to investigate the handshaking between mirror and cache and thus need the logs of the mirror. 

```bash
# the name of our mirror
MIRROR_NAME=registry.docker.io.mirror.test 
# the mirror will be accessable by its host name in the kind network
DOCKER_KIND_NETWORK=kind
```

## The registry needs TLS 

```bash
# create a temporary directory
mkdir registry-certs
```

```bash
# cert config
cat <<EOF>openssl-${MIRROR_NAME}.cnf
[req]
default_bits        = 2048
default_keyfile     = domain.key
distinguished_name  = req_distinguished_name
x509_extensions     = v3_ca
req_extensions      = v3_ca
prompt              = no

[req_distinguished_name]
countryName         = DE
stateOrProvinceName = SomeState
localityName        = SomeCity
organizationName    = MyCompany
organizationalUnitName = IT
commonName          = ${MIRROR_NAME}

[v3_ca]
subjectAltName      = @alt_names

[alt_names]
DNS.1 = ${MIRROR_NAME}
EOF
```

```bash
# create self signed cert
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout registry-certs/${MIRROR_NAME}.key -out registry-certs/${MIRROR_NAME}.crt -config openssl-${MIRROR_NAME}.cnf
```

### Now run the registry

```bash
# run registry as mirror
docker run -d \
    --name ${MIRROR_NAME} \
    --network $DOCKER_KIND_NETWORK \
    -p 443:443 \
    -v $(pwd)/registry-certs:/certs \
    -e REGISTRY_HTTP_ADDR=0.0.0.0:443 \
    -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/${MIRROR_NAME}.crt \
    -e REGISTRY_HTTP_TLS_KEY=/certs/${MIRROR_NAME}.key \
    -e REGISTRY_PROXY_REMOTEURL="https://registry-1.docker.io" \
    registry:2
```

### Next run the kind cluster

```bash
# create kind cluster
cat <<EOF | kind create cluster --name cluster-with-registry-mirror --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri"]
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["${MIRROR_NAME}"]
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."${MIRROR_NAME}".tls]
          insecure_skip_verify = true
EOF
```

### Log the registry and do a deployment

```bash
# in another terminal
docker logs -f ${MIRROR_NAME}
```

```bash
# check images in the cluster before deployment
docker exec -it cluster-with-registry-mirror-control-plane crictl image ls

# do deployment
kubectl run busybox --image=busybox -- /bin/sh -c "sleep 3600"

# check images in the cluster again, you should see busybox
docker exec -it cluster-with-registry-mirror-control-plane crictl image ls
```

## journalctl

You also can check the containerd logs:

```bash
docker exec -it cluster-with-registry-mirror-control-plane journalctl -u containerd
```

See also:
* Logging variants: https://www.baeldung.com/ops/containerd-check-logs
* Monitoring containerd: https://collabnix.com/monitoring-containerd/

### debug journalctl

* https://gvisor.dev/docs/
* https://gvisor.dev/docs/user_guide/containerd/configuration/

```bash
cat <<EOF | kind create cluster --name cluster-with-registry-mirror --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
  [debug]
    level="debug"
  [plugins."io.containerd.grpc.v1.cri"]
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["${MIRROR_NAME}"]
EOF
```

## Integrate registry proxy

If you have a running registry proxy which also proxies the mirror, e.g. started like

```bash
CACHE_PROXY_NAME=docker_registry_proxy

docker run -itd \
    --restart always \
    --name $CACHE_PROXY_NAME \
    --network $DOCKER_KIND_NETWORK \
    --hostname $CACHE_PROXY_NAME \
    -p 0.0.0.0:${HOST_PORT:-3128}:3128 \
    -e ENABLE_MANIFEST_CACHE=true \
    -v $(pwd)/docker_mirror_cache:/docker_mirror_cache \
    -v $(pwd)/docker_mirror_certs:/ca \
    -e REGISTRIES="$MIRROR_NAME k8s.gcr.io gcr.io quay.io docker.elastic.co" \
    rpardini/docker-registry-proxy:0.6.2
```

then you need to make the proxy aware of the mirror's certificate.

### set mirror ca in proxy

The proxy is ssl veryfying upstreams. So we need to place the ca of the mirror.

```bash
docker cp registry-certs/registry-1.docker.io.mirror.test.crt docker_registry_proxy:/
docker exec -it docker_registry_proxy bash -c 'cat /registry-1.docker.io.mirror.test.crt >> /etc/ssl/certs/ca-certificates.crt'
docker exec -it docker_registry_proxy bash -c 'kill -SIGHUP $(cat /run/nginx.pid)'

```