Static manifest generation uses kustomize instead of python (#8099)

* regenerate at 4.0.12

* bash for loop and static values files

* add .tool-versions

* fixup static manifests with kustomize instead of python

* remove spec.replicas where set

* generate manifests for all supported versions

* update docs

* remove all versions except default (1.20) for now

* update to 1.1.1/4.0.15
This commit is contained in:
Alastair Firth 2022-01-18 00:28:56 +01:00 committed by GitHub
parent a665a409da
commit d16e0dec48
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
32 changed files with 4566 additions and 4722 deletions

View file

@ -230,40 +230,20 @@ Promoting the images basically means that images, that were pushed to staging co
- Prepare to use a script to update the edit the static manifests and set the "image", "digest", "version" etc. fields to the desired value.
- This script depends on kustomize and helm. The versions are pinned in `hack/.tool-versions` and you can use [asdf](https://github.com/asdf-vm/asdf#asdf) to install them
- This script depends on python and a specific python package `pip3 install ruamel.yaml`
- Execute the script to update static manifests using that script [hack/generate-deploy-scripts.sh](https://github.com/kubernetes/ingress-nginx/blob/main/hack/generate-deploy-scripts.sh)
- Execute the script to update static manifests using that script [generate-deploy-scripts.sh](https://github.com/kubernetes/ingress-nginx/blob/main/hack/generate-deploy-scripts.sh)
- Open some of the manifests and check if the script worked properly
- Use grep -ir to search for any misses by the script or undesired changes
- The script should properly set the image and the digest fields to the desired tag and semver
- Manually fix one problem that the script can not take care of.
- This problem is wrong formatting of a snippet in the file [deploy-tls-termination.yaml](https://github.com/kubernetes/ingress-nginx/blob/main/deploy/static/provider/aws/deploy-tls-termination.yaml)
- In the configMap section, for the configMap named ingress-nginx-controller, the "configMap.data" spec has a snippet
- This snippet becomes a single line, formatted with the newline character "\n"
- That single line formatted with "\n" needs to be changed as it does not meet yaml requirements
- At the time of writing this doc, the 'configMap.data' spec is at line number 39.
- So editing begins at line 40 (at the time of writing this doc)
- Make that snippet look like this ;
```
data:
http-snippet:|
server{
listen 2443;
return 308 https://$host$request_uri;
}
```
### f. Edit the changelog
[Changelog.md](https://github.com/kubernetes/ingress-nginx/blob/main/Changelog.md)
- Each time a release is made, a new section is added to the Changelog.md file
- A new section in the Changelog.md file consists of 3 components listed below
@ -277,6 +257,7 @@ Promoting the images basically means that images, that were pushed to staging co
- One process to generate this list of PRs is already described above in step 4c. So if you are following this document, then you have done this already and very likely have retained the file containing the list of PRs, in the format that is needed.
### g. Edit the Documentation:
- Update the version in [docs/deploy/index.md](docs/deploy/index.md)
- Update Supported versions in the Support Versions table in the README.md

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,11 @@
# NOTE: kustomize is not supported. This file exists only to be able to reference it from bases.
# https://kubectl.docs.kubernetes.io/references/kustomize/bases/
#
# ```
# namespace: ingress-nginx
# bases:
# - github.com/kubernetes/ingress-nginx/deploy/static/provider/aws/nlb-with-tls-termination?ref=master
# ```
resources:
- deploy.yaml

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -55,8 +55,11 @@ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/cont
!!! info
The YAML manifest in the command above was generated with `helm template`, so you will end up with almost the same resources as if you had used Helm to install the controller.
If you are running an old version of Kubernetes (1.18 or earlier), please read
[this paragraph](#running-on-Kubernetes-versions-older-than-1.19) for specific instructions.
!!! attention
If you are running an old version of Kubernetes (1.18 or earlier), please read
[this paragraph](#running-on-Kubernetes-versions-older-than-1.19) for specific instructions.
Because of api deprecations, the default manifest may not work on your cluster.
Specific manifests for supported Kubernetes versions are available within a subfolder of each provider.
### Pre-flight check
@ -186,12 +189,13 @@ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/cont
##### TLS termination in AWS Load Balancer (NLB)
By default, TLS is terminated in the ingress controller. But it is also possible to terminate TLS in the Load Balancer. This section explains how to do that on AWS with using an NLB.
By default, TLS is terminated in the ingress controller. But it is also possible to terminate TLS in the Load Balancer. This section explains how to do that on AWS using an NLB.
1. Download the the [deploy-tls-termination.yaml](https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/aws/deploy-tls-termination.yaml) template:
```console
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/aws/deploy-tls-termination.yaml
```
1. Download the [deploy.yaml](https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/aws/nlb-with-tls-termination/deploy.yaml) template
```console
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/aws/nlb-with-tls-termination/deploy.yaml
```
2. Edit the file and change the VPC CIDR in use for the Kubernetes cluster:
```
@ -205,7 +209,7 @@ By default, TLS is terminated in the ingress controller. But it is also possible
4. Deploy the manifest:
```console
kubectl apply -f deploy-tls-termination.yaml
kubectl apply -f deploy.yaml
```
##### NLB Idle Timeouts

2
hack/.tool-versions Normal file
View file

@ -0,0 +1,2 @@
kustomize 4.1.3
helm 3.7.1

View file

@ -22,194 +22,51 @@ set -o errexit
set -o nounset
set -o pipefail
# for backwards compatibility, the default version of 1.20 is copied to the root of the variant
# with enough docs updates, this could be removed
# see # DEFAULT VERSION HANDLING
K8S_DEFAULT_VERSION=1.20
# K8S_TARGET_VERSIONS=("1.19" "1.20" "1.21" "1.22") TODO @afirth revert for #8000
K8S_TARGET_VERSIONS=("1.20")
DIR=$(cd $(dirname "${BASH_SOURCE}")/.. && pwd -P)
RELEASE_NAME=ingress-nginx
NAMESPACE=ingress-nginx
# clean
rm -rf ${DIR}/deploy/static/provider/*
NAMESPACE_VAR="
apiVersion: v1
kind: Namespace
metadata:
name: $NAMESPACE
labels:
app.kubernetes.io/name: $RELEASE_NAME
app.kubernetes.io/instance: ingress-nginx
"
TEMPLATE_DIR="${DIR}/hack/manifest-templates"
# Baremetal
OUTPUT_FILE="${DIR}/deploy/static/provider/baremetal/deploy.yaml"
cat << EOF | helm template $RELEASE_NAME ${DIR}/charts/ingress-nginx --namespace $NAMESPACE --values - | $DIR/hack/add-namespace.py $NAMESPACE > ${OUTPUT_FILE}
controller:
service:
type: NodePort
# each helm values file `values.yaml` under `hack/manifest-templates/provider` will be generated as provider/<provider>[/variant][/kube-version]/deploy.yaml
# TARGET is provider/<provider>[/variant]
TARGETS=$(dirname $(cd $DIR/hack/manifest-templates/ && find . -type f -name "values.yaml" ) | cut -d'/' -f2-)
for K8S_VERSION in "${K8S_TARGET_VERSIONS[@]}"
do
for TARGET in ${TARGETS}
do
TARGET_DIR="${TEMPLATE_DIR}/${TARGET}"
MANIFEST="${TEMPLATE_DIR}/common/manifest.yaml" # intermediate manifest
OUTPUT_DIR="${DIR}/deploy/static/${TARGET}/${K8S_VERSION}"
echo $OUTPUT_DIR
publishService:
enabled: false
EOF
mkdir -p ${OUTPUT_DIR}
cd ${TARGET_DIR}
helm template ingress-nginx ${DIR}/charts/ingress-nginx \
--values values.yaml \
--namespace ingress-nginx \
--kube-version ${K8S_VERSION} \
> $MANIFEST
kustomize --load-restrictor=LoadRestrictionsNone build . > ${OUTPUT_DIR}/deploy.yaml
rm $MANIFEST
cd ~-
# automatically generate the (unsupported) kustomization.yaml for each target
sed "s_{TARGET}_${TARGET}_" $TEMPLATE_DIR/static-kustomization-template.yaml > ${OUTPUT_DIR}/kustomization.yaml
echo "${NAMESPACE_VAR}
$(cat ${OUTPUT_FILE})" > ${OUTPUT_FILE}
# Cloud - generic
OUTPUT_FILE="${DIR}/deploy/static/provider/cloud/deploy.yaml"
cat << EOF | helm template $RELEASE_NAME ${DIR}/charts/ingress-nginx --namespace $NAMESPACE --values - | $DIR/hack/add-namespace.py $NAMESPACE > ${OUTPUT_FILE}
controller:
service:
type: LoadBalancer
externalTrafficPolicy: Local
EOF
echo "${NAMESPACE_VAR}
$(cat ${OUTPUT_FILE})" > ${OUTPUT_FILE}
# AWS - NLB
OUTPUT_FILE="${DIR}/deploy/static/provider/aws/deploy.yaml"
cat << EOF | helm template $RELEASE_NAME ${DIR}/charts/ingress-nginx --namespace $NAMESPACE --values - | $DIR/hack/add-namespace.py $NAMESPACE > ${OUTPUT_FILE}
controller:
service:
type: LoadBalancer
externalTrafficPolicy: Local
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
EOF
echo "${NAMESPACE_VAR}
$(cat ${OUTPUT_FILE})" > ${OUTPUT_FILE}
OUTPUT_FILE="${DIR}/deploy/static/provider/aws/deploy-tls-termination.yaml"
cat << EOF | helm template $RELEASE_NAME ${DIR}/charts/ingress-nginx --namespace $NAMESPACE --values - | $DIR/hack/add-namespace.py $NAMESPACE > ${OUTPUT_FILE}
controller:
service:
type: LoadBalancer
externalTrafficPolicy: Local
annotations:
# This example is for legacy in-tree service load balancer controller for AWS NLB,
# that has been phased out from Kubernetes mainline.
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX"
service.beta.kubernetes.io/aws-load-balancer-type: nlb
# Ensure the ELB idle timeout is less than nginx keep-alive timeout. By default,
# NGINX keep-alive is set to 75s. If using WebSockets, the value will need to be
# increased to '3600' to avoid any potential issues.
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
targetPorts:
http: tohttps
https: http
# Configures the ports the nginx-controller listens on
containerPort:
http: 80
https: 80
tohttps: 2443
config:
proxy-real-ip-cidr: XXX.XXX.XXX/XX
use-forwarded-headers: "true"
http-snippet: |
server {
listen 2443;
return 308 https://\$host\$request_uri;
}
EOF
echo "${NAMESPACE_VAR}
$(cat ${OUTPUT_FILE})" > ${OUTPUT_FILE}
# Kind - https://kind.sigs.k8s.io/docs/user/ingress/
OUTPUT_FILE="${DIR}/deploy/static/provider/kind/deploy.yaml"
cat << EOF | helm template $RELEASE_NAME ${DIR}/charts/ingress-nginx --namespace $NAMESPACE --values - | $DIR/hack/add-namespace.py $NAMESPACE > ${OUTPUT_FILE}
controller:
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
hostPort:
enabled: true
terminationGracePeriodSeconds: 0
service:
type: NodePort
watchIngressWithoutClass: true
nodeSelector:
ingress-ready: "true"
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Equal"
effect: "NoSchedule"
publishService:
enabled: false
extraArgs:
publish-status-address: localhost
EOF
# Digital Ocean
echo "${NAMESPACE_VAR}
$(cat ${OUTPUT_FILE})" > ${OUTPUT_FILE}
OUTPUT_FILE="${DIR}/deploy/static/provider/do/deploy.yaml"
cat << EOF | helm template $RELEASE_NAME ${DIR}/charts/ingress-nginx --namespace $NAMESPACE --values - | $DIR/hack/add-namespace.py $NAMESPACE > ${OUTPUT_FILE}
controller:
service:
type: LoadBalancer
externalTrafficPolicy: Local
annotations:
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
config:
use-proxy-protocol: "true"
admissionWebhooks:
timeoutSeconds: 29
EOF
# Scaleway
echo "${NAMESPACE_VAR}
$(cat ${OUTPUT_FILE})" > ${OUTPUT_FILE}
OUTPUT_FILE="${DIR}/deploy/static/provider/scw/deploy.yaml"
cat << EOF | helm template $RELEASE_NAME ${DIR}/charts/ingress-nginx --namespace $NAMESPACE --values - | $DIR/hack/add-namespace.py $NAMESPACE > ${OUTPUT_FILE}
controller:
service:
type: LoadBalancer
externalTrafficPolicy: Local
annotations:
service.beta.kubernetes.io/scw-loadbalancer-proxy-protocol-v2: "true"
config:
use-proxy-protocol: "true"
EOF
# Exoscale
echo "${NAMESPACE_VAR}
$(cat ${OUTPUT_FILE})" > ${OUTPUT_FILE}
OUTPUT_FILE="${DIR}/deploy/static/provider/exoscale/deploy.yaml"
cat << EOF | helm template $RELEASE_NAME ${DIR}/charts/ingress-nginx --namespace $NAMESPACE --values - | $DIR/hack/add-namespace.py $NAMESPACE > ${OUTPUT_FILE}
controller:
kind: DaemonSet
service:
type: LoadBalancer
externalTrafficPolicy: Local
annotations:
service.beta.kubernetes.io/exoscale-loadbalancer-name: "nginx-ingress-controller"
service.beta.kubernetes.io/exoscale-loadbalancer-description: "NGINX Ingress Controller load balancer"
service.beta.kubernetes.io/exoscale-loadbalancer-service-strategy: "source-hash"
service.beta.kubernetes.io/exoscale-loadbalancer-service-healthcheck-mode: "http"
service.beta.kubernetes.io/exoscale-loadbalancer-service-healthcheck-uri: "/"
service.beta.kubernetes.io/exoscale-loadbalancer-service-healthcheck-interval: "10s"
service.beta.kubernetes.io/exoscale-loadbalancer-service-healthcheck-timeout: "3s"
service.beta.kubernetes.io/exoscale-loadbalancer-service-healthcheck-retries: "1"
publishService:
enabled: true
EOF
echo "${NAMESPACE_VAR}
$(cat ${OUTPUT_FILE})" > ${OUTPUT_FILE}
# DEFAULT VERSION HANDLING
if [[ ${K8S_VERSION} = ${K8S_DEFAULT_VERSION} ]]
then
cp ${OUTPUT_DIR}/*.yaml ${OUTPUT_DIR}/../
sed -i "1s/^/#GENERATED FOR K8S ${K8S_VERSION}\n/" ${OUTPUT_DIR}/../deploy.yaml
rm -rf ${OUTPUT_DIR} # TODO @afirth remove for #8000 - this avoids the duplicate files for easier review of the build script changes
fi
done
done

View file

@ -0,0 +1,14 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- manifest.yaml
- namespace.yaml
patches:
- target:
group: apps
version: v1
kind: Deployment
patch: |-
- op: remove
path: /spec/replicas

View file

@ -0,0 +1,7 @@
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx

View file

@ -0,0 +1,4 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../common

View file

@ -0,0 +1,4 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../../common

View file

@ -0,0 +1,36 @@
# AWS NLB with TLS termination
controller:
service:
type: LoadBalancer
externalTrafficPolicy: Local
annotations:
# This example is for legacy in-tree service load balancer controller for AWS NLB,
# that has been phased out from Kubernetes mainline.
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX"
service.beta.kubernetes.io/aws-load-balancer-type: nlb
# Ensure the ELB idle timeout is less than nginx keep-alive timeout. By default,
# NGINX keep-alive is set to 75s. If using WebSockets, the value will need to be
# increased to '3600' to avoid any potential issues.
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
targetPorts:
http: tohttps
https: http
# Configures the ports the nginx-controller listens on
containerPort:
http: 80
https: 80
tohttps: 2443
config:
proxy-real-ip-cidr: XXX.XXX.XXX/XX
use-forwarded-headers: "true"
http-snippet: |
server {
listen 2443;
return 308 https://$host$request_uri;
}

View file

@ -0,0 +1,9 @@
# AWS - NLB
controller:
service:
type: LoadBalancer
externalTrafficPolicy: Local
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"

View file

@ -0,0 +1,4 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../common

View file

@ -0,0 +1,7 @@
# Baremetal
controller:
service:
type: NodePort
publishService:
enabled: false

View file

@ -0,0 +1,4 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../common

View file

@ -0,0 +1,4 @@
controller:
service:
type: LoadBalancer
externalTrafficPolicy: Local

View file

@ -0,0 +1,4 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../common

View file

@ -0,0 +1,11 @@
# Digital Ocean
controller:
service:
type: LoadBalancer
externalTrafficPolicy: Local
annotations:
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
config:
use-proxy-protocol: "true"
admissionWebhooks:
timeoutSeconds: 29

View file

@ -0,0 +1,4 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../common

View file

@ -0,0 +1,17 @@
# Exoscale
controller:
kind: DaemonSet
service:
type: LoadBalancer
externalTrafficPolicy: Local
annotations:
service.beta.kubernetes.io/exoscale-loadbalancer-name: "nginx-ingress-controller"
service.beta.kubernetes.io/exoscale-loadbalancer-description: "NGINX Ingress Controller load balancer"
service.beta.kubernetes.io/exoscale-loadbalancer-service-strategy: "source-hash"
service.beta.kubernetes.io/exoscale-loadbalancer-service-healthcheck-mode: "http"
service.beta.kubernetes.io/exoscale-loadbalancer-service-healthcheck-uri: "/"
service.beta.kubernetes.io/exoscale-loadbalancer-service-healthcheck-interval: "10s"
service.beta.kubernetes.io/exoscale-loadbalancer-service-healthcheck-timeout: "3s"
service.beta.kubernetes.io/exoscale-loadbalancer-service-healthcheck-retries: "1"
publishService:
enabled: true

View file

@ -0,0 +1,4 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../common

View file

@ -0,0 +1,24 @@
# Kind - https://kind.sigs.k8s.io/docs/user/ingress/
controller:
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
hostPort:
enabled: true
terminationGracePeriodSeconds: 0
service:
type: NodePort
watchIngressWithoutClass: true
nodeSelector:
ingress-ready: "true"
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Equal"
effect: "NoSchedule"
publishService:
enabled: false
extraArgs:
publish-status-address: localhost

View file

@ -0,0 +1,4 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../common

View file

@ -0,0 +1,9 @@
# Scaleway
controller:
service:
type: LoadBalancer
externalTrafficPolicy: Local
annotations:
service.beta.kubernetes.io/scw-loadbalancer-proxy-protocol-v2: "true"
config:
use-proxy-protocol: "true"

View file

@ -0,0 +1,11 @@
# NOTE: kustomize is not supported. This file exists only to be able to reference it from bases.
# https://kubectl.docs.kubernetes.io/references/kustomize/bases/
#
# ```
# namespace: ingress-nginx
# bases:
# - github.com/kubernetes/ingress-nginx/deploy/static/{TARGET}?ref=master
# ```
resources:
- deploy.yaml