From 451a01bb0a2a470bf9ea8b44312769aa4f3bd51e Mon Sep 17 00:00:00 2001 From: Aarni Koskela Date: Wed, 2 May 2018 17:11:20 +0300 Subject: [PATCH 01/11] TLS.md: Move the TLS secret misc bit to the TLS document --- docs/user-guide/miscellaneous.md | 6 ------ docs/user-guide/tls.md | 18 ++++++++++++++++++ 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/docs/user-guide/miscellaneous.md b/docs/user-guide/miscellaneous.md index d8b06ce78..d184bc2b1 100644 --- a/docs/user-guide/miscellaneous.md +++ b/docs/user-guide/miscellaneous.md @@ -1,11 +1,5 @@ # Miscellaneous -## Conventions - -Anytime we reference a tls secret, we mean (x509, pem encoded, RSA 2048, etc). You can generate such a certificate with: -`openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ${KEY_FILE} -out ${CERT_FILE} -subj "/CN=${HOST}/O=${HOST}"` -and create the secret via `kubectl create secret tls ${CERT_NAME} --key ${KEY_FILE} --cert ${CERT_FILE}` - ## Requirements The default backend is a service which handles all url paths and hosts the nginx controller doesn't understand (i.e., all the requests that are not mapped with an Ingress). diff --git a/docs/user-guide/tls.md b/docs/user-guide/tls.md index 93db6499d..b8fdf88b5 100644 --- a/docs/user-guide/tls.md +++ b/docs/user-guide/tls.md @@ -9,6 +9,24 @@ - [Default TLS Version and Ciphers](#default-tls-version-and-ciphers) - [Legacy TLS](#legacy-tls) +## TLS Secrets + +Anytime we reference a TLS secret, we mean a PEM-encoded X.509, RSA (2048) secret. + +You can generate a self-signed certificate and private key with with: + +```bash +$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ${KEY_FILE} -out ${CERT_FILE} -subj "/CN=${HOST}/O=${HOST}"` +``` + +Then create the secret in the cluster via: + +```bash +kubectl create secret tls ${CERT_NAME} --key ${KEY_FILE} --cert ${CERT_FILE} +``` + +The resulting secret will be of type `kubernetes.io/tls`. + ## Default SSL Certificate NGINX provides the option to configure a server as a catch-all with [server_name](http://nginx.org/en/docs/http/server_names.html) for requests that do not match any of the configured server names. This configuration works without issues for HTTP traffic. From f65c8f0aaaa3cbcdd8cc685055969c8bb9e1cc30 Mon Sep 17 00:00:00 2001 From: Aarni Koskela Date: Wed, 2 May 2018 17:11:50 +0300 Subject: [PATCH 02/11] TLS.md: Clarify how to set --default-ssl-certificate --- docs/user-guide/tls.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/user-guide/tls.md b/docs/user-guide/tls.md index b8fdf88b5..c00c40259 100644 --- a/docs/user-guide/tls.md +++ b/docs/user-guide/tls.md @@ -31,7 +31,9 @@ The resulting secret will be of type `kubernetes.io/tls`. NGINX provides the option to configure a server as a catch-all with [server_name](http://nginx.org/en/docs/http/server_names.html) for requests that do not match any of the configured server names. This configuration works without issues for HTTP traffic. In case of HTTPS, NGINX requires a certificate. -For this reason the Ingress controller provides the flag `--default-ssl-certificate`. The secret behind this flag contains the default certificate to be used in the mentioned scenario. If this flag is not provided NGINX will use a self signed certificate. +For this reason the Ingress controller provides the flag `--default-ssl-certificate`. The secret referred to by this flag contains the default certificate to be used when accessing the catch-all server. If this flag is not provided NGINX will use a self-signed certificate. + +For instance, if you have a TLS secret `foo-tls` in the `default` namespace, add `--default-ssl-certificate=default/foo-tls` in the `nginx-controller` deployment. Running without the flag `--default-ssl-certificate`: From aca5097a5621d833c9c0c281f666d2af92a65266 Mon Sep 17 00:00:00 2001 From: Aarni Koskela Date: Wed, 2 May 2018 17:12:21 +0300 Subject: [PATCH 03/11] TLS.md: Remove the frankly useless curl output in the default certificate section --- docs/user-guide/tls.md | 98 ------------------------------------------ 1 file changed, 98 deletions(-) diff --git a/docs/user-guide/tls.md b/docs/user-guide/tls.md index c00c40259..e9353f8cd 100644 --- a/docs/user-guide/tls.md +++ b/docs/user-guide/tls.md @@ -35,104 +35,6 @@ For this reason the Ingress controller provides the flag `--default-ssl-certific For instance, if you have a TLS secret `foo-tls` in the `default` namespace, add `--default-ssl-certificate=default/foo-tls` in the `nginx-controller` deployment. -Running without the flag `--default-ssl-certificate`: - -```console -$ curl -v https://10.2.78.7:443 -k -* Rebuilt URL to: https://10.2.78.7:443/ -* Trying 10.2.78.4... -* Connected to 10.2.78.7 (10.2.78.7) port 443 (#0) -* ALPN, offering http/1.1 -* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH -* successfully set certificate verify locations: -* CAfile: /etc/ssl/certs/ca-certificates.crt - CApath: /etc/ssl/certs -* TLSv1.2 (OUT), TLS header, Certificate Status (22): -* TLSv1.2 (OUT), TLS handshake, Client hello (1): -* TLSv1.2 (IN), TLS handshake, Server hello (2): -* TLSv1.2 (IN), TLS handshake, Certificate (11): -* TLSv1.2 (IN), TLS handshake, Server key exchange (12): -* TLSv1.2 (IN), TLS handshake, Server finished (14): -* TLSv1.2 (OUT), TLS handshake, Client key exchange (16): -* TLSv1.2 (OUT), TLS change cipher, Client hello (1): -* TLSv1.2 (OUT), TLS handshake, Finished (20): -* TLSv1.2 (IN), TLS change cipher, Client hello (1): -* TLSv1.2 (IN), TLS handshake, Finished (20): -* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 -* ALPN, server accepted to use http/1.1 -* Server certificate: -* subject: CN=foo.bar.com -* start date: Apr 13 00:50:56 2016 GMT -* expire date: Apr 13 00:50:56 2017 GMT -* issuer: CN=foo.bar.com -* SSL certificate verify result: self signed certificate (18), continuing anyway. -> GET / HTTP/1.1 -> Host: 10.2.78.7 -> User-Agent: curl/7.47.1 -> Accept: */* -> -< HTTP/1.1 404 Not Found -< Server: nginx/1.11.1 -< Date: Thu, 21 Jul 2016 15:38:46 GMT -< Content-Type: text/html -< Transfer-Encoding: chunked -< Connection: keep-alive -< Strict-Transport-Security: max-age=15724800; includeSubDomains; preload -< -The page you're looking for could not be found. - -* Connection #0 to host 10.2.78.7 left intact -``` - -Specifying `--default-ssl-certificate=default/foo-tls`: - -```console -core@localhost ~ $ curl -v https://10.2.78.7:443 -k -* Rebuilt URL to: https://10.2.78.7:443/ -* Trying 10.2.78.7... -* Connected to 10.2.78.7 (10.2.78.7) port 443 (#0) -* ALPN, offering http/1.1 -* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH -* successfully set certificate verify locations: -* CAfile: /etc/ssl/certs/ca-certificates.crt - CApath: /etc/ssl/certs -* TLSv1.2 (OUT), TLS header, Certificate Status (22): -* TLSv1.2 (OUT), TLS handshake, Client hello (1): -* TLSv1.2 (IN), TLS handshake, Server hello (2): -* TLSv1.2 (IN), TLS handshake, Certificate (11): -* TLSv1.2 (IN), TLS handshake, Server key exchange (12): -* TLSv1.2 (IN), TLS handshake, Server finished (14): -* TLSv1.2 (OUT), TLS handshake, Client key exchange (16): -* TLSv1.2 (OUT), TLS change cipher, Client hello (1): -* TLSv1.2 (OUT), TLS handshake, Finished (20): -* TLSv1.2 (IN), TLS change cipher, Client hello (1): -* TLSv1.2 (IN), TLS handshake, Finished (20): -* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 -* ALPN, server accepted to use http/1.1 -* Server certificate: -* subject: CN=foo.bar.com -* start date: Apr 13 00:50:56 2016 GMT -* expire date: Apr 13 00:50:56 2017 GMT -* issuer: CN=foo.bar.com -* SSL certificate verify result: self signed certificate (18), continuing anyway. -> GET / HTTP/1.1 -> Host: 10.2.78.7 -> User-Agent: curl/7.47.1 -> Accept: */* -> -< HTTP/1.1 404 Not Found -< Server: nginx/1.11.1 -< Date: Mon, 18 Jul 2016 21:02:59 GMT -< Content-Type: text/html -< Transfer-Encoding: chunked -< Connection: keep-alive -< Strict-Transport-Security: max-age=15724800; includeSubDomains; preload -< -The page you're looking for could not be found. - -* Connection #0 to host 10.2.78.7 left intact -``` - ## SSL Passthrough The flag `--enable-ssl-passthrough` enables SSL passthrough feature. From ec56200ee021fca15ee8fbd891de6a687b4e758a Mon Sep 17 00:00:00 2001 From: Aarni Koskela Date: Wed, 2 May 2018 17:32:45 +0300 Subject: [PATCH 04/11] TLS.md: Reformat and grammar check --- docs/user-guide/tls.md | 89 +++++++++++++++++++++++++++++++----------- 1 file changed, 66 insertions(+), 23 deletions(-) diff --git a/docs/user-guide/tls.md b/docs/user-guide/tls.md index e9353f8cd..188344b8d 100644 --- a/docs/user-guide/tls.md +++ b/docs/user-guide/tls.md @@ -29,56 +29,91 @@ The resulting secret will be of type `kubernetes.io/tls`. ## Default SSL Certificate -NGINX provides the option to configure a server as a catch-all with [server_name](http://nginx.org/en/docs/http/server_names.html) for requests that do not match any of the configured server names. This configuration works without issues for HTTP traffic. -In case of HTTPS, NGINX requires a certificate. -For this reason the Ingress controller provides the flag `--default-ssl-certificate`. The secret referred to by this flag contains the default certificate to be used when accessing the catch-all server. If this flag is not provided NGINX will use a self-signed certificate. +NGINX provides the option to configure a server as a catch-all with +[server_name](http://nginx.org/en/docs/http/server_names.html) +for requests that do not match any of the configured server names. +This configuration works without out-of-the-box for HTTP traffic. +For HTTPS, a certificate is naturally required. -For instance, if you have a TLS secret `foo-tls` in the `default` namespace, add `--default-ssl-certificate=default/foo-tls` in the `nginx-controller` deployment. +For this reason the Ingress controller provides the flag `--default-ssl-certificate`. +The secret referred to by this flag contains the default certificate to be used when +accessing the catch-all server. +If this flag is not provided NGINX will use a self-signed certificate. + +For instance, if you have a TLS secret `foo-tls` in the `default` namespace, +add `--default-ssl-certificate=default/foo-tls` in the `nginx-controller` deployment. ## SSL Passthrough -The flag `--enable-ssl-passthrough` enables SSL passthrough feature. -By default this feature is disabled +The flag `--enable-ssl-passthrough` enables the SSL passthrough feature. +By default this feature is disabled. + +This is required to enable passthrough backends in Ingress configurations. + +TODO: Improve this documentation. ## HTTP Strict Transport Security -HTTP Strict Transport Security (HSTS) is an opt-in security enhancement specified through the use of a special response header. Once a supported browser receives this header that browser will prevent any communications from being sent over HTTP to the specified domain and will instead send all communications over HTTPS. +HTTP Strict Transport Security (HSTS) is an opt-in security enhancement specified +through the use of a special response header. Once a supported browser receives +this header that browser will prevent any communications from being sent over +HTTP to the specified domain and will instead send all communications over HTTPS. -By default the controller redirects (301) to HTTPS if there is a TLS Ingress rule. +HSTS is enabled by default. -To disable this behavior use `hsts: "false"` in the configuration ConfigMap. +To disable this behavior use `hsts: "false"` in the configuration [ConfigMap][ConfigMap]. ## Server-side HTTPS enforcement through redirect -By default the controller redirects (301) to `HTTPS` if TLS is enabled for that ingress. If you want to disable that behavior globally, you can use `ssl-redirect: "false"` in the NGINX config map. +By default the controller redirects HTTP clients to the HTTPS port +443 using a 301 Moved Permanently response if TLS is enabled for that Ingress. -To configure this feature for specific ingress resources, you can use the `nginx.ingress.kubernetes.io/ssl-redirect: "false"` annotation in the particular resource. +This can be disabled globally using `ssl-redirect: "false"` in the NGINX [config map][ConfigMap], +or per-Ingress with the `nginx.ingress.kubernetes.io/ssl-redirect: "false"` +annotation in the particular resource. -When using SSL offloading outside of cluster (e.g. AWS ELB) it may be useful to enforce a redirect to `HTTPS` even when there is not TLS cert available. This can be achieved by using the `nginx.ingress.kubernetes.io/force-ssl-redirect: "true"` annotation in the particular resource. +!!! tip + + When using SSL offloading outside of cluster (e.g. AWS ELB) it may be useful to enforce a + redirect to HTTPS even when there is no TLS certificate available. + This can be achieved by using the `nginx.ingress.kubernetes.io/force-ssl-redirect: "true"` + annotation in the particular resource. ## Automated Certificate Management with Kube-Lego -[Kube-Lego] automatically requests missing or expired certificates from [Let's Encrypt] by monitoring ingress resources and their referenced secrets. To enable this for an ingress resource you have to add an annotation: +!!! tip + Kube-Lego has reached end-of-life and is being + replaced by [cert-manager](https://github.com/jetstack/cert-manager/). + +[Kube-Lego] automatically requests missing or expired certificates from [Let's Encrypt] +by monitoring ingress resources and their referenced secrets. + +To enable this for an ingress resource you have to add an annotation: ```console kubectl annotate ing ingress-demo kubernetes.io/tls-acme="true" ``` -To setup Kube-Lego you can take a look at this [full example]. The first -version to fully support Kube-Lego is nginx Ingress controller 0.8. - -[full example]:https://github.com/jetstack/kube-lego/tree/master/examples -[Kube-Lego]:https://github.com/jetstack/kube-lego -[Let's Encrypt]:https://letsencrypt.org +To setup Kube-Lego you can take a look at this [full example][full-kube-lego-example]. +The first version to fully support Kube-Lego is Nginx Ingress controller 0.8. ## Default TLS Version and Ciphers -To provide the most secure baseline configuration possible, nginx-ingress defaults to using TLS 1.2 and a [secure set of TLS ciphers](https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/configmap.md#ssl-ciphers) +To provide the most secure baseline configuration possible, -## Legacy TLS -The default configuration, though secure, does not support some older browsers and operating systems. For instance, 20% of Android phones in use today are not compatible with nginx-ingress's default configuration. To change this default behavior, use a [ConfigMap](https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/configmap.md#ssl-ciphers). +nginx-ingress defaults to using TLS 1.2 only and a [secure set of TLS ciphers][ssl-ciphers]. -A sample ConfigMap to allow these older clients connect could look something like the following: +### Legacy TLS + +The default configuration, though secure, does not support some older browsers and operating systems. + +For instance, TLS 1.1+ is only enabled by default from Android 5.0 on. At the time of writing, +May 2018, [approximately 15% of Android devices](https://developer.android.com/about/dashboards/#Platform) +are not compatible with nginx-ingress's default configuration. + +To change this default behavior, use a [ConfigMap]. + +A sample ConfigMap fragment to allow these older clients to connect could look something like the following: ``` kind: ConfigMap @@ -89,3 +124,11 @@ data: ssl-ciphers: "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA" ssl-protocols: "TLSv1 TLSv1.1 TLSv1.2" ``` + + + +[full-kube-lego-example]:https://github.com/jetstack/kube-lego/tree/master/examples +[Kube-Lego]:https://github.com/jetstack/kube-lego +[Let's Encrypt]:https://letsencrypt.org +[ConfigMap]: ./nginx-configuration/configmap.md +[ssl-ciphers]: ./nginx-configuration/configmap.md#ssl-ciphers \ No newline at end of file From ed48199b3034cdd19409b826275c82900297d040 Mon Sep 17 00:00:00 2001 From: Aarni Koskela Date: Wed, 2 May 2018 17:33:08 +0300 Subject: [PATCH 05/11] TLS.md: Remove useless manual TOC --- docs/user-guide/tls.md | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/docs/user-guide/tls.md b/docs/user-guide/tls.md index 188344b8d..5f344cb9e 100644 --- a/docs/user-guide/tls.md +++ b/docs/user-guide/tls.md @@ -1,13 +1,4 @@ -# TLS - -- [Default SSL Certificate](#default-ssl-certificate) -- [SSL Passthrough](#ssl-passthrough) -- [HTTPS enforcement](#server-side-https-enforcement) -- [HSTS](#http-strict-transport-security) -- [Server-side HTTPS enforcement through redirect](#server-side-https-enforcement-through-redirect) -- [Kube-Lego](#automated-certificate-management-with-kube-lego) -- [Default TLS Version and Ciphers](#default-tls-version-and-ciphers) -- [Legacy TLS](#legacy-tls) +# TLS/HTTPS ## TLS Secrets From 572aac442170a147711a4e215a9896763fdcc7b0 Mon Sep 17 00:00:00 2001 From: Aarni Koskela Date: Wed, 2 May 2018 17:57:56 +0300 Subject: [PATCH 06/11] multiple-ingress.md: rework page for clarity and less repetition --- docs/user-guide/multiple-ingress.md | 59 +++++++++++++++-------------- 1 file changed, 30 insertions(+), 29 deletions(-) diff --git a/docs/user-guide/multiple-ingress.md b/docs/user-guide/multiple-ingress.md index 6f7dc7e71..0d6a0f043 100644 --- a/docs/user-guide/multiple-ingress.md +++ b/docs/user-guide/multiple-ingress.md @@ -1,30 +1,10 @@ -# Multiple ingress controllers +# Multiple Ingress controllers -## Running multiple ingress controllers - -If you're running multiple ingress controllers, or running on a cloud provider that natively handles ingress, you need to specify the annotation `kubernetes.io/ingress.class: "nginx"` in all ingresses that you would like this controller to claim. - -This mechanism also provides users the ability to run _multiple_ NGINX ingress controllers (e.g. one which serves public traffic, one which serves "internal" traffic). When utilizing this functionality the option `--ingress-class` should be changed to a value unique for the cluster within the definition of the replication controller. Here is a partial example: - -``` -spec: - template: - spec: - containers: - - name: nginx-ingress-internal-controller - args: - - /nginx-ingress-controller - - '--default-backend-service=ingress/nginx-ingress-default-backend' - - '--election-id=ingress-controller-leader-internal' - - '--ingress-class=nginx-internal' - - '--configmap=ingress/nginx-ingress-internal-controller' -``` +If you're running multiple ingress controllers, or running on a cloud provider that natively handles ingress such as GKE, +you need to specify the annotation `kubernetes.io/ingress.class: "nginx"` in all ingresses that you would like the ingress-nginx controller to claim. -## Annotation ingress.class - -If you have multiple Ingress controllers in a single cluster, you can pick one by specifying the `ingress.class` -annotation, eg creating an Ingress with an annotation like +For instance, ```yaml metadata: @@ -44,11 +24,32 @@ metadata: will target the nginx controller, forcing the GCE controller to ignore it. -__Note__: Deploying multiple ingress controller and not specifying the annotation will result in both controllers fighting to satisfy the Ingress. - -## Disabling NGINX ingress controller - -Setting the annotation `kubernetes.io/ingress.class` to any other value which does not match a valid ingress class will force the NGINX Ingress controller to ignore your Ingress. If you are only running a single NGINX ingress controller, this can be achieved by setting this to any value except "nginx" or an empty string. +To reiterate, setting the annotation to any value which does not match a valid ingress class will force the NGINX Ingress controller to ignore your Ingress. +If you are only running a single NGINX ingress controller, this can be achieved by setting the annotation to any value except "nginx" or an empty string. Do this if you wish to use one of the other Ingress controllers at the same time as the NGINX controller. +!!! important + Deploying multiple Ingress controllers and not specifying a class annotation will + result in both or all controllers fighting to satisfy the Ingress, and all of them + updating the Ingress status field in confusing ways. + +## Multiple ingress-nginx controllers + +This mechanism also provides users the ability to run _multiple_ NGINX ingress controllers (e.g. one which serves public traffic, one which serves "internal" traffic). +To do this, the option `--ingress-class` must be changed to a value unique for the cluster within the definition of the replication controller. +Here is a partial example: + +```yaml +spec: + template: + spec: + containers: + - name: nginx-ingress-internal-controller + args: + - /nginx-ingress-controller + - '--default-backend-service=ingress/nginx-ingress-default-backend' + - '--election-id=ingress-controller-leader-internal' + - '--ingress-class=nginx-internal' + - '--configmap=ingress/nginx-ingress-internal-controller' +``` From f1f3785fbd4762d21c916c60138dcac4708a8b0c Mon Sep 17 00:00:00 2001 From: Aarni Koskela Date: Thu, 3 May 2018 10:20:39 +0300 Subject: [PATCH 07/11] Add upgrade documentation Closes #2458 --- docs/deploy/upgrade.md | 50 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 docs/deploy/upgrade.md diff --git a/docs/deploy/upgrade.md b/docs/deploy/upgrade.md new file mode 100644 index 000000000..fa269a708 --- /dev/null +++ b/docs/deploy/upgrade.md @@ -0,0 +1,50 @@ +# Upgrading + +!!! important + No matter the method you use for upgrading, *if you use template overrides, + make sure your templates are compatible with the new version of ingress-nginx*. + +## Without Helm + +To upgrade your ingress-nginx installation, it should be enough to change the version of the image +in the controller Deployment. + +I.e. if your deployment resource looks like (partial example): + +```yaml +kind: Deployment +metadata: + name: nginx-ingress-controller + namespace: ingress-nginx +spec: + replicas: 1 + selector: ... + template: + metadata: ... + spec: + containers: + - name: nginx-ingress-controller + image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0 + args: ... +``` + +simply change the `0.9.0` tag to the version you wish to upgrade to. +The easiest way to do this is e.g. (do note you may need to change the name parameter according to your installation): + +``` +kubectl set image deployment/nginx-ingress-controller \ + nginx-ingress-controller=nginx:quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0 +``` + +For interactive editing, use `kubectl edit deployment nginx-ingress-controller`. + + +## With Helm + +If you installed ingress-nginx using the Helm command in the deployment docs so its name is `ngx-ingress`, +you should be able to upgrade using + +```shell +helm upgrade --reuse-values ngx-ingress stable/nginx-ingress +``` + From b66efdd8bd76c1991f948b31c2af66fc8e5651d9 Mon Sep 17 00:00:00 2001 From: Aarni Koskela Date: Thu, 3 May 2018 10:44:21 +0300 Subject: [PATCH 08/11] Reformat log-format.md --- .../nginx-configuration/log-format.md | 56 ++++++++++--------- 1 file changed, 30 insertions(+), 26 deletions(-) diff --git a/docs/user-guide/nginx-configuration/log-format.md b/docs/user-guide/nginx-configuration/log-format.md index 6185f4e7c..8cd64ae38 100644 --- a/docs/user-guide/nginx-configuration/log-format.md +++ b/docs/user-guide/nginx-configuration/log-format.md @@ -1,34 +1,38 @@ # Log format -The default configuration uses a custom logging format to add additional information about upstreams, response time and status +The default configuration uses a custom logging format to add additional information about upstreams, response time and status. ``` - log_format upstreaminfo '{{ if $cfg.useProxyProtocol }}$proxy_protocol_addr{{ else }}$remote_addr{{ end }} - ' - '[$the_real_ip] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" ' - '$request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status'; +log_format upstreaminfo + '{{ if $cfg.useProxyProtocol }}$proxy_protocol_addr{{ else }}$remote_addr{{ end }} - ' + '[$the_real_ip] - $remote_user [$time_local] "$request" ' + '$status $body_bytes_sent "$http_referer" "$http_user_agent" ' + '$request_length $request_time [$proxy_upstream_name] $upstream_addr ' + '$upstream_response_length $upstream_response_time $upstream_status'; ``` +| Placeholder | Description | +|-------------|-------------| +| `$proxy_protocol_addr` | remote address if proxy protocol is enabled | +| `$remote_addr` | remote address if proxy protocol is disabled (default) | +| `$the_real_ip` | the source IP address of the client | +| `$remote_user` | user name supplied with the Basic authentication | +| `$time_local` | local time in the Common Log Format | +| `$request` | full original request line | +| `$status` | response status | +| `$body_bytes_sent` | number of bytes sent to a client, not counting the response header | +| `$http_referer` | value of the Referer header | +| `$http_user_agent` | value of User-Agent header | +| `$request_length` | request length (including request line, header, and request body) | +| `$request_time` | time elapsed since the first bytes were read from the client | +| `$proxy_upstream_name` | name of the upstream. The format is `upstream---` | +| `$upstream_addr` | the IP address and port (or the path to the domain socket) of the upstream server. If several servers were contacted during request processing, their addresses are separated by commas. | +| `$upstream_response_length` | the length of the response obtained from the upstream server | +| `$upstream_response_time` | time spent on receiving the response from the upstream server as seconds with millisecond resolution | +| `$upstream_status` | status code of the response obtained from the upstream server | + + Sources: -- [upstream variables](http://nginx.org/en/docs/http/ngx_http_upstream_module.html#variables) -- [embedded variables](http://nginx.org/en/docs/http/ngx_http_core_module.html#variables) - -Description: - -- `$proxy_protocol_addr`: if PROXY protocol is enabled -- `$remote_addr`: if PROXY protocol is disabled (default) -- `$the_real_ip`: the source IP address of the client -- `$remote_user`: user name supplied with the Basic authentication -- `$time_local`: local time in the Common Log Format -- `$request`: full original request line -- `$status`: response status -- `$body_bytes_sent`: number of bytes sent to a client, not counting the response header -- `$http_referer`: value of the Referer header -- `$http_user_agent`: value of User-Agent header -- `$request_length`: request length (including request line, header, and request body) -- `$request_time`: time elapsed since the first bytes were read from the client -- `$proxy_upstream_name`: name of the upstream. The format is `upstream---` -- `$upstream_addr`: keeps the IP address and port, or the path to the UNIX-domain socket of the upstream server. If several servers were contacted during request processing, their addresses are separated by commas -- `$upstream_response_length`: keeps the length of the response obtained from the upstream server -- `$upstream_response_time`: keeps time spent on receiving the response from the upstream server; the time is kept in seconds with millisecond resolution -- `$upstream_status`: keeps status code of the response obtained from the upstream server \ No newline at end of file +- [Upstream variables](http://nginx.org/en/docs/http/ngx_http_upstream_module.html#variables) +- [Embedded variables](http://nginx.org/en/docs/http/ngx_http_core_module.html#variables) \ No newline at end of file From 4b85ef9c9c7eedab8ae31502a5630ef06d1bfe81 Mon Sep 17 00:00:00 2001 From: Aarni Koskela Date: Thu, 3 May 2018 12:23:59 +0300 Subject: [PATCH 09/11] Add note about changing annotation prefixes --- docs/user-guide/nginx-configuration/annotations.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/user-guide/nginx-configuration/annotations.md b/docs/user-guide/nginx-configuration/annotations.md index 4e776ccd7..3cfd35d46 100644 --- a/docs/user-guide/nginx-configuration/annotations.md +++ b/docs/user-guide/nginx-configuration/annotations.md @@ -7,6 +7,11 @@ You can add these Kubernetes annotations to specific Ingress objects to customiz Other types, such as boolean or numeric values must be quoted, i.e. `"true"`, `"false"`, `"100"`. +!!! note + The annotation prefix can be changed using the + [`--annotations-prefix` command line argument](../cli-arguments.md), + but the default is `nginx.ingress.kubernetes.io`, as described in the + table below. |Name | type | |---------------------------|------| From f1e5c9b2dda274d98ed3eab5925d344d06e6ebce Mon Sep 17 00:00:00 2001 From: Aarni Koskela Date: Thu, 3 May 2018 12:46:39 +0300 Subject: [PATCH 10/11] Clean up annotations.md; extract default backend from miscellaneous --- docs/user-guide/default-backend.md | 17 + docs/user-guide/miscellaneous.md | 10 - .../nginx-configuration/annotations.md | 297 ++++++++++-------- docs/user-guide/tls.md | 4 +- 4 files changed, 183 insertions(+), 145 deletions(-) create mode 100644 docs/user-guide/default-backend.md diff --git a/docs/user-guide/default-backend.md b/docs/user-guide/default-backend.md new file mode 100644 index 000000000..a9fca1a81 --- /dev/null +++ b/docs/user-guide/default-backend.md @@ -0,0 +1,17 @@ +# Default backend + +The default backend is a service which handles all URL paths and hosts the nginx controller doesn't understand +(i.e., all the requests that are not mapped with an Ingress). + +Basically a default backend exposes two URLs: + +- `/healthz` that returns 200 +- `/` that returns 404 + +!!! example + The sub-directory [`/images/404-server`](https://github.com/kubernetes/ingress-nginx/tree/master/images/404-server) + provides a service which satisfies the requirements for a default backend. + +!!! example + The sub-directory [`/images/custom-error-pages`](https://github.com/kubernetes/ingress-nginx/tree/master/images/custom-error-pages) + provides an additional service for the purpose of customizing the error pages served via the default backend. diff --git a/docs/user-guide/miscellaneous.md b/docs/user-guide/miscellaneous.md index d184bc2b1..222b04ee2 100644 --- a/docs/user-guide/miscellaneous.md +++ b/docs/user-guide/miscellaneous.md @@ -1,15 +1,5 @@ # Miscellaneous -## Requirements - -The default backend is a service which handles all url paths and hosts the nginx controller doesn't understand (i.e., all the requests that are not mapped with an Ingress). -Basically a default backend exposes two URLs: - -- `/healthz` that returns 200 -- `/` that returns 404 - -The sub-directory [`/images/404-server`](https://github.com/kubernetes/ingress-nginx/tree/master/images/404-server) provides a service which satisfies the requirements for a default backend. The sub-directory [`/images/custom-error-pages`](https://github.com/kubernetes/ingress-nginx/tree/master/images/custom-error-pages) provides an additional service for the purpose of customizing the error pages served via the default backend. - ## Source IP address By default NGINX uses the content of the header `X-Forwarded-For` as the source of truth to get information about the client IP address. This works without issues in L7 **if we configure the setting `proxy-real-ip-cidr`** with the correct information of the IP/network address of trusted external load balancer. diff --git a/docs/user-guide/nginx-configuration/annotations.md b/docs/user-guide/nginx-configuration/annotations.md index 3cfd35d46..42b8fce38 100644 --- a/docs/user-guide/nginx-configuration/annotations.md +++ b/docs/user-guide/nginx-configuration/annotations.md @@ -89,14 +89,31 @@ If the scheme of [`base` tag](https://developer.mozilla.org/en/docs/Web/HTML/Ele If the Application Root is exposed in a different path and needs to be redirected, set the annotation `nginx.ingress.kubernetes.io/app-root` to redirect requests for `/`. -Please check the [rewrite](../../examples/rewrite/README.md) example. +!!! example + Please check the [rewrite](../../examples/rewrite/README.md) example. ### Session Affinity The annotation `nginx.ingress.kubernetes.io/affinity` enables and sets the affinity type in all Upstreams of an Ingress. This way, a request will always be directed to the same upstream server. The only affinity type available for NGINX is `cookie`. -Please check the [affinity](../../examples/affinity/cookie/README.md) example. +!!! example + Please check the [affinity](../../examples/affinity/cookie/README.md) example. + +#### Cookie affinity + +If you use the ``cookie`` affinity type you can also specify the name of the cookie that will be used to route the requests with the annotation `nginx.ingress.kubernetes.io/session-cookie-name`. The default is to create a cookie named 'INGRESSCOOKIE'. + +In case of NGINX the annotation `nginx.ingress.kubernetes.io/session-cookie-hash` defines which algorithm will be used to hash the used upstream. Default value is `md5` and possible values are `md5`, `sha1` and `index`. + +!!! attention + The `index` option is not an actual hash; an in-memory index is used instead, which has less overhead. + However, with `index`, matching against a changing upstream server list is inconsistent. + So, at reload, if upstream servers have changed, index values are not guaranteed to correspond to the same server as before! + **Use `index` with caution** and only if you need to! + +In NGINX this feature is implemented by the third party module [nginx-sticky-module-ng](https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng). The workflow used to define which upstream server will be used is explained [here](https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng/raw/08a395c66e425540982c00482f55034e1fee67b6/docs/sticky.pdf) + ### Authentication @@ -120,7 +137,8 @@ This annotation also accepts the alternative form "namespace/secretName", in whi nginx.ingress.kubernetes.io/auth-realm: "realm string" ``` -Please check the [auth](../../examples/auth/basic/README.md) example. +!!! example + Please check the [auth](../../examples/auth/basic/README.md) example. ### Custom NGINX upstream checks @@ -138,10 +156,12 @@ To use custom values in an Ingress rule define these annotations: In NGINX, backend server pools are called "[upstreams](http://nginx.org/en/docs/http/ngx_http_upstream_module.html)". Each upstream contains the endpoints for a service. An upstream is created for each service that has Ingress rules defined. -!!! Important - All Ingress rules using the same service will use the same upstream. Only one of the Ingress rules should define annotations to configure the upstream servers. +!!! attention + All Ingress rules using the same service will use the same upstream. + Only one of the Ingress rules should define annotations to configure the upstream servers. -Please check the [custom upstream check](../../examples/customization/custom-upstream-check/README.md) example. +!!! example + Please check the [custom upstream check](../../examples/customization/custom-upstream-check/README.md) example. ### Custom NGINX upstream hashing @@ -165,42 +185,24 @@ This configuration setting allows you to control the value for host in the follo It is possible to enable Client Certificate Authentication using additional annotations in Ingress Rule. The annotations are: -``` -nginx.ingress.kubernetes.io/auth-tls-secret: secretName -``` -The name of the Secret that contains the full Certificate Authority chain `ca.crt` that is enabled to authenticate against this Ingress. -This annotation also accepts the alternative form "namespace/secretName", in which case the Secret lookup is performed in the referenced namespace instead of the Ingress namespace. +* `nginx.ingress.kubernetes.io/auth-tls-secret: secretName`: + The name of the Secret that contains the full Certificate Authority chain `ca.crt` that is enabled to authenticate against this Ingress. + This annotation also accepts the alternative form "namespace/secretName", in which case the Secret lookup is performed in the referenced namespace instead of the Ingress namespace. +* `nginx.ingress.kubernetes.io/auth-tls-verify-depth`: + The validation depth between the provided client certificate and the Certification Authority chain. +* `nginx.ingress.kubernetes.io/auth-tls-verify-client`: + Enables verification of client certificates. +* `nginx.ingress.kubernetes.io/auth-tls-error-page`: + The URL/Page that user should be redirected in case of a Certificate Authentication Error +* `nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream`: + Indicates if the received certificates should be passed or not to the upstream server. By default this is disabled. -``` -nginx.ingress.kubernetes.io/auth-tls-verify-depth -``` +!!! example + Please check the [client-certs](../../examples/auth/client-certs/README.md) example. -The validation depth between the provided client certificate and the Certification Authority chain. - -``` -nginx.ingress.kubernetes.io/auth-tls-verify-client -``` - -Enables verification of client certificates. - -``` -nginx.ingress.kubernetes.io/auth-tls-error-page -``` - -The URL/Page that user should be redirected in case of a Certificate Authentication Error - -``` -nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream -``` - -Indicates if the received certificates should be passed or not to the upstream server. -By default this is disabled. - -Please check the [client-certs](../../examples/auth/client-certs/README.md) example. - -!!! Important - TLS with Client Authentication is NOT possible in Cloudflare as is not allowed it and might result in unexpected behavior. +!!! attention + TLS with Client Authentication is **not** possible in Cloudflare and might result in unexpected behavior. Cloudflare only allows Authenticated Origin Pulls and is required to use their own certificate: [https://blog.cloudflare.com/protecting-the-origin-with-tls-authenticated-origin-pulls/](https://blog.cloudflare.com/protecting-the-origin-with-tls-authenticated-origin-pulls/) @@ -218,47 +220,55 @@ nginx.ingress.kubernetes.io/configuration-snippet: | ### Default Backend -The ingress controller requires a default backend. This service handles the response when the service in the Ingress rule does not have endpoints. +The ingress controller requires a [default backend](../default-backend.md). +This service handles the response when the service in the Ingress rule does not have endpoints. This is a global configuration for the ingress controller. In some cases could be required to return a custom content or format. In this scenario we can use the annotation `nginx.ingress.kubernetes.io/default-backend: ` to specify a custom default backend. ### Enable CORS -To enable Cross-Origin Resource Sharing (CORS) in an Ingress rule add the annotation `nginx.ingress.kubernetes.io/enable-cors: "true"`. This will add a section in the server location enabling this functionality. +To enable Cross-Origin Resource Sharing (CORS) in an Ingress rule, +add the annotation `nginx.ingress.kubernetes.io/enable-cors: "true"`. +This will add a section in the server location enabling this functionality. CORS can be controlled with the following annotations: -* `nginx.ingress.kubernetes.io/cors-allow-methods` controls which methods are accepted. This is a multi-valued field, separated by ',' and accepts only letters (upper and lower case). +* `nginx.ingress.kubernetes.io/cors-allow-methods` + controls which methods are accepted. + This is a multi-valued field, separated by ',' and accepts only letters (upper and lower case). + Example: `nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS"` -Example: `nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS"` +* `nginx.ingress.kubernetes.io/cors-allow-headers` + controls which headers are accepted. + This is a multi-valued field, separated by ',' and accepts letters, numbers, _ and -. + Example: `nginx.ingress.kubernetes.io/cors-allow-headers: "X-Forwarded-For, X-app123-XPTO"` -* `nginx.ingress.kubernetes.io/cors-allow-headers` controls which headers are accepted. This is a multi-valued field, separated by ',' and accepts letters, numbers, _ and -. +* `nginx.ingress.kubernetes.io/cors-allow-origin` + controls what's the accepted Origin for CORS and defaults to '*'. + This is a single field value, with the following format: `http(s)://origin-site.com` or `http(s)://origin-site.com:port` + Example: `nginx.ingress.kubernetes.io/cors-allow-origin: "https://origin-site.com:4443"` -Example: `nginx.ingress.kubernetes.io/cors-allow-headers: "X-Forwarded-For, X-app123-XPTO"` +* `nginx.ingress.kubernetes.io/cors-allow-credentials` + controls if credentials can be passed during CORS operations. + Example: `nginx.ingress.kubernetes.io/cors-allow-credentials: "true"` -* `nginx.ingress.kubernetes.io/cors-allow-origin` controls what's the accepted Origin for CORS and defaults to '*'. This is a single field value, with the following format: http(s)://origin-site.com or http(s)://origin-site.com:port +* `nginx.ingress.kubernetes.io/cors-max-age` + controls how long preflight requests can be cached. + Example: `nginx.ingress.kubernetes.io/cors-max-age: 600` -Example: `nginx.ingress.kubernetes.io/cors-allow-origin: "https://origin-site.com:4443"` - -* `nginx.ingress.kubernetes.io/cors-allow-credentials` controls if credentials can be passed during CORS operations. - -Example: `nginx.ingress.kubernetes.io/cors-allow-credentials: "true"` - -* `nginx.ingress.kubernetes.io/cors-max-age` controls how long preflight requests can be cached. - -Example: `nginx.ingress.kubernetes.io/cors-max-age: 600` - - -For more information please see [https://enable-cors.org](https://enable-cors.org/server_nginx.html) +!!! note + For more information please see [https://enable-cors.org](https://enable-cors.org/server_nginx.html) ### Server Alias To add Server Aliases to an Ingress rule add the annotation `nginx.ingress.kubernetes.io/server-alias: ""`. -This will create a server with the same configuration, but a different server_name as the provided host. +This will create a server with the same configuration, but a different `server_name` as the provided host. !!! Note - A server-alias name cannot conflict with the hostname of an existing server. If it does the server-alias annotation will be ignored. If a server-alias is created and later a new server with the same hostname is created the new server configuration will take place over the alias configuration. + A server-alias name cannot conflict with the hostname of an existing server. If it does the server-alias annotation will be ignored. + If a server-alias is created and later a new server with the same hostname is created, + the new server configuration will take place over the alias configuration. -For more information please see [http://nginx.org](http://nginx.org/en/docs/http/ngx_http_core_module.html#server_name) +For more information please see [the `server_name` documentation](http://nginx.org/en/docs/http/ngx_http_core_module.html#server_name). ### Server snippet @@ -281,8 +291,8 @@ if ( $agentflag = 1 ) { } ``` -!!! Important - This annotation can be used only once per host +!!! attention + This annotation can be used only once per host. ### Client Body Buffer Size @@ -291,14 +301,16 @@ the whole body or only its part is written to a temporary file. By default, buff This is 8K on x86, other 32-bit platforms, and x86-64. It is usually 16K on other 64-bit platforms. This annotation is applied to each location provided in the ingress rule. -__Note:__ The annotation value must be given in a valid format otherwise the -For example to set the client-body-buffer-size the following can be done: +!!! note + The annotation value must be given in a format understood by Nginx. -* `nginx.ingress.kubernetes.io/client-body-buffer-size: "1000"` # 1000 bytes -* `nginx.ingress.kubernetes.io/client-body-buffer-size: 1k` # 1 kilobyte -* `nginx.ingress.kubernetes.io/client-body-buffer-size: 1K` # 1 kilobyte -* `nginx.ingress.kubernetes.io/client-body-buffer-size: 1m` # 1 megabyte -* `nginx.ingress.kubernetes.io/client-body-buffer-size: 1M` # 1 megabyte +!!! example + + * `nginx.ingress.kubernetes.io/client-body-buffer-size: "1000"` # 1000 bytes + * `nginx.ingress.kubernetes.io/client-body-buffer-size: 1k` # 1 kilobyte + * `nginx.ingress.kubernetes.io/client-body-buffer-size: 1K` # 1 kilobyte + * `nginx.ingress.kubernetes.io/client-body-buffer-size: 1m` # 1 megabyte + * `nginx.ingress.kubernetes.io/client-body-buffer-size: 1M` # 1 megabyte For more information please see [http://nginx.org](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size) @@ -312,25 +324,28 @@ nginx.ingress.kubernetes.io/auth-url: "URL to the authentication service" Additionally it is possible to set: -`nginx.ingress.kubernetes.io/auth-method`: `` to specify the HTTP method to use. +* `nginx.ingress.kubernetes.io/auth-method`: + `` to specify the HTTP method to use. +* `nginx.ingress.kubernetes.io/auth-signin`: + `` to specify the location of the error page. +* `nginx.ingress.kubernetes.io/auth-response-headers`: + `` to specify headers to pass to backend once authorization request completes. +* `nginx.ingress.kubernetes.io/auth-request-redirect`: + `` to specify the X-Auth-Request-Redirect header value. -`nginx.ingress.kubernetes.io/auth-signin`: `` to specify the location of the error page. - -`nginx.ingress.kubernetes.io/auth-response-headers`: `` to specify headers to pass to backend once authorization request completes. - -`nginx.ingress.kubernetes.io/auth-request-redirect`: `` to specify the X-Auth-Request-Redirect header value. - -Please check the [external-auth](../../examples/auth/external-auth/README.md) example. +!!! example + Please check the [external-auth](../../examples/auth/external-auth/README.md) example. ### Rate limiting -The annotations `nginx.ingress.kubernetes.io/limit-connections`, `nginx.ingress.kubernetes.io/limit-rps`, and `nginx.ingress.kubernetes.io/limit-rpm` define a limit on the connections that can be opened by a single client IP address. This can be used to mitigate [DDoS Attacks](https://www.nginx.com/blog/mitigating-ddos-attacks-with-nginx-and-nginx-plus). +These annotations define a limit on the connections that can be opened by a single client IP address. +This can be used to mitigate [DDoS Attacks](https://www.nginx.com/blog/mitigating-ddos-attacks-with-nginx-and-nginx-plus). -`nginx.ingress.kubernetes.io/limit-connections`: number of concurrent connections allowed from a single IP address. - -`nginx.ingress.kubernetes.io/limit-rps`: number of connections that may be accepted from a given IP each second. - -`nginx.ingress.kubernetes.io/limit-rpm`: number of connections that may be accepted from a given IP each minute. +* `nginx.ingress.kubernetes.io/limit-connections`: number of concurrent connections allowed from a single IP address. +* `nginx.ingress.kubernetes.io/limit-rps`: number of connections that may be accepted from a given IP each second. +* `nginx.ingress.kubernetes.io/limit-rpm`: number of connections that may be accepted from a given IP each minute. +* `nginx.ingress.kubernetes.io/limit-rate-after`: sets the initial amount after which the further transmission of a response to a client will be rate limited. +* `nginx.ingress.kubernetes.io/limit-rate`: rate of request that accepted from a client each second. You can specify the client IP source ranges to be excluded from rate-limiting through the `nginx.ingress.kubernetes.io/limit-whitelist` annotation. The value is a comma separated list of CIDRs. @@ -338,34 +353,44 @@ If you specify multiple annotations in a single Ingress rule, `limit-rpm`, and t The annotation `nginx.ingress.kubernetes.io/limit-rate`, `nginx.ingress.kubernetes.io/limit-rate-after` define a limit the rate of response transmission to a client. The rate is specified in bytes per second. The zero value disables rate limiting. The limit is set per a request, and so if a client simultaneously opens two connections, the overall rate will be twice as much as the specified limit. -`nginx.ingress.kubernetes.io/limit-rate-after`: sets the initial amount after which the further transmission of a response to a client will be rate limited. - -`nginx.ingress.kubernetes.io/limit-rate`: rate of request that accepted from a client each second. - -To configure this setting globally for all Ingress rules, the `limit-rate-after` and `limit-rate` value may be set in the NGINX ConfigMap. if you set the value in ingress annotation will cover global setting. +To configure this setting globally for all Ingress rules, the `limit-rate-after` and `limit-rate` value may be set in the [NGINX ConfigMap][configmap]. if you set the value in ingress annotation will cover global setting. ### Permanent Redirect + This annotation allows to return a permanent redirect instead of sending data to the upstream. For example `nginx.ingress.kubernetes.io/permanent-redirect: https://www.google.com` would redirect everything to Google. ### SSL Passthrough The annotation `nginx.ingress.kubernetes.io/ssl-passthrough` allows to configure TLS termination in the pod and not in NGINX. -!!! Important - - Using the annotation `nginx.ingress.kubernetes.io/ssl-passthrough` invalidates all the other available annotations. This is because SSL Passthrough works in L4 (TCP). +!!! attention + Using the annotation `nginx.ingress.kubernetes.io/ssl-passthrough` invalidates all the other available annotations. + This is because SSL Passthrough works on level 4 of the OSI stack (TCP), not on the HTTP/HTTPS level. - - The use of this annotation requires Proxy Protocol to be enabled in the load-balancer. For example enabling Proxy Protocol for AWS ELB is described [here](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-proxy-protocol.html). If you're using ingress-controller without load balancer then the flag `--enable-ssl-passthrough` is required (by default it is disabled). +!!! attention + The use of this annotation requires the Proxy Protocol to be enabled in the front-end load-balancer. + For example enabling Proxy Protocol for AWS ELB is described [here](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-proxy-protocol.html). + If you're using ingress-controller without load balancer then the flag + `--enable-ssl-passthrough` is required (by default it is disabled). ### Secure backends -By default NGINX uses `http` to reach the services. Adding the annotation `nginx.ingress.kubernetes.io/secure-backends: "true"` in the Ingress rule changes the protocol to `https`. +By default NGINX uses plain HTTP to reach the services. +Adding the annotation `nginx.ingress.kubernetes.io/secure-backends: "true"` in the Ingress rule changes the protocol to HTTPS. If you want to validate the upstream against a specific certificate, you can create a secret with it and reference the secret with the annotation `nginx.ingress.kubernetes.io/secure-verify-ca-secret`. -> Note that if an invalid or non-existent secret is given, the NGINX ingress controller will ignore the `secure-backends` annotation. +!!! attention + + Note that if an invalid or non-existent secret is given, + the ingress controller will ignore the `secure-backends` annotation. ### Service Upstream -By default the NGINX ingress controller uses a list of all endpoints (Pod IP/port) in the NGINX upstream configuration. This annotation disables that behavior and instead uses a single upstream in NGINX, the service's Cluster IP and port. This can be desirable for things like zero-downtime deployments as it reduces the need to reload NGINX configuration when Pods come up and down. See issue [#257](https://github.com/kubernetes/ingress-nginx/issues/257). +By default the NGINX ingress controller uses a list of all endpoints (Pod IP/port) in the NGINX upstream configuration. + +The `nginx.ingress.kubernetes.io/service-upstream` annotation disables that behavior and instead uses a single upstream in NGINX, the service's Cluster IP and port. + +This can be desirable for things like zero-downtime deployments as it reduces the need to reload NGINX configuration when Pods come up and down. See issue [#257](https://github.com/kubernetes/ingress-nginx/issues/257). #### Known Issues @@ -376,36 +401,33 @@ If the `service-upstream` annotation is specified the following things should be ### Server-side HTTPS enforcement through redirect -By default the controller redirects (301) to `HTTPS` if TLS is enabled for that ingress. If you want to disable that behavior globally, you can use `ssl-redirect: "false"` in the NGINX config map. +By default the controller redirects (308) to HTTPS if TLS is enabled for that ingress. +If you want to disable this behavior globally, you can use `ssl-redirect: "false"` in the NGINX [config map][configmap]. -To configure this feature for specific ingress resources, you can use the `nginx.ingress.kubernetes.io/ssl-redirect: "false"` annotation in the particular resource. +To configure this feature for specific ingress resources, you can use the `nginx.ingress.kubernetes.io/ssl-redirect: "false"` +annotation in the particular resource. -When using SSL offloading outside of cluster (e.g. AWS ELB) it may be useful to enforce a redirect to `HTTPS` even when there is not TLS cert available. This can be achieved by using the `nginx.ingress.kubernetes.io/force-ssl-redirect: "true"` annotation in the particular resource. +When using SSL offloading outside of cluster (e.g. AWS ELB) it may be useful to enforce a redirect to HTTPS +even when there is no TLS certificate available. +This can be achieved by using the `nginx.ingress.kubernetes.io/force-ssl-redirect: "true"` annotation in the particular resource. -### Redirect from to www +### Redirect from/to www. -In some scenarios is required to redirect from `www.domain.com` to `domain.com` or viceversa. +In some scenarios is required to redirect from `www.domain.com` to `domain.com` or vice versa. To enable this feature use the annotation `nginx.ingress.kubernetes.io/from-to-www-redirect: "true"` -!!! Important +!!! attention If at some point a new Ingress is created with a host equal to one of the options (like `domain.com`) the annotation will be omitted. ### Whitelist source range -You can specify the allowed client IP source ranges through the `nginx.ingress.kubernetes.io/whitelist-source-range` annotation. The value is a comma separated list of [CIDRs](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing), e.g. `10.0.0.0/24,172.10.0.1`. +You can specify allowed client IP source ranges through the `nginx.ingress.kubernetes.io/whitelist-source-range` annotation. +The value is a comma separated list of [CIDRs](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing), e.g. `10.0.0.0/24,172.10.0.1`. -To configure this setting globally for all Ingress rules, the `whitelist-source-range` value may be set in the NGINX ConfigMap. +To configure this setting globally for all Ingress rules, the `whitelist-source-range` value may be set in the [NGINX ConfigMap][configmap]. -__Note:__ Adding an annotation to an Ingress rule overrides any global restriction. - -### Cookie affinity - -If you use the ``cookie`` type you can also specify the name of the cookie that will be used to route the requests with the annotation `nginx.ingress.kubernetes.io/session-cookie-name`. The default is to create a cookie named 'INGRESSCOOKIE'. - -In case of NGINX the annotation `nginx.ingress.kubernetes.io/session-cookie-hash` defines which algorithm will be used to 'hash' the used upstream. Default value is `md5` and possible values are `md5`, `sha1` and `index`. -The `index` option is not hashed, an in-memory index is used instead, it's quicker and the overhead is shorter Warning: the matching against upstream servers list is inconsistent. So, at reload, if upstreams servers has changed, index values are not guaranteed to correspond to the same server as before! **USE IT WITH CAUTION** and only if you need to! - -In NGINX this feature is implemented by the third party module [nginx-sticky-module-ng](https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng). The workflow used to define which upstream server will be used is explained [here](https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng/raw/08a395c66e425540982c00482f55034e1fee67b6/docs/sticky.pdf) +!!! note + Adding an annotation to an Ingress rule overrides any global restriction. ### Custom timeouts @@ -422,15 +444,16 @@ In some scenarios is required to have different values. To allow this we provide ### Proxy redirect With the annotations `nginx.ingress.kubernetes.io/proxy-redirect-from` and `nginx.ingress.kubernetes.io/proxy-redirect-to` it is possible to set the text that should be changed in the `Location` and `Refresh` header fields of a proxied server response (http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_redirect) -Setting "off" or "default" in the annotation `nginx.ingress.kubernetes.io/proxy-redirect-from` disables `nginx.ingress.kubernetes.io/proxy-redirect-to` -Both annotations will be used in any other case -By default the value is "off". + +Setting "off" or "default" in the annotation `nginx.ingress.kubernetes.io/proxy-redirect-from` disables `nginx.ingress.kubernetes.io/proxy-redirect-to`. + +Both annotations will be used in any other case. By default the value is "off". ### Custom max body size -For NGINX, 413 error will be returned to the client when the size in a request exceeds the maximum allowed size of the client request body. This size can be configured by the parameter [`client_max_body_size`](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size). +For NGINX, an 413 error will be returned to the client when the size in a request exceeds the maximum allowed size of the client request body. This size can be configured by the parameter [`client_max_body_size`](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size). -To configure this setting globally for all Ingress rules, the `proxy-body-size` value may be set in the NGINX ConfigMap. +To configure this setting globally for all Ingress rules, the `proxy-body-size` value may be set in the [NGINX ConfigMap][configmap]. To use custom values in an Ingress rule define these annotation: ```yaml @@ -440,9 +463,9 @@ nginx.ingress.kubernetes.io/proxy-body-size: 8m ### Proxy buffering Enable or disable proxy buffering [`proxy_buffering`](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering). -By default proxy buffering is disabled in the nginx config. +By default proxy buffering is disabled in the NGINX config. -To configure this setting globally for all Ingress rules, the `proxy-buffering` value may be set in the NGINX ConfigMap. +To configure this setting globally for all Ingress rules, the `proxy-buffering` value may be set in the [NGINX ConfigMap][configmap]. To use custom values in an Ingress rule define these annotation: ```yaml @@ -460,7 +483,9 @@ nginx.ingress.kubernetes.io/ssl-ciphers: "ALL:!aNULL:!EXPORT56:RC4+RSA:+HIGH:+ME ``` ### Connection proxy header -Using this annotation will override the default connection header set by nginx. To use custom values in an Ingress rule, define the annotation: + +Using this annotation will override the default connection header set by NGINX. +To use custom values in an Ingress rule, define the annotation: ```yaml nginx.ingress.kubernetes.io/connection-proxy-header: "keep-alive" @@ -468,7 +493,8 @@ nginx.ingress.kubernetes.io/connection-proxy-header: "keep-alive" ### Enable Access Log -In some scenarios could be required to disable NGINX access logs. To enable this feature use the annotation: +Access logs are enabled by default, but in some scenarios access logs might be required to be disabled for a given +ingress. To do this, use the annotation: ```yaml nginx.ingress.kubernetes.io/enable-access-log: "false" @@ -476,7 +502,8 @@ nginx.ingress.kubernetes.io/enable-access-log: "false" ### Enable Rewrite Log -In some scenarios it could be required to enable NGINX rewrite logs. Note that rewrite logs are sent to the error_log file at the notice level. To enable this feature use the annotation: +Rewrite logs are not enabled by default. In some scenarios it could be required to enable NGINX rewrite logs. +Note that rewrite logs are sent to the error_log file at the notice level. To enable this feature use the annotation: ```yaml nginx.ingress.kubernetes.io/enable-rewrite-log: "true" @@ -484,19 +511,21 @@ nginx.ingress.kubernetes.io/enable-rewrite-log: "true" ### Lua Resty WAF -Using `lua-resty-waf-*` annotations we can enable and control [lua-resty-waf](https://github.com/p0pr0ck5/lua-resty-waf) per location. -Following configuration will enable WAF for the paths defined in the corresponding ingress: +Using `lua-resty-waf-*` annotations we can enable and control the [lua-resty-waf](https://github.com/p0pr0ck5/lua-resty-waf) +Web Application Firewall per location. + +Following configuration will enable the WAF for the paths defined in the corresponding ingress: ```yaml nginx.ingress.kubernetes.io/lua-resty-waf: "active" ``` In order to run it in debugging mode you can set `nginx.ingress.kubernetes.io/lua-resty-waf-debug` to `"true"` in addition to the above configuration. -The other possible values for `nginx.ingress.kubernetes.io/lua-resty-waf` are `inactive` and `simulate`. In `inactive` mode WAF won't do anything, whereas -in `simulate` mode it will log a warning message if there's a matching WAF rule for given request. This is useful to debug a rule and eliminate possible false positives before fully deploying it. +The other possible values for `nginx.ingress.kubernetes.io/lua-resty-waf` are `inactive` and `simulate`. +In `inactive` mode WAF won't do anything, whereas in `simulate` mode it will log a warning message if there's a matching WAF rule for given request. This is useful to debug a rule and eliminate possible false positives before fully deploying it. `lua-resty-waf` comes with predefined set of rules [https://github.com/p0pr0ck5/lua-resty-waf/tree/84b4f40362500dd0cb98b9e71b5875cb1a40f1ad/rules](https://github.com/p0pr0ck5/lua-resty-waf/tree/84b4f40362500dd0cb98b9e71b5875cb1a40f1ad/rules) that covers ModSecurity CRS. -You can use `nginx.ingress.kubernetes.io/lua-resty-waf-ignore-rulesets` to ignore subset of those rulesets. For an example: +You can use `nginx.ingress.kubernetes.io/lua-resty-waf-ignore-rulesets` to ignore a subset of those rulesets. For an example: ```yaml nginx.ingress.kubernetes.io/lua-resty-waf-ignore-rulesets: "41000_sqli, 42000_xss" @@ -504,8 +533,7 @@ nginx.ingress.kubernetes.io/lua-resty-waf-ignore-rulesets: "41000_sqli, 42000_xs will ignore the two mentioned rulesets. -It is also possible to configure custom WAF rules per ingress using `nginx.ingress.kubernetes.io/lua-resty-waf-extra-rules` annotation. For an example the following snippet will -configure a WAF rule to deny requests with query string value that contains word `foo`: +It is also possible to configure custom WAF rules per ingress using the `nginx.ingress.kubernetes.io/lua-resty-waf-extra-rules` annotation. For an example the following snippet will configure a WAF rule to deny requests with query string value that contains word `foo`: ```yaml @@ -518,8 +546,11 @@ For details on how to write WAF rules, please refer to [https://github.com/p0pr0 Since NGINX 1.13.10 it is possible to expose [gRPC services natively](http://nginx.org/en/docs/http/ngx_http_grpc_module.html) -You only need to add the annotation `nginx.ingress.kubernetes.io/grpc-backend: "true"` to enable this feature. Additionally, if the gRPC service requires TLS `nginx.ingress.kubernetes.io/secure-backends: "true"` +You only need to add the annotation `nginx.ingress.kubernetes.io/grpc-backend: "true"` to enable this feature. +Additionally, if the gRPC service requires TLS, add `nginx.ingress.kubernetes.io/secure-backends: "true"`. -!!! Important +!!! attention This feature requires HTTP2 to work which means we need to expose this service using HTTPS. -Exposing a gRPC service using HTTP is not supported. + Exposing a gRPC service using HTTP is not supported. + +[configmap]: ./configmap.md \ No newline at end of file diff --git a/docs/user-guide/tls.md b/docs/user-guide/tls.md index 5f344cb9e..250b0a5b4 100644 --- a/docs/user-guide/tls.md +++ b/docs/user-guide/tls.md @@ -57,7 +57,7 @@ To disable this behavior use `hsts: "false"` in the configuration [ConfigMap][Co ## Server-side HTTPS enforcement through redirect By default the controller redirects HTTP clients to the HTTPS port -443 using a 301 Moved Permanently response if TLS is enabled for that Ingress. +443 using a 308 Permanent Redirect response if TLS is enabled for that Ingress. This can be disabled globally using `ssl-redirect: "false"` in the NGINX [config map][ConfigMap], or per-Ingress with the `nginx.ingress.kubernetes.io/ssl-redirect: "false"` @@ -102,7 +102,7 @@ For instance, TLS 1.1+ is only enabled by default from Android 5.0 on. At the ti May 2018, [approximately 15% of Android devices](https://developer.android.com/about/dashboards/#Platform) are not compatible with nginx-ingress's default configuration. -To change this default behavior, use a [ConfigMap]. +To change this default behavior, use a [ConfigMap][ConfigMap]. A sample ConfigMap fragment to allow these older clients to connect could look something like the following: From a59b811e7d9279e1908db23471ea9db2e491950d Mon Sep 17 00:00:00 2001 From: Aarni Koskela Date: Thu, 3 May 2018 13:41:12 +0300 Subject: [PATCH 11/11] Index all examples and fix their titles --- docs/examples/README.md | 32 ------------------ .../oauth-external-auth}/README.md | 0 .../dashboard-ingress.yaml | 0 .../oauth-external-auth}/images/dashboard.png | Bin .../images/github-auth.png | Bin .../images/oauth-login.png | Bin .../images/register-oauth-app-2.png | Bin .../images/register-oauth-app.png | Bin .../oauth-external-auth}/oauth2-proxy.yaml | 0 .../custom-vts-metrics-prometheus/README.md | 2 +- .../customization/ssl-dh-param/README.md | 2 +- docs/examples/index.md | 26 ++++++++++++++ 12 files changed, 28 insertions(+), 34 deletions(-) delete mode 100644 docs/examples/README.md rename docs/examples/{external-auth => auth/oauth-external-auth}/README.md (100%) rename docs/examples/{external-auth => auth/oauth-external-auth}/dashboard-ingress.yaml (100%) rename docs/examples/{external-auth => auth/oauth-external-auth}/images/dashboard.png (100%) rename docs/examples/{external-auth => auth/oauth-external-auth}/images/github-auth.png (100%) rename docs/examples/{external-auth => auth/oauth-external-auth}/images/oauth-login.png (100%) rename docs/examples/{external-auth => auth/oauth-external-auth}/images/register-oauth-app-2.png (100%) rename docs/examples/{external-auth => auth/oauth-external-auth}/images/register-oauth-app.png (100%) rename docs/examples/{external-auth => auth/oauth-external-auth}/oauth2-proxy.yaml (100%) create mode 100644 docs/examples/index.md diff --git a/docs/examples/README.md b/docs/examples/README.md deleted file mode 100644 index 06043ab01..000000000 --- a/docs/examples/README.md +++ /dev/null @@ -1,32 +0,0 @@ -# Ingress examples - -This directory contains a catalog of examples on how to run, configure and -scale Ingress. Please review the [prerequisites](PREREQUISITES.md) before -trying them. - -## Scaling - -Name | Description | Complexity Level ------| ----------- | ---------------- -[Static-ip](static-ip/README.md) | a single ingress gets a single static ip | Intermediate - -## Algorithms - -Name | Description | Complexity Level ------| ----------- | ---------------- -[Session stickyness](affinity/cookie/README.md) | route requests consistently to the same endpoint | Advanced - -## Auth - -Name | Description | Complexity Level ------| ----------- | ---------------- -[Basic auth](auth/basic/README.md) | password protect your website | nginx | Intermediate -[Client certificate authentication](auth/client-certs/README.md) | secure your website with client certificate authentication | nginx | Intermediate -[External auth plugin](external-auth/README.md) | defer to an external auth service | Intermediate - -## Customization - -Name | Description | Complexity Level ------| ----------- | ---------------- -[configuration-snippets](customization/configuration-snippets/README.md) | customize nginx location configuration using annotations | Advanced -[custom-headers](customization/custom-headers/README.md) | set custom headers before send traffic to backends | Advanced diff --git a/docs/examples/external-auth/README.md b/docs/examples/auth/oauth-external-auth/README.md similarity index 100% rename from docs/examples/external-auth/README.md rename to docs/examples/auth/oauth-external-auth/README.md diff --git a/docs/examples/external-auth/dashboard-ingress.yaml b/docs/examples/auth/oauth-external-auth/dashboard-ingress.yaml similarity index 100% rename from docs/examples/external-auth/dashboard-ingress.yaml rename to docs/examples/auth/oauth-external-auth/dashboard-ingress.yaml diff --git a/docs/examples/external-auth/images/dashboard.png b/docs/examples/auth/oauth-external-auth/images/dashboard.png similarity index 100% rename from docs/examples/external-auth/images/dashboard.png rename to docs/examples/auth/oauth-external-auth/images/dashboard.png diff --git a/docs/examples/external-auth/images/github-auth.png b/docs/examples/auth/oauth-external-auth/images/github-auth.png similarity index 100% rename from docs/examples/external-auth/images/github-auth.png rename to docs/examples/auth/oauth-external-auth/images/github-auth.png diff --git a/docs/examples/external-auth/images/oauth-login.png b/docs/examples/auth/oauth-external-auth/images/oauth-login.png similarity index 100% rename from docs/examples/external-auth/images/oauth-login.png rename to docs/examples/auth/oauth-external-auth/images/oauth-login.png diff --git a/docs/examples/external-auth/images/register-oauth-app-2.png b/docs/examples/auth/oauth-external-auth/images/register-oauth-app-2.png similarity index 100% rename from docs/examples/external-auth/images/register-oauth-app-2.png rename to docs/examples/auth/oauth-external-auth/images/register-oauth-app-2.png diff --git a/docs/examples/external-auth/images/register-oauth-app.png b/docs/examples/auth/oauth-external-auth/images/register-oauth-app.png similarity index 100% rename from docs/examples/external-auth/images/register-oauth-app.png rename to docs/examples/auth/oauth-external-auth/images/register-oauth-app.png diff --git a/docs/examples/external-auth/oauth2-proxy.yaml b/docs/examples/auth/oauth-external-auth/oauth2-proxy.yaml similarity index 100% rename from docs/examples/external-auth/oauth2-proxy.yaml rename to docs/examples/auth/oauth-external-auth/oauth2-proxy.yaml diff --git a/docs/examples/customization/custom-vts-metrics-prometheus/README.md b/docs/examples/customization/custom-vts-metrics-prometheus/README.md index fe5deb7dc..c85213905 100644 --- a/docs/examples/customization/custom-vts-metrics-prometheus/README.md +++ b/docs/examples/customization/custom-vts-metrics-prometheus/README.md @@ -1,4 +1,4 @@ -# Deploying the Nginx Ingress controller +# Custom VTS metrics with Prometheus This example aims to demonstrate the deployment of an nginx ingress controller and use a ConfigMap to enable [nginx vts module](https://github.com/vozlt/nginx-module-vts ) to export metrics in prometheus format. diff --git a/docs/examples/customization/ssl-dh-param/README.md b/docs/examples/customization/ssl-dh-param/README.md index 90801867a..f7305e77a 100644 --- a/docs/examples/customization/ssl-dh-param/README.md +++ b/docs/examples/customization/ssl-dh-param/README.md @@ -1,4 +1,4 @@ -# Deploying the Nginx Ingress controller +# Custom DH parameters for perfect forward secrecy This example aims to demonstrate the deployment of an nginx ingress controller and use a ConfigMap to configure custom Diffie-Hellman parameters file to help with diff --git a/docs/examples/index.md b/docs/examples/index.md new file mode 100644 index 000000000..feec494cd --- /dev/null +++ b/docs/examples/index.md @@ -0,0 +1,26 @@ +# Ingress examples + +This directory contains a catalog of examples on how to run, configure and scale Ingress. +Please review the [prerequisites](PREREQUISITES.md) before trying them. + +Category | Name | Description | Complexity Level +---------| ---- | ----------- | ---------------- +Apps | [Docker Registry](docker-registry/README.md) | TODO | TODO +Auth | [Basic authentication](auth/basic/README.md) | password protect your website | Intermediate +Auth | [Client certificate authentication](auth/client-certs/README.md) | secure your website with client certificate authentication | Intermediate +Auth | [External authentication plugin](auth/external-auth/README.md) | defer to an external authentication service | Intermediate +Auth | [OAuth external auth](auth/oauth-external-auth/README.md) | TODO | TODO +Customization | [Configuration snippets](customization/configuration-snippets/README.md) | customize nginx location configuration using annotations | Advanced +Customization | [Custom configuration](customization/custom-configuration/README.md) | TODO | TODO +Customization | [Custom DH parameters for perfect forward secrecy](customization/ssl-dh-param/README.md) | TODO | TODO +Customization | [Custom errors](customization/custom-errors/README.md) | TODO | TODO +Customization | [Custom headers](customization/custom-headers/README.md) | set custom headers before sending traffic to backends | Advanced +Customization | [Custom upstream check](customization/custom-upstream-check/README.md) | TODO | TODO +Customization | [Custom VTS metrics with Prometheus](customization/custom-vts-metrics-prometheus/README.md) | TODO | TODO +Customization | [External authentication with response header propagation](customization/external-auth-headers/README.md) | TODO | TODO +Customization | [Sysctl tuning](customization/sysctl/README.md) | TODO | TODO +Features | [Rewrite](rewrite/README.md) | TODO | TODO +Features | [Session stickiness](affinity/cookie/README.md) | route requests consistently to the same endpoint | Advanced +Scaling | [Static IP](static-ip/README.md) | a single ingress gets a single static IP | Intermediate +TLS | [Multi TLS certificate termination](multi-tls/README.md) | TODO | TODO +TLS | [TLS termination](tls-termination/README.md) | TODO | TODO