diff --git a/index.html b/index.html
index 0fc873025..f9b510a72 100644
--- a/index.html
+++ b/index.html
@@ -77,6 +77,7 @@ args:
 </code></pre></div></li> <li>Now, install an additional instance of the ingress-NGINX controller like this: <div class=highlight><pre><span></span><code>helm install ingress-nginx-2 ingress-nginx/ingress-nginx  \
 --namespace ingress-nginx-2 \
 --set controller.ingressClassResource.name=nginx-two \
+--set controller.ingressClass=nginx-two \
 --set controller.ingressClassResource.controllerValue=&quot;example.com/ingress-nginx-2&quot; \
 --set controller.ingressClassResource.enabled=true \
 --set controller.ingressClassByName=true
@@ -84,10 +85,11 @@ args:
 --namespace kube-system \
 --set controller.electionID=nginx-two-leader \
 --set controller.ingressClassResource.name=nginx-two \
+--set controller.ingressClass=nginx-two \
 --set controller.ingressClassResource.controllerValue=&quot;example.com/ingress-nginx-2&quot; \
 --set controller.ingressClassResource.enabled=true \
 --set controller.ingressClassByName=true
-</code></pre></div></li> </ul> </article> </div> </div> </main> <footer class=md-footer> <div class=md-footer-nav> <nav class="md-footer-nav__inner md-grid" aria-label=Footer> <a href=how-it-works/ class="md-footer-nav__link md-footer-nav__link--next" rel=next> <div class=md-footer-nav__title> <div class=md-ellipsis> <span class=md-footer-nav__direction> Next </span> How it works </div> </div> <div class="md-footer-nav__button md-icon"> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M4 11v2h12l-5.5 5.5 1.42 1.42L19.84 12l-7.92-7.92L10.5 5.5 16 11H4z"/></svg> </div> </a> </nav> </div> <div class="md-footer-meta md-typeset"> <div class="md-footer-meta__inner md-grid"> <div class=md-footer-copyright> Made with <a href=https://squidfunk.github.io/mkdocs-material/ target=_blank rel=noopener> Material for MkDocs </a> </div> </div> </div> </footer> </div> <script src=assets/javascripts/vendor.93c04032.min.js></script> <script src=assets/javascripts/bundle.83e5331e.min.js></script><script id=__lang type=application/json>{"clipboard.copy": "Copy to clipboard", "clipboard.copied": "Copied to clipboard", "search.config.lang": "en", "search.config.pipeline": "trimmer, stopWordFilter", "search.config.separator": "[\\s\\-]+", "search.placeholder": "Search", "search.result.placeholder": "Type to start searching", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.term.missing": "Missing"}</script> <script>
+</code></pre></div></li> <li>Note, controller.ingressClassResource.name and controller.ingressClass have to be set with the value of the new class as the first is to create the IngressClass object and the other is to modify the deployment of the actuall ingress controller pod.</li> </ul> </article> </div> </div> </main> <footer class=md-footer> <div class=md-footer-nav> <nav class="md-footer-nav__inner md-grid" aria-label=Footer> <a href=how-it-works/ class="md-footer-nav__link md-footer-nav__link--next" rel=next> <div class=md-footer-nav__title> <div class=md-ellipsis> <span class=md-footer-nav__direction> Next </span> How it works </div> </div> <div class="md-footer-nav__button md-icon"> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M4 11v2h12l-5.5 5.5 1.42 1.42L19.84 12l-7.92-7.92L10.5 5.5 16 11H4z"/></svg> </div> </a> </nav> </div> <div class="md-footer-meta md-typeset"> <div class="md-footer-meta__inner md-grid"> <div class=md-footer-copyright> Made with <a href=https://squidfunk.github.io/mkdocs-material/ target=_blank rel=noopener> Material for MkDocs </a> </div> </div> </div> </footer> </div> <script src=assets/javascripts/vendor.93c04032.min.js></script> <script src=assets/javascripts/bundle.83e5331e.min.js></script><script id=__lang type=application/json>{"clipboard.copy": "Copy to clipboard", "clipboard.copied": "Copied to clipboard", "search.config.lang": "en", "search.config.pipeline": "trimmer, stopWordFilter", "search.config.separator": "[\\s\\-]+", "search.placeholder": "Search", "search.result.placeholder": "Type to start searching", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.term.missing": "Missing"}</script> <script>
         app = initialize({
           base: ".",
           features: ['navigation.tabs', 'navigation.tabs.sticky', 'navigation.instant', 'navigation.sections'],
diff --git a/search/search_index.json b/search/search_index.json
index 8d02ec346..62965ed8e 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Overview \u00b6 This is the documentation for the Ingress NGINX Controller. It is built around the Kubernetes Ingress resource , using a ConfigMap to store the controller configuration. You can learn more about using Ingress in the official Kubernetes documentation . Getting Started \u00b6 See Deployment for a whirlwind tour that will get you started. FAQ - Migration to apiVersion networking.k8s.io/v1 \u00b6 If you are using Ingress objects in your cluster (running Kubernetes older than v1.22), and you plan to upgrade to Kubernetes v1.22, this section is relevant to you. Please read this official blog on deprecated Ingress API versions Please read this official documentation on the IngressClass object What is an IngressClass and why is it important for users of Ingress-NGINX controller now ? \u00b6 IngressClass is a Kubernetes resource. See the description below. It's important because until now, a default install of the Ingress-NGINX controller did not require any IngressClass object. From version 1.0.0 of the Ingress-NGINX Controller, an IngressClass object is required. On clusters with more than one instance of the Ingress-NGINX controller, all instances of the controllers must be aware of which Ingress objects they serve. The ingressClassName field of an Ingress is the way to let the controller know about that. kubectl explain ingressclass KIND: IngressClass VERSION: networking.k8s.io/v1 DESCRIPTION: IngressClass represents the class of the Ingress, referenced by the Ingress Spec. The `ingressclass.kubernetes.io/is-default-class` annotation can be used to indicate that an IngressClass should be considered default. When a single IngressClass resource has this annotation set to true, new Ingress resources without a class specified will be assigned this default class. FIELDS: apiVersion <string> APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources kind <string> Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds metadata <Object> Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata spec <Object> Spec is the desired state of the IngressClass. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status` What has caused this change in behavior? \u00b6 There are 2 reasons primarily. Reason #1 \u00b6 Until K8s version 1.21, it was possible to create an Ingress resource using deprecated versions of the Ingress API, such as: extensions/v1beta1 networking.k8s.io/v1beta1 You would get a message about deprecation, but the Ingress resource would get created. From K8s version 1.22 onwards, you can only access the Ingress API via the stable, networking.k8s.io/v1 API. The reason is explained in the official blog on deprecated ingress API versions . Reason #2 \u00b6 If you are already using the Ingress-NGINX controller and then upgrade to K8s version v1.22 , there are several scenarios where your existing Ingress objects will not work how you expect. Read this FAQ to check which scenario matches your use case. What is ingressClassName field ? \u00b6 ingressClassName is a field in the specs of an Ingress object. kubectl explain ingress.spec.ingressClassName KIND: Ingress VERSION: networking.k8s.io/v1 FIELD: ingressClassName <string> DESCRIPTION: IngressClassName is the name of the IngressClass cluster resource. The associated IngressClass defines which controller will implement the resource. This replaces the deprecated `kubernetes.io/ingress.class` annotation. For backwards compatibility, when that annotation is set, it must be given precedence over this field. The controller may emit a warning if the field and annotation have different values. Implementations of this API should ignore Ingresses without a class specified. An IngressClass resource may be marked as default, which can be used to set a default value for this field. For more information, refer to the IngressClass documentation. The .spec.ingressClassName behavior has precedence over the deprecated kubernetes.io/ingress.class annotation. I have only one ingress controller in my cluster. What should I do? \u00b6 If a single instance of the Ingress-NGINX controller is the sole Ingress controller running in your cluster, you should add the annotation \"ingressclass.kubernetes.io/is-default-class\" in your IngressClass, so any new Ingress objects will have this one as default IngressClass. When using Helm, you can enable this annotation by setting .controller.ingressClassResource.default: true in your Helm chart installation's values file. If you have any old Ingress objects remaining without an IngressClass set, you can do one or more of the following to make the Ingress-NGINX controller aware of the old objects: You can manually set the .spec.ingressClassName field in the manifest of your own Ingress resources. You can re-create them after setting the ingressclass.kubernetes.io/is-default-class annotation to true on the IngressClass Alternatively you can make the Ingress-NGINX controller watch Ingress objects without the ingressClassName field set by starting your Ingress-NGINX with the flag --watch-ingress-without-class=true . When using Helm, you can configure your Helm chart installation's values file with .controller.watchIngressWithoutClass: true You can configure your Helm chart installation's values file with .controller.watchIngressWithoutClass: true . We recommend that you create the IngressClass as shown below: --- apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: labels: app.kubernetes.io/component: controller name: nginx annotations: ingressclass.kubernetes.io/is-default-class: \"true\" spec: controller: k8s.io/ingress-nginx And add the value spec.ingressClassName=nginx in your Ingress objects. I have multiple ingress objects in my cluster. What should I do ? \u00b6 If you have lot of ingress objects without ingressClass configuration, you can run the ingress-controller with the flag --watch-ingress-without-class=true . What is the flag '--watch-ingress-without-class' ? \u00b6 Its a flag that is passed,as an argument, to the nginx-ingress-controller executable. In the configuration, it looks like this: ... ... args: - /nginx-ingress-controller - --watch-ingress-without-class=true - --publish-service=$(POD_NAMESPACE)/ingress-nginx-dev-v1-test-controller - --election-id=ingress-controller-leader - --controller-class=k8s.io/ingress-nginx - --configmap=$(POD_NAMESPACE)/ingress-nginx-dev-v1-test-controller - --validating-webhook=:8443 - --validating-webhook-certificate=/usr/local/certificates/cert - --validating-webhook-key=/usr/local/certificates/key ... ... I have more than one controller in my cluster and already use the annotation ? \u00b6 No problem. This should still keep working, but we highly recommend you to test! Even though kubernetes.io/ingress.class is deprecated, the Ingress-NGINX controller still understands that annotation. If you want to follow good practice, you should consider migrating to use IngressClass and .spec.ingressClassName . I have more than one controller running in my cluster, and I want to use the new API ? \u00b6 In this scenario, you need to create multiple IngressClasses (see example one). But be aware that IngressClass works in a very specific way: you will need to change the .spec.controller value in your IngressClass and configure the controller to expect the exact same value. Let's see some example, supposing that you have three IngressClasses: IngressClass ingress-nginx-one , with .spec.controller equal to example.com/ingress-nginx1 IngressClass ingress-nginx-two , with .spec.controller equal to example.com/ingress-nginx2 IngressClass ingress-nginx-three , with .spec.controller equal to example.com/ingress-nginx1 (for private use, you can also use a controller name that doesn't contain a / ; for example: ingress-nginx1 ) When deploying your ingress controllers, you will have to change the --controller-class field as follows: Ingress-Nginx A, configured to use controller class name example.com/ingress-nginx1 Ingress-Nginx B, configured to use controller class name example.com/ingress-nginx2 Then, when you create an Ingress object with its ingressClassName set to ingress-nginx-two , only controllers looking for the example.com/ingress-nginx2 controller class pay attention to the new object. Given that Ingress-Nginx B is set up that way, it will serve that object, whereas Ingress-Nginx A ignores the new Ingress. Bear in mind that, if you start Ingress-Nginx B with the command line argument --watch-ingress-without-class=true , then it will serve: Ingresses without any ingressClassName set Ingresses where the deprecated annotation ( kubernetes.io/ingress.class ) matches the value set in the command line argument --ingress-class Ingresses that refer to any IngressClass that has the same spec.controller as configured in --controller-class If you start Ingress-Nginx B with the command line argument --watch-ingress-without-class=true and you run Ingress-Nginx A with the command line argument --watch-ingress-without-class=false then this is a supported configuration. If you have two Ingress-NGINX controllers for the same cluster, both running with --watch-ingress-without-class=true then there is likely to be a conflict. I am seeing this error message in the logs of the Ingress-NGINX controller: \"ingress class annotation is not equal to the expected by Ingress Controller\". Why ? \u00b6 It is highly likely that you will also see the name of the ingress resource in the same error message. This error message has been observed on use the deprecated annotation ( kubernetes.io/ingress.class ) in a Ingress resource manifest. It is recommended to use the .spec.ingressClassName field of the Ingress resource, to specify the name of the IngressClass of the Ingress you are defining. How to easily install multiple instances of the ingress-NGINX controller in the same cluster ? \u00b6 Create a new namespace kubectl create namespace ingress-nginx-2 Use Helm to install the additional instance of the ingress controller Ensure you have Helm working (refer to the Helm documentation ) We have to assume that you have the helm repo for the ingress-NGINX controller already added to your Helm config. But, if you have not added the helm repo then you can do this to add the repo to your helm config; helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx Make sure you have updated the helm repo data; helm repo update Now, install an additional instance of the ingress-NGINX controller like this: helm install ingress-nginx-2 ingress-nginx/ingress-nginx \\ --namespace ingress-nginx-2 \\ --set controller.ingressClassResource.name=nginx-two \\ --set controller.ingressClassResource.controllerValue=\"example.com/ingress-nginx-2\" \\ --set controller.ingressClassResource.enabled=true \\ --set controller.ingressClassByName=true If you need to install yet another instance, then repeat the procedure to create a new namespace, change the values such as names & namespaces (for example from \"-2\" to \"-3\"), or anything else that meets your needs. If you need to install all instances in the same namespace, then you need to specify a different election id , like this: helm install ingress-nginx-2 ingress-nginx/ingress-nginx \\ --namespace kube-system \\ --set controller.electionID=nginx-two-leader \\ --set controller.ingressClassResource.name=nginx-two \\ --set controller.ingressClassResource.controllerValue=\"example.com/ingress-nginx-2\" \\ --set controller.ingressClassResource.enabled=true \\ --set controller.ingressClassByName=true","title":"Welcome"},{"location":"#overview","text":"This is the documentation for the Ingress NGINX Controller. It is built around the Kubernetes Ingress resource , using a ConfigMap to store the controller configuration. You can learn more about using Ingress in the official Kubernetes documentation .","title":"Overview"},{"location":"#getting-started","text":"See Deployment for a whirlwind tour that will get you started.","title":"Getting Started"},{"location":"#faq-migration-to-apiversion-networkingk8siov1","text":"If you are using Ingress objects in your cluster (running Kubernetes older than v1.22), and you plan to upgrade to Kubernetes v1.22, this section is relevant to you. Please read this official blog on deprecated Ingress API versions Please read this official documentation on the IngressClass object","title":"FAQ - Migration to apiVersion networking.k8s.io/v1"},{"location":"#what-is-an-ingressclass-and-why-is-it-important-for-users-of-ingress-nginx-controller-now","text":"IngressClass is a Kubernetes resource. See the description below. It's important because until now, a default install of the Ingress-NGINX controller did not require any IngressClass object. From version 1.0.0 of the Ingress-NGINX Controller, an IngressClass object is required. On clusters with more than one instance of the Ingress-NGINX controller, all instances of the controllers must be aware of which Ingress objects they serve. The ingressClassName field of an Ingress is the way to let the controller know about that. kubectl explain ingressclass KIND: IngressClass VERSION: networking.k8s.io/v1 DESCRIPTION: IngressClass represents the class of the Ingress, referenced by the Ingress Spec. The `ingressclass.kubernetes.io/is-default-class` annotation can be used to indicate that an IngressClass should be considered default. When a single IngressClass resource has this annotation set to true, new Ingress resources without a class specified will be assigned this default class. FIELDS: apiVersion <string> APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources kind <string> Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds metadata <Object> Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata spec <Object> Spec is the desired state of the IngressClass. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status`","title":"What is an IngressClass and why is it important for users of Ingress-NGINX controller now ?"},{"location":"#what-has-caused-this-change-in-behavior","text":"There are 2 reasons primarily.","title":"What has caused this change in behavior?"},{"location":"#reason-1","text":"Until K8s version 1.21, it was possible to create an Ingress resource using deprecated versions of the Ingress API, such as: extensions/v1beta1 networking.k8s.io/v1beta1 You would get a message about deprecation, but the Ingress resource would get created. From K8s version 1.22 onwards, you can only access the Ingress API via the stable, networking.k8s.io/v1 API. The reason is explained in the official blog on deprecated ingress API versions .","title":"Reason #1"},{"location":"#reason-2","text":"If you are already using the Ingress-NGINX controller and then upgrade to K8s version v1.22 , there are several scenarios where your existing Ingress objects will not work how you expect. Read this FAQ to check which scenario matches your use case.","title":"Reason #2"},{"location":"#what-is-ingressclassname-field","text":"ingressClassName is a field in the specs of an Ingress object. kubectl explain ingress.spec.ingressClassName KIND: Ingress VERSION: networking.k8s.io/v1 FIELD: ingressClassName <string> DESCRIPTION: IngressClassName is the name of the IngressClass cluster resource. The associated IngressClass defines which controller will implement the resource. This replaces the deprecated `kubernetes.io/ingress.class` annotation. For backwards compatibility, when that annotation is set, it must be given precedence over this field. The controller may emit a warning if the field and annotation have different values. Implementations of this API should ignore Ingresses without a class specified. An IngressClass resource may be marked as default, which can be used to set a default value for this field. For more information, refer to the IngressClass documentation. The .spec.ingressClassName behavior has precedence over the deprecated kubernetes.io/ingress.class annotation.","title":"What is ingressClassName field ?"},{"location":"#i-have-only-one-ingress-controller-in-my-cluster-what-should-i-do","text":"If a single instance of the Ingress-NGINX controller is the sole Ingress controller running in your cluster, you should add the annotation \"ingressclass.kubernetes.io/is-default-class\" in your IngressClass, so any new Ingress objects will have this one as default IngressClass. When using Helm, you can enable this annotation by setting .controller.ingressClassResource.default: true in your Helm chart installation's values file. If you have any old Ingress objects remaining without an IngressClass set, you can do one or more of the following to make the Ingress-NGINX controller aware of the old objects: You can manually set the .spec.ingressClassName field in the manifest of your own Ingress resources. You can re-create them after setting the ingressclass.kubernetes.io/is-default-class annotation to true on the IngressClass Alternatively you can make the Ingress-NGINX controller watch Ingress objects without the ingressClassName field set by starting your Ingress-NGINX with the flag --watch-ingress-without-class=true . When using Helm, you can configure your Helm chart installation's values file with .controller.watchIngressWithoutClass: true You can configure your Helm chart installation's values file with .controller.watchIngressWithoutClass: true . We recommend that you create the IngressClass as shown below: --- apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: labels: app.kubernetes.io/component: controller name: nginx annotations: ingressclass.kubernetes.io/is-default-class: \"true\" spec: controller: k8s.io/ingress-nginx And add the value spec.ingressClassName=nginx in your Ingress objects.","title":"I have only one ingress controller in my cluster. What should I do?"},{"location":"#i-have-multiple-ingress-objects-in-my-cluster-what-should-i-do","text":"If you have lot of ingress objects without ingressClass configuration, you can run the ingress-controller with the flag --watch-ingress-without-class=true .","title":"I have multiple ingress objects in my cluster. What should I do ?"},{"location":"#what-is-the-flag-watch-ingress-without-class","text":"Its a flag that is passed,as an argument, to the nginx-ingress-controller executable. In the configuration, it looks like this: ... ... args: - /nginx-ingress-controller - --watch-ingress-without-class=true - --publish-service=$(POD_NAMESPACE)/ingress-nginx-dev-v1-test-controller - --election-id=ingress-controller-leader - --controller-class=k8s.io/ingress-nginx - --configmap=$(POD_NAMESPACE)/ingress-nginx-dev-v1-test-controller - --validating-webhook=:8443 - --validating-webhook-certificate=/usr/local/certificates/cert - --validating-webhook-key=/usr/local/certificates/key ... ...","title":"What is the flag '--watch-ingress-without-class' ?"},{"location":"#i-have-more-than-one-controller-in-my-cluster-and-already-use-the-annotation","text":"No problem. This should still keep working, but we highly recommend you to test! Even though kubernetes.io/ingress.class is deprecated, the Ingress-NGINX controller still understands that annotation. If you want to follow good practice, you should consider migrating to use IngressClass and .spec.ingressClassName .","title":"I have more than one controller in my cluster and already use the annotation ?"},{"location":"#i-have-more-than-one-controller-running-in-my-cluster-and-i-want-to-use-the-new-api","text":"In this scenario, you need to create multiple IngressClasses (see example one). But be aware that IngressClass works in a very specific way: you will need to change the .spec.controller value in your IngressClass and configure the controller to expect the exact same value. Let's see some example, supposing that you have three IngressClasses: IngressClass ingress-nginx-one , with .spec.controller equal to example.com/ingress-nginx1 IngressClass ingress-nginx-two , with .spec.controller equal to example.com/ingress-nginx2 IngressClass ingress-nginx-three , with .spec.controller equal to example.com/ingress-nginx1 (for private use, you can also use a controller name that doesn't contain a / ; for example: ingress-nginx1 ) When deploying your ingress controllers, you will have to change the --controller-class field as follows: Ingress-Nginx A, configured to use controller class name example.com/ingress-nginx1 Ingress-Nginx B, configured to use controller class name example.com/ingress-nginx2 Then, when you create an Ingress object with its ingressClassName set to ingress-nginx-two , only controllers looking for the example.com/ingress-nginx2 controller class pay attention to the new object. Given that Ingress-Nginx B is set up that way, it will serve that object, whereas Ingress-Nginx A ignores the new Ingress. Bear in mind that, if you start Ingress-Nginx B with the command line argument --watch-ingress-without-class=true , then it will serve: Ingresses without any ingressClassName set Ingresses where the deprecated annotation ( kubernetes.io/ingress.class ) matches the value set in the command line argument --ingress-class Ingresses that refer to any IngressClass that has the same spec.controller as configured in --controller-class If you start Ingress-Nginx B with the command line argument --watch-ingress-without-class=true and you run Ingress-Nginx A with the command line argument --watch-ingress-without-class=false then this is a supported configuration. If you have two Ingress-NGINX controllers for the same cluster, both running with --watch-ingress-without-class=true then there is likely to be a conflict.","title":"I have more than one controller running in my cluster, and I want to use the new API ?"},{"location":"#i-am-seeing-this-error-message-in-the-logs-of-the-ingress-nginx-controller-ingress-class-annotation-is-not-equal-to-the-expected-by-ingress-controller-why","text":"It is highly likely that you will also see the name of the ingress resource in the same error message. This error message has been observed on use the deprecated annotation ( kubernetes.io/ingress.class ) in a Ingress resource manifest. It is recommended to use the .spec.ingressClassName field of the Ingress resource, to specify the name of the IngressClass of the Ingress you are defining.","title":"I am seeing this error message in the logs of the Ingress-NGINX controller: \"ingress class annotation is not equal to the expected by Ingress Controller\". Why ?"},{"location":"#how-to-easily-install-multiple-instances-of-the-ingress-nginx-controller-in-the-same-cluster","text":"Create a new namespace kubectl create namespace ingress-nginx-2 Use Helm to install the additional instance of the ingress controller Ensure you have Helm working (refer to the Helm documentation ) We have to assume that you have the helm repo for the ingress-NGINX controller already added to your Helm config. But, if you have not added the helm repo then you can do this to add the repo to your helm config; helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx Make sure you have updated the helm repo data; helm repo update Now, install an additional instance of the ingress-NGINX controller like this: helm install ingress-nginx-2 ingress-nginx/ingress-nginx \\ --namespace ingress-nginx-2 \\ --set controller.ingressClassResource.name=nginx-two \\ --set controller.ingressClassResource.controllerValue=\"example.com/ingress-nginx-2\" \\ --set controller.ingressClassResource.enabled=true \\ --set controller.ingressClassByName=true If you need to install yet another instance, then repeat the procedure to create a new namespace, change the values such as names & namespaces (for example from \"-2\" to \"-3\"), or anything else that meets your needs. If you need to install all instances in the same namespace, then you need to specify a different election id , like this: helm install ingress-nginx-2 ingress-nginx/ingress-nginx \\ --namespace kube-system \\ --set controller.electionID=nginx-two-leader \\ --set controller.ingressClassResource.name=nginx-two \\ --set controller.ingressClassResource.controllerValue=\"example.com/ingress-nginx-2\" \\ --set controller.ingressClassResource.enabled=true \\ --set controller.ingressClassByName=true","title":"How to easily install multiple instances of the ingress-NGINX controller in the same cluster ?"},{"location":"e2e-tests/","text":"e2e test suite for Ingress NGINX Controller \u00b6 Geoip2 \u00b6 should include geoip2 line in config when enabled and db file exists should only allow requests from specific countries [Security] global-auth-url \u00b6 should return status code 401 when request any protected service should return status code 200 when request whitelisted (via no-auth-locations) service and 401 when request protected service should return status code 200 when request whitelisted (via ingress annotation) service and 401 when request protected service should still return status code 200 after auth backend is deleted using cache should proxy_method method when global-auth-method is configured should add custom error page when global-auth-signin url is configured should add auth headers when global-auth-response-headers is configured should set request-redirect when global-auth-request-redirect is configured should set snippet when global external auth is configured user retains cookie by default user does not retain cookie if upstream returns error status code user with global-auth-always-set-cookie key in configmap retains cookie if upstream returns error status code [Security] Pod Security Policies \u00b6 should be running with a Pod Security Policy log-format-* \u00b6 should disable the log-format-escape-json by default should enable the log-format-escape-json should disable the log-format-escape-json log-format-escape-json enabled log-format-escape-json disabled server-tokens \u00b6 should not exists Server header in the response should exists Server header in the response when is enabled proxy-connect-timeout \u00b6 should set valid proxy timeouts using configmap values should not set invalid proxy timeouts using configmap values ssl-ciphers \u00b6 Add ssl ciphers use-proxy-protocol \u00b6 should respect port passed by the PROXY Protocol should respect proto passed by the PROXY Protocol server port should enable PROXY Protocol for HTTPS should enable PROXY Protocol for TCP plugins \u00b6 should exist a x-hello-world header configmap server-snippet \u00b6 should add value of server-snippet setting to all ingress config should add global server-snippet and drop annotations per admin config [Flag] disable-catch-all \u00b6 should ignore catch all Ingress with backend should ignore catch all Ingress with backend and rules should delete Ingress updated to catch-all should allow Ingress with rules enable-real-ip \u00b6 trusts X-Forwarded-For header only when setting is true should not trust X-Forwarded-For header when setting is false keep-alive keep-alive-requests \u00b6 should set keepalive_timeout should set keepalive_requests should set keepalive connection to upstream server should set keep alive connection timeout to upstream server should set the request count to upstream server through one keep alive connection enable-multi-accept \u00b6 should be enabled by default should be enabled when set to true should be disabled when set to false [Flag] watch namespace selector \u00b6 should ingore Ingress of namespace without label foo=bar and accept those of namespace with label foo=bar [Flag] disable-service-external-name \u00b6 should ignore services of external-name type Configure OpenTracing \u00b6 should not exists opentracing directive should exists opentracing directive when is enabled should include opentracing_trust_incoming_span off directive when disabled should not exists opentracing_operation_name directive when is empty should exists opentracing_operation_name directive when is configured should not exists opentracing_location_operation_name directive when is empty should exists opentracing_location_operation_name directive when is configured should enable opentracing using zipkin should enable opentracing using jaeger should enable opentracing using jaeger with sampler host should propagate the w3c header when configured with jaeger should enable opentracing using jaeger with an HTTP endpoint should enable opentracing using datadog Configmap change \u00b6 should reload after an update in the configuration Configmap - limit-rate \u00b6 Check limit-rate config access-log \u00b6 use the default configuration use the specified configuration use the specified configuration use the specified configuration use the specified configuration global-options \u00b6 should have worker_rlimit_nofile option should have worker_rlimit_nofile option and be independent on amount of worker processes [Flag] ingress-class \u00b6 should ignore Ingress with a different class annotation should ignore Ingress with different controller class should accept both Ingresses with default IngressClassName and IngressClass annotation should ignore Ingress without IngressClass configuration should delete Ingress when class is removed should serve Ingress when class is added should serve Ingress when class is updated between annotation and ingressClassName should ignore Ingress with no class and accept the correctly configured Ingresses should watch Ingress with no class and ignore ingress with a different class should watch Ingress that uses the class name even if spec is different should watch Ingress with correct annotation should ignore Ingress with only IngressClassName reuse-port \u00b6 reuse port should be enabled by default reuse port should be disabled reuse port should be enabled proxy-send-timeout \u00b6 should set valid proxy send timeouts using configmap values should not set invalid proxy send timeouts using configmap values [SSL] [Flag] default-ssl-certificate \u00b6 uses default ssl certificate for catch-all ingress uses default ssl certificate for host based ingress when configured certificate does not match host brotli \u00b6 condition [Security] Pod Security Policies with volumes \u00b6 should be running with a Pod Security Policy Dynamic $proxy_host \u00b6 should exist a proxy_host should exist a proxy_host using the upstream-vhost annotation value [Security] block-* \u00b6 should block CIDRs defined in the ConfigMap should block User-Agents defined in the ConfigMap should block Referers defined in the ConfigMap settings-global-rate-limit \u00b6 generates correct NGINX configuration Add no tls redirect locations \u00b6 Check no tls redirect locations config main-snippet \u00b6 should add value of main-snippet setting to nginx config [Lua] lua-shared-dicts \u00b6 configures lua shared dicts Bad annotation values \u00b6 [BAD_ANNOTATIONS] should drop an ingress if there is an invalid character in some annotation [BAD_ANNOTATIONS] should drop an ingress if there is a forbidden word in some annotation [BAD_ANNOTATIONS] should allow an ingress if there is a default blocklist config in place [BAD_ANNOTATIONS] should drop an ingress if there is a custom blocklist config in place and allow others to pass [SSL] TLS protocols, ciphers and headers) \u00b6 setting cipher suite enforcing TLS v1.0 setting max-age parameter setting includeSubDomains parameter setting preload parameter overriding what's set from the upstream should not use ports during the HTTP to HTTPS redirection should not use ports or X-Forwarded-Host during the HTTP to HTTPS redirection [Security] modsecurity-snippet \u00b6 should add value of modsecurity-snippet setting to nginx config OCSP \u00b6 should enable OCSP and contain stapling information in the connection configmap stream-snippet \u00b6 should add value of stream-snippet via config map to nginx config proxy-read-timeout \u00b6 should set valid proxy read timeouts using configmap values should not set invalid proxy read timeouts using configmap values proxy-next-upstream \u00b6 should build proxy next upstream using configmap values hash size \u00b6 should set server_names_hash_bucket_size should set server_names_hash_max_size should set proxy-headers-hash-bucket-size should set proxy-headers-hash-max-size should set variables-hash-bucket-size should set variables-hash-max-size should set vmap-hash-bucket-size [Security] no-auth-locations \u00b6 should return status code 401 when accessing '/' unauthentication should return status code 200 when accessing '/' authentication should return status code 200 when accessing '/noauth' unauthenticated [Flag] custom HTTP and HTTPS ports \u00b6 should set X-Forwarded-Port headers accordingly when listening on a non-default HTTP port should set X-Forwarded-Port header to 443 should set the X-Forwarded-Port header to 443 use-forwarded-headers \u00b6 should trust X-Forwarded headers when setting is true should not trust X-Forwarded headers when setting is false add-headers \u00b6 Add a custom header Add multiple custom headers [Load Balancer] round-robin \u00b6 should evenly distribute requests with round-robin (default algorithm) [Load Balancer] EWMA \u00b6 does not fail requests [Load Balancer] load-balance \u00b6 should apply the configmap load-balance setting [SSL] redirect to HTTPS \u00b6 should redirect from HTTP to HTTPS when secret is missing [SSL] secret update \u00b6 should not appear references to secret updates not used in ingress rules should return the fake SSL certificate if the secret is invalid [Service] Type ExternalName \u00b6 works with external name set to incomplete fqdn should return 200 for service type=ExternalName without a port defined should return 200 for service type=ExternalName with a port defined should return status 502 for service type=ExternalName with an invalid host should return 200 for service type=ExternalName using a port name should return 200 for service type=ExternalName using FQDN with trailing dot should update the external name after a service update should sync ingress on external name service addition/deletion [Service] Nil Service Backend \u00b6 should return 404 when backend service is nil [Service] backend status code 503 \u00b6 should return 503 when backend service does not exist should return 503 when all backend service endpoints are unavailable","title":"E2e tests"},{"location":"e2e-tests/#e2e-test-suite-for-ingress-nginx-controller","text":"","title":"e2e test suite for Ingress NGINX Controller"},{"location":"e2e-tests/#geoip2","text":"should include geoip2 line in config when enabled and db file exists should only allow requests from specific countries","title":"Geoip2"},{"location":"e2e-tests/#security-global-auth-url","text":"should return status code 401 when request any protected service should return status code 200 when request whitelisted (via no-auth-locations) service and 401 when request protected service should return status code 200 when request whitelisted (via ingress annotation) service and 401 when request protected service should still return status code 200 after auth backend is deleted using cache should proxy_method method when global-auth-method is configured should add custom error page when global-auth-signin url is configured should add auth headers when global-auth-response-headers is configured should set request-redirect when global-auth-request-redirect is configured should set snippet when global external auth is configured user retains cookie by default user does not retain cookie if upstream returns error status code user with global-auth-always-set-cookie key in configmap retains cookie if upstream returns error status code","title":"[Security] global-auth-url"},{"location":"e2e-tests/#security-pod-security-policies","text":"should be running with a Pod Security Policy","title":"[Security] Pod Security Policies"},{"location":"e2e-tests/#log-format-","text":"should disable the log-format-escape-json by default should enable the log-format-escape-json should disable the log-format-escape-json log-format-escape-json enabled log-format-escape-json disabled","title":"log-format-*"},{"location":"e2e-tests/#server-tokens","text":"should not exists Server header in the response should exists Server header in the response when is enabled","title":"server-tokens"},{"location":"e2e-tests/#proxy-connect-timeout","text":"should set valid proxy timeouts using configmap values should not set invalid proxy timeouts using configmap values","title":"proxy-connect-timeout"},{"location":"e2e-tests/#ssl-ciphers","text":"Add ssl ciphers","title":"ssl-ciphers"},{"location":"e2e-tests/#use-proxy-protocol","text":"should respect port passed by the PROXY Protocol should respect proto passed by the PROXY Protocol server port should enable PROXY Protocol for HTTPS should enable PROXY Protocol for TCP","title":"use-proxy-protocol"},{"location":"e2e-tests/#plugins","text":"should exist a x-hello-world header","title":"plugins"},{"location":"e2e-tests/#configmap-server-snippet","text":"should add value of server-snippet setting to all ingress config should add global server-snippet and drop annotations per admin config","title":"configmap server-snippet"},{"location":"e2e-tests/#flag-disable-catch-all","text":"should ignore catch all Ingress with backend should ignore catch all Ingress with backend and rules should delete Ingress updated to catch-all should allow Ingress with rules","title":"[Flag] disable-catch-all"},{"location":"e2e-tests/#enable-real-ip","text":"trusts X-Forwarded-For header only when setting is true should not trust X-Forwarded-For header when setting is false","title":"enable-real-ip"},{"location":"e2e-tests/#keep-alive-keep-alive-requests","text":"should set keepalive_timeout should set keepalive_requests should set keepalive connection to upstream server should set keep alive connection timeout to upstream server should set the request count to upstream server through one keep alive connection","title":"keep-alive keep-alive-requests"},{"location":"e2e-tests/#enable-multi-accept","text":"should be enabled by default should be enabled when set to true should be disabled when set to false","title":"enable-multi-accept"},{"location":"e2e-tests/#flag-watch-namespace-selector","text":"should ingore Ingress of namespace without label foo=bar and accept those of namespace with label foo=bar","title":"[Flag] watch namespace selector"},{"location":"e2e-tests/#flag-disable-service-external-name","text":"should ignore services of external-name type","title":"[Flag] disable-service-external-name"},{"location":"e2e-tests/#configure-opentracing","text":"should not exists opentracing directive should exists opentracing directive when is enabled should include opentracing_trust_incoming_span off directive when disabled should not exists opentracing_operation_name directive when is empty should exists opentracing_operation_name directive when is configured should not exists opentracing_location_operation_name directive when is empty should exists opentracing_location_operation_name directive when is configured should enable opentracing using zipkin should enable opentracing using jaeger should enable opentracing using jaeger with sampler host should propagate the w3c header when configured with jaeger should enable opentracing using jaeger with an HTTP endpoint should enable opentracing using datadog","title":"Configure OpenTracing"},{"location":"e2e-tests/#configmap-change","text":"should reload after an update in the configuration","title":"Configmap change"},{"location":"e2e-tests/#configmap-limit-rate","text":"Check limit-rate config","title":"Configmap - limit-rate"},{"location":"e2e-tests/#access-log","text":"use the default configuration use the specified configuration use the specified configuration use the specified configuration use the specified configuration","title":"access-log"},{"location":"e2e-tests/#global-options","text":"should have worker_rlimit_nofile option should have worker_rlimit_nofile option and be independent on amount of worker processes","title":"global-options"},{"location":"e2e-tests/#flag-ingress-class","text":"should ignore Ingress with a different class annotation should ignore Ingress with different controller class should accept both Ingresses with default IngressClassName and IngressClass annotation should ignore Ingress without IngressClass configuration should delete Ingress when class is removed should serve Ingress when class is added should serve Ingress when class is updated between annotation and ingressClassName should ignore Ingress with no class and accept the correctly configured Ingresses should watch Ingress with no class and ignore ingress with a different class should watch Ingress that uses the class name even if spec is different should watch Ingress with correct annotation should ignore Ingress with only IngressClassName","title":"[Flag] ingress-class"},{"location":"e2e-tests/#reuse-port","text":"reuse port should be enabled by default reuse port should be disabled reuse port should be enabled","title":"reuse-port"},{"location":"e2e-tests/#proxy-send-timeout","text":"should set valid proxy send timeouts using configmap values should not set invalid proxy send timeouts using configmap values","title":"proxy-send-timeout"},{"location":"e2e-tests/#ssl-flag-default-ssl-certificate","text":"uses default ssl certificate for catch-all ingress uses default ssl certificate for host based ingress when configured certificate does not match host","title":"[SSL] [Flag] default-ssl-certificate"},{"location":"e2e-tests/#brotli","text":"condition","title":"brotli"},{"location":"e2e-tests/#security-pod-security-policies-with-volumes","text":"should be running with a Pod Security Policy","title":"[Security] Pod Security Policies with volumes"},{"location":"e2e-tests/#dynamic-proxy_host","text":"should exist a proxy_host should exist a proxy_host using the upstream-vhost annotation value","title":"Dynamic $proxy_host"},{"location":"e2e-tests/#security-block-","text":"should block CIDRs defined in the ConfigMap should block User-Agents defined in the ConfigMap should block Referers defined in the ConfigMap","title":"[Security] block-*"},{"location":"e2e-tests/#settings-global-rate-limit","text":"generates correct NGINX configuration","title":"settings-global-rate-limit"},{"location":"e2e-tests/#add-no-tls-redirect-locations","text":"Check no tls redirect locations config","title":"Add no tls redirect locations"},{"location":"e2e-tests/#main-snippet","text":"should add value of main-snippet setting to nginx config","title":"main-snippet"},{"location":"e2e-tests/#lua-lua-shared-dicts","text":"configures lua shared dicts","title":"[Lua] lua-shared-dicts"},{"location":"e2e-tests/#bad-annotation-values","text":"[BAD_ANNOTATIONS] should drop an ingress if there is an invalid character in some annotation [BAD_ANNOTATIONS] should drop an ingress if there is a forbidden word in some annotation [BAD_ANNOTATIONS] should allow an ingress if there is a default blocklist config in place [BAD_ANNOTATIONS] should drop an ingress if there is a custom blocklist config in place and allow others to pass","title":"Bad annotation values"},{"location":"e2e-tests/#ssl-tls-protocols-ciphers-and-headers","text":"setting cipher suite enforcing TLS v1.0 setting max-age parameter setting includeSubDomains parameter setting preload parameter overriding what's set from the upstream should not use ports during the HTTP to HTTPS redirection should not use ports or X-Forwarded-Host during the HTTP to HTTPS redirection","title":"[SSL] TLS protocols, ciphers and headers)"},{"location":"e2e-tests/#security-modsecurity-snippet","text":"should add value of modsecurity-snippet setting to nginx config","title":"[Security] modsecurity-snippet"},{"location":"e2e-tests/#ocsp","text":"should enable OCSP and contain stapling information in the connection","title":"OCSP"},{"location":"e2e-tests/#configmap-stream-snippet","text":"should add value of stream-snippet via config map to nginx config","title":"configmap stream-snippet"},{"location":"e2e-tests/#proxy-read-timeout","text":"should set valid proxy read timeouts using configmap values should not set invalid proxy read timeouts using configmap values","title":"proxy-read-timeout"},{"location":"e2e-tests/#proxy-next-upstream","text":"should build proxy next upstream using configmap values","title":"proxy-next-upstream"},{"location":"e2e-tests/#hash-size","text":"should set server_names_hash_bucket_size should set server_names_hash_max_size should set proxy-headers-hash-bucket-size should set proxy-headers-hash-max-size should set variables-hash-bucket-size should set variables-hash-max-size should set vmap-hash-bucket-size","title":"hash size"},{"location":"e2e-tests/#security-no-auth-locations","text":"should return status code 401 when accessing '/' unauthentication should return status code 200 when accessing '/' authentication should return status code 200 when accessing '/noauth' unauthenticated","title":"[Security] no-auth-locations"},{"location":"e2e-tests/#flag-custom-http-and-https-ports","text":"should set X-Forwarded-Port headers accordingly when listening on a non-default HTTP port should set X-Forwarded-Port header to 443 should set the X-Forwarded-Port header to 443","title":"[Flag] custom HTTP and HTTPS ports"},{"location":"e2e-tests/#use-forwarded-headers","text":"should trust X-Forwarded headers when setting is true should not trust X-Forwarded headers when setting is false","title":"use-forwarded-headers"},{"location":"e2e-tests/#add-headers","text":"Add a custom header Add multiple custom headers","title":"add-headers"},{"location":"e2e-tests/#load-balancer-round-robin","text":"should evenly distribute requests with round-robin (default algorithm)","title":"[Load Balancer] round-robin"},{"location":"e2e-tests/#load-balancer-ewma","text":"does not fail requests","title":"[Load Balancer] EWMA"},{"location":"e2e-tests/#load-balancer-load-balance","text":"should apply the configmap load-balance setting","title":"[Load Balancer] load-balance"},{"location":"e2e-tests/#ssl-redirect-to-https","text":"should redirect from HTTP to HTTPS when secret is missing","title":"[SSL] redirect to HTTPS"},{"location":"e2e-tests/#ssl-secret-update","text":"should not appear references to secret updates not used in ingress rules should return the fake SSL certificate if the secret is invalid","title":"[SSL] secret update"},{"location":"e2e-tests/#service-type-externalname","text":"works with external name set to incomplete fqdn should return 200 for service type=ExternalName without a port defined should return 200 for service type=ExternalName with a port defined should return status 502 for service type=ExternalName with an invalid host should return 200 for service type=ExternalName using a port name should return 200 for service type=ExternalName using FQDN with trailing dot should update the external name after a service update should sync ingress on external name service addition/deletion","title":"[Service] Type ExternalName"},{"location":"e2e-tests/#service-nil-service-backend","text":"should return 404 when backend service is nil","title":"[Service] Nil Service Backend"},{"location":"e2e-tests/#service-backend-status-code-503","text":"should return 503 when backend service does not exist should return 503 when all backend service endpoints are unavailable","title":"[Service] backend status code 503"},{"location":"how-it-works/","text":"How it works \u00b6 The objective of this document is to explain how the Ingress-NGINX controller works, in particular how the NGINX model is built and why we need one. NGINX configuration \u00b6 The goal of this Ingress controller is the assembly of a configuration file (nginx.conf). The main implication of this requirement is the need to reload NGINX after any change in the configuration file. Though it is important to note that we don't reload Nginx on changes that impact only an upstream configuration (i.e Endpoints change when you deploy your app) . We use lua-nginx-module to achieve this. Check below to learn more about how it's done. NGINX model \u00b6 Usually, a Kubernetes Controller utilizes the synchronization loop pattern to check if the desired state in the controller is updated or a change is required. To this purpose, we need to build a model using different objects from the cluster, in particular (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster. To get this object from the cluster, we use Kubernetes Informers , in particular, FilteredSharedInformer . This informers allows reacting to changes in using callbacks to individual changes when a new object is added, modified or removed. Unfortunately, there is no way to know if a particular change is going to affect the final configuration file. Therefore on every change, we have to rebuild a new model from scratch based on the state of cluster and compare it to the current model. If the new model equals to the current one, then we avoid generating a new NGINX configuration and triggering a reload. Otherwise, we check if the difference is only about Endpoints. If so we then send the new list of Endpoints to a Lua handler running inside Nginx using HTTP POST request and again avoid generating a new NGINX configuration and triggering a reload. If the difference between running and new model is about more than just Endpoints we create a new NGINX configuration based on the new model, replace the current model and trigger a reload. One of the uses of the model is to avoid unnecessary reloads when there's no change in the state and to detect conflicts in definitions. The final representation of the NGINX configuration is generated from a Go template using the new model as input for the variables required by the template. Building the NGINX model \u00b6 Building a model is an expensive operation, for this reason, the use of the synchronization loop is a must. By using a work queue it is possible to not lose changes and remove the use of sync.Mutex to force a single execution of the sync loop and additionally it is possible to create a time window between the start and end of the sync loop that allows us to discard unnecessary updates. It is important to understand that any change in the cluster could generate events that the informer will send to the controller and one of the reasons for the work queue . Operations to build the model: Order Ingress rules by CreationTimestamp field, i.e., old rules first. If the same path for the same host is defined in more than one Ingress, the oldest rule wins. If more than one Ingress contains a TLS section for the same host, the oldest rule wins. If multiple Ingresses define an annotation that affects the configuration of the Server block, the oldest rule wins. Create a list of NGINX Servers (per hostname) Create a list of NGINX Upstreams If multiple Ingresses define different paths for the same host, the ingress controller will merge the definitions. Annotations are applied to all the paths in the Ingress. Multiple Ingresses can define different annotations. These definitions are not shared between Ingresses. When a reload is required \u00b6 The next list describes the scenarios when a reload is required: New Ingress Resource Created. TLS section is added to existing Ingress. Change in Ingress annotations that impacts more than just upstream configuration. For instance load-balance annotation does not require a reload. A path is added/removed from an Ingress. An Ingress, Service, Secret is removed. Some missing referenced object from the Ingress is available, like a Service or Secret. A Secret is updated. Avoiding reloads \u00b6 In some cases, it is possible to avoid reloads, in particular when there is a change in the endpoints, i.e., a pod is started or replaced. It is out of the scope of this Ingress controller to remove reloads completely. This would require an incredible amount of work and at some point makes no sense. This can change only if NGINX changes the way new configurations are read, basically, new changes do not replace worker processes. Avoiding reloads on Endpoints changes \u00b6 On every endpoint change the controller fetches endpoints from all the services it sees and generates corresponding Backend objects. It then sends these objects to a Lua handler running inside Nginx. The Lua code in turn stores those backends in a shared memory zone. Then for every request Lua code running in balancer_by_lua context detects what endpoints it should choose upstream peer from and applies the configured load balancing algorithm to choose the peer. Then Nginx takes care of the rest. This way we avoid reloading Nginx on endpoint changes. Note that this includes annotation changes that affects only upstream configuration in Nginx as well. In a relatively big cluster with frequently deploying apps this feature saves significant number of Nginx reloads which can otherwise affect response latency, load balancing quality (after every reload Nginx resets the state of load balancing) and so on. Avoiding outage from wrong configuration \u00b6 Because the ingress controller works using the synchronization loop pattern , it is applying the configuration for all matching objects. In case some Ingress objects have a broken configuration, for example a syntax error in the nginx.ingress.kubernetes.io/configuration-snippet annotation, the generated configuration becomes invalid, does not reload and hence no more ingresses will be taken into account. To prevent this situation to happen, the nginx ingress controller optionally exposes a validating admission webhook server to ensure the validity of incoming ingress objects. This webhook appends the incoming ingress objects to the list of ingresses, generates the configuration and calls nginx to ensure the configuration has no syntax errors.","title":"How it works"},{"location":"how-it-works/#how-it-works","text":"The objective of this document is to explain how the Ingress-NGINX controller works, in particular how the NGINX model is built and why we need one.","title":"How it works"},{"location":"how-it-works/#nginx-configuration","text":"The goal of this Ingress controller is the assembly of a configuration file (nginx.conf). The main implication of this requirement is the need to reload NGINX after any change in the configuration file. Though it is important to note that we don't reload Nginx on changes that impact only an upstream configuration (i.e Endpoints change when you deploy your app) . We use lua-nginx-module to achieve this. Check below to learn more about how it's done.","title":"NGINX configuration"},{"location":"how-it-works/#nginx-model","text":"Usually, a Kubernetes Controller utilizes the synchronization loop pattern to check if the desired state in the controller is updated or a change is required. To this purpose, we need to build a model using different objects from the cluster, in particular (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster. To get this object from the cluster, we use Kubernetes Informers , in particular, FilteredSharedInformer . This informers allows reacting to changes in using callbacks to individual changes when a new object is added, modified or removed. Unfortunately, there is no way to know if a particular change is going to affect the final configuration file. Therefore on every change, we have to rebuild a new model from scratch based on the state of cluster and compare it to the current model. If the new model equals to the current one, then we avoid generating a new NGINX configuration and triggering a reload. Otherwise, we check if the difference is only about Endpoints. If so we then send the new list of Endpoints to a Lua handler running inside Nginx using HTTP POST request and again avoid generating a new NGINX configuration and triggering a reload. If the difference between running and new model is about more than just Endpoints we create a new NGINX configuration based on the new model, replace the current model and trigger a reload. One of the uses of the model is to avoid unnecessary reloads when there's no change in the state and to detect conflicts in definitions. The final representation of the NGINX configuration is generated from a Go template using the new model as input for the variables required by the template.","title":"NGINX model"},{"location":"how-it-works/#building-the-nginx-model","text":"Building a model is an expensive operation, for this reason, the use of the synchronization loop is a must. By using a work queue it is possible to not lose changes and remove the use of sync.Mutex to force a single execution of the sync loop and additionally it is possible to create a time window between the start and end of the sync loop that allows us to discard unnecessary updates. It is important to understand that any change in the cluster could generate events that the informer will send to the controller and one of the reasons for the work queue . Operations to build the model: Order Ingress rules by CreationTimestamp field, i.e., old rules first. If the same path for the same host is defined in more than one Ingress, the oldest rule wins. If more than one Ingress contains a TLS section for the same host, the oldest rule wins. If multiple Ingresses define an annotation that affects the configuration of the Server block, the oldest rule wins. Create a list of NGINX Servers (per hostname) Create a list of NGINX Upstreams If multiple Ingresses define different paths for the same host, the ingress controller will merge the definitions. Annotations are applied to all the paths in the Ingress. Multiple Ingresses can define different annotations. These definitions are not shared between Ingresses.","title":"Building the NGINX model"},{"location":"how-it-works/#when-a-reload-is-required","text":"The next list describes the scenarios when a reload is required: New Ingress Resource Created. TLS section is added to existing Ingress. Change in Ingress annotations that impacts more than just upstream configuration. For instance load-balance annotation does not require a reload. A path is added/removed from an Ingress. An Ingress, Service, Secret is removed. Some missing referenced object from the Ingress is available, like a Service or Secret. A Secret is updated.","title":"When a reload is required"},{"location":"how-it-works/#avoiding-reloads","text":"In some cases, it is possible to avoid reloads, in particular when there is a change in the endpoints, i.e., a pod is started or replaced. It is out of the scope of this Ingress controller to remove reloads completely. This would require an incredible amount of work and at some point makes no sense. This can change only if NGINX changes the way new configurations are read, basically, new changes do not replace worker processes.","title":"Avoiding reloads"},{"location":"how-it-works/#avoiding-reloads-on-endpoints-changes","text":"On every endpoint change the controller fetches endpoints from all the services it sees and generates corresponding Backend objects. It then sends these objects to a Lua handler running inside Nginx. The Lua code in turn stores those backends in a shared memory zone. Then for every request Lua code running in balancer_by_lua context detects what endpoints it should choose upstream peer from and applies the configured load balancing algorithm to choose the peer. Then Nginx takes care of the rest. This way we avoid reloading Nginx on endpoint changes. Note that this includes annotation changes that affects only upstream configuration in Nginx as well. In a relatively big cluster with frequently deploying apps this feature saves significant number of Nginx reloads which can otherwise affect response latency, load balancing quality (after every reload Nginx resets the state of load balancing) and so on.","title":"Avoiding reloads on Endpoints changes"},{"location":"how-it-works/#avoiding-outage-from-wrong-configuration","text":"Because the ingress controller works using the synchronization loop pattern , it is applying the configuration for all matching objects. In case some Ingress objects have a broken configuration, for example a syntax error in the nginx.ingress.kubernetes.io/configuration-snippet annotation, the generated configuration becomes invalid, does not reload and hence no more ingresses will be taken into account. To prevent this situation to happen, the nginx ingress controller optionally exposes a validating admission webhook server to ensure the validity of incoming ingress objects. This webhook appends the incoming ingress objects to the list of ingresses, generates the configuration and calls nginx to ensure the configuration has no syntax errors.","title":"Avoiding outage from wrong configuration"},{"location":"kubectl-plugin/","text":"The ingress-nginx kubectl plugin \u00b6 Installation \u00b6 Install krew , then run kubectl krew install ingress-nginx to install the plugin. Then run kubectl ingress-nginx --help to make sure the plugin is properly installed and to get a list of commands: kubectl ingress-nginx --help A kubectl plugin for inspecting your ingress-nginx deployments Usage: ingress-nginx [command] Available Commands: backends Inspect the dynamic backend information of an ingress-nginx instance certs Output the certificate data stored in an ingress-nginx pod conf Inspect the generated nginx.conf exec Execute a command inside an ingress-nginx pod general Inspect the other dynamic ingress-nginx information help Help about any command info Show information about the ingress-nginx service ingresses Provide a short summary of all of the ingress definitions lint Inspect kubernetes resources for possible issues logs Get the kubernetes logs for an ingress-nginx pod ssh ssh into a running ingress-nginx pod Flags: --as string Username to impersonate for the operation --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. --cache-dir string Default HTTP cache directory (default \"/Users/alexkursell/.kube/http-cache\") --certificate-authority string Path to a cert file for the certificate authority --client-certificate string Path to a client certificate file for TLS --client-key string Path to a client key file for TLS --cluster string The name of the kubeconfig cluster to use --context string The name of the kubeconfig context to use -h, --help help for ingress-nginx --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure --kubeconfig string Path to the kubeconfig file to use for CLI requests. -n, --namespace string If present, the namespace scope for this CLI request --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\") -s, --server string The address and port of the Kubernetes API server --token string Bearer token for authentication to the API server --user string The name of the kubeconfig user to use Use \"ingress-nginx [command] --help\" for more information about a command. Common Flags \u00b6 Every subcommand supports the basic kubectl configuration flags like --namespace , --context , --client-key and so on. Subcommands that act on a particular ingress-nginx pod ( backends , certs , conf , exec , general , logs , ssh ), support the --deployment <deployment> and --pod <pod> flags to select either a pod from a deployment with the given name, or a pod with the given name. The --deployment flag defaults to ingress-nginx-controller . Subcommands that inspect resources ( ingresses , lint ) support the --all-namespaces flag, which causes them to inspect resources in every namespace. Subcommands \u00b6 Note that backends , general , certs , and conf require ingress-nginx version 0.23.0 or higher. backends \u00b6 Run kubectl ingress-nginx backends to get a JSON array of the backends that an ingress-nginx controller currently knows about: $ kubectl ingress-nginx backends -n ingress-nginx [ { \"name\": \"default-apple-service-5678\", \"service\": { \"metadata\": { \"creationTimestamp\": null }, \"spec\": { \"ports\": [ { \"protocol\": \"TCP\", \"port\": 5678, \"targetPort\": 5678 } ], \"selector\": { \"app\": \"apple\" }, \"clusterIP\": \"10.97.230.121\", \"type\": \"ClusterIP\", \"sessionAffinity\": \"None\" }, \"status\": { \"loadBalancer\": {} } }, \"port\": 0, \"sslPassthrough\": false, \"endpoints\": [ { \"address\": \"10.1.3.86\", \"port\": \"5678\" } ], \"sessionAffinityConfig\": { \"name\": \"\", \"cookieSessionAffinity\": { \"name\": \"\" } }, \"upstreamHashByConfig\": { \"upstream-hash-by-subset-size\": 3 }, \"noServer\": false, \"trafficShapingPolicy\": { \"weight\": 0, \"header\": \"\", \"headerValue\": \"\", \"cookie\": \"\" } }, { \"name\": \"default-echo-service-8080\", ... }, { \"name\": \"upstream-default-backend\", ... } ] Add the --list option to show only the backend names. Add the --backend <backend> option to show only the backend with the given name. certs \u00b6 Use kubectl ingress-nginx certs --host <hostname> to dump the SSL cert/key information for a given host. WARNING: This command will dump sensitive private key information. Don't blindly share the output, and certainly don't log it anywhere. $ kubectl ingress-nginx certs -n ingress-nginx --host testaddr.local -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN RSA PRIVATE KEY----- <REDACTED! DO NOT SHARE THIS!> -----END RSA PRIVATE KEY----- conf \u00b6 Use kubectl ingress-nginx conf to dump the generated nginx.conf file. Add the --host <hostname> option to view only the server block for that host: kubectl ingress-nginx conf -n ingress-nginx --host testaddr.local server { server_name testaddr.local ; listen 80; set $proxy_upstream_name \"-\"; set $pass_access_scheme $scheme; set $pass_server_port $server_port; set $best_http_host $http_host; set $pass_port $pass_server_port; location / { set $namespace \"\"; set $ingress_name \"\"; set $service_name \"\"; set $service_port \"0\"; set $location_path \"/\"; ... exec \u00b6 kubectl ingress-nginx exec is exactly the same as kubectl exec , with the same command flags. It will automatically choose an ingress-nginx pod to run the command in. $ kubectl ingress-nginx exec -i -n ingress-nginx -- ls /etc/nginx fastcgi_params geoip lua mime.types modsecurity modules nginx.conf opentracing.json owasp-modsecurity-crs template info \u00b6 Shows the internal and external IP/CNAMES for an ingress-nginx service. $ kubectl ingress-nginx info -n ingress-nginx Service cluster IP address: 10.187.253.31 LoadBalancer IP|CNAME: 35.123.123.123 Use the --service <service> flag if your ingress-nginx LoadBalancer service is not named ingress-nginx . ingresses \u00b6 kubectl ingress-nginx ingresses , alternately kubectl ingress-nginx ing , shows a more detailed view of the ingress definitions in a namespace. Compare: $ kubectl get ingresses --all-namespaces NAMESPACE NAME HOSTS ADDRESS PORTS AGE default example-ingress1 testaddr.local,testaddr2.local localhost 80 5d default test-ingress-2 * localhost 80 5d vs. $ kubectl ingress-nginx ingresses --all-namespaces NAMESPACE INGRESS NAME HOST+PATH ADDRESSES TLS SERVICE SERVICE PORT ENDPOINTS default example-ingress1 testaddr.local/etameta localhost NO pear-service 5678 5 default example-ingress1 testaddr2.local/otherpath localhost NO apple-service 5678 1 default example-ingress1 testaddr2.local/otherotherpath localhost NO pear-service 5678 5 default test-ingress-2 * localhost NO echo-service 8080 2 lint \u00b6 kubectl ingress-nginx lint can check a namespace or entire cluster for potential configuration issues. This command is especially useful when upgrading between ingress-nginx versions. $ kubectl ingress-nginx lint --all-namespaces --verbose Checking ingresses... \u2717 anamespace/this-nginx - Contains the removed session-cookie-hash annotation. Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3743 \u2717 othernamespace/ingress-definition-blah - The rewrite-target annotation value does not reference a capture group Lint added for version 0.22.0 https://github.com/kubernetes/ingress-nginx/issues/3174 Checking deployments... \u2717 namespace2/ingress-nginx-controller - Uses removed config flag --sort-backends Lint added for version 0.22.0 https://github.com/kubernetes/ingress-nginx/issues/3655 - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808 To show the lints added only for a particular ingress-nginx release, use the --from-version and --to-version flags: $ kubectl ingress-nginx lint --all-namespaces --verbose --from-version 0 .24.0 --to-version 0 .24.0 Checking ingresses... \u2717 anamespace/this-nginx - Contains the removed session-cookie-hash annotation. Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3743 Checking deployments... \u2717 namespace2/ingress-nginx-controller - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808 logs \u00b6 kubectl ingress-nginx logs is almost the same as kubectl logs , with fewer flags. It will automatically choose an ingress-nginx pod to read logs from. $ kubectl ingress-nginx logs -n ingress-nginx ------------------------------------------------------------------------------- NGINX Ingress controller Release: dev Build: git-48dc3a867 Repository: git@github.com:kubernetes/ingress-nginx.git ------------------------------------------------------------------------------- W0405 16:53:46.061589 7 flags.go:214] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false) nginx version: nginx/1.15.9 W0405 16:53:46.070093 7 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0405 16:53:46.070499 7 main.go:205] Creating API client for https://10.96.0.1:443 I0405 16:53:46.077784 7 main.go:249] Running in Kubernetes cluster version v1.10 (v1.10.11) - git (clean) commit 637c7e288581ee40ab4ca210618a89a555b6e7e9 - platform linux/amd64 I0405 16:53:46.183359 7 nginx.go:265] Starting NGINX Ingress controller I0405 16:53:46.193913 7 event.go:209] Event(v1.ObjectReference{Kind:\"ConfigMap\", Namespace:\"ingress-nginx\", Name:\"udp-services\", UID:\"82258915-563e-11e9-9c52-025000000001\", APIVersion:\"v1\", ResourceVersion:\"494\", FieldPath:\"\"}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services ... ssh \u00b6 kubectl ingress-nginx ssh is exactly the same as kubectl ingress-nginx exec -it -- /bin/bash . Use it when you want to quickly be dropped into a shell inside a running ingress-nginx container. $ kubectl ingress-nginx ssh -n ingress-nginx www-data@ingress-nginx-controller-7cbf77c976-wx5pn:/etc/nginx$","title":"kubectl plugin"},{"location":"kubectl-plugin/#the-ingress-nginx-kubectl-plugin","text":"","title":"The ingress-nginx kubectl plugin"},{"location":"kubectl-plugin/#installation","text":"Install krew , then run kubectl krew install ingress-nginx to install the plugin. Then run kubectl ingress-nginx --help to make sure the plugin is properly installed and to get a list of commands: kubectl ingress-nginx --help A kubectl plugin for inspecting your ingress-nginx deployments Usage: ingress-nginx [command] Available Commands: backends Inspect the dynamic backend information of an ingress-nginx instance certs Output the certificate data stored in an ingress-nginx pod conf Inspect the generated nginx.conf exec Execute a command inside an ingress-nginx pod general Inspect the other dynamic ingress-nginx information help Help about any command info Show information about the ingress-nginx service ingresses Provide a short summary of all of the ingress definitions lint Inspect kubernetes resources for possible issues logs Get the kubernetes logs for an ingress-nginx pod ssh ssh into a running ingress-nginx pod Flags: --as string Username to impersonate for the operation --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. --cache-dir string Default HTTP cache directory (default \"/Users/alexkursell/.kube/http-cache\") --certificate-authority string Path to a cert file for the certificate authority --client-certificate string Path to a client certificate file for TLS --client-key string Path to a client key file for TLS --cluster string The name of the kubeconfig cluster to use --context string The name of the kubeconfig context to use -h, --help help for ingress-nginx --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure --kubeconfig string Path to the kubeconfig file to use for CLI requests. -n, --namespace string If present, the namespace scope for this CLI request --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\") -s, --server string The address and port of the Kubernetes API server --token string Bearer token for authentication to the API server --user string The name of the kubeconfig user to use Use \"ingress-nginx [command] --help\" for more information about a command.","title":"Installation"},{"location":"kubectl-plugin/#common-flags","text":"Every subcommand supports the basic kubectl configuration flags like --namespace , --context , --client-key and so on. Subcommands that act on a particular ingress-nginx pod ( backends , certs , conf , exec , general , logs , ssh ), support the --deployment <deployment> and --pod <pod> flags to select either a pod from a deployment with the given name, or a pod with the given name. The --deployment flag defaults to ingress-nginx-controller . Subcommands that inspect resources ( ingresses , lint ) support the --all-namespaces flag, which causes them to inspect resources in every namespace.","title":"Common Flags"},{"location":"kubectl-plugin/#subcommands","text":"Note that backends , general , certs , and conf require ingress-nginx version 0.23.0 or higher.","title":"Subcommands"},{"location":"kubectl-plugin/#backends","text":"Run kubectl ingress-nginx backends to get a JSON array of the backends that an ingress-nginx controller currently knows about: $ kubectl ingress-nginx backends -n ingress-nginx [ { \"name\": \"default-apple-service-5678\", \"service\": { \"metadata\": { \"creationTimestamp\": null }, \"spec\": { \"ports\": [ { \"protocol\": \"TCP\", \"port\": 5678, \"targetPort\": 5678 } ], \"selector\": { \"app\": \"apple\" }, \"clusterIP\": \"10.97.230.121\", \"type\": \"ClusterIP\", \"sessionAffinity\": \"None\" }, \"status\": { \"loadBalancer\": {} } }, \"port\": 0, \"sslPassthrough\": false, \"endpoints\": [ { \"address\": \"10.1.3.86\", \"port\": \"5678\" } ], \"sessionAffinityConfig\": { \"name\": \"\", \"cookieSessionAffinity\": { \"name\": \"\" } }, \"upstreamHashByConfig\": { \"upstream-hash-by-subset-size\": 3 }, \"noServer\": false, \"trafficShapingPolicy\": { \"weight\": 0, \"header\": \"\", \"headerValue\": \"\", \"cookie\": \"\" } }, { \"name\": \"default-echo-service-8080\", ... }, { \"name\": \"upstream-default-backend\", ... } ] Add the --list option to show only the backend names. Add the --backend <backend> option to show only the backend with the given name.","title":"backends"},{"location":"kubectl-plugin/#certs","text":"Use kubectl ingress-nginx certs --host <hostname> to dump the SSL cert/key information for a given host. WARNING: This command will dump sensitive private key information. Don't blindly share the output, and certainly don't log it anywhere. $ kubectl ingress-nginx certs -n ingress-nginx --host testaddr.local -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN RSA PRIVATE KEY----- <REDACTED! DO NOT SHARE THIS!> -----END RSA PRIVATE KEY-----","title":"certs"},{"location":"kubectl-plugin/#conf","text":"Use kubectl ingress-nginx conf to dump the generated nginx.conf file. Add the --host <hostname> option to view only the server block for that host: kubectl ingress-nginx conf -n ingress-nginx --host testaddr.local server { server_name testaddr.local ; listen 80; set $proxy_upstream_name \"-\"; set $pass_access_scheme $scheme; set $pass_server_port $server_port; set $best_http_host $http_host; set $pass_port $pass_server_port; location / { set $namespace \"\"; set $ingress_name \"\"; set $service_name \"\"; set $service_port \"0\"; set $location_path \"/\"; ...","title":"conf"},{"location":"kubectl-plugin/#exec","text":"kubectl ingress-nginx exec is exactly the same as kubectl exec , with the same command flags. It will automatically choose an ingress-nginx pod to run the command in. $ kubectl ingress-nginx exec -i -n ingress-nginx -- ls /etc/nginx fastcgi_params geoip lua mime.types modsecurity modules nginx.conf opentracing.json owasp-modsecurity-crs template","title":"exec"},{"location":"kubectl-plugin/#info","text":"Shows the internal and external IP/CNAMES for an ingress-nginx service. $ kubectl ingress-nginx info -n ingress-nginx Service cluster IP address: 10.187.253.31 LoadBalancer IP|CNAME: 35.123.123.123 Use the --service <service> flag if your ingress-nginx LoadBalancer service is not named ingress-nginx .","title":"info"},{"location":"kubectl-plugin/#ingresses","text":"kubectl ingress-nginx ingresses , alternately kubectl ingress-nginx ing , shows a more detailed view of the ingress definitions in a namespace. Compare: $ kubectl get ingresses --all-namespaces NAMESPACE NAME HOSTS ADDRESS PORTS AGE default example-ingress1 testaddr.local,testaddr2.local localhost 80 5d default test-ingress-2 * localhost 80 5d vs. $ kubectl ingress-nginx ingresses --all-namespaces NAMESPACE INGRESS NAME HOST+PATH ADDRESSES TLS SERVICE SERVICE PORT ENDPOINTS default example-ingress1 testaddr.local/etameta localhost NO pear-service 5678 5 default example-ingress1 testaddr2.local/otherpath localhost NO apple-service 5678 1 default example-ingress1 testaddr2.local/otherotherpath localhost NO pear-service 5678 5 default test-ingress-2 * localhost NO echo-service 8080 2","title":"ingresses"},{"location":"kubectl-plugin/#lint","text":"kubectl ingress-nginx lint can check a namespace or entire cluster for potential configuration issues. This command is especially useful when upgrading between ingress-nginx versions. $ kubectl ingress-nginx lint --all-namespaces --verbose Checking ingresses... \u2717 anamespace/this-nginx - Contains the removed session-cookie-hash annotation. Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3743 \u2717 othernamespace/ingress-definition-blah - The rewrite-target annotation value does not reference a capture group Lint added for version 0.22.0 https://github.com/kubernetes/ingress-nginx/issues/3174 Checking deployments... \u2717 namespace2/ingress-nginx-controller - Uses removed config flag --sort-backends Lint added for version 0.22.0 https://github.com/kubernetes/ingress-nginx/issues/3655 - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808 To show the lints added only for a particular ingress-nginx release, use the --from-version and --to-version flags: $ kubectl ingress-nginx lint --all-namespaces --verbose --from-version 0 .24.0 --to-version 0 .24.0 Checking ingresses... \u2717 anamespace/this-nginx - Contains the removed session-cookie-hash annotation. Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3743 Checking deployments... \u2717 namespace2/ingress-nginx-controller - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808","title":"lint"},{"location":"kubectl-plugin/#logs","text":"kubectl ingress-nginx logs is almost the same as kubectl logs , with fewer flags. It will automatically choose an ingress-nginx pod to read logs from. $ kubectl ingress-nginx logs -n ingress-nginx ------------------------------------------------------------------------------- NGINX Ingress controller Release: dev Build: git-48dc3a867 Repository: git@github.com:kubernetes/ingress-nginx.git ------------------------------------------------------------------------------- W0405 16:53:46.061589 7 flags.go:214] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false) nginx version: nginx/1.15.9 W0405 16:53:46.070093 7 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0405 16:53:46.070499 7 main.go:205] Creating API client for https://10.96.0.1:443 I0405 16:53:46.077784 7 main.go:249] Running in Kubernetes cluster version v1.10 (v1.10.11) - git (clean) commit 637c7e288581ee40ab4ca210618a89a555b6e7e9 - platform linux/amd64 I0405 16:53:46.183359 7 nginx.go:265] Starting NGINX Ingress controller I0405 16:53:46.193913 7 event.go:209] Event(v1.ObjectReference{Kind:\"ConfigMap\", Namespace:\"ingress-nginx\", Name:\"udp-services\", UID:\"82258915-563e-11e9-9c52-025000000001\", APIVersion:\"v1\", ResourceVersion:\"494\", FieldPath:\"\"}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services ...","title":"logs"},{"location":"kubectl-plugin/#ssh","text":"kubectl ingress-nginx ssh is exactly the same as kubectl ingress-nginx exec -it -- /bin/bash . Use it when you want to quickly be dropped into a shell inside a running ingress-nginx container. $ kubectl ingress-nginx ssh -n ingress-nginx www-data@ingress-nginx-controller-7cbf77c976-wx5pn:/etc/nginx$","title":"ssh"},{"location":"troubleshooting/","text":"Troubleshooting \u00b6 Ingress-Controller Logs and Events \u00b6 There are many ways to troubleshoot the ingress-controller. The following are basic troubleshooting methods to obtain more information. Check the Ingress Resource Events \u00b6 $ kubectl get ing -n <namespace-of-ingress-resource> NAME HOSTS ADDRESS PORTS AGE cafe-ingress cafe.com 10.0.2.15 80 25s $ kubectl describe ing <ingress-resource-name> -n <namespace-of-ingress-resource> Name: cafe-ingress Namespace: default Address: 10.0.2.15 Default backend: default-http-backend:80 (172.17.0.5:8080) Rules: Host Path Backends ---- ---- -------- cafe.com /tea tea-svc:80 (<none>) /coffee coffee-svc:80 (<none>) Annotations: kubectl.kubernetes.io/last-applied-configuration: {\"apiVersion\":\"networking.k8s.io/v1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{},\"name\":\"cafe-ingress\",\"namespace\":\"default\",\"selfLink\":\"/apis/networking/v1/namespaces/default/ingresses/cafe-ingress\"},\"spec\":{\"rules\":[{\"host\":\"cafe.com\",\"http\":{\"paths\":[{\"backend\":{\"serviceName\":\"tea-svc\",\"servicePort\":80},\"path\":\"/tea\"},{\"backend\":{\"serviceName\":\"coffee-svc\",\"servicePort\":80},\"path\":\"/coffee\"}]}}]},\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"169.48.142.110\"}]}}} Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal CREATE 1m ingress-nginx-controller Ingress default/cafe-ingress Normal UPDATE 58s ingress-nginx-controller Ingress default/cafe-ingress Check the Ingress Controller Logs \u00b6 $ kubectl get pods -n <namespace-of-ingress-controller> NAME READY STATUS RESTARTS AGE ingress-nginx-controller-67956bf89d-fv58j 1/1 Running 0 1m $ kubectl logs -n <namespace> ingress-nginx-controller-67956bf89d-fv58j ------------------------------------------------------------------------------- NGINX Ingress controller Release: 0.14.0 Build: git-734361d Repository: https://github.com/kubernetes/ingress-nginx ------------------------------------------------------------------------------- .... Check the Nginx Configuration \u00b6 $ kubectl get pods -n <namespace-of-ingress-controller> NAME READY STATUS RESTARTS AGE ingress-nginx-controller-67956bf89d-fv58j 1/1 Running 0 1m $ kubectl exec -it -n <namespace-of-ingress-controller> ingress-nginx-controller-67956bf89d-fv58j -- cat /etc/nginx/nginx.conf daemon off; worker_processes 2; pid /run/nginx.pid; worker_rlimit_nofile 523264; worker_shutdown_timeout 240s; events { multi_accept on; worker_connections 16384; use epoll; } http { .... Check if used Services Exist \u00b6 $ kubectl get svc --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default coffee-svc ClusterIP 10.106.154.35 <none> 80/TCP 18m default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 30m default tea-svc ClusterIP 10.104.172.12 <none> 80/TCP 18m kube-system default-http-backend NodePort 10.108.189.236 <none> 80:30001/TCP 30m kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 30m kube-system kubernetes-dashboard NodePort 10.103.128.17 <none> 80:30000/TCP 30m Debug Logging \u00b6 Using the flag --v=XX it is possible to increase the level of logging. This is performed by editing the deployment. $ kubectl get deploy -n <namespace-of-ingress-controller> NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE default-http-backend 1 1 1 1 35m ingress-nginx-controller 1 1 1 1 35m $ kubectl edit deploy -n <namespace-of-ingress-controller> ingress-nginx-controller # Add --v = X to \"- args\" , where X is an integer --v=2 shows details using diff about the changes in the configuration in nginx --v=3 shows details about the service, Ingress rule, endpoint changes and it dumps the nginx configuration in JSON format --v=5 configures NGINX in debug mode Authentication to the Kubernetes API Server \u00b6 A number of components are involved in the authentication process and the first step is to narrow down the source of the problem, namely whether it is a problem with service authentication or with the kubeconfig file. Both authentications must work: +-------------+ service +------------+ | | authentication | | + apiserver +<-------------------+ ingress | | | | controller | +-------------+ +------------+ Service authentication The Ingress controller needs information from apiserver. Therefore, authentication is required, which can be achieved in a couple of ways: Service Account: This is recommended, because nothing has to be configured. The Ingress controller will use information provided by the system to communicate with the API server. See 'Service Account' section for details. Kubeconfig file: In some Kubernetes environments service accounts are not available. In this case a manual configuration is required. The Ingress controller binary can be started with the --kubeconfig flag. The value of the flag is a path to a file specifying how to connect to the API server. Using the --kubeconfig does not requires the flag --apiserver-host . The format of the file is identical to ~/.kube/config which is used by kubectl to connect to the API server. See 'kubeconfig' section for details. Using the flag --apiserver-host : Using this flag --apiserver-host=http://localhost:8080 it is possible to specify an unsecured API server or reach a remote kubernetes cluster using kubectl proxy . Please do not use this approach in production. In the diagram below you can see the full authentication flow with all options, starting with the browser on the lower left hand side. Kubernetes Workstation +---------------------------------------------------+ +------------------+ | | | | | +-----------+ apiserver +------------+ | | +------------+ | | | | proxy | | | | | | | | | apiserver | | ingress | | | | ingress | | | | | | controller | | | | controller | | | | | | | | | | | | | | | | | | | | | | | | | service account/ | | | | | | | | | | kubeconfig | | | | | | | | | +<-------------------+ | | | | | | | | | | | | | | | | | +------+----+ kubeconfig +------+-----+ | | +------+-----+ | | |<--------------------------------------------------------| | | | | | +---------------------------------------------------+ +------------------+ Service Account \u00b6 If using a service account to connect to the API server, the ingress-controller expects the file /var/run/secrets/kubernetes.io/serviceaccount/token to be present. It provides a secret token that is required to authenticate with the API server. Verify with the following commands: # start a container that contains curl $ kubectl run -it --rm test --image = curlimages/curl --restart = Never -- /bin/sh # check if secret exists / $ ls /var/run/secrets/kubernetes.io/serviceaccount/ ca.crt namespace token / $ # check base connectivity from cluster inside / $ curl -k https://kubernetes.default.svc.cluster.local { \"kind\": \"Status\", \"apiVersion\": \"v1\", \"metadata\": { }, \"status\": \"Failure\", \"message\": \"forbidden: User \\\"system:anonymous\\\" cannot get path \\\"/\\\"\", \"reason\": \"Forbidden\", \"details\": { }, \"code\": 403 }/ $ # connect using tokens }/ $ curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H \"Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://kubernetes.default.svc.cluster.local && echo { \"paths\": [ \"/api\", \"/api/v1\", \"/apis\", \"/apis/\", ... TRUNCATED \"/readyz/shutdown\", \"/version\" ] } / $ # when you type ` exit ` or ` ^D ` the test pod will be deleted. If it is not working, there are two possible reasons: The contents of the tokens are invalid. Find the secret name with kubectl get secrets | grep service-account and delete it with kubectl delete secret <name> . It will automatically be recreated. You have a non-standard Kubernetes installation and the file containing the token may not be present. The API server will mount a volume containing this file, but only if the API server is configured to use the ServiceAccount admission controller. If you experience this error, verify that your API server is using the ServiceAccount admission controller. If you are configuring the API server by hand, you can set this with the --admission-control parameter. Note that you should use other admission controllers as well. Before configuring this option, you should read about admission controllers. More information: User Guide: Service Accounts Cluster Administrator Guide: Managing Service Accounts Kube-Config \u00b6 If you want to use a kubeconfig file for authentication, follow the deploy procedure and add the flag --kubeconfig=/etc/kubernetes/kubeconfig.yaml to the args section of the deployment. Using GDB with Nginx \u00b6 Gdb can be used to with nginx to perform a configuration dump. This allows us to see which configuration is being used, as well as older configurations. Note: The below is based on the nginx documentation . SSH into the worker $ ssh user@workerIP Obtain the Docker Container Running nginx $ docker ps | grep ingress-nginx-controller CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d9e1d243156a registry.k8s.io/ingress-nginx/controller \"/usr/bin/dumb-init \u2026\" 19 minutes ago Up 19 minutes k8s_ingress-nginx-controller_ingress-nginx-controller-67956bf89d-mqxzt_kube-system_079f31ec-aa37-11e8-ad39-080027a227db_0 Exec into the container $ docker exec -it --user = 0 --privileged d9e1d243156a bash Make sure nginx is running in --with-debug $ nginx -V 2 > & 1 | grep -- '--with-debug' Get list of processes running on container $ ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 20:23 ? 00:00:00 /usr/bin/dumb-init /nginx-ingres root 5 1 0 20:23 ? 00:00:05 /ingress-nginx-controller --defa root 21 5 0 20:23 ? 00:00:00 nginx: master process /usr/sbin/ nobody 106 21 0 20:23 ? 00:00:00 nginx: worker process nobody 107 21 0 20:23 ? 00:00:00 nginx: worker process root 172 0 0 20:43 pts/0 00:00:00 bash Attach gdb to the nginx master process $ gdb -p 21 .... Attaching to process 21 Reading symbols from /usr/sbin/nginx...done. .... (gdb) Copy and paste the following: set $cd = ngx_cycle->config_dump set $nelts = $cd.nelts set $elts = (ngx_conf_dump_t*)($cd.elts) while ($nelts-- > 0) set $name = $elts[$nelts]->name.data printf \"Dumping %s to nginx_conf.txt\\n\", $name append memory nginx_conf.txt \\ $ elts [ $nelts ] ->buffer.start $elts [ $nelts ] ->buffer.end end Quit GDB by pressing CTRL+D Open nginx_conf.txt cat nginx_conf.txt Image related issues faced on Nginx 4.2.5 or other versions (Helm chart versions) \u00b6 Incase you face below error while installing Nginx using helm chart (either by helm commands or helm_release terraform provider ) Warning Failed 5m5s (x4 over 6m34s) kubelet Failed to pull image \"registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47\": rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47\": failed to resolve reference \"registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47\": failed to do request: Head \"https://eu.gcr.io/v2/k8s-artifacts-prod/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47\": EOF Then please follow the below steps. During troubleshooting you can also execute the below commands to test the connectivities from you local machines and repositories details a. curl registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 > /dev/null (\u2388 |myprompt)\u279c ~ curl registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 > /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 (\u2388 |myprompt)\u279c ~ b. curl -I https://eu.gcr.io/v2/k8s-artifacts-prod/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 (\u2388 |myprompt)\u279c ~ curl -I https://eu.gcr.io/v2/k8s-artifacts-prod/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 HTTP/2 200 docker-distribution-api-version: registry/2.0 content-type: application/vnd.docker.distribution.manifest.list.v2+json docker-content-digest: sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 content-length: 1384 date: Wed, 28 Sep 2022 16:46:28 GMT server: Docker Registry x-xss-protection: 0 x-frame-options: SAMEORIGIN alt-svc: h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\" (\u2388 |myprompt)\u279c ~ Redirection in the proxy is implemented to ensure the pulling of the images. This is the solution recommended to whitelist the below image repositories : *.appspot.com *.k8s.io *.pkg.dev *.gcr.io More details about the above repos : a. *.k8s.io -> To ensure you can pull any images from registry.k8s.io b. *.gcr.io -> GCP services are used for image hosting. This is part of the domains suggested by GCP to allow and ensure users can pull images from their container registry services. c. *.appspot.com -> This a Google domain. part of the domain used for GCR. Unable to listen on port (80/443) \u00b6 One possible reason for this error is lack of permission to bind to the port. Ports 80, 443, and any other port < 1024 are Linux privileged ports which historically could only be bound by root. The ingress-nginx-controller uses the CAP_NET_BIND_SERVICE linux capability to allow binding these ports as a normal user (www-data / 101). This involves two components: 1. In the image, the /nginx-ingress-controller file has the cap_net_bind_service capability added (e.g. via setcap ) 2. The NET_BIND_SERVICE capability is added to the container in the containerSecurityContext of the deployment. If encountering this on one/some node(s) and not on others, try to purge and pull a fresh copy of the image to the affected node(s), in case there has been corruption of the underlying layers to lose the capability on the executable. Create a test pod \u00b6 The /nginx-ingress-controller process exits/crashes when encountering this error, making it difficult to troubleshoot what is happening inside the container. To get around this, start an equivalent container running \"sleep 3600\", and exec into it for further troubleshooting. For example: apiVersion : v1 kind : Pod metadata : name : ingress-nginx-sleep namespace : default labels : app : nginx spec : containers : - name : nginx image : ##_CONTROLLER_IMAGE_## resources : requests : memory : \"512Mi\" cpu : \"500m\" limits : memory : \"1Gi\" cpu : \"1\" command : [ \"sleep\" ] args : [ \"3600\" ] ports : - containerPort : 80 name : http protocol : TCP - containerPort : 443 name : https protocol : TCP securityContext : allowPrivilegeEscalation : true capabilities : add : - NET_BIND_SERVICE drop : - ALL runAsUser : 101 restartPolicy : Never nodeSelector : kubernetes.io/hostname : ##_NODE_NAME_## tolerations : - key : \"node.kubernetes.io/unschedulable\" operator : \"Exists\" effect : NoSchedule * update the namespace if applicable/desired * replace ##_NODE_NAME_## with the problematic node (or remove nodeSelector section if problem is not confined to one node) * replace ##_CONTROLLER_IMAGE_## with the same image as in use by your ingress-nginx deployment * confirm the securityContext section matches what is in place for ingress-nginx-controller pods in your cluster Apply the YAML and open a shell into the pod. Try to manually run the controller process: $ /nginx-ingress-controller You should get the same error as from the ingress controller pod logs. Confirm the capabilities are properly surfacing into the pod: $ grep CapBnd /proc/1/status CapBnd: 0000000000000400 The above value has only net_bind_service enabled (per security context in YAML which adds that and drops all). If you get a different value, then you can decode it on another linux box (capsh not available in this container) like below, and then figure out why specified capabilities are not propagating into the pod/container. $ capsh --decode = 0000000000000400 0x0000000000000400=cap_net_bind_service Create a test pod as root \u00b6 (Note, this may be restricted by PodSecurityPolicy, PodSecurityAdmission/Standards, OPA Gatekeeper, etc. in which case you will need to do the appropriate workaround for testing, e.g. deploy in a new namespace without the restrictions.) To test further you may want to install additional utilities, etc. Modify the pod yaml by: * changing runAsUser from 101 to 0 * removing the \"drop..ALL\" section from the capabilities. Some things to try after shelling into this container: Try running the controller as the www-data (101) user: $ chmod 4755 /nginx-ingress-controller $ /nginx-ingress-controller Examine the errors to see if there is still an issue listening on the port or if it passed that and moved on to other expected errors due to running out of context. Install the libcap package and check capabilities on the file: $ apk add libcap (1/1) Installing libcap (2.50-r0) Executing busybox-1.33.1-r7.trigger OK: 26 MiB in 41 packages $ getcap /nginx-ingress-controller /nginx-ingress-controller cap_net_bind_service=ep (if missing, see above about purging image on the server and re-pulling) Strace the executable to see what system calls are being executed when it fails: $ apk add strace (1/1) Installing strace (5.12-r0) Executing busybox-1.33.1-r7.trigger OK: 28 MiB in 42 packages $ strace /nginx-ingress-controller execve(\"/nginx-ingress-controller\", [\"/nginx-ingress-controller\"], 0x7ffeb9eb3240 /* 131 vars */) = 0 arch_prctl(ARCH_SET_FS, 0x29ea690) = 0 ...","title":"Troubleshooting"},{"location":"troubleshooting/#troubleshooting","text":"","title":"Troubleshooting"},{"location":"troubleshooting/#ingress-controller-logs-and-events","text":"There are many ways to troubleshoot the ingress-controller. The following are basic troubleshooting methods to obtain more information.","title":"Ingress-Controller Logs and Events"},{"location":"troubleshooting/#check-the-ingress-resource-events","text":"$ kubectl get ing -n <namespace-of-ingress-resource> NAME HOSTS ADDRESS PORTS AGE cafe-ingress cafe.com 10.0.2.15 80 25s $ kubectl describe ing <ingress-resource-name> -n <namespace-of-ingress-resource> Name: cafe-ingress Namespace: default Address: 10.0.2.15 Default backend: default-http-backend:80 (172.17.0.5:8080) Rules: Host Path Backends ---- ---- -------- cafe.com /tea tea-svc:80 (<none>) /coffee coffee-svc:80 (<none>) Annotations: kubectl.kubernetes.io/last-applied-configuration: {\"apiVersion\":\"networking.k8s.io/v1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{},\"name\":\"cafe-ingress\",\"namespace\":\"default\",\"selfLink\":\"/apis/networking/v1/namespaces/default/ingresses/cafe-ingress\"},\"spec\":{\"rules\":[{\"host\":\"cafe.com\",\"http\":{\"paths\":[{\"backend\":{\"serviceName\":\"tea-svc\",\"servicePort\":80},\"path\":\"/tea\"},{\"backend\":{\"serviceName\":\"coffee-svc\",\"servicePort\":80},\"path\":\"/coffee\"}]}}]},\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"169.48.142.110\"}]}}} Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal CREATE 1m ingress-nginx-controller Ingress default/cafe-ingress Normal UPDATE 58s ingress-nginx-controller Ingress default/cafe-ingress","title":"Check the Ingress Resource Events"},{"location":"troubleshooting/#check-the-ingress-controller-logs","text":"$ kubectl get pods -n <namespace-of-ingress-controller> NAME READY STATUS RESTARTS AGE ingress-nginx-controller-67956bf89d-fv58j 1/1 Running 0 1m $ kubectl logs -n <namespace> ingress-nginx-controller-67956bf89d-fv58j ------------------------------------------------------------------------------- NGINX Ingress controller Release: 0.14.0 Build: git-734361d Repository: https://github.com/kubernetes/ingress-nginx ------------------------------------------------------------------------------- ....","title":"Check the Ingress Controller Logs"},{"location":"troubleshooting/#check-the-nginx-configuration","text":"$ kubectl get pods -n <namespace-of-ingress-controller> NAME READY STATUS RESTARTS AGE ingress-nginx-controller-67956bf89d-fv58j 1/1 Running 0 1m $ kubectl exec -it -n <namespace-of-ingress-controller> ingress-nginx-controller-67956bf89d-fv58j -- cat /etc/nginx/nginx.conf daemon off; worker_processes 2; pid /run/nginx.pid; worker_rlimit_nofile 523264; worker_shutdown_timeout 240s; events { multi_accept on; worker_connections 16384; use epoll; } http { ....","title":"Check the Nginx Configuration"},{"location":"troubleshooting/#check-if-used-services-exist","text":"$ kubectl get svc --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default coffee-svc ClusterIP 10.106.154.35 <none> 80/TCP 18m default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 30m default tea-svc ClusterIP 10.104.172.12 <none> 80/TCP 18m kube-system default-http-backend NodePort 10.108.189.236 <none> 80:30001/TCP 30m kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 30m kube-system kubernetes-dashboard NodePort 10.103.128.17 <none> 80:30000/TCP 30m","title":"Check if used Services Exist"},{"location":"troubleshooting/#debug-logging","text":"Using the flag --v=XX it is possible to increase the level of logging. This is performed by editing the deployment. $ kubectl get deploy -n <namespace-of-ingress-controller> NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE default-http-backend 1 1 1 1 35m ingress-nginx-controller 1 1 1 1 35m $ kubectl edit deploy -n <namespace-of-ingress-controller> ingress-nginx-controller # Add --v = X to \"- args\" , where X is an integer --v=2 shows details using diff about the changes in the configuration in nginx --v=3 shows details about the service, Ingress rule, endpoint changes and it dumps the nginx configuration in JSON format --v=5 configures NGINX in debug mode","title":"Debug Logging"},{"location":"troubleshooting/#authentication-to-the-kubernetes-api-server","text":"A number of components are involved in the authentication process and the first step is to narrow down the source of the problem, namely whether it is a problem with service authentication or with the kubeconfig file. Both authentications must work: +-------------+ service +------------+ | | authentication | | + apiserver +<-------------------+ ingress | | | | controller | +-------------+ +------------+ Service authentication The Ingress controller needs information from apiserver. Therefore, authentication is required, which can be achieved in a couple of ways: Service Account: This is recommended, because nothing has to be configured. The Ingress controller will use information provided by the system to communicate with the API server. See 'Service Account' section for details. Kubeconfig file: In some Kubernetes environments service accounts are not available. In this case a manual configuration is required. The Ingress controller binary can be started with the --kubeconfig flag. The value of the flag is a path to a file specifying how to connect to the API server. Using the --kubeconfig does not requires the flag --apiserver-host . The format of the file is identical to ~/.kube/config which is used by kubectl to connect to the API server. See 'kubeconfig' section for details. Using the flag --apiserver-host : Using this flag --apiserver-host=http://localhost:8080 it is possible to specify an unsecured API server or reach a remote kubernetes cluster using kubectl proxy . Please do not use this approach in production. In the diagram below you can see the full authentication flow with all options, starting with the browser on the lower left hand side. Kubernetes Workstation +---------------------------------------------------+ +------------------+ | | | | | +-----------+ apiserver +------------+ | | +------------+ | | | | proxy | | | | | | | | | apiserver | | ingress | | | | ingress | | | | | | controller | | | | controller | | | | | | | | | | | | | | | | | | | | | | | | | service account/ | | | | | | | | | | kubeconfig | | | | | | | | | +<-------------------+ | | | | | | | | | | | | | | | | | +------+----+ kubeconfig +------+-----+ | | +------+-----+ | | |<--------------------------------------------------------| | | | | | +---------------------------------------------------+ +------------------+","title":"Authentication to the Kubernetes API Server"},{"location":"troubleshooting/#service-account","text":"If using a service account to connect to the API server, the ingress-controller expects the file /var/run/secrets/kubernetes.io/serviceaccount/token to be present. It provides a secret token that is required to authenticate with the API server. Verify with the following commands: # start a container that contains curl $ kubectl run -it --rm test --image = curlimages/curl --restart = Never -- /bin/sh # check if secret exists / $ ls /var/run/secrets/kubernetes.io/serviceaccount/ ca.crt namespace token / $ # check base connectivity from cluster inside / $ curl -k https://kubernetes.default.svc.cluster.local { \"kind\": \"Status\", \"apiVersion\": \"v1\", \"metadata\": { }, \"status\": \"Failure\", \"message\": \"forbidden: User \\\"system:anonymous\\\" cannot get path \\\"/\\\"\", \"reason\": \"Forbidden\", \"details\": { }, \"code\": 403 }/ $ # connect using tokens }/ $ curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H \"Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://kubernetes.default.svc.cluster.local && echo { \"paths\": [ \"/api\", \"/api/v1\", \"/apis\", \"/apis/\", ... TRUNCATED \"/readyz/shutdown\", \"/version\" ] } / $ # when you type ` exit ` or ` ^D ` the test pod will be deleted. If it is not working, there are two possible reasons: The contents of the tokens are invalid. Find the secret name with kubectl get secrets | grep service-account and delete it with kubectl delete secret <name> . It will automatically be recreated. You have a non-standard Kubernetes installation and the file containing the token may not be present. The API server will mount a volume containing this file, but only if the API server is configured to use the ServiceAccount admission controller. If you experience this error, verify that your API server is using the ServiceAccount admission controller. If you are configuring the API server by hand, you can set this with the --admission-control parameter. Note that you should use other admission controllers as well. Before configuring this option, you should read about admission controllers. More information: User Guide: Service Accounts Cluster Administrator Guide: Managing Service Accounts","title":"Service Account"},{"location":"troubleshooting/#kube-config","text":"If you want to use a kubeconfig file for authentication, follow the deploy procedure and add the flag --kubeconfig=/etc/kubernetes/kubeconfig.yaml to the args section of the deployment.","title":"Kube-Config"},{"location":"troubleshooting/#using-gdb-with-nginx","text":"Gdb can be used to with nginx to perform a configuration dump. This allows us to see which configuration is being used, as well as older configurations. Note: The below is based on the nginx documentation . SSH into the worker $ ssh user@workerIP Obtain the Docker Container Running nginx $ docker ps | grep ingress-nginx-controller CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d9e1d243156a registry.k8s.io/ingress-nginx/controller \"/usr/bin/dumb-init \u2026\" 19 minutes ago Up 19 minutes k8s_ingress-nginx-controller_ingress-nginx-controller-67956bf89d-mqxzt_kube-system_079f31ec-aa37-11e8-ad39-080027a227db_0 Exec into the container $ docker exec -it --user = 0 --privileged d9e1d243156a bash Make sure nginx is running in --with-debug $ nginx -V 2 > & 1 | grep -- '--with-debug' Get list of processes running on container $ ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 20:23 ? 00:00:00 /usr/bin/dumb-init /nginx-ingres root 5 1 0 20:23 ? 00:00:05 /ingress-nginx-controller --defa root 21 5 0 20:23 ? 00:00:00 nginx: master process /usr/sbin/ nobody 106 21 0 20:23 ? 00:00:00 nginx: worker process nobody 107 21 0 20:23 ? 00:00:00 nginx: worker process root 172 0 0 20:43 pts/0 00:00:00 bash Attach gdb to the nginx master process $ gdb -p 21 .... Attaching to process 21 Reading symbols from /usr/sbin/nginx...done. .... (gdb) Copy and paste the following: set $cd = ngx_cycle->config_dump set $nelts = $cd.nelts set $elts = (ngx_conf_dump_t*)($cd.elts) while ($nelts-- > 0) set $name = $elts[$nelts]->name.data printf \"Dumping %s to nginx_conf.txt\\n\", $name append memory nginx_conf.txt \\ $ elts [ $nelts ] ->buffer.start $elts [ $nelts ] ->buffer.end end Quit GDB by pressing CTRL+D Open nginx_conf.txt cat nginx_conf.txt","title":"Using GDB with Nginx"},{"location":"troubleshooting/#image-related-issues-faced-on-nginx-425-or-other-versions-helm-chart-versions","text":"Incase you face below error while installing Nginx using helm chart (either by helm commands or helm_release terraform provider ) Warning Failed 5m5s (x4 over 6m34s) kubelet Failed to pull image \"registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47\": rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47\": failed to resolve reference \"registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47\": failed to do request: Head \"https://eu.gcr.io/v2/k8s-artifacts-prod/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47\": EOF Then please follow the below steps. During troubleshooting you can also execute the below commands to test the connectivities from you local machines and repositories details a. curl registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 > /dev/null (\u2388 |myprompt)\u279c ~ curl registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 > /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 (\u2388 |myprompt)\u279c ~ b. curl -I https://eu.gcr.io/v2/k8s-artifacts-prod/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 (\u2388 |myprompt)\u279c ~ curl -I https://eu.gcr.io/v2/k8s-artifacts-prod/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 HTTP/2 200 docker-distribution-api-version: registry/2.0 content-type: application/vnd.docker.distribution.manifest.list.v2+json docker-content-digest: sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 content-length: 1384 date: Wed, 28 Sep 2022 16:46:28 GMT server: Docker Registry x-xss-protection: 0 x-frame-options: SAMEORIGIN alt-svc: h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\" (\u2388 |myprompt)\u279c ~ Redirection in the proxy is implemented to ensure the pulling of the images. This is the solution recommended to whitelist the below image repositories : *.appspot.com *.k8s.io *.pkg.dev *.gcr.io More details about the above repos : a. *.k8s.io -> To ensure you can pull any images from registry.k8s.io b. *.gcr.io -> GCP services are used for image hosting. This is part of the domains suggested by GCP to allow and ensure users can pull images from their container registry services. c. *.appspot.com -> This a Google domain. part of the domain used for GCR.","title":"Image related issues faced on Nginx 4.2.5 or other versions (Helm chart versions)"},{"location":"troubleshooting/#unable-to-listen-on-port-80443","text":"One possible reason for this error is lack of permission to bind to the port. Ports 80, 443, and any other port < 1024 are Linux privileged ports which historically could only be bound by root. The ingress-nginx-controller uses the CAP_NET_BIND_SERVICE linux capability to allow binding these ports as a normal user (www-data / 101). This involves two components: 1. In the image, the /nginx-ingress-controller file has the cap_net_bind_service capability added (e.g. via setcap ) 2. The NET_BIND_SERVICE capability is added to the container in the containerSecurityContext of the deployment. If encountering this on one/some node(s) and not on others, try to purge and pull a fresh copy of the image to the affected node(s), in case there has been corruption of the underlying layers to lose the capability on the executable.","title":"Unable to listen on port (80/443)"},{"location":"troubleshooting/#create-a-test-pod","text":"The /nginx-ingress-controller process exits/crashes when encountering this error, making it difficult to troubleshoot what is happening inside the container. To get around this, start an equivalent container running \"sleep 3600\", and exec into it for further troubleshooting. For example: apiVersion : v1 kind : Pod metadata : name : ingress-nginx-sleep namespace : default labels : app : nginx spec : containers : - name : nginx image : ##_CONTROLLER_IMAGE_## resources : requests : memory : \"512Mi\" cpu : \"500m\" limits : memory : \"1Gi\" cpu : \"1\" command : [ \"sleep\" ] args : [ \"3600\" ] ports : - containerPort : 80 name : http protocol : TCP - containerPort : 443 name : https protocol : TCP securityContext : allowPrivilegeEscalation : true capabilities : add : - NET_BIND_SERVICE drop : - ALL runAsUser : 101 restartPolicy : Never nodeSelector : kubernetes.io/hostname : ##_NODE_NAME_## tolerations : - key : \"node.kubernetes.io/unschedulable\" operator : \"Exists\" effect : NoSchedule * update the namespace if applicable/desired * replace ##_NODE_NAME_## with the problematic node (or remove nodeSelector section if problem is not confined to one node) * replace ##_CONTROLLER_IMAGE_## with the same image as in use by your ingress-nginx deployment * confirm the securityContext section matches what is in place for ingress-nginx-controller pods in your cluster Apply the YAML and open a shell into the pod. Try to manually run the controller process: $ /nginx-ingress-controller You should get the same error as from the ingress controller pod logs. Confirm the capabilities are properly surfacing into the pod: $ grep CapBnd /proc/1/status CapBnd: 0000000000000400 The above value has only net_bind_service enabled (per security context in YAML which adds that and drops all). If you get a different value, then you can decode it on another linux box (capsh not available in this container) like below, and then figure out why specified capabilities are not propagating into the pod/container. $ capsh --decode = 0000000000000400 0x0000000000000400=cap_net_bind_service","title":"Create a test pod"},{"location":"troubleshooting/#create-a-test-pod-as-root","text":"(Note, this may be restricted by PodSecurityPolicy, PodSecurityAdmission/Standards, OPA Gatekeeper, etc. in which case you will need to do the appropriate workaround for testing, e.g. deploy in a new namespace without the restrictions.) To test further you may want to install additional utilities, etc. Modify the pod yaml by: * changing runAsUser from 101 to 0 * removing the \"drop..ALL\" section from the capabilities. Some things to try after shelling into this container: Try running the controller as the www-data (101) user: $ chmod 4755 /nginx-ingress-controller $ /nginx-ingress-controller Examine the errors to see if there is still an issue listening on the port or if it passed that and moved on to other expected errors due to running out of context. Install the libcap package and check capabilities on the file: $ apk add libcap (1/1) Installing libcap (2.50-r0) Executing busybox-1.33.1-r7.trigger OK: 26 MiB in 41 packages $ getcap /nginx-ingress-controller /nginx-ingress-controller cap_net_bind_service=ep (if missing, see above about purging image on the server and re-pulling) Strace the executable to see what system calls are being executed when it fails: $ apk add strace (1/1) Installing strace (5.12-r0) Executing busybox-1.33.1-r7.trigger OK: 28 MiB in 42 packages $ strace /nginx-ingress-controller execve(\"/nginx-ingress-controller\", [\"/nginx-ingress-controller\"], 0x7ffeb9eb3240 /* 131 vars */) = 0 arch_prctl(ARCH_SET_FS, 0x29ea690) = 0 ...","title":"Create a test pod as root"},{"location":"deploy/","text":"Installation Guide \u00b6 There are multiple ways to install the NGINX ingress controller: with Helm , using the project repository chart; with kubectl apply , using YAML manifests; with specific addons (e.g. for minikube or MicroK8s ). On most Kubernetes clusters, the ingress controller will work without requiring any extra configuration. If you want to get started as fast as possible, you can check the quick start instructions. However, in many environments, you can improve the performance or get better logs by enabling extra features. We recommend that you check the environment-specific instructions for details about optimizing the ingress controller for your particular environment or cloud provider. Contents \u00b6 Quick start Environment-specific instructions ... Docker Desktop ... Rancher Desktop ... minikube ... MicroK8s ... AWS ... GCE - GKE ... Azure ... Digital Ocean ... Scaleway ... Exoscale ... Oracle Cloud Infrastructure ... OVHcloud ... Bare-metal Miscellaneous Quick start \u00b6 If you have Helm, you can deploy the ingress controller with the following command: helm upgrade --install ingress-nginx ingress-nginx \\ --repo https://kubernetes.github.io/ingress-nginx \\ --namespace ingress-nginx --create-namespace It will install the controller in the ingress-nginx namespace, creating that namespace if it doesn't already exist. Info This command is idempotent : if the ingress controller is not installed, it will install it, if the ingress controller is already installed, it will upgrade it. If you don't have Helm or if you prefer to use a YAML manifest, you can run the following command instead: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml Info The YAML manifest in the command above was generated with helm template , so you will end up with almost the same resources as if you had used Helm to install the controller. Attention If you are running an old version of Kubernetes (1.18 or earlier), please read this paragraph for specific instructions. Because of api deprecations, the default manifest may not work on your cluster. Specific manifests for supported Kubernetes versions are available within a sub-folder of each provider. Pre-flight check \u00b6 A few pods should start in the ingress-nginx namespace: kubectl get pods --namespace=ingress-nginx After a while, they should all be running. The following command will wait for the ingress controller pod to be up, running, and ready: kubectl wait --namespace ingress-nginx \\ --for=condition=ready pod \\ --selector=app.kubernetes.io/component=controller \\ --timeout=120s Local testing \u00b6 Let's create a simple web server and the associated service: kubectl create deployment demo --image=httpd --port=80 kubectl expose deployment demo Then create an ingress resource. The following example uses a host that maps to localhost : kubectl create ingress demo-localhost --class=nginx \\ --rule=\"demo.localdev.me/*=demo:80\" Now, forward a local port to the ingress controller: kubectl port-forward --namespace=ingress-nginx service/ingress-nginx-controller 8080:80 At this point, if you access http://demo.localdev.me:8080/, you should see an HTML page telling you \"It works!\". Online testing \u00b6 If your Kubernetes cluster is a \"real\" cluster that supports services of type LoadBalancer , it will have allocated an external IP address or FQDN to the ingress controller. You can see that IP address or FQDN with the following command: kubectl get service ingress-nginx-controller --namespace=ingress-nginx It will be the EXTERNAL-IP field. If that field shows <pending> , this means that your Kubernetes cluster wasn't able to provision the load balancer (generally, this is because it doesn't support services of type LoadBalancer ). Once you have the external IP address (or FQDN), set up a DNS record pointing to it. Then you can create an ingress resource. The following example assumes that you have set up a DNS record for www.demo.io : kubectl create ingress demo --class=nginx \\ --rule=\"www.demo.io/*=demo:80\" Alternatively, the above command can be rewritten as follows for the --rule command and below. kubectl create ingress demo --class=nginx \\ --rule www.demo.io/=demo:80 You should then be able to see the \"It works!\" page when you connect to http://www.demo.io/. Congratulations, you are serving a public website hosted on a Kubernetes cluster! \ud83c\udf89 Environment-specific instructions \u00b6 Local development clusters \u00b6 minikube \u00b6 The ingress controller can be installed through minikube's addons system: minikube addons enable ingress MicroK8s \u00b6 The ingress controller can be installed through MicroK8s's addons system: microk8s enable ingress Please check the MicroK8s documentation page for details. Docker Desktop \u00b6 Kubernetes is available in Docker Desktop: Mac, from version 18.06.0-ce Windows, from version 18.06.0-ce First, make sure that Kubernetes is enabled in the Docker settings. The command kubectl get nodes should show a single node called docker-desktop . The ingress controller can be installed on Docker Desktop using the default quick start instructions. On most systems, if you don't have any other service of type LoadBalancer bound to port 80, the ingress controller will be assigned the EXTERNAL-IP of localhost , which means that it will be reachable on localhost:80. If that doesn't work, you might have to fall back to the kubectl port-forward method described in the local testing section . Rancher Desktop \u00b6 Rancher Desktop provides Kubernetes and Container Management on the desktop. Kubernetes is enabled by default in Rancher Desktop. Rancher Desktop uses K3s under the hood, which in turn uses Traefik as the default ingress controller for the Kubernetes cluster. To use NGINX ingress controller in place of the default Traefik, disable Traefik from Preference > Kubernetes menu. Once traefik is disabled, the NGINX ingress controller can be installed on Rancher Desktop using the default quick start instructions. Follow the instructions described in the local testing section to try a sample. Cloud deployments \u00b6 If the load balancers of your cloud provider do active healthchecks on their backends (most do), you can change the externalTrafficPolicy of the ingress controller Service to Local (instead of the default Cluster ) to save an extra hop in some cases. If you're installing with Helm, this can be done by adding --set controller.service.externalTrafficPolicy=Local to the helm install or helm upgrade command. Furthermore, if the load balancers of your cloud provider support the PROXY protocol, you can enable it, and it will let the ingress controller see the real IP address of the clients. Otherwise, it will generally see the IP address of the upstream load balancer. This must be done both in the ingress controller (with e.g. --set controller.config.use-proxy-protocol=true ) and in the cloud provider's load balancer configuration to function correctly. In the following sections, we provide YAML manifests that enable these options when possible, using the specific options of various cloud providers. AWS \u00b6 In AWS, we use a Network load balancer (NLB) to expose the NGINX Ingress controller behind a Service of Type=LoadBalancer . Info The provided templates illustrate the setup for legacy in-tree service load balancer for AWS NLB. AWS provides the documentation on how to use Network load balancing on Amazon EKS with AWS Load Balancer Controller . Network Load Balancer (NLB) \u00b6 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/aws/deploy.yaml TLS termination in AWS Load Balancer (NLB) \u00b6 By default, TLS is terminated in the ingress controller. But it is also possible to terminate TLS in the Load Balancer. This section explains how to do that on AWS using an NLB. Download the deploy.yaml template wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/aws/nlb-with-tls-termination/deploy.yaml Edit the file and change the VPC CIDR in use for the Kubernetes cluster: proxy-real-ip-cidr: XXX.XXX.XXX/XX Change the AWS Certificate Manager (ACM) ID as well: arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX Deploy the manifest: kubectl apply -f deploy.yaml NLB Idle Timeouts \u00b6 Idle timeout value for TCP flows is 350 seconds and cannot be modified . For this reason, you need to ensure the keepalive_timeout value is configured less than 350 seconds to work as expected. By default, NGINX keepalive_timeout is set to 75s . More information with regard to timeouts can be found in the official AWS documentation GCE-GKE \u00b6 First, your user needs to have cluster-admin permissions on the cluster. This can be done with the following command: kubectl create clusterrolebinding cluster-admin-binding \\ --clusterrole cluster-admin \\ --user $(gcloud config get-value account) Then, the ingress controller can be installed like this: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml Warning For private clusters, you will need to either add a firewall rule that allows master nodes access to port 8443/tcp on worker nodes, or change the existing rule that allows access to port 80/tcp , 443/tcp and 10254/tcp to also allow access to port 8443/tcp . More information can be found in the Official GCP Documentation . See the GKE documentation on adding rules and the Kubernetes issue for more detail. Proxy-protocol is supported in GCE check the Official Documentations on how to enable. Azure \u00b6 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml More information with regard to Azure annotations for ingress controller can be found in the official AKS documentation . Digital Ocean \u00b6 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/do/deploy.yaml - By default the service object of the ingress-nginx-controller for Digital-Ocean, only configures one annotation. Its this one service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: \"true\" . While this makes the service functional, it was reported that the Digital-Ocean LoadBalancer graphs shows no data , unless a few other annotations are also configured. Some of these other annotations require values that can not be generic and hence not forced in a out-of-the-box installation. These annotations and a discussion on them is well documented in this issue . Please refer to the issue to add annotations, with values specific to user, to get graphs of the DO-LB populated with data. Scaleway \u00b6 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/scw/deploy.yaml Exoscale \u00b6 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/exoscale/deploy.yaml The full list of annotations supported by Exoscale is available in the Exoscale Cloud Controller Manager documentation . Oracle Cloud Infrastructure \u00b6 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml A complete list of available annotations for Oracle Cloud Infrastructure can be found in the OCI Cloud Controller Manager documentation. OVHcloud \u00b6 helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update helm -n ingress-nginx install ingress-nginx ingress-nginx/ingress-nginx --create-namespace You can find the complete tutorial . Bare metal clusters \u00b6 This section is applicable to Kubernetes clusters deployed on bare metal servers, as well as \"raw\" VMs where Kubernetes was installed manually, using generic Linux distros (like CentOS, Ubuntu...) For quick testing, you can use a NodePort . This should work on almost every cluster, but it will typically use a port in the range 30000-32767. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/baremetal/deploy.yaml For more information about bare metal deployments (and how to use port 80 instead of a random port in the 30000-32767 range), see bare-metal considerations . Miscellaneous \u00b6 Checking ingress controller version \u00b6 Run /nginx-ingress-controller --version within the pod, for instance with kubectl exec : POD_NAMESPACE=ingress-nginx POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx --field-selector=status.phase=Running -o name) kubectl exec $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version Scope \u00b6 By default, the controller watches Ingress objects from all namespaces. If you want to change this behavior, use the flag --watch-namespace or check the Helm chart value controller.scope to limit the controller to a single namespace. See also \u201cHow to easily install multiple instances of the Ingress NGINX controller in the same cluster\u201d for more details. Webhook network access \u00b6 Warning The controller uses an admission webhook to validate Ingress definitions. Make sure that you don't have Network policies or additional firewalls preventing connections from the API server to the ingress-nginx-controller-admission service. Certificate generation \u00b6 Attention The first time the ingress controller starts, two Jobs create the SSL Certificate used by the admission webhook. This can cause an initial delay of up to two minutes until it is possible to create and validate Ingress definitions. You can wait until it is ready to run the next command: kubectl wait --namespace ingress-nginx \\ --for=condition=ready pod \\ --selector=app.kubernetes.io/component=controller \\ --timeout=120s Running on Kubernetes versions older than 1.19 \u00b6 Ingress resources evolved over time. They started with apiVersion: extensions/v1beta1 , then moved to apiVersion: networking.k8s.io/v1beta1 and more recently to apiVersion: networking.k8s.io/v1 . Here is how these Ingress versions are supported in Kubernetes: - before Kubernetes 1.19, only v1beta1 Ingress resources are supported - from Kubernetes 1.19 to 1.21, both v1beta1 and v1 Ingress resources are supported - in Kubernetes 1.22 and above, only v1 Ingress resources are supported And here is how these Ingress versions are supported in NGINX Ingress Controller: - before version 1.0, only v1beta1 Ingress resources are supported - in version 1.0 and above, only v1 Ingress resources are As a result, if you're running Kubernetes 1.19 or later, you should be able to use the latest version of the NGINX Ingress Controller; but if you're using an old version of Kubernetes (1.18 or earlier) you will have to use version 0.X of the NGINX Ingress Controller (e.g. version 0.49). The Helm chart of the NGINX Ingress Controller switched to version 1 in version 4 of the chart. In other words, if you're running Kubernetes 1.19 or earlier, you should use version 3.X of the chart (this can be done by adding --version='<4' to the helm install command).","title":"Installation Guide"},{"location":"deploy/#installation-guide","text":"There are multiple ways to install the NGINX ingress controller: with Helm , using the project repository chart; with kubectl apply , using YAML manifests; with specific addons (e.g. for minikube or MicroK8s ). On most Kubernetes clusters, the ingress controller will work without requiring any extra configuration. If you want to get started as fast as possible, you can check the quick start instructions. However, in many environments, you can improve the performance or get better logs by enabling extra features. We recommend that you check the environment-specific instructions for details about optimizing the ingress controller for your particular environment or cloud provider.","title":"Installation Guide"},{"location":"deploy/#contents","text":"Quick start Environment-specific instructions ... Docker Desktop ... Rancher Desktop ... minikube ... MicroK8s ... AWS ... GCE - GKE ... Azure ... Digital Ocean ... Scaleway ... Exoscale ... Oracle Cloud Infrastructure ... OVHcloud ... Bare-metal Miscellaneous","title":"Contents"},{"location":"deploy/#quick-start","text":"If you have Helm, you can deploy the ingress controller with the following command: helm upgrade --install ingress-nginx ingress-nginx \\ --repo https://kubernetes.github.io/ingress-nginx \\ --namespace ingress-nginx --create-namespace It will install the controller in the ingress-nginx namespace, creating that namespace if it doesn't already exist. Info This command is idempotent : if the ingress controller is not installed, it will install it, if the ingress controller is already installed, it will upgrade it. If you don't have Helm or if you prefer to use a YAML manifest, you can run the following command instead: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml Info The YAML manifest in the command above was generated with helm template , so you will end up with almost the same resources as if you had used Helm to install the controller. Attention If you are running an old version of Kubernetes (1.18 or earlier), please read this paragraph for specific instructions. Because of api deprecations, the default manifest may not work on your cluster. Specific manifests for supported Kubernetes versions are available within a sub-folder of each provider.","title":"Quick start"},{"location":"deploy/#pre-flight-check","text":"A few pods should start in the ingress-nginx namespace: kubectl get pods --namespace=ingress-nginx After a while, they should all be running. The following command will wait for the ingress controller pod to be up, running, and ready: kubectl wait --namespace ingress-nginx \\ --for=condition=ready pod \\ --selector=app.kubernetes.io/component=controller \\ --timeout=120s","title":"Pre-flight check"},{"location":"deploy/#local-testing","text":"Let's create a simple web server and the associated service: kubectl create deployment demo --image=httpd --port=80 kubectl expose deployment demo Then create an ingress resource. The following example uses a host that maps to localhost : kubectl create ingress demo-localhost --class=nginx \\ --rule=\"demo.localdev.me/*=demo:80\" Now, forward a local port to the ingress controller: kubectl port-forward --namespace=ingress-nginx service/ingress-nginx-controller 8080:80 At this point, if you access http://demo.localdev.me:8080/, you should see an HTML page telling you \"It works!\".","title":"Local testing"},{"location":"deploy/#online-testing","text":"If your Kubernetes cluster is a \"real\" cluster that supports services of type LoadBalancer , it will have allocated an external IP address or FQDN to the ingress controller. You can see that IP address or FQDN with the following command: kubectl get service ingress-nginx-controller --namespace=ingress-nginx It will be the EXTERNAL-IP field. If that field shows <pending> , this means that your Kubernetes cluster wasn't able to provision the load balancer (generally, this is because it doesn't support services of type LoadBalancer ). Once you have the external IP address (or FQDN), set up a DNS record pointing to it. Then you can create an ingress resource. The following example assumes that you have set up a DNS record for www.demo.io : kubectl create ingress demo --class=nginx \\ --rule=\"www.demo.io/*=demo:80\" Alternatively, the above command can be rewritten as follows for the --rule command and below. kubectl create ingress demo --class=nginx \\ --rule www.demo.io/=demo:80 You should then be able to see the \"It works!\" page when you connect to http://www.demo.io/. Congratulations, you are serving a public website hosted on a Kubernetes cluster! \ud83c\udf89","title":"Online testing"},{"location":"deploy/#environment-specific-instructions","text":"","title":"Environment-specific instructions"},{"location":"deploy/#local-development-clusters","text":"","title":"Local development clusters"},{"location":"deploy/#minikube","text":"The ingress controller can be installed through minikube's addons system: minikube addons enable ingress","title":"minikube"},{"location":"deploy/#microk8s","text":"The ingress controller can be installed through MicroK8s's addons system: microk8s enable ingress Please check the MicroK8s documentation page for details.","title":"MicroK8s"},{"location":"deploy/#docker-desktop","text":"Kubernetes is available in Docker Desktop: Mac, from version 18.06.0-ce Windows, from version 18.06.0-ce First, make sure that Kubernetes is enabled in the Docker settings. The command kubectl get nodes should show a single node called docker-desktop . The ingress controller can be installed on Docker Desktop using the default quick start instructions. On most systems, if you don't have any other service of type LoadBalancer bound to port 80, the ingress controller will be assigned the EXTERNAL-IP of localhost , which means that it will be reachable on localhost:80. If that doesn't work, you might have to fall back to the kubectl port-forward method described in the local testing section .","title":"Docker Desktop"},{"location":"deploy/#rancher-desktop","text":"Rancher Desktop provides Kubernetes and Container Management on the desktop. Kubernetes is enabled by default in Rancher Desktop. Rancher Desktop uses K3s under the hood, which in turn uses Traefik as the default ingress controller for the Kubernetes cluster. To use NGINX ingress controller in place of the default Traefik, disable Traefik from Preference > Kubernetes menu. Once traefik is disabled, the NGINX ingress controller can be installed on Rancher Desktop using the default quick start instructions. Follow the instructions described in the local testing section to try a sample.","title":"Rancher Desktop"},{"location":"deploy/#cloud-deployments","text":"If the load balancers of your cloud provider do active healthchecks on their backends (most do), you can change the externalTrafficPolicy of the ingress controller Service to Local (instead of the default Cluster ) to save an extra hop in some cases. If you're installing with Helm, this can be done by adding --set controller.service.externalTrafficPolicy=Local to the helm install or helm upgrade command. Furthermore, if the load balancers of your cloud provider support the PROXY protocol, you can enable it, and it will let the ingress controller see the real IP address of the clients. Otherwise, it will generally see the IP address of the upstream load balancer. This must be done both in the ingress controller (with e.g. --set controller.config.use-proxy-protocol=true ) and in the cloud provider's load balancer configuration to function correctly. In the following sections, we provide YAML manifests that enable these options when possible, using the specific options of various cloud providers.","title":"Cloud deployments"},{"location":"deploy/#aws","text":"In AWS, we use a Network load balancer (NLB) to expose the NGINX Ingress controller behind a Service of Type=LoadBalancer . Info The provided templates illustrate the setup for legacy in-tree service load balancer for AWS NLB. AWS provides the documentation on how to use Network load balancing on Amazon EKS with AWS Load Balancer Controller .","title":"AWS"},{"location":"deploy/#network-load-balancer-nlb","text":"kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/aws/deploy.yaml","title":"Network Load Balancer (NLB)"},{"location":"deploy/#tls-termination-in-aws-load-balancer-nlb","text":"By default, TLS is terminated in the ingress controller. But it is also possible to terminate TLS in the Load Balancer. This section explains how to do that on AWS using an NLB. Download the deploy.yaml template wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/aws/nlb-with-tls-termination/deploy.yaml Edit the file and change the VPC CIDR in use for the Kubernetes cluster: proxy-real-ip-cidr: XXX.XXX.XXX/XX Change the AWS Certificate Manager (ACM) ID as well: arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX Deploy the manifest: kubectl apply -f deploy.yaml","title":"TLS termination in AWS Load Balancer (NLB)"},{"location":"deploy/#nlb-idle-timeouts","text":"Idle timeout value for TCP flows is 350 seconds and cannot be modified . For this reason, you need to ensure the keepalive_timeout value is configured less than 350 seconds to work as expected. By default, NGINX keepalive_timeout is set to 75s . More information with regard to timeouts can be found in the official AWS documentation","title":"NLB Idle Timeouts"},{"location":"deploy/#gce-gke","text":"First, your user needs to have cluster-admin permissions on the cluster. This can be done with the following command: kubectl create clusterrolebinding cluster-admin-binding \\ --clusterrole cluster-admin \\ --user $(gcloud config get-value account) Then, the ingress controller can be installed like this: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml Warning For private clusters, you will need to either add a firewall rule that allows master nodes access to port 8443/tcp on worker nodes, or change the existing rule that allows access to port 80/tcp , 443/tcp and 10254/tcp to also allow access to port 8443/tcp . More information can be found in the Official GCP Documentation . See the GKE documentation on adding rules and the Kubernetes issue for more detail. Proxy-protocol is supported in GCE check the Official Documentations on how to enable.","title":"GCE-GKE"},{"location":"deploy/#azure","text":"kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml More information with regard to Azure annotations for ingress controller can be found in the official AKS documentation .","title":"Azure"},{"location":"deploy/#digital-ocean","text":"kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/do/deploy.yaml - By default the service object of the ingress-nginx-controller for Digital-Ocean, only configures one annotation. Its this one service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: \"true\" . While this makes the service functional, it was reported that the Digital-Ocean LoadBalancer graphs shows no data , unless a few other annotations are also configured. Some of these other annotations require values that can not be generic and hence not forced in a out-of-the-box installation. These annotations and a discussion on them is well documented in this issue . Please refer to the issue to add annotations, with values specific to user, to get graphs of the DO-LB populated with data.","title":"Digital Ocean"},{"location":"deploy/#scaleway","text":"kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/scw/deploy.yaml","title":"Scaleway"},{"location":"deploy/#exoscale","text":"kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/exoscale/deploy.yaml The full list of annotations supported by Exoscale is available in the Exoscale Cloud Controller Manager documentation .","title":"Exoscale"},{"location":"deploy/#oracle-cloud-infrastructure","text":"kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml A complete list of available annotations for Oracle Cloud Infrastructure can be found in the OCI Cloud Controller Manager documentation.","title":"Oracle Cloud Infrastructure"},{"location":"deploy/#ovhcloud","text":"helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update helm -n ingress-nginx install ingress-nginx ingress-nginx/ingress-nginx --create-namespace You can find the complete tutorial .","title":"OVHcloud"},{"location":"deploy/#bare-metal-clusters","text":"This section is applicable to Kubernetes clusters deployed on bare metal servers, as well as \"raw\" VMs where Kubernetes was installed manually, using generic Linux distros (like CentOS, Ubuntu...) For quick testing, you can use a NodePort . This should work on almost every cluster, but it will typically use a port in the range 30000-32767. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/baremetal/deploy.yaml For more information about bare metal deployments (and how to use port 80 instead of a random port in the 30000-32767 range), see bare-metal considerations .","title":"Bare metal clusters"},{"location":"deploy/#miscellaneous","text":"","title":"Miscellaneous"},{"location":"deploy/#checking-ingress-controller-version","text":"Run /nginx-ingress-controller --version within the pod, for instance with kubectl exec : POD_NAMESPACE=ingress-nginx POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx --field-selector=status.phase=Running -o name) kubectl exec $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version","title":"Checking ingress controller version"},{"location":"deploy/#scope","text":"By default, the controller watches Ingress objects from all namespaces. If you want to change this behavior, use the flag --watch-namespace or check the Helm chart value controller.scope to limit the controller to a single namespace. See also \u201cHow to easily install multiple instances of the Ingress NGINX controller in the same cluster\u201d for more details.","title":"Scope"},{"location":"deploy/#webhook-network-access","text":"Warning The controller uses an admission webhook to validate Ingress definitions. Make sure that you don't have Network policies or additional firewalls preventing connections from the API server to the ingress-nginx-controller-admission service.","title":"Webhook network access"},{"location":"deploy/#certificate-generation","text":"Attention The first time the ingress controller starts, two Jobs create the SSL Certificate used by the admission webhook. This can cause an initial delay of up to two minutes until it is possible to create and validate Ingress definitions. You can wait until it is ready to run the next command: kubectl wait --namespace ingress-nginx \\ --for=condition=ready pod \\ --selector=app.kubernetes.io/component=controller \\ --timeout=120s","title":"Certificate generation"},{"location":"deploy/#running-on-kubernetes-versions-older-than-119","text":"Ingress resources evolved over time. They started with apiVersion: extensions/v1beta1 , then moved to apiVersion: networking.k8s.io/v1beta1 and more recently to apiVersion: networking.k8s.io/v1 . Here is how these Ingress versions are supported in Kubernetes: - before Kubernetes 1.19, only v1beta1 Ingress resources are supported - from Kubernetes 1.19 to 1.21, both v1beta1 and v1 Ingress resources are supported - in Kubernetes 1.22 and above, only v1 Ingress resources are supported And here is how these Ingress versions are supported in NGINX Ingress Controller: - before version 1.0, only v1beta1 Ingress resources are supported - in version 1.0 and above, only v1 Ingress resources are As a result, if you're running Kubernetes 1.19 or later, you should be able to use the latest version of the NGINX Ingress Controller; but if you're using an old version of Kubernetes (1.18 or earlier) you will have to use version 0.X of the NGINX Ingress Controller (e.g. version 0.49). The Helm chart of the NGINX Ingress Controller switched to version 1 in version 4 of the chart. In other words, if you're running Kubernetes 1.19 or earlier, you should use version 3.X of the chart (this can be done by adding --version='<4' to the helm install command).","title":"Running on Kubernetes versions older than 1.19"},{"location":"deploy/baremetal/","text":"Bare-metal considerations \u00b6 In traditional cloud environments, where network load balancers are available on-demand, a single Kubernetes manifest suffices to provide a single point of contact to the NGINX Ingress controller to external clients and, indirectly, to any application running inside the cluster. Bare-metal environments lack this commodity, requiring a slightly different setup to offer the same kind of access to external consumers. The rest of this document describes a few recommended approaches to deploying the NGINX Ingress controller inside a Kubernetes cluster running on bare-metal. A pure software solution: MetalLB \u00b6 MetalLB provides a network load-balancer implementation for Kubernetes clusters that do not run on a supported cloud provider, effectively allowing the usage of LoadBalancer Services within any cluster. This section demonstrates how to use the Layer 2 configuration mode of MetalLB together with the NGINX Ingress controller in a Kubernetes cluster that has publicly accessible nodes . In this mode, one node attracts all the traffic for the ingress-nginx Service IP. See Traffic policies for more details. Note The description of other supported configuration modes is off-scope for this document. Warning MetalLB is currently in beta . Read about the Project maturity and make sure you inform yourself by reading the official documentation thoroughly. MetalLB can be deployed either with a simple Kubernetes manifest or with Helm. The rest of this example assumes MetalLB was deployed following the Installation instructions, and that the NGINX Ingress controller was installed using the steps described in the quickstart section of the installation guide . MetalLB requires a pool of IP addresses in order to be able to take ownership of the ingress-nginx Service. This pool can be defined through IPAddressPool objects in the same namespace as the MetalLB controller. This pool of IPs must be dedicated to MetalLB's use, you can't reuse the Kubernetes node IPs or IPs handed out by a DHCP server. Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 After creating the following objects, MetalLB takes ownership of one of the IP addresses in the pool and updates the loadBalancer IP field of the ingress-nginx Service accordingly. --- apiVersion : metallb.io/v1beta1 kind : IPAddressPool metadata : name : default namespace : metallb-system spec : addresses : - 203.0.113.10-203.0.113.15 autoAssign : true --- apiVersion : metallb.io/v1beta1 kind : L2Advertisement metadata : name : default namespace : metallb-system spec : ipAddressPools : - default $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 <none> 80/TCP ingress-nginx LoadBalancer 10.0.220.217 203.0.113.10 80:30100/TCP,443:30101/TCP As soon as MetalLB sets the external IP address of the ingress-nginx LoadBalancer Service, the corresponding entries are created in the iptables NAT table and the node with the selected IP address starts responding to HTTP requests on the ports configured in the LoadBalancer Service: $ curl -D- http://203.0.113.10 -H 'Host: myapp.example.com' HTTP/1.1 200 OK Server: nginx/1.15.2 Tip In order to preserve the source IP address in HTTP requests sent to NGINX, it is necessary to use the Local traffic policy. Traffic policies are described in more details in Traffic policies as well as in the next section. Over a NodePort Service \u00b6 Due to its simplicity, this is the setup a user will deploy by default when following the steps described in the installation guide . Info A Service of type NodePort exposes, via the kube-proxy component, the same unprivileged port (default: 30000-32767) on every Kubernetes node, masters included. For more information, see Services . In this configuration, the NGINX container remains isolated from the host network. As a result, it can safely bind to any port, including the standard HTTP ports 80 and 443. However, due to the container namespace isolation, a client located outside the cluster network (e.g. on the public internet) is not able to access Ingress hosts directly on ports 80 and 443. Instead, the external client must append the NodePort allocated to the ingress-nginx Service to HTTP requests. Example Given the NodePort 30100 allocated to the ingress-nginx Service $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 80/TCP ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP and a Kubernetes node with the public IP address 203.0.113.2 (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 a client would reach an Ingress with host: myapp.example.com at http://myapp.example.com:30100 , where the myapp.example.com subdomain resolves to the 203.0.113.2 IP address. Impact on the host system While it may sound tempting to reconfigure the NodePort range using the --service-node-port-range API server flag to include unprivileged ports and be able to expose ports 80 and 443, doing so may result in unexpected issues including (but not limited to) the use of ports otherwise reserved to system daemons and the necessity to grant kube-proxy privileges it may otherwise not require. This practice is therefore discouraged . See the other approaches proposed in this page for alternatives. This approach has a few other limitations one ought to be aware of: Source IP address Services of type NodePort perform source address translation by default. This means the source IP of a HTTP request is always the IP address of the Kubernetes node that received the request from the perspective of NGINX. The recommended way to preserve the source IP in a NodePort setup is to set the value of the externalTrafficPolicy field of the ingress-nginx Service spec to Local ( example ). Warning This setting effectively drops packets sent to Kubernetes nodes which are not running any instance of the NGINX Ingress controller. Consider assigning NGINX Pods to specific nodes in order to control on what nodes the NGINX Ingress controller should be scheduled or not scheduled. Example In a Kubernetes cluster composed of 3 nodes (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 with a ingress-nginx-controller Deployment composed of 2 replicas $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 ingress-nginx-controller-cf9ff8c96-8vvf8 1/1 Running 172.17.0.3 host-3 ingress-nginx-controller-cf9ff8c96-pxsds 1/1 Running 172.17.1.4 host-2 Requests sent to host-2 and host-3 would be forwarded to NGINX and original client's IP would be preserved, while requests to host-1 would get dropped because there is no NGINX replica running on that node. Ingress status Because NodePort Services do not get a LoadBalancerIP assigned by definition, the NGINX Ingress controller does not update the status of Ingress objects it manages . $ kubectl get ingress NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 80 Despite the fact there is no load balancer providing a public IP address to the NGINX Ingress controller, it is possible to force the status update of all managed Ingress objects by setting the externalIPs field of the ingress-nginx Service. Warning There is more to setting externalIPs than just enabling the NGINX Ingress controller to update the status of Ingress objects. Please read about this option in the Services page of official Kubernetes documentation as well as the section about External IPs in this document for more information. Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 one could edit the ingress-nginx Service and add the following field to the object spec spec : externalIPs : - 203.0.113.1 - 203.0.113.2 - 203.0.113.3 which would in turn be reflected on Ingress objects as follows: $ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.1,203.0.113.2,203.0.113.3 80 Redirects As NGINX is not aware of the port translation operated by the NodePort Service , backend applications are responsible for generating redirect URLs that take into account the URL used by external clients, including the NodePort. Example Redirects generated by NGINX, for instance HTTP to HTTPS or domain to www.domain , are generated without NodePort: $ curl -D- http://myapp.example.com:30100 ` HTTP/1.1 308 Permanent Redirect Server: nginx/1.15.2 Location: https://myapp.example.com/ #-> missing NodePort in HTTPS redirect Via the host network \u00b6 In a setup where there is no external load balancer available but using NodePorts is not an option, one can configure ingress-nginx Pods to use the network of the host they run on instead of a dedicated network namespace. The benefit of this approach is that the NGINX Ingress controller can bind ports 80 and 443 directly to Kubernetes nodes' network interfaces, without the extra network translation imposed by NodePort Services. Note This approach does not leverage any Service object to expose the NGINX Ingress controller. If the ingress-nginx Service exists in the target cluster, it is recommended to delete it . This can be achieved by enabling the hostNetwork option in the Pods' spec. template : spec : hostNetwork : true Security considerations Enabling this option exposes every system daemon to the NGINX Ingress controller on any network interface, including the host's loopback. Please evaluate the impact this may have on the security of your system carefully. Example Consider this ingress-nginx-controller Deployment composed of 2 replicas, NGINX Pods inherit from the IP address of their host instead of an internal Pod IP. $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 ingress-nginx-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3 ingress-nginx-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2 One major limitation of this deployment approach is that only a single NGINX Ingress controller Pod may be scheduled on each cluster node, because binding the same port multiple times on the same network interface is technically impossible. Pods that are unschedulable due to such situation fail with the following event: $ kubectl -n ingress-nginx describe pod <unschedulable-ingress-nginx-controller-pod> ... Events: Type Reason From Message ---- ------ ---- ------- Warning FailedScheduling default-scheduler 0/3 nodes are available: 3 node(s) didn't have free ports for the requested pod ports. One way to ensure only schedulable Pods are created is to deploy the NGINX Ingress controller as a DaemonSet instead of a traditional Deployment. Info A DaemonSet schedules exactly one type of Pod per cluster node, masters included, unless a node is configured to repel those Pods . For more information, see DaemonSet . Because most properties of DaemonSet objects are identical to Deployment objects, this documentation page leaves the configuration of the corresponding manifest at the user's discretion. Like with NodePorts, this approach has a few quirks it is important to be aware of. DNS resolution Pods configured with hostNetwork: true do not use the internal DNS resolver (i.e. kube-dns or CoreDNS ), unless their dnsPolicy spec field is set to ClusterFirstWithHostNet . Consider using this setting if NGINX is expected to resolve internal names for any reason. Ingress status Because there is no Service exposing the NGINX Ingress controller in a configuration using the host network, the default --publish-service flag used in standard cloud setups does not apply and the status of all Ingress objects remains blank. $ kubectl get ingress NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 80 Instead, and because bare-metal nodes usually don't have an ExternalIP, one has to enable the --report-node-internal-ip-address flag, which sets the status of all Ingress objects to the internal IP address of all nodes running the NGINX Ingress controller. Example Given a ingress-nginx-controller DaemonSet composed of 2 replicas $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 ingress-nginx-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3 ingress-nginx-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2 the controller sets the status of all Ingress objects it manages to the following value: $ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.2,203.0.113.3 80 Note Alternatively, it is possible to override the address written to Ingress objects using the --publish-status-address flag. See Command line arguments . Using a self-provisioned edge \u00b6 Similarly to cloud environments, this deployment approach requires an edge network component providing a public entrypoint to the Kubernetes cluster. This edge component can be either hardware (e.g. vendor appliance) or software (e.g. HAproxy ) and is usually managed outside of the Kubernetes landscape by operations teams. Such deployment builds upon the NodePort Service described above in Over a NodePort Service , with one significant difference: external clients do not access cluster nodes directly, only the edge component does. This is particularly suitable for private Kubernetes clusters where none of the nodes has a public IP address. On the edge side, the only prerequisite is to dedicate a public IP address that forwards all HTTP traffic to Kubernetes nodes and/or masters. Incoming traffic on TCP ports 80 and 443 is forwarded to the corresponding HTTP and HTTPS NodePort on the target nodes as shown in the diagram below: External IPs \u00b6 Source IP address This method does not allow preserving the source IP of HTTP requests in any manner, it is therefore not recommended to use it despite its apparent simplicity. The externalIPs Service option was previously mentioned in the NodePort section. As per the Services page of the official Kubernetes documentation, the externalIPs option causes kube-proxy to route traffic sent to arbitrary IP addresses and on the Service ports to the endpoints of that Service. These IP addresses must belong to the target node . Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 and the following ingress-nginx NodePort Service $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP One could set the following external IPs in the Service spec, and NGINX would become available on both the NodePort and the Service port: spec : externalIPs : - 203.0.113.2 - 203.0.113.3 $ curl -D- http://myapp.example.com:30100 HTTP/1.1 200 OK Server: nginx/1.15.2 $ curl -D- http://myapp.example.com HTTP/1.1 200 OK Server: nginx/1.15.2 We assume the myapp.example.com subdomain above resolves to both 203.0.113.2 and 203.0.113.3 IP addresses.","title":"Bare-metal considerations"},{"location":"deploy/baremetal/#bare-metal-considerations","text":"In traditional cloud environments, where network load balancers are available on-demand, a single Kubernetes manifest suffices to provide a single point of contact to the NGINX Ingress controller to external clients and, indirectly, to any application running inside the cluster. Bare-metal environments lack this commodity, requiring a slightly different setup to offer the same kind of access to external consumers. The rest of this document describes a few recommended approaches to deploying the NGINX Ingress controller inside a Kubernetes cluster running on bare-metal.","title":"Bare-metal considerations"},{"location":"deploy/baremetal/#a-pure-software-solution-metallb","text":"MetalLB provides a network load-balancer implementation for Kubernetes clusters that do not run on a supported cloud provider, effectively allowing the usage of LoadBalancer Services within any cluster. This section demonstrates how to use the Layer 2 configuration mode of MetalLB together with the NGINX Ingress controller in a Kubernetes cluster that has publicly accessible nodes . In this mode, one node attracts all the traffic for the ingress-nginx Service IP. See Traffic policies for more details. Note The description of other supported configuration modes is off-scope for this document. Warning MetalLB is currently in beta . Read about the Project maturity and make sure you inform yourself by reading the official documentation thoroughly. MetalLB can be deployed either with a simple Kubernetes manifest or with Helm. The rest of this example assumes MetalLB was deployed following the Installation instructions, and that the NGINX Ingress controller was installed using the steps described in the quickstart section of the installation guide . MetalLB requires a pool of IP addresses in order to be able to take ownership of the ingress-nginx Service. This pool can be defined through IPAddressPool objects in the same namespace as the MetalLB controller. This pool of IPs must be dedicated to MetalLB's use, you can't reuse the Kubernetes node IPs or IPs handed out by a DHCP server. Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 After creating the following objects, MetalLB takes ownership of one of the IP addresses in the pool and updates the loadBalancer IP field of the ingress-nginx Service accordingly. --- apiVersion : metallb.io/v1beta1 kind : IPAddressPool metadata : name : default namespace : metallb-system spec : addresses : - 203.0.113.10-203.0.113.15 autoAssign : true --- apiVersion : metallb.io/v1beta1 kind : L2Advertisement metadata : name : default namespace : metallb-system spec : ipAddressPools : - default $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 <none> 80/TCP ingress-nginx LoadBalancer 10.0.220.217 203.0.113.10 80:30100/TCP,443:30101/TCP As soon as MetalLB sets the external IP address of the ingress-nginx LoadBalancer Service, the corresponding entries are created in the iptables NAT table and the node with the selected IP address starts responding to HTTP requests on the ports configured in the LoadBalancer Service: $ curl -D- http://203.0.113.10 -H 'Host: myapp.example.com' HTTP/1.1 200 OK Server: nginx/1.15.2 Tip In order to preserve the source IP address in HTTP requests sent to NGINX, it is necessary to use the Local traffic policy. Traffic policies are described in more details in Traffic policies as well as in the next section.","title":"A pure software solution: MetalLB"},{"location":"deploy/baremetal/#over-a-nodeport-service","text":"Due to its simplicity, this is the setup a user will deploy by default when following the steps described in the installation guide . Info A Service of type NodePort exposes, via the kube-proxy component, the same unprivileged port (default: 30000-32767) on every Kubernetes node, masters included. For more information, see Services . In this configuration, the NGINX container remains isolated from the host network. As a result, it can safely bind to any port, including the standard HTTP ports 80 and 443. However, due to the container namespace isolation, a client located outside the cluster network (e.g. on the public internet) is not able to access Ingress hosts directly on ports 80 and 443. Instead, the external client must append the NodePort allocated to the ingress-nginx Service to HTTP requests. Example Given the NodePort 30100 allocated to the ingress-nginx Service $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 80/TCP ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP and a Kubernetes node with the public IP address 203.0.113.2 (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 a client would reach an Ingress with host: myapp.example.com at http://myapp.example.com:30100 , where the myapp.example.com subdomain resolves to the 203.0.113.2 IP address. Impact on the host system While it may sound tempting to reconfigure the NodePort range using the --service-node-port-range API server flag to include unprivileged ports and be able to expose ports 80 and 443, doing so may result in unexpected issues including (but not limited to) the use of ports otherwise reserved to system daemons and the necessity to grant kube-proxy privileges it may otherwise not require. This practice is therefore discouraged . See the other approaches proposed in this page for alternatives. This approach has a few other limitations one ought to be aware of: Source IP address Services of type NodePort perform source address translation by default. This means the source IP of a HTTP request is always the IP address of the Kubernetes node that received the request from the perspective of NGINX. The recommended way to preserve the source IP in a NodePort setup is to set the value of the externalTrafficPolicy field of the ingress-nginx Service spec to Local ( example ). Warning This setting effectively drops packets sent to Kubernetes nodes which are not running any instance of the NGINX Ingress controller. Consider assigning NGINX Pods to specific nodes in order to control on what nodes the NGINX Ingress controller should be scheduled or not scheduled. Example In a Kubernetes cluster composed of 3 nodes (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 with a ingress-nginx-controller Deployment composed of 2 replicas $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 ingress-nginx-controller-cf9ff8c96-8vvf8 1/1 Running 172.17.0.3 host-3 ingress-nginx-controller-cf9ff8c96-pxsds 1/1 Running 172.17.1.4 host-2 Requests sent to host-2 and host-3 would be forwarded to NGINX and original client's IP would be preserved, while requests to host-1 would get dropped because there is no NGINX replica running on that node. Ingress status Because NodePort Services do not get a LoadBalancerIP assigned by definition, the NGINX Ingress controller does not update the status of Ingress objects it manages . $ kubectl get ingress NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 80 Despite the fact there is no load balancer providing a public IP address to the NGINX Ingress controller, it is possible to force the status update of all managed Ingress objects by setting the externalIPs field of the ingress-nginx Service. Warning There is more to setting externalIPs than just enabling the NGINX Ingress controller to update the status of Ingress objects. Please read about this option in the Services page of official Kubernetes documentation as well as the section about External IPs in this document for more information. Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 one could edit the ingress-nginx Service and add the following field to the object spec spec : externalIPs : - 203.0.113.1 - 203.0.113.2 - 203.0.113.3 which would in turn be reflected on Ingress objects as follows: $ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.1,203.0.113.2,203.0.113.3 80 Redirects As NGINX is not aware of the port translation operated by the NodePort Service , backend applications are responsible for generating redirect URLs that take into account the URL used by external clients, including the NodePort. Example Redirects generated by NGINX, for instance HTTP to HTTPS or domain to www.domain , are generated without NodePort: $ curl -D- http://myapp.example.com:30100 ` HTTP/1.1 308 Permanent Redirect Server: nginx/1.15.2 Location: https://myapp.example.com/ #-> missing NodePort in HTTPS redirect","title":"Over a NodePort Service"},{"location":"deploy/baremetal/#via-the-host-network","text":"In a setup where there is no external load balancer available but using NodePorts is not an option, one can configure ingress-nginx Pods to use the network of the host they run on instead of a dedicated network namespace. The benefit of this approach is that the NGINX Ingress controller can bind ports 80 and 443 directly to Kubernetes nodes' network interfaces, without the extra network translation imposed by NodePort Services. Note This approach does not leverage any Service object to expose the NGINX Ingress controller. If the ingress-nginx Service exists in the target cluster, it is recommended to delete it . This can be achieved by enabling the hostNetwork option in the Pods' spec. template : spec : hostNetwork : true Security considerations Enabling this option exposes every system daemon to the NGINX Ingress controller on any network interface, including the host's loopback. Please evaluate the impact this may have on the security of your system carefully. Example Consider this ingress-nginx-controller Deployment composed of 2 replicas, NGINX Pods inherit from the IP address of their host instead of an internal Pod IP. $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 ingress-nginx-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3 ingress-nginx-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2 One major limitation of this deployment approach is that only a single NGINX Ingress controller Pod may be scheduled on each cluster node, because binding the same port multiple times on the same network interface is technically impossible. Pods that are unschedulable due to such situation fail with the following event: $ kubectl -n ingress-nginx describe pod <unschedulable-ingress-nginx-controller-pod> ... Events: Type Reason From Message ---- ------ ---- ------- Warning FailedScheduling default-scheduler 0/3 nodes are available: 3 node(s) didn't have free ports for the requested pod ports. One way to ensure only schedulable Pods are created is to deploy the NGINX Ingress controller as a DaemonSet instead of a traditional Deployment. Info A DaemonSet schedules exactly one type of Pod per cluster node, masters included, unless a node is configured to repel those Pods . For more information, see DaemonSet . Because most properties of DaemonSet objects are identical to Deployment objects, this documentation page leaves the configuration of the corresponding manifest at the user's discretion. Like with NodePorts, this approach has a few quirks it is important to be aware of. DNS resolution Pods configured with hostNetwork: true do not use the internal DNS resolver (i.e. kube-dns or CoreDNS ), unless their dnsPolicy spec field is set to ClusterFirstWithHostNet . Consider using this setting if NGINX is expected to resolve internal names for any reason. Ingress status Because there is no Service exposing the NGINX Ingress controller in a configuration using the host network, the default --publish-service flag used in standard cloud setups does not apply and the status of all Ingress objects remains blank. $ kubectl get ingress NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 80 Instead, and because bare-metal nodes usually don't have an ExternalIP, one has to enable the --report-node-internal-ip-address flag, which sets the status of all Ingress objects to the internal IP address of all nodes running the NGINX Ingress controller. Example Given a ingress-nginx-controller DaemonSet composed of 2 replicas $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 ingress-nginx-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3 ingress-nginx-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2 the controller sets the status of all Ingress objects it manages to the following value: $ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.2,203.0.113.3 80 Note Alternatively, it is possible to override the address written to Ingress objects using the --publish-status-address flag. See Command line arguments .","title":"Via the host network"},{"location":"deploy/baremetal/#using-a-self-provisioned-edge","text":"Similarly to cloud environments, this deployment approach requires an edge network component providing a public entrypoint to the Kubernetes cluster. This edge component can be either hardware (e.g. vendor appliance) or software (e.g. HAproxy ) and is usually managed outside of the Kubernetes landscape by operations teams. Such deployment builds upon the NodePort Service described above in Over a NodePort Service , with one significant difference: external clients do not access cluster nodes directly, only the edge component does. This is particularly suitable for private Kubernetes clusters where none of the nodes has a public IP address. On the edge side, the only prerequisite is to dedicate a public IP address that forwards all HTTP traffic to Kubernetes nodes and/or masters. Incoming traffic on TCP ports 80 and 443 is forwarded to the corresponding HTTP and HTTPS NodePort on the target nodes as shown in the diagram below:","title":"Using a self-provisioned edge"},{"location":"deploy/baremetal/#external-ips","text":"Source IP address This method does not allow preserving the source IP of HTTP requests in any manner, it is therefore not recommended to use it despite its apparent simplicity. The externalIPs Service option was previously mentioned in the NodePort section. As per the Services page of the official Kubernetes documentation, the externalIPs option causes kube-proxy to route traffic sent to arbitrary IP addresses and on the Service ports to the endpoints of that Service. These IP addresses must belong to the target node . Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 and the following ingress-nginx NodePort Service $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP One could set the following external IPs in the Service spec, and NGINX would become available on both the NodePort and the Service port: spec : externalIPs : - 203.0.113.2 - 203.0.113.3 $ curl -D- http://myapp.example.com:30100 HTTP/1.1 200 OK Server: nginx/1.15.2 $ curl -D- http://myapp.example.com HTTP/1.1 200 OK Server: nginx/1.15.2 We assume the myapp.example.com subdomain above resolves to both 203.0.113.2 and 203.0.113.3 IP addresses.","title":"External IPs"},{"location":"deploy/hardening-guide/","text":"Hardening Guide \u00b6 Overview \u00b6 There are several ways to do hardening and securing of nginx. In this documentation two guides are used, the guides are overlapping in some points: nginx CIS Benchmark cipherlist.eu (one of many forks of the now dead project cipherli.st) This guide describes, what of the different configurations described in those guides is already implemented as default in the nginx implementation of kubernetes ingress, what needs to be configured, what is obsolete due to the fact that the nginx is running as container (the CIS benchmark relates to a non-containerized installation) and what is difficult or not possible. Be aware that this is only a guide and you are responsible for your own implementation. Some of the configurations may lead to have specific clients unable to reach your site or similar consequences. This guide refers to chapters in the CIS Benchmark. For full explanation you should refer to the benchmark document itself Configuration Guide \u00b6 Chapter in CIS benchmark Status Default Action to do if not default 1 Initial Setup 1.1 Installation 1.1.1 Ensure NGINX is installed (Scored) OK done through helm charts / following documentation to deploy nginx ingress 1.1.2 Ensure NGINX is installed from source (Not Scored) OK done through helm charts / following documentation to deploy nginx ingress 1.2 Configure Software Updates 1.2.1 Ensure package manager repositories are properly configured (Not Scored) OK done via helm, nginx version could be overwritten, however compatibility is not ensured then 1.2.2 Ensure the latest software package is installed (Not Scored) ACTION NEEDED done via helm, nginx version could be overwritten, however compatibility is not ensured then Plan for periodic updates 2 Basic Configuration 2.1 Minimize NGINX Modules 2.1.1 Ensure only required modules are installed (Not Scored) OK Already only needed modules are installed, however proposals for further reduction are welcome 2.1.2 Ensure HTTP WebDAV module is not installed (Scored) OK 2.1.3 Ensure modules with gzip functionality are disabled (Scored) OK 2.1.4 Ensure the autoindex module is disabled (Scored) OK No autoindex configs so far in ingress defaults 2.2 Account Security 2.2.1 Ensure that NGINX is run using a non-privileged, dedicated service account (Not Scored) OK Pod configured as user www-data: See this line in helm chart values . Compiled with user www-data: See this line in build script 2.2.2 Ensure the NGINX service account is locked (Scored) OK Docker design ensures this 2.2.3 Ensure the NGINX service account has an invalid shell (Scored) OK Shell is nologin: see this line in build script 2.3 Permissions and Ownership 2.3.1 Ensure NGINX directories and files are owned by root (Scored) OK Obsolete through docker-design and ingress controller needs to update the configs dynamically 2.3.2 Ensure access to NGINX directories and files is restricted (Scored) OK See previous answer 2.3.3 Ensure the NGINX process ID (PID) file is secured (Scored) OK No PID-File due to docker design 2.3.4 Ensure the core dump directory is secured (Not Scored) OK No working_directory configured by default 2.4 Network Configuration 2.4.1 Ensure NGINX only listens for network connections on authorized ports (Not Scored) OK Ensured by automatic nginx.conf configuration 2.4.2 Ensure requests for unknown host names are rejected (Not Scored) OK They are not rejected but send to the \"default backend\" delivering appropriate errors (mostly 404) 2.4.3 Ensure keepalive_timeout is 10 seconds or less, but not 0 (Scored) ACTION NEEDED Default is 75s configure keep-alive to 10 seconds according to this documentation 2.4.4 Ensure send_timeout is set to 10 seconds or less, but not 0 (Scored) RISK TO BE ACCEPTED Not configured, however the nginx default is 60s Not configurable 2.5 Information Disclosure 2.5.1 Ensure server_tokens directive is set to off (Scored) OK server_tokens is configured to off by default 2.5.2 Ensure default error and index.html pages do not reference NGINX (Scored) ACTION NEEDED 404 shows no version at all, 503 and 403 show \"nginx\", which is hardcoded see this line in nginx source code configure custom error pages at least for 403, 404 and 503 and 500 2.5.3 Ensure hidden file serving is disabled (Not Scored) ACTION NEEDED config not set configure a config.server-snippet Snippet, but beware of .well-known challenges or similar. Refer to the benchmark here please 2.5.4 Ensure the NGINX reverse proxy does not enable information disclosure (Scored) ACTION NEEDED hide not configured configure hide-headers with array of \"X-Powered-By\" and \"Server\": according to this documentation 3 Logging 3.1 Ensure detailed logging is enabled (Not Scored) OK nginx ingress has a very detailed log format by default 3.2 Ensure access logging is enabled (Scored) OK Access log is enabled by default 3.3 Ensure error logging is enabled and set to the info logging level (Scored) OK Error log is configured by default. The log level does not matter, because it is all sent to STDOUT anyway 3.4 Ensure log files are rotated (Scored) OBSOLETE Log file handling is not part of the nginx ingress and should be handled separately 3.5 Ensure error logs are sent to a remote syslog server (Not Scored) OBSOLETE See previous answer 3.6 Ensure access logs are sent to a remote syslog server (Not Scored) OBSOLETE See previous answer 3.7 Ensure proxies pass source IP information (Scored) OK Headers are set by default 4 Encryption 4.1 TLS / SSL Configuration 4.1.1 Ensure HTTP is redirected to HTTPS (Scored) OK Redirect to TLS is default 4.1.2 Ensure a trusted certificate and trust chain is installed (Not Scored) ACTION NEEDED For installing certs there are enough manuals in the web. A good way is to use lets encrypt through cert-manager Install proper certificates or use lets encrypt with cert-manager 4.1.3 Ensure private key permissions are restricted (Scored) ACTION NEEDED See previous answer 4.1.4 Ensure only modern TLS protocols are used (Scored) OK/ACTION NEEDED Default is TLS 1.2 + 1.3, while this is okay for CIS Benchmark, cipherlist.eu only recommends 1.3. This may cut off old OS's Set controller.config.ssl-protocols to \"TLSv1.3\" 4.1.5 Disable weak ciphers (Scored) ACTION NEEDED Default ciphers are already good, but cipherlist.eu recommends even stronger ciphers Set controller.config.ssl-ciphers to \"EECDH+AESGCM:EDH+AESGCM\" 4.1.6 Ensure custom Diffie-Hellman parameters are used (Scored) ACTION NEEDED No custom DH parameters are generated Generate dh parameters for each ingress deployment you use - see here for a how to 4.1.7 Ensure Online Certificate Status Protocol (OCSP) stapling is enabled (Scored) ACTION NEEDED Not enabled set via this configuration parameter 4.1.8 Ensure HTTP Strict Transport Security (HSTS) is enabled (Scored) OK HSTS is enabled by default 4.1.9 Ensure HTTP Public Key Pinning is enabled (Not Scored) ACTION NEEDED / RISK TO BE ACCEPTED HKPK not enabled by default If lets encrypt is not used, set correct HPKP header. There are several ways to implement this - with the helm charts it works via controller.add-headers. If lets encrypt is used, this is complicated, a solution here is yet unknown 4.1.10 Ensure upstream server traffic is authenticated with a client certificate (Scored) DEPENDS ON BACKEND Highly dependent on backends, not every backend allows configuring this, can also be mitigated via a service mesh If backend allows it, manual is here 4.1.11 Ensure the upstream traffic server certificate is trusted (Not Scored) DEPENDS ON BACKEND Highly dependent on backends, not every backend allows configuring this, can also be mitigated via a service mesh If backend allows it, see configuration here 4.1.12 Ensure your domain is preloaded (Not Scored) ACTION NEEDED Preload is not active by default Set controller.config.hsts-preload to true 4.1.13 Ensure session resumption is disabled to enable perfect forward security (Scored) OK Session tickets are disabled by default 4.1.14 Ensure HTTP/2.0 is used (Not Scored) OK http2 is set by default 5 Request Filtering and Restrictions 5.1 Access Control 5.1.1 Ensure allow and deny filters limit access to specific IP addresses (Not Scored) OK/ACTION NEEDED Depends on use case, geo ip module is compiled into nginx ingress controller, there are several ways to use it If needed set IP restrictions via annotations or work with config snippets (be careful with lets-encrypt-http-challenge!) 5.1.2 Ensure only whitelisted HTTP methods are allowed (Not Scored) OK/ACTION NEEDED Depends on use case If required it can be set via config snippet 5.2 Request Limits 5.2.1 Ensure timeout values for reading the client header and body are set correctly (Scored) ACTION NEEDED Default timeout is 60s Set via this configuration parameter and respective body equivalent 5.2.2 Ensure the maximum request body size is set correctly (Scored) ACTION NEEDED Default is 1m set via this configuration parameter 5.2.3 Ensure the maximum buffer size for URIs is defined (Scored) ACTION NEEDED Default is 4 8k Set via this configuration parameter 5.2.4 Ensure the number of connections per IP address is limited (Not Scored) OK/ACTION NEEDED No limit set Depends on use case, limit can be set via these annotations 5.2.5 Ensure rate limits by IP address are set (Not Scored) OK/ACTION NEEDED No limit set Depends on use case, limit can be set via these annotations 5.3 Browser Security 5.3.1 Ensure X-Frame-Options header is configured and enabled (Scored) ACTION NEEDED Header not set by default Several ways to implement this - with the helm charts it works via controller.add-headers 5.3.2 Ensure X-Content-Type-Options header is configured and enabled (Scored) ACTION NEEDED See previous answer See previous answer 5.3.3 Ensure the X-XSS-Protection Header is enabled and configured properly (Scored) ACTION NEEDED See previous answer See previous answer 5.3.4 Ensure that Content Security Policy (CSP) is enabled and configured properly (Not Scored) ACTION NEEDED See previous answer See previous answer 5.3.5 Ensure the Referrer Policy is enabled and configured properly (Not Scored) ACTION NEEDED Depends on application. It should be handled in the applications webserver itself, not in the load balancing ingress check backend webserver 6 Mandatory Access Control n/a too high level, depends on backends @media only screen and (min-width: 768px) { td:nth-child(1){ white-space:normal !important; } .md-typeset table:not([class]) td { padding: .2rem .3rem; } }","title":"Hardening guide"},{"location":"deploy/hardening-guide/#hardening-guide","text":"","title":"Hardening Guide"},{"location":"deploy/hardening-guide/#overview","text":"There are several ways to do hardening and securing of nginx. In this documentation two guides are used, the guides are overlapping in some points: nginx CIS Benchmark cipherlist.eu (one of many forks of the now dead project cipherli.st) This guide describes, what of the different configurations described in those guides is already implemented as default in the nginx implementation of kubernetes ingress, what needs to be configured, what is obsolete due to the fact that the nginx is running as container (the CIS benchmark relates to a non-containerized installation) and what is difficult or not possible. Be aware that this is only a guide and you are responsible for your own implementation. Some of the configurations may lead to have specific clients unable to reach your site or similar consequences. This guide refers to chapters in the CIS Benchmark. For full explanation you should refer to the benchmark document itself","title":"Overview"},{"location":"deploy/hardening-guide/#configuration-guide","text":"Chapter in CIS benchmark Status Default Action to do if not default 1 Initial Setup 1.1 Installation 1.1.1 Ensure NGINX is installed (Scored) OK done through helm charts / following documentation to deploy nginx ingress 1.1.2 Ensure NGINX is installed from source (Not Scored) OK done through helm charts / following documentation to deploy nginx ingress 1.2 Configure Software Updates 1.2.1 Ensure package manager repositories are properly configured (Not Scored) OK done via helm, nginx version could be overwritten, however compatibility is not ensured then 1.2.2 Ensure the latest software package is installed (Not Scored) ACTION NEEDED done via helm, nginx version could be overwritten, however compatibility is not ensured then Plan for periodic updates 2 Basic Configuration 2.1 Minimize NGINX Modules 2.1.1 Ensure only required modules are installed (Not Scored) OK Already only needed modules are installed, however proposals for further reduction are welcome 2.1.2 Ensure HTTP WebDAV module is not installed (Scored) OK 2.1.3 Ensure modules with gzip functionality are disabled (Scored) OK 2.1.4 Ensure the autoindex module is disabled (Scored) OK No autoindex configs so far in ingress defaults 2.2 Account Security 2.2.1 Ensure that NGINX is run using a non-privileged, dedicated service account (Not Scored) OK Pod configured as user www-data: See this line in helm chart values . Compiled with user www-data: See this line in build script 2.2.2 Ensure the NGINX service account is locked (Scored) OK Docker design ensures this 2.2.3 Ensure the NGINX service account has an invalid shell (Scored) OK Shell is nologin: see this line in build script 2.3 Permissions and Ownership 2.3.1 Ensure NGINX directories and files are owned by root (Scored) OK Obsolete through docker-design and ingress controller needs to update the configs dynamically 2.3.2 Ensure access to NGINX directories and files is restricted (Scored) OK See previous answer 2.3.3 Ensure the NGINX process ID (PID) file is secured (Scored) OK No PID-File due to docker design 2.3.4 Ensure the core dump directory is secured (Not Scored) OK No working_directory configured by default 2.4 Network Configuration 2.4.1 Ensure NGINX only listens for network connections on authorized ports (Not Scored) OK Ensured by automatic nginx.conf configuration 2.4.2 Ensure requests for unknown host names are rejected (Not Scored) OK They are not rejected but send to the \"default backend\" delivering appropriate errors (mostly 404) 2.4.3 Ensure keepalive_timeout is 10 seconds or less, but not 0 (Scored) ACTION NEEDED Default is 75s configure keep-alive to 10 seconds according to this documentation 2.4.4 Ensure send_timeout is set to 10 seconds or less, but not 0 (Scored) RISK TO BE ACCEPTED Not configured, however the nginx default is 60s Not configurable 2.5 Information Disclosure 2.5.1 Ensure server_tokens directive is set to off (Scored) OK server_tokens is configured to off by default 2.5.2 Ensure default error and index.html pages do not reference NGINX (Scored) ACTION NEEDED 404 shows no version at all, 503 and 403 show \"nginx\", which is hardcoded see this line in nginx source code configure custom error pages at least for 403, 404 and 503 and 500 2.5.3 Ensure hidden file serving is disabled (Not Scored) ACTION NEEDED config not set configure a config.server-snippet Snippet, but beware of .well-known challenges or similar. Refer to the benchmark here please 2.5.4 Ensure the NGINX reverse proxy does not enable information disclosure (Scored) ACTION NEEDED hide not configured configure hide-headers with array of \"X-Powered-By\" and \"Server\": according to this documentation 3 Logging 3.1 Ensure detailed logging is enabled (Not Scored) OK nginx ingress has a very detailed log format by default 3.2 Ensure access logging is enabled (Scored) OK Access log is enabled by default 3.3 Ensure error logging is enabled and set to the info logging level (Scored) OK Error log is configured by default. The log level does not matter, because it is all sent to STDOUT anyway 3.4 Ensure log files are rotated (Scored) OBSOLETE Log file handling is not part of the nginx ingress and should be handled separately 3.5 Ensure error logs are sent to a remote syslog server (Not Scored) OBSOLETE See previous answer 3.6 Ensure access logs are sent to a remote syslog server (Not Scored) OBSOLETE See previous answer 3.7 Ensure proxies pass source IP information (Scored) OK Headers are set by default 4 Encryption 4.1 TLS / SSL Configuration 4.1.1 Ensure HTTP is redirected to HTTPS (Scored) OK Redirect to TLS is default 4.1.2 Ensure a trusted certificate and trust chain is installed (Not Scored) ACTION NEEDED For installing certs there are enough manuals in the web. A good way is to use lets encrypt through cert-manager Install proper certificates or use lets encrypt with cert-manager 4.1.3 Ensure private key permissions are restricted (Scored) ACTION NEEDED See previous answer 4.1.4 Ensure only modern TLS protocols are used (Scored) OK/ACTION NEEDED Default is TLS 1.2 + 1.3, while this is okay for CIS Benchmark, cipherlist.eu only recommends 1.3. This may cut off old OS's Set controller.config.ssl-protocols to \"TLSv1.3\" 4.1.5 Disable weak ciphers (Scored) ACTION NEEDED Default ciphers are already good, but cipherlist.eu recommends even stronger ciphers Set controller.config.ssl-ciphers to \"EECDH+AESGCM:EDH+AESGCM\" 4.1.6 Ensure custom Diffie-Hellman parameters are used (Scored) ACTION NEEDED No custom DH parameters are generated Generate dh parameters for each ingress deployment you use - see here for a how to 4.1.7 Ensure Online Certificate Status Protocol (OCSP) stapling is enabled (Scored) ACTION NEEDED Not enabled set via this configuration parameter 4.1.8 Ensure HTTP Strict Transport Security (HSTS) is enabled (Scored) OK HSTS is enabled by default 4.1.9 Ensure HTTP Public Key Pinning is enabled (Not Scored) ACTION NEEDED / RISK TO BE ACCEPTED HKPK not enabled by default If lets encrypt is not used, set correct HPKP header. There are several ways to implement this - with the helm charts it works via controller.add-headers. If lets encrypt is used, this is complicated, a solution here is yet unknown 4.1.10 Ensure upstream server traffic is authenticated with a client certificate (Scored) DEPENDS ON BACKEND Highly dependent on backends, not every backend allows configuring this, can also be mitigated via a service mesh If backend allows it, manual is here 4.1.11 Ensure the upstream traffic server certificate is trusted (Not Scored) DEPENDS ON BACKEND Highly dependent on backends, not every backend allows configuring this, can also be mitigated via a service mesh If backend allows it, see configuration here 4.1.12 Ensure your domain is preloaded (Not Scored) ACTION NEEDED Preload is not active by default Set controller.config.hsts-preload to true 4.1.13 Ensure session resumption is disabled to enable perfect forward security (Scored) OK Session tickets are disabled by default 4.1.14 Ensure HTTP/2.0 is used (Not Scored) OK http2 is set by default 5 Request Filtering and Restrictions 5.1 Access Control 5.1.1 Ensure allow and deny filters limit access to specific IP addresses (Not Scored) OK/ACTION NEEDED Depends on use case, geo ip module is compiled into nginx ingress controller, there are several ways to use it If needed set IP restrictions via annotations or work with config snippets (be careful with lets-encrypt-http-challenge!) 5.1.2 Ensure only whitelisted HTTP methods are allowed (Not Scored) OK/ACTION NEEDED Depends on use case If required it can be set via config snippet 5.2 Request Limits 5.2.1 Ensure timeout values for reading the client header and body are set correctly (Scored) ACTION NEEDED Default timeout is 60s Set via this configuration parameter and respective body equivalent 5.2.2 Ensure the maximum request body size is set correctly (Scored) ACTION NEEDED Default is 1m set via this configuration parameter 5.2.3 Ensure the maximum buffer size for URIs is defined (Scored) ACTION NEEDED Default is 4 8k Set via this configuration parameter 5.2.4 Ensure the number of connections per IP address is limited (Not Scored) OK/ACTION NEEDED No limit set Depends on use case, limit can be set via these annotations 5.2.5 Ensure rate limits by IP address are set (Not Scored) OK/ACTION NEEDED No limit set Depends on use case, limit can be set via these annotations 5.3 Browser Security 5.3.1 Ensure X-Frame-Options header is configured and enabled (Scored) ACTION NEEDED Header not set by default Several ways to implement this - with the helm charts it works via controller.add-headers 5.3.2 Ensure X-Content-Type-Options header is configured and enabled (Scored) ACTION NEEDED See previous answer See previous answer 5.3.3 Ensure the X-XSS-Protection Header is enabled and configured properly (Scored) ACTION NEEDED See previous answer See previous answer 5.3.4 Ensure that Content Security Policy (CSP) is enabled and configured properly (Not Scored) ACTION NEEDED See previous answer See previous answer 5.3.5 Ensure the Referrer Policy is enabled and configured properly (Not Scored) ACTION NEEDED Depends on application. It should be handled in the applications webserver itself, not in the load balancing ingress check backend webserver 6 Mandatory Access Control n/a too high level, depends on backends @media only screen and (min-width: 768px) { td:nth-child(1){ white-space:normal !important; } .md-typeset table:not([class]) td { padding: .2rem .3rem; } }","title":"Configuration Guide"},{"location":"deploy/rbac/","text":"Role Based Access Control (RBAC) \u00b6 Overview \u00b6 This example applies to ingress-nginx-controllers being deployed in an environment with RBAC enabled. Role Based Access Control is comprised of four layers: ClusterRole - permissions assigned to a role that apply to an entire cluster ClusterRoleBinding - binding a ClusterRole to a specific account Role - permissions assigned to a role that apply to a specific namespace RoleBinding - binding a Role to a specific account In order for RBAC to be applied to an ingress-nginx-controller, that controller should be assigned to a ServiceAccount . That ServiceAccount should be bound to the Role s and ClusterRole s defined for the ingress-nginx-controller. Service Accounts created in this example \u00b6 One ServiceAccount is created in this example, ingress-nginx . Permissions Granted in this example \u00b6 There are two sets of permissions defined in this example. Cluster-wide permissions defined by the ClusterRole named ingress-nginx , and namespace specific permissions defined by the Role named ingress-nginx . Cluster Permissions \u00b6 These permissions are granted in order for the ingress-nginx-controller to be able to function as an ingress across the cluster. These permissions are granted to the ClusterRole named ingress-nginx configmaps , endpoints , nodes , pods , secrets : list, watch nodes : get services , ingresses : get, list, watch events : create, patch ingresses/status : update Namespace Permissions \u00b6 These permissions are granted specific to the ingress-nginx namespace. These permissions are granted to the Role named ingress-nginx configmaps , pods , secrets : get endpoints : get Furthermore to support leader-election, the ingress-nginx-controller needs to have access to a configmap using the resourceName ingress-controller-leader-nginx Note that resourceNames can NOT be used to limit requests using the \u201ccreate\u201d verb because authorizers only have access to information that can be obtained from the request URL, method, and headers (resource names in a \u201ccreate\u201d request are part of the request body). configmaps : get, update (for resourceName ingress-controller-leader-nginx ) configmaps : create This resourceName is the concatenation of the election-id and the ingress-class as defined by the ingress-controller, which defaults to: election-id : ingress-controller-leader ingress-class : nginx resourceName : <election-id>-<ingress-class> Please adapt accordingly if you overwrite either parameter when launching the ingress-nginx-controller. Bindings \u00b6 The ServiceAccount ingress-nginx is bound to the Role ingress-nginx and the ClusterRole ingress-nginx . The serviceAccountName associated with the containers in the deployment must match the serviceAccount. The namespace references in the Deployment metadata, container arguments, and POD_NAMESPACE should be in the ingress-nginx namespace.","title":"Role Based Access Control (RBAC)"},{"location":"deploy/rbac/#role-based-access-control-rbac","text":"","title":"Role Based Access Control (RBAC)"},{"location":"deploy/rbac/#overview","text":"This example applies to ingress-nginx-controllers being deployed in an environment with RBAC enabled. Role Based Access Control is comprised of four layers: ClusterRole - permissions assigned to a role that apply to an entire cluster ClusterRoleBinding - binding a ClusterRole to a specific account Role - permissions assigned to a role that apply to a specific namespace RoleBinding - binding a Role to a specific account In order for RBAC to be applied to an ingress-nginx-controller, that controller should be assigned to a ServiceAccount . That ServiceAccount should be bound to the Role s and ClusterRole s defined for the ingress-nginx-controller.","title":"Overview"},{"location":"deploy/rbac/#service-accounts-created-in-this-example","text":"One ServiceAccount is created in this example, ingress-nginx .","title":"Service Accounts created in this example"},{"location":"deploy/rbac/#permissions-granted-in-this-example","text":"There are two sets of permissions defined in this example. Cluster-wide permissions defined by the ClusterRole named ingress-nginx , and namespace specific permissions defined by the Role named ingress-nginx .","title":"Permissions Granted in this example"},{"location":"deploy/rbac/#cluster-permissions","text":"These permissions are granted in order for the ingress-nginx-controller to be able to function as an ingress across the cluster. These permissions are granted to the ClusterRole named ingress-nginx configmaps , endpoints , nodes , pods , secrets : list, watch nodes : get services , ingresses : get, list, watch events : create, patch ingresses/status : update","title":"Cluster Permissions"},{"location":"deploy/rbac/#namespace-permissions","text":"These permissions are granted specific to the ingress-nginx namespace. These permissions are granted to the Role named ingress-nginx configmaps , pods , secrets : get endpoints : get Furthermore to support leader-election, the ingress-nginx-controller needs to have access to a configmap using the resourceName ingress-controller-leader-nginx Note that resourceNames can NOT be used to limit requests using the \u201ccreate\u201d verb because authorizers only have access to information that can be obtained from the request URL, method, and headers (resource names in a \u201ccreate\u201d request are part of the request body). configmaps : get, update (for resourceName ingress-controller-leader-nginx ) configmaps : create This resourceName is the concatenation of the election-id and the ingress-class as defined by the ingress-controller, which defaults to: election-id : ingress-controller-leader ingress-class : nginx resourceName : <election-id>-<ingress-class> Please adapt accordingly if you overwrite either parameter when launching the ingress-nginx-controller.","title":"Namespace Permissions"},{"location":"deploy/rbac/#bindings","text":"The ServiceAccount ingress-nginx is bound to the Role ingress-nginx and the ClusterRole ingress-nginx . The serviceAccountName associated with the containers in the deployment must match the serviceAccount. The namespace references in the Deployment metadata, container arguments, and POD_NAMESPACE should be in the ingress-nginx namespace.","title":"Bindings"},{"location":"deploy/upgrade/","text":"Upgrading \u00b6 Important No matter the method you use for upgrading, if you use template overrides, make sure your templates are compatible with the new version of ingress-nginx . Without Helm \u00b6 To upgrade your ingress-nginx installation, it should be enough to change the version of the image in the controller Deployment. I.e. if your deployment resource looks like (partial example): kind : Deployment metadata : name : ingress-nginx-controller namespace : ingress-nginx spec : replicas : 1 selector : ... template : metadata : ... spec : containers : - name : ingress-nginx-controller image : registry.k8s.io/ingress-nginx/controller:v1.0.4@sha256:545cff00370f28363dad31e3b59a94ba377854d3a11f18988f5f9e56841ef9ef args : ... simply change the v1.0.4 tag to the version you wish to upgrade to. The easiest way to do this is e.g. (do note you may need to change the name parameter according to your installation): kubectl set image deployment/ingress-nginx-controller \\ controller=registry.k8s.io/ingress-nginx/controller:v1.0.5@sha256:55a1fcda5b7657c372515fe402c3e39ad93aa59f6e4378e82acd99912fe6028d \\ -n ingress-nginx For interactive editing, use kubectl edit deployment ingress-nginx-controller -n ingress-nginx . With Helm \u00b6 If you installed ingress-nginx using the Helm command in the deployment docs so its name is ingress-nginx , you should be able to upgrade using helm upgrade --reuse-values ingress-nginx ingress-nginx/ingress-nginx Migrating from stable/nginx-ingress \u00b6 See detailed steps in the upgrading section of the ingress-nginx chart README .","title":"Upgrade"},{"location":"deploy/upgrade/#upgrading","text":"Important No matter the method you use for upgrading, if you use template overrides, make sure your templates are compatible with the new version of ingress-nginx .","title":"Upgrading"},{"location":"deploy/upgrade/#without-helm","text":"To upgrade your ingress-nginx installation, it should be enough to change the version of the image in the controller Deployment. I.e. if your deployment resource looks like (partial example): kind : Deployment metadata : name : ingress-nginx-controller namespace : ingress-nginx spec : replicas : 1 selector : ... template : metadata : ... spec : containers : - name : ingress-nginx-controller image : registry.k8s.io/ingress-nginx/controller:v1.0.4@sha256:545cff00370f28363dad31e3b59a94ba377854d3a11f18988f5f9e56841ef9ef args : ... simply change the v1.0.4 tag to the version you wish to upgrade to. The easiest way to do this is e.g. (do note you may need to change the name parameter according to your installation): kubectl set image deployment/ingress-nginx-controller \\ controller=registry.k8s.io/ingress-nginx/controller:v1.0.5@sha256:55a1fcda5b7657c372515fe402c3e39ad93aa59f6e4378e82acd99912fe6028d \\ -n ingress-nginx For interactive editing, use kubectl edit deployment ingress-nginx-controller -n ingress-nginx .","title":"Without Helm"},{"location":"deploy/upgrade/#with-helm","text":"If you installed ingress-nginx using the Helm command in the deployment docs so its name is ingress-nginx , you should be able to upgrade using helm upgrade --reuse-values ingress-nginx ingress-nginx/ingress-nginx","title":"With Helm"},{"location":"deploy/upgrade/#migrating-from-stablenginx-ingress","text":"See detailed steps in the upgrading section of the ingress-nginx chart README .","title":"Migrating from stable/nginx-ingress"},{"location":"developer-guide/code-overview/","text":"Ingress NGINX - Code Overview \u00b6 This document provides an overview of Ingress NGINX code. Core Golang code \u00b6 This part of the code is responsible for the main logic of Ingress NGINX. It contains all the logics that parses Ingress Objects , annotations , watches Endpoints and turn them into usable nginx.conf configuration. Core Sync Logics: \u00b6 Ingress-nginx has an internal model of the ingresses, secrets and endpoints in a given cluster. It maintains two copies of that: One copy is the currently running configuration model Second copy is the one generated in response to some changes in the cluster The sync logic diffs the two models and if there's a change it tries to converge the running configuration to the new one. There are static and dynamic configuration changes. All endpoints and certificate changes are handled dynamically by posting the payload to an internal NGINX endpoint that is handled by Lua. The following parts of the code can be found: Entrypoint \u00b6 The main package is responsible for starting ingress-nginx program, which can be found in cmd/nginx directory. Version \u00b6 Is the package of the code responsible for adding version subcommand, and can be found in version directory. Internal code \u00b6 This part of the code contains the internal logics that compose Ingress NGINX Controller, and it's split into: Admission Controller \u00b6 Contains the code of Kubernetes Admission Controller which validates the syntax of ingress objects before accepting it. This code can be found in internal/admission/controller directory. File functions \u00b6 Contains auxiliary codes that deal with files, such as generating the SHA1 checksum of a file, or creating required directories. This code can be found in internal/file directory. Ingress functions \u00b6 Contains all the logics from NGINX Ingress Controller, with some examples being: Expected Golang structures that will be used in templates and other parts of the code - internal/ingress/types.go . supported annotations and its parsing logics - internal/ingress/annotations . reconciliation loops and logics - internal/ingress/controller defaults - define the default struct - internal/ingress/defaults . Error interface and types implementation - internal/ingress/errors Metrics collectors for Prometheus exporting - internal/ingress/metric . Resolver - Extracts information from a controller - internal/ingress/resolver . Ingress Object status publisher - internal/ingress/status . And other parts of the code that will be written in this document in a future. K8s functions \u00b6 Contains helper functions for parsing Kubernetes objects. This part of the code can be found in internal/k8s directory. Networking functions \u00b6 Contains helper functions for networking, such as IPv4 and IPv6 parsing, SSL certificate parsing, etc. This part of the code can be found in internal/net directory. NGINX functions \u00b6 Contains helper function to deal with NGINX, such as verify if it's running and reading it's configuration file parts. This part of the code can be found in internal/nginx directory. Tasks / Queue \u00b6 Contains the functions responsible for the sync queue part of the controller. This part of the code can be found in internal/task directory. Other parts of internal \u00b6 Other parts of internal code might not be covered here, like runtime and watch but they can be added in a future. E2E Test \u00b6 The e2e tests code is in test directory. Other programs \u00b6 Describe here kubectl plugin , dbg , waitshutdown and cover the hack scripts. kubectl plugin \u00b6 It containes kubectl plugin for inspecting your ingress-nginx deployments. This part of code can be found in cmd/plugin directory Detail functions flow and available flow can be found in kubectl-plugin Deploy files \u00b6 This directory contains the yaml deploy files used as examples or references in the docs to deploy Ingress NGINX and other components. Those files are in deploy directory. Helm Chart \u00b6 Used to generate the Helm chart published. Code is in charts/ingress-nginx . Documentation/Website \u00b6 The documentation used to generate the website https://kubernetes.github.io/ingress-nginx/ This code is available in docs and it's main \"language\" is Markdown , used by mkdocs file to generate static pages. Container Images \u00b6 Container images used to run ingress-nginx, or to build the final image. Base Images \u00b6 Contains the Dockerfiles and scripts used to build base images that are used in other parts of the repo. They are present in images repo. Some examples: * nginx - The base NGINX image ingress-nginx uses is not a vanilla NGINX. It bundles many libraries together and it is a job in itself to maintain that and keep things up-to-date. * custom-error-pages - Used on the custom error page examples. There are other images inside this directory. Ingress Controller Image \u00b6 The image used to build the final ingress controller, used in deploy scripts and Helm charts. This is NGINX with some Lua enhancement. We do dynamic certificate, endpoints handling, canary traffic split, custom load balancing etc at this component. One can also add new functionalities using Lua plugin system. The files are in rootfs directory and contains: The Dockerfile nginx config Ingress NGINX Lua Scripts \u00b6 Ingress NGINX uses Lua Scripts to enable features like hot reloading, rate limiting and monitoring. Some are written using the OpenResty helper. The directory containing Lua scripts is rootfs/etc/nginx/lua . Nginx Go template file \u00b6 One of the functions of Ingress NGINX is to turn Ingress objects into nginx.conf file. To do so, the final step is to apply those configurations in nginx.tmpl turning it into a final nginx.conf file.","title":"Code Overview"},{"location":"developer-guide/code-overview/#ingress-nginx-code-overview","text":"This document provides an overview of Ingress NGINX code.","title":"Ingress NGINX - Code Overview"},{"location":"developer-guide/code-overview/#core-golang-code","text":"This part of the code is responsible for the main logic of Ingress NGINX. It contains all the logics that parses Ingress Objects , annotations , watches Endpoints and turn them into usable nginx.conf configuration.","title":"Core Golang code"},{"location":"developer-guide/code-overview/#core-sync-logics","text":"Ingress-nginx has an internal model of the ingresses, secrets and endpoints in a given cluster. It maintains two copies of that: One copy is the currently running configuration model Second copy is the one generated in response to some changes in the cluster The sync logic diffs the two models and if there's a change it tries to converge the running configuration to the new one. There are static and dynamic configuration changes. All endpoints and certificate changes are handled dynamically by posting the payload to an internal NGINX endpoint that is handled by Lua. The following parts of the code can be found:","title":"Core Sync Logics:"},{"location":"developer-guide/code-overview/#entrypoint","text":"The main package is responsible for starting ingress-nginx program, which can be found in cmd/nginx directory.","title":"Entrypoint"},{"location":"developer-guide/code-overview/#version","text":"Is the package of the code responsible for adding version subcommand, and can be found in version directory.","title":"Version"},{"location":"developer-guide/code-overview/#internal-code","text":"This part of the code contains the internal logics that compose Ingress NGINX Controller, and it's split into:","title":"Internal code"},{"location":"developer-guide/code-overview/#admission-controller","text":"Contains the code of Kubernetes Admission Controller which validates the syntax of ingress objects before accepting it. This code can be found in internal/admission/controller directory.","title":"Admission Controller"},{"location":"developer-guide/code-overview/#file-functions","text":"Contains auxiliary codes that deal with files, such as generating the SHA1 checksum of a file, or creating required directories. This code can be found in internal/file directory.","title":"File functions"},{"location":"developer-guide/code-overview/#ingress-functions","text":"Contains all the logics from NGINX Ingress Controller, with some examples being: Expected Golang structures that will be used in templates and other parts of the code - internal/ingress/types.go . supported annotations and its parsing logics - internal/ingress/annotations . reconciliation loops and logics - internal/ingress/controller defaults - define the default struct - internal/ingress/defaults . Error interface and types implementation - internal/ingress/errors Metrics collectors for Prometheus exporting - internal/ingress/metric . Resolver - Extracts information from a controller - internal/ingress/resolver . Ingress Object status publisher - internal/ingress/status . And other parts of the code that will be written in this document in a future.","title":"Ingress functions"},{"location":"developer-guide/code-overview/#k8s-functions","text":"Contains helper functions for parsing Kubernetes objects. This part of the code can be found in internal/k8s directory.","title":"K8s functions"},{"location":"developer-guide/code-overview/#networking-functions","text":"Contains helper functions for networking, such as IPv4 and IPv6 parsing, SSL certificate parsing, etc. This part of the code can be found in internal/net directory.","title":"Networking functions"},{"location":"developer-guide/code-overview/#nginx-functions","text":"Contains helper function to deal with NGINX, such as verify if it's running and reading it's configuration file parts. This part of the code can be found in internal/nginx directory.","title":"NGINX functions"},{"location":"developer-guide/code-overview/#tasks-queue","text":"Contains the functions responsible for the sync queue part of the controller. This part of the code can be found in internal/task directory.","title":"Tasks / Queue"},{"location":"developer-guide/code-overview/#other-parts-of-internal","text":"Other parts of internal code might not be covered here, like runtime and watch but they can be added in a future.","title":"Other parts of internal"},{"location":"developer-guide/code-overview/#e2e-test","text":"The e2e tests code is in test directory.","title":"E2E Test"},{"location":"developer-guide/code-overview/#other-programs","text":"Describe here kubectl plugin , dbg , waitshutdown and cover the hack scripts.","title":"Other programs"},{"location":"developer-guide/code-overview/#kubectl-plugin","text":"It containes kubectl plugin for inspecting your ingress-nginx deployments. This part of code can be found in cmd/plugin directory Detail functions flow and available flow can be found in kubectl-plugin","title":"kubectl plugin"},{"location":"developer-guide/code-overview/#deploy-files","text":"This directory contains the yaml deploy files used as examples or references in the docs to deploy Ingress NGINX and other components. Those files are in deploy directory.","title":"Deploy files"},{"location":"developer-guide/code-overview/#helm-chart","text":"Used to generate the Helm chart published. Code is in charts/ingress-nginx .","title":"Helm Chart"},{"location":"developer-guide/code-overview/#documentationwebsite","text":"The documentation used to generate the website https://kubernetes.github.io/ingress-nginx/ This code is available in docs and it's main \"language\" is Markdown , used by mkdocs file to generate static pages.","title":"Documentation/Website"},{"location":"developer-guide/code-overview/#container-images","text":"Container images used to run ingress-nginx, or to build the final image.","title":"Container Images"},{"location":"developer-guide/code-overview/#base-images","text":"Contains the Dockerfiles and scripts used to build base images that are used in other parts of the repo. They are present in images repo. Some examples: * nginx - The base NGINX image ingress-nginx uses is not a vanilla NGINX. It bundles many libraries together and it is a job in itself to maintain that and keep things up-to-date. * custom-error-pages - Used on the custom error page examples. There are other images inside this directory.","title":"Base Images"},{"location":"developer-guide/code-overview/#ingress-controller-image","text":"The image used to build the final ingress controller, used in deploy scripts and Helm charts. This is NGINX with some Lua enhancement. We do dynamic certificate, endpoints handling, canary traffic split, custom load balancing etc at this component. One can also add new functionalities using Lua plugin system. The files are in rootfs directory and contains: The Dockerfile nginx config","title":"Ingress Controller Image"},{"location":"developer-guide/code-overview/#ingress-nginx-lua-scripts","text":"Ingress NGINX uses Lua Scripts to enable features like hot reloading, rate limiting and monitoring. Some are written using the OpenResty helper. The directory containing Lua scripts is rootfs/etc/nginx/lua .","title":"Ingress NGINX Lua Scripts"},{"location":"developer-guide/code-overview/#nginx-go-template-file","text":"One of the functions of Ingress NGINX is to turn Ingress objects into nginx.conf file. To do so, the final step is to apply those configurations in nginx.tmpl turning it into a final nginx.conf file.","title":"Nginx Go template file"},{"location":"developer-guide/getting-started/","text":"Developing for NGINX Ingress Controller This document explains how to get started with developing for NGINX Ingress controller. For the really new contributors, who want to contribute to the INGRESS-NGINX project, but need help with understanding some basic concepts, that are needed to work with the Kubernetes ingress resource, here is a link to the New Contributors Guide . This guide contains tips on how a http/https request travels, from a browser or a curl command, to the webserver process running inside a container, in a pod, in a Kubernetes cluster, but enters the cluster via a ingress resource. For those who are familiar with those basic networking concepts like routing of a packet with regards to a http request, termination of connection, reverseproxy etc. etc., you can skip this and move on to the sections below. (or read it anyways just for context and also provide feedbacks if any) Prerequisites \u00b6 Install Go 1.14 or later. Note The project uses Go Modules Install Docker (v19.03.0 or later with experimental feature on) Important The majority of make tasks run as docker containers Quick Start \u00b6 Fork the repository Clone the repository to any location in your work station Add a GO111MODULE environment variable with export GO111MODULE=on Run go mod download to install dependencies Local build \u00b6 Start a local Kubernetes cluster using kind , build and deploy the ingress controller make dev-env - If you are working on the v1.x.x version of this controller, and you want to create a cluster with kubernetes version 1.22, then please visit the documentation for kind , and look for how to set a custom image for the kind node (image: kindest/node...), in the kind config file. Testing \u00b6 Run go unit tests make test Run unit-tests for lua code make lua-test Lua tests are located in the directory rootfs/etc/nginx/lua/test Important Test files must follow the naming convention <mytest>_test.lua or it will be ignored Run e2e test suite make kind-e2e-test To limit the scope of the tests to execute, we can use the environment variable FOCUS FOCUS=\"no-auth-locations\" make kind-e2e-test Note The variable FOCUS defines Ginkgo Focused Specs Valid values are defined in the describe definition of the e2e tests like Default Backend The complete list of tests can be found here Custom docker image \u00b6 In some cases, it can be useful to build a docker image and publish such an image to a private or custom registry location. This can be done setting two environment variables, REGISTRY and TAG export TAG=\"dev\" export REGISTRY=\"$USER\" make build image and then publish such version with docker push $REGISTRY/controller:$TAG","title":"Getting Started"},{"location":"developer-guide/getting-started/#prerequisites","text":"Install Go 1.14 or later. Note The project uses Go Modules Install Docker (v19.03.0 or later with experimental feature on) Important The majority of make tasks run as docker containers","title":"Prerequisites"},{"location":"developer-guide/getting-started/#quick-start","text":"Fork the repository Clone the repository to any location in your work station Add a GO111MODULE environment variable with export GO111MODULE=on Run go mod download to install dependencies","title":"Quick Start"},{"location":"developer-guide/getting-started/#local-build","text":"Start a local Kubernetes cluster using kind , build and deploy the ingress controller make dev-env - If you are working on the v1.x.x version of this controller, and you want to create a cluster with kubernetes version 1.22, then please visit the documentation for kind , and look for how to set a custom image for the kind node (image: kindest/node...), in the kind config file.","title":"Local build"},{"location":"developer-guide/getting-started/#testing","text":"Run go unit tests make test Run unit-tests for lua code make lua-test Lua tests are located in the directory rootfs/etc/nginx/lua/test Important Test files must follow the naming convention <mytest>_test.lua or it will be ignored Run e2e test suite make kind-e2e-test To limit the scope of the tests to execute, we can use the environment variable FOCUS FOCUS=\"no-auth-locations\" make kind-e2e-test Note The variable FOCUS defines Ginkgo Focused Specs Valid values are defined in the describe definition of the e2e tests like Default Backend The complete list of tests can be found here","title":"Testing"},{"location":"developer-guide/getting-started/#custom-docker-image","text":"In some cases, it can be useful to build a docker image and publish such an image to a private or custom registry location. This can be done setting two environment variables, REGISTRY and TAG export TAG=\"dev\" export REGISTRY=\"$USER\" make build image and then publish such version with docker push $REGISTRY/controller:$TAG","title":"Custom docker image"},{"location":"enhancements/","text":"Kubernetes Enhancement Proposals (KEPs) \u00b6 A Kubernetes Enhancement Proposal (KEP) is a way to propose, communicate and coordinate on new efforts for the Kubernetes project. For this reason, the ingress-nginx project is adopting it. Quick start for the KEP process \u00b6 Follow the process outlined in the KEP template Do I have to use the KEP process? \u00b6 No... but we hope that you will. Over time having a rich set of KEPs in one place will make it easier for people to track what is going on in the community and find a structured historic record. KEPs are only required when the changes are wide ranging and impact most of the project. Why would I want to use the KEP process? \u00b6 Our aim with KEPs is to clearly communicate new efforts to the Kubernetes contributor community. As such, we want to build a well curated set of clear proposals in a common format with useful metadata. Benefits to KEP users (in the limit): Exposure on a kubernetes blessed web site that is findable via web search engines. Cross indexing of KEPs so that users can find connections and the current status of any KEP. A clear process with approvers and reviewers for making decisions. This will lead to more structured decisions that stick as there is a discoverable record around the decisions. We are inspired by IETF RFCs, Python PEPs, and Rust RFCs.","title":"Kubernetes Enhancement Proposals (KEPs)"},{"location":"enhancements/#kubernetes-enhancement-proposals-keps","text":"A Kubernetes Enhancement Proposal (KEP) is a way to propose, communicate and coordinate on new efforts for the Kubernetes project. For this reason, the ingress-nginx project is adopting it.","title":"Kubernetes Enhancement Proposals (KEPs)"},{"location":"enhancements/#quick-start-for-the-kep-process","text":"Follow the process outlined in the KEP template","title":"Quick start for the KEP process"},{"location":"enhancements/#do-i-have-to-use-the-kep-process","text":"No... but we hope that you will. Over time having a rich set of KEPs in one place will make it easier for people to track what is going on in the community and find a structured historic record. KEPs are only required when the changes are wide ranging and impact most of the project.","title":"Do I have to use the KEP process?"},{"location":"enhancements/#why-would-i-want-to-use-the-kep-process","text":"Our aim with KEPs is to clearly communicate new efforts to the Kubernetes contributor community. As such, we want to build a well curated set of clear proposals in a common format with useful metadata. Benefits to KEP users (in the limit): Exposure on a kubernetes blessed web site that is findable via web search engines. Cross indexing of KEPs so that users can find connections and the current status of any KEP. A clear process with approvers and reviewers for making decisions. This will lead to more structured decisions that stick as there is a discoverable record around the decisions. We are inspired by IETF RFCs, Python PEPs, and Rust RFCs.","title":"Why would I want to use the KEP process?"},{"location":"enhancements/20190724-only-dynamic-ssl/","text":"Remove static SSL configuration mode \u00b6 Table of Contents \u00b6 Summary Motivation Goals Non-Goals Proposal Implementation Details/Notes/Constraints Drawbacks Alternatives Summary \u00b6 Since release 0.19.0 is possible to configure SSL certificates without the need of NGINX reloads (thanks to lua) and after release 0.24.0 the default enabled mode is dynamic. Motivation \u00b6 The static configuration implies reloads, something that affects the majority of the users. Goals \u00b6 Deprecation of the flag --enable-dynamic-certificates . Cleanup of the codebase. Non-Goals \u00b6 Features related to certificate authentication are not changed in any way. Proposal \u00b6 Remove static SSL configuration Implementation Details/Notes/Constraints \u00b6 Deprecate the flag Move the directives ssl_certificate and ssl_certificate_key from each server block to the http section. These settings are required to avoid NGINX errors in the logs. Remove any action of the flag --enable-dynamic-certificates Drawbacks \u00b6 Alternatives \u00b6 Keep both implementations","title":"Remove static SSL configuration mode"},{"location":"enhancements/20190724-only-dynamic-ssl/#remove-static-ssl-configuration-mode","text":"","title":"Remove static SSL configuration mode"},{"location":"enhancements/20190724-only-dynamic-ssl/#table-of-contents","text":"Summary Motivation Goals Non-Goals Proposal Implementation Details/Notes/Constraints Drawbacks Alternatives","title":"Table of Contents"},{"location":"enhancements/20190724-only-dynamic-ssl/#summary","text":"Since release 0.19.0 is possible to configure SSL certificates without the need of NGINX reloads (thanks to lua) and after release 0.24.0 the default enabled mode is dynamic.","title":"Summary"},{"location":"enhancements/20190724-only-dynamic-ssl/#motivation","text":"The static configuration implies reloads, something that affects the majority of the users.","title":"Motivation"},{"location":"enhancements/20190724-only-dynamic-ssl/#goals","text":"Deprecation of the flag --enable-dynamic-certificates . Cleanup of the codebase.","title":"Goals"},{"location":"enhancements/20190724-only-dynamic-ssl/#non-goals","text":"Features related to certificate authentication are not changed in any way.","title":"Non-Goals"},{"location":"enhancements/20190724-only-dynamic-ssl/#proposal","text":"Remove static SSL configuration","title":"Proposal"},{"location":"enhancements/20190724-only-dynamic-ssl/#implementation-detailsnotesconstraints","text":"Deprecate the flag Move the directives ssl_certificate and ssl_certificate_key from each server block to the http section. These settings are required to avoid NGINX errors in the logs. Remove any action of the flag --enable-dynamic-certificates","title":"Implementation Details/Notes/Constraints"},{"location":"enhancements/20190724-only-dynamic-ssl/#drawbacks","text":"","title":"Drawbacks"},{"location":"enhancements/20190724-only-dynamic-ssl/#alternatives","text":"Keep both implementations","title":"Alternatives"},{"location":"enhancements/20190815-zone-aware-routing/","text":"Availability zone aware routing \u00b6 Table of Contents \u00b6 Availability zone aware routing Table of Contents Summary Motivation Goals Non-Goals Proposal Implementation History Drawbacks [optional] Summary \u00b6 Teach ingress-nginx about availability zones where endpoints are running in. This way ingress-nginx pod will do its best to proxy to zone-local endpoint. Motivation \u00b6 When users run their services across multiple availability zones they usually pay for egress traffic between zones. Providers such as GCP, and Amazon EC2 usually charge extra for this feature. ingress-nginx when picking an endpoint to route request to does not consider whether the endpoint is in a different zone or the same one. That means it's at least equally likely that it will pick an endpoint from another zone and proxy the request to it. In this situation response from the endpoint to the ingress-nginx pod is considered inter-zone traffic and usually costs extra money. At the time of this writing, GCP charges $0.01 per GB of inter-zone egress traffic according to https://cloud.google.com/compute/network-pricing. According to https://datapath.io/resources/blog/what-are-aws-data-transfer-costs-and-how-to-minimize-them/ Amazon also charges the same amount of money as GCP for cross-zone, egress traffic. This can be a lot of money depending on once's traffic. By teaching ingress-nginx about zones we can eliminate or at least decrease this cost. Arguably inter-zone network latency should also be better than cross-zone. Goals \u00b6 Given a regional cluster running ingress-nginx, ingress-nginx should do best-effort to pick a zone-local endpoint when proxying This should not impact canary feature ingress-nginx should be able to operate successfully if there are no zonal endpoints Non-Goals \u00b6 This feature inherently assumes that endpoints are distributed across zones in a way that they can handle all the traffic from ingress-nginx pod(s) in that zone This feature will be relying on https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#failure-domainbetakubernetesiozone, it is not this KEP's goal to support other cases Proposal \u00b6 The idea here is to have the controller part of ingress-nginx (1) detect what zone its current pod is running in and (2) detect the zone for every endpoint it knows about. After that, it will post that data as part of endpoints to Lua land. When picking an endpoint, the Lua balancer will try to pick zone-local endpoint first and if there is no zone-local endpoint then it will fall back to current behavior. Initially, this feature should be optional since it is going to make it harder to reason about the load balancing and not everyone might want that. How does controller know what zone it runs in? We can have the pod spec pass the node name using downward API as an environment variable. Upon startup, the controller can get node details from the API based on the node name. Once the node details are obtained we can extract the zone from the failure-domain.beta.kubernetes.io/zone annotation. Then we can pass that value to Lua land through Nginx configuration when loading lua_ingress.lua module in init_by_lua phase. How do we extract zones for endpoints? We can have the controller watch create and update events on nodes in the entire cluster and based on that keep the map of nodes to zones in the memory. And when we generate endpoints list, we can access node name using .subsets.addresses[i].nodeName and based on that fetch zone from the map in memory and store it as a field on the endpoint. This solution assumes failure-domain.beta.kubernetes.io/zone annotation does not change until the end of the node's life. Otherwise, we have to watch update events as well on the nodes and that'll add even more overhead. Alternatively, we can get the list of nodes only when there's no node in the memory for the given node name. This is probably a better solution because then we would avoid watching for API changes on node resources. We can eagerly fetch all the nodes and build node name to zone mapping on start. From there on, it will sync during endpoint building in the main event loop if there's no existing entry for the node of an endpoint. This means an extra API call in case cluster has expanded. How do we make sure we do our best to choose zone-local endpoint? This will be done on the Lua side. For every backend, we will initialize two balancer instances: (1) with all endpoints (2) with all endpoints corresponding to the current zone for the backend. Then given the request once we choose what backend needs to serve the request, we will first try to use a zonal balancer for that backend. If a zonal balancer does not exist (i.e. there's no zonal endpoint) then we will use a general balancer. In case of zonal outages, we assume that the readiness probe will fail and the controller will see no endpoints for the backend and therefore we will use a general balancer. We can enable the feature using a configmap setting. Doing it this way makes it easier to rollback in case of a problem. Implementation History \u00b6 initial version of KEP is shipped proposal and implementation details are done Drawbacks [optional] \u00b6 More load on the Kubernetes API server.","title":"Availability zone aware routing"},{"location":"enhancements/20190815-zone-aware-routing/#availability-zone-aware-routing","text":"","title":"Availability zone aware routing"},{"location":"enhancements/20190815-zone-aware-routing/#table-of-contents","text":"Availability zone aware routing Table of Contents Summary Motivation Goals Non-Goals Proposal Implementation History Drawbacks [optional]","title":"Table of Contents"},{"location":"enhancements/20190815-zone-aware-routing/#summary","text":"Teach ingress-nginx about availability zones where endpoints are running in. This way ingress-nginx pod will do its best to proxy to zone-local endpoint.","title":"Summary"},{"location":"enhancements/20190815-zone-aware-routing/#motivation","text":"When users run their services across multiple availability zones they usually pay for egress traffic between zones. Providers such as GCP, and Amazon EC2 usually charge extra for this feature. ingress-nginx when picking an endpoint to route request to does not consider whether the endpoint is in a different zone or the same one. That means it's at least equally likely that it will pick an endpoint from another zone and proxy the request to it. In this situation response from the endpoint to the ingress-nginx pod is considered inter-zone traffic and usually costs extra money. At the time of this writing, GCP charges $0.01 per GB of inter-zone egress traffic according to https://cloud.google.com/compute/network-pricing. According to https://datapath.io/resources/blog/what-are-aws-data-transfer-costs-and-how-to-minimize-them/ Amazon also charges the same amount of money as GCP for cross-zone, egress traffic. This can be a lot of money depending on once's traffic. By teaching ingress-nginx about zones we can eliminate or at least decrease this cost. Arguably inter-zone network latency should also be better than cross-zone.","title":"Motivation"},{"location":"enhancements/20190815-zone-aware-routing/#goals","text":"Given a regional cluster running ingress-nginx, ingress-nginx should do best-effort to pick a zone-local endpoint when proxying This should not impact canary feature ingress-nginx should be able to operate successfully if there are no zonal endpoints","title":"Goals"},{"location":"enhancements/20190815-zone-aware-routing/#non-goals","text":"This feature inherently assumes that endpoints are distributed across zones in a way that they can handle all the traffic from ingress-nginx pod(s) in that zone This feature will be relying on https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#failure-domainbetakubernetesiozone, it is not this KEP's goal to support other cases","title":"Non-Goals"},{"location":"enhancements/20190815-zone-aware-routing/#proposal","text":"The idea here is to have the controller part of ingress-nginx (1) detect what zone its current pod is running in and (2) detect the zone for every endpoint it knows about. After that, it will post that data as part of endpoints to Lua land. When picking an endpoint, the Lua balancer will try to pick zone-local endpoint first and if there is no zone-local endpoint then it will fall back to current behavior. Initially, this feature should be optional since it is going to make it harder to reason about the load balancing and not everyone might want that. How does controller know what zone it runs in? We can have the pod spec pass the node name using downward API as an environment variable. Upon startup, the controller can get node details from the API based on the node name. Once the node details are obtained we can extract the zone from the failure-domain.beta.kubernetes.io/zone annotation. Then we can pass that value to Lua land through Nginx configuration when loading lua_ingress.lua module in init_by_lua phase. How do we extract zones for endpoints? We can have the controller watch create and update events on nodes in the entire cluster and based on that keep the map of nodes to zones in the memory. And when we generate endpoints list, we can access node name using .subsets.addresses[i].nodeName and based on that fetch zone from the map in memory and store it as a field on the endpoint. This solution assumes failure-domain.beta.kubernetes.io/zone annotation does not change until the end of the node's life. Otherwise, we have to watch update events as well on the nodes and that'll add even more overhead. Alternatively, we can get the list of nodes only when there's no node in the memory for the given node name. This is probably a better solution because then we would avoid watching for API changes on node resources. We can eagerly fetch all the nodes and build node name to zone mapping on start. From there on, it will sync during endpoint building in the main event loop if there's no existing entry for the node of an endpoint. This means an extra API call in case cluster has expanded. How do we make sure we do our best to choose zone-local endpoint? This will be done on the Lua side. For every backend, we will initialize two balancer instances: (1) with all endpoints (2) with all endpoints corresponding to the current zone for the backend. Then given the request once we choose what backend needs to serve the request, we will first try to use a zonal balancer for that backend. If a zonal balancer does not exist (i.e. there's no zonal endpoint) then we will use a general balancer. In case of zonal outages, we assume that the readiness probe will fail and the controller will see no endpoints for the backend and therefore we will use a general balancer. We can enable the feature using a configmap setting. Doing it this way makes it easier to rollback in case of a problem.","title":"Proposal"},{"location":"enhancements/20190815-zone-aware-routing/#implementation-history","text":"initial version of KEP is shipped proposal and implementation details are done","title":"Implementation History"},{"location":"enhancements/20190815-zone-aware-routing/#drawbacks-optional","text":"More load on the Kubernetes API server.","title":"Drawbacks [optional]"},{"location":"enhancements/YYYYMMDD-kep-template/","text":"Title \u00b6 This is the title of the KEP. Keep it simple and descriptive. A good title can help communicate what the KEP is and should be considered as part of any review. The title should be lowercased and spaces/punctuation should be replaced with - . To get started with this template: Make a copy of this template. Create a copy of this template and name it YYYYMMDD-my-title.md , where YYYYMMDD is the date the KEP was first drafted. Fill out the \"overview\" sections. This includes the Summary and Motivation sections. These should be easy if you've preflighted the idea of the KEP in an issue. Create a PR. Assign it to folks that are sponsoring this process. Create an issue When filing an enhancement tracking issue, please ensure to complete all fields in the template. Merge early. Avoid getting hung up on specific details and instead aim to get the goal of the KEP merged quickly. The best way to do this is to just start with the \"Overview\" sections and fill out details incrementally in follow on PRs. View anything marked as a provisional as a working document and subject to change. Aim for single topic PRs to keep discussions focused. If you disagree with what is already in a document, open a new PR with suggested changes. The canonical place for the latest set of instructions (and the likely source of this file) is here . The Metadata section above is intended to support the creation of tooling around the KEP process. This will be a YAML section that is fenced as a code block. See the KEP process for details on each of these items. Table of Contents \u00b6 A table of contents is helpful for quickly jumping to sections of a KEP and for highlighting any additional information provided beyond the standard KEP template. Ensure the TOC is wrapped with <!-- toc --&rt;<!-- /toc --&rt; tags, and then generate with hack/update-toc.sh . Summary Motivation Goals Non-Goals Proposal User Stories [optional] Story 1 Story 2 Implementation Details/Notes/Constraints [optional] Risks and Mitigations Design Details Test Plan Removing a deprecated flag Implementation History Drawbacks [optional] Alternatives [optional] Summary \u00b6 The Summary section is incredibly important for producing high quality user-focused documentation such as release notes or a development roadmap. It should be possible to collect this information before implementation begins in order to avoid requiring implementers to split their attention between writing release notes and implementing the feature itself. A good summary is probably at least a paragraph in length. Motivation \u00b6 This section is for explicitly listing the motivation, goals and non-goals of this KEP. Describe why the change is important and the benefits to users. The motivation section can optionally provide links to experience reports to demonstrate the interest in a KEP within the wider Kubernetes community. Goals \u00b6 List the specific goals of the KEP. How will we know that this has succeeded? Non-Goals \u00b6 What is out of scope for this KEP? Listing non-goals helps to focus discussion and make progress. Proposal \u00b6 This is where we get down to the nitty gritty of what the proposal actually is. User Stories [optional] \u00b6 Detail the things that people will be able to do if this KEP is implemented. Include as much detail as possible so that people can understand the \"how\" of the system. The goal here is to make this feel real for users without getting bogged down. Story 1 \u00b6 Story 2 \u00b6 Implementation Details/Notes/Constraints [optional] \u00b6 What are the caveats to the implementation? What are some important details that didn't come across above. Go in to as much detail as necessary here. This might be a good place to talk about core concepts and how they relate. Risks and Mitigations \u00b6 What are the risks of this proposal and how do we mitigate. Think broadly. For example, consider both security and how this will impact the larger kubernetes ecosystem. How will security be reviewed and by whom? How will UX be reviewed and by whom? Consider including folks that also work outside project. Design Details \u00b6 Test Plan \u00b6 Note: Section not required until targeted at a release. Consider the following in developing a test plan for this enhancement: Will there be e2e and integration tests, in addition to unit tests? How will it be tested in isolation vs with other components? No need to outline all of the test cases, just the general strategy. Anything that would count as tricky in the implementation and anything particularly challenging to test should be called out. All code is expected to have adequate tests (eventually with coverage expectations). Please adhere to the Kubernetes testing guidelines when drafting this test plan. Removing a deprecated flag \u00b6 Announce deprecation and support policy of the existing flag Two versions passed since introducing the functionality which deprecates the flag (to address version skew) Address feedback on usage/changed behavior, provided on GitHub issues Deprecate the flag Implementation History \u00b6 Major milestones in the life cycle of a KEP should be tracked in Implementation History . Major milestones might include the Summary and Motivation sections being merged signaling acceptance the Proposal section being merged signaling agreement on a proposed design the date implementation started the first Kubernetes release where an initial version of the KEP was available the version of Kubernetes where the KEP graduated to general availability when the KEP was retired or superseded Drawbacks [optional] \u00b6 Why should this KEP not be implemented. Alternatives [optional] \u00b6 Similar to the Drawbacks section the Alternatives section is used to highlight and record other possible approaches to delivering the value proposed by a KEP.","title":"KEP Template"},{"location":"enhancements/YYYYMMDD-kep-template/#title","text":"This is the title of the KEP. Keep it simple and descriptive. A good title can help communicate what the KEP is and should be considered as part of any review. The title should be lowercased and spaces/punctuation should be replaced with - . To get started with this template: Make a copy of this template. Create a copy of this template and name it YYYYMMDD-my-title.md , where YYYYMMDD is the date the KEP was first drafted. Fill out the \"overview\" sections. This includes the Summary and Motivation sections. These should be easy if you've preflighted the idea of the KEP in an issue. Create a PR. Assign it to folks that are sponsoring this process. Create an issue When filing an enhancement tracking issue, please ensure to complete all fields in the template. Merge early. Avoid getting hung up on specific details and instead aim to get the goal of the KEP merged quickly. The best way to do this is to just start with the \"Overview\" sections and fill out details incrementally in follow on PRs. View anything marked as a provisional as a working document and subject to change. Aim for single topic PRs to keep discussions focused. If you disagree with what is already in a document, open a new PR with suggested changes. The canonical place for the latest set of instructions (and the likely source of this file) is here . The Metadata section above is intended to support the creation of tooling around the KEP process. This will be a YAML section that is fenced as a code block. See the KEP process for details on each of these items.","title":"Title"},{"location":"enhancements/YYYYMMDD-kep-template/#table-of-contents","text":"A table of contents is helpful for quickly jumping to sections of a KEP and for highlighting any additional information provided beyond the standard KEP template. Ensure the TOC is wrapped with <!-- toc --&rt;<!-- /toc --&rt; tags, and then generate with hack/update-toc.sh . Summary Motivation Goals Non-Goals Proposal User Stories [optional] Story 1 Story 2 Implementation Details/Notes/Constraints [optional] Risks and Mitigations Design Details Test Plan Removing a deprecated flag Implementation History Drawbacks [optional] Alternatives [optional]","title":"Table of Contents"},{"location":"enhancements/YYYYMMDD-kep-template/#summary","text":"The Summary section is incredibly important for producing high quality user-focused documentation such as release notes or a development roadmap. It should be possible to collect this information before implementation begins in order to avoid requiring implementers to split their attention between writing release notes and implementing the feature itself. A good summary is probably at least a paragraph in length.","title":"Summary"},{"location":"enhancements/YYYYMMDD-kep-template/#motivation","text":"This section is for explicitly listing the motivation, goals and non-goals of this KEP. Describe why the change is important and the benefits to users. The motivation section can optionally provide links to experience reports to demonstrate the interest in a KEP within the wider Kubernetes community.","title":"Motivation"},{"location":"enhancements/YYYYMMDD-kep-template/#goals","text":"List the specific goals of the KEP. How will we know that this has succeeded?","title":"Goals"},{"location":"enhancements/YYYYMMDD-kep-template/#non-goals","text":"What is out of scope for this KEP? Listing non-goals helps to focus discussion and make progress.","title":"Non-Goals"},{"location":"enhancements/YYYYMMDD-kep-template/#proposal","text":"This is where we get down to the nitty gritty of what the proposal actually is.","title":"Proposal"},{"location":"enhancements/YYYYMMDD-kep-template/#user-stories-optional","text":"Detail the things that people will be able to do if this KEP is implemented. Include as much detail as possible so that people can understand the \"how\" of the system. The goal here is to make this feel real for users without getting bogged down.","title":"User Stories [optional]"},{"location":"enhancements/YYYYMMDD-kep-template/#story-1","text":"","title":"Story 1"},{"location":"enhancements/YYYYMMDD-kep-template/#story-2","text":"","title":"Story 2"},{"location":"enhancements/YYYYMMDD-kep-template/#implementation-detailsnotesconstraints-optional","text":"What are the caveats to the implementation? What are some important details that didn't come across above. Go in to as much detail as necessary here. This might be a good place to talk about core concepts and how they relate.","title":"Implementation Details/Notes/Constraints [optional]"},{"location":"enhancements/YYYYMMDD-kep-template/#risks-and-mitigations","text":"What are the risks of this proposal and how do we mitigate. Think broadly. For example, consider both security and how this will impact the larger kubernetes ecosystem. How will security be reviewed and by whom? How will UX be reviewed and by whom? Consider including folks that also work outside project.","title":"Risks and Mitigations"},{"location":"enhancements/YYYYMMDD-kep-template/#design-details","text":"","title":"Design Details"},{"location":"enhancements/YYYYMMDD-kep-template/#test-plan","text":"Note: Section not required until targeted at a release. Consider the following in developing a test plan for this enhancement: Will there be e2e and integration tests, in addition to unit tests? How will it be tested in isolation vs with other components? No need to outline all of the test cases, just the general strategy. Anything that would count as tricky in the implementation and anything particularly challenging to test should be called out. All code is expected to have adequate tests (eventually with coverage expectations). Please adhere to the Kubernetes testing guidelines when drafting this test plan.","title":"Test Plan"},{"location":"enhancements/YYYYMMDD-kep-template/#removing-a-deprecated-flag","text":"Announce deprecation and support policy of the existing flag Two versions passed since introducing the functionality which deprecates the flag (to address version skew) Address feedback on usage/changed behavior, provided on GitHub issues Deprecate the flag","title":"Removing a deprecated flag"},{"location":"enhancements/YYYYMMDD-kep-template/#implementation-history","text":"Major milestones in the life cycle of a KEP should be tracked in Implementation History . Major milestones might include the Summary and Motivation sections being merged signaling acceptance the Proposal section being merged signaling agreement on a proposed design the date implementation started the first Kubernetes release where an initial version of the KEP was available the version of Kubernetes where the KEP graduated to general availability when the KEP was retired or superseded","title":"Implementation History"},{"location":"enhancements/YYYYMMDD-kep-template/#drawbacks-optional","text":"Why should this KEP not be implemented.","title":"Drawbacks [optional]"},{"location":"enhancements/YYYYMMDD-kep-template/#alternatives-optional","text":"Similar to the Drawbacks section the Alternatives section is used to highlight and record other possible approaches to delivering the value proposed by a KEP.","title":"Alternatives [optional]"},{"location":"examples/","text":"Ingress examples \u00b6 This directory contains a catalog of examples on how to run, configure and scale Ingress. Please review the prerequisites before trying them. The examples on these pages include the spec.ingressClassName field which replaces the deprecated kubernetes.io/ingress.class: nginx annotation. Users of ingress-nginx < 1.0.0 (Helm chart < 4.0.0) should use the legacy documentation . For more information, check out the Migration to apiVersion networking.k8s.io/v1 guide. Category Name Description Complexity Level Apps Docker Registry TODO TODO Auth Basic authentication password protect your website Intermediate Auth Client certificate authentication secure your website with client certificate authentication Intermediate Auth External authentication plugin defer to an external authentication service Intermediate Auth OAuth external auth TODO TODO Customization Configuration snippets customize nginx location configuration using annotations Advanced Customization Custom configuration TODO TODO Customization Custom DH parameters for perfect forward secrecy TODO TODO Customization Custom errors serve custom error pages from the default backend Intermediate Customization Custom headers set custom headers before sending traffic to backends Advanced Customization External authentication with response header propagation TODO TODO Customization Sysctl tuning TODO TODO Features Rewrite TODO TODO Features Session stickiness route requests consistently to the same endpoint Advanced Scaling Static IP a single ingress gets a single static IP Intermediate TLS Multi TLS certificate termination TODO TODO TLS TLS termination TODO TODO","title":"Introduction"},{"location":"examples/#ingress-examples","text":"This directory contains a catalog of examples on how to run, configure and scale Ingress. Please review the prerequisites before trying them. The examples on these pages include the spec.ingressClassName field which replaces the deprecated kubernetes.io/ingress.class: nginx annotation. Users of ingress-nginx < 1.0.0 (Helm chart < 4.0.0) should use the legacy documentation . For more information, check out the Migration to apiVersion networking.k8s.io/v1 guide. Category Name Description Complexity Level Apps Docker Registry TODO TODO Auth Basic authentication password protect your website Intermediate Auth Client certificate authentication secure your website with client certificate authentication Intermediate Auth External authentication plugin defer to an external authentication service Intermediate Auth OAuth external auth TODO TODO Customization Configuration snippets customize nginx location configuration using annotations Advanced Customization Custom configuration TODO TODO Customization Custom DH parameters for perfect forward secrecy TODO TODO Customization Custom errors serve custom error pages from the default backend Intermediate Customization Custom headers set custom headers before sending traffic to backends Advanced Customization External authentication with response header propagation TODO TODO Customization Sysctl tuning TODO TODO Features Rewrite TODO TODO Features Session stickiness route requests consistently to the same endpoint Advanced Scaling Static IP a single ingress gets a single static IP Intermediate TLS Multi TLS certificate termination TODO TODO TLS TLS termination TODO TODO","title":"Ingress examples"},{"location":"examples/PREREQUISITES/","text":"Prerequisites \u00b6 Many of the examples in this directory have common prerequisites. TLS certificates \u00b6 Unless otherwise mentioned, the TLS secret used in examples is a 2048 bit RSA key/cert pair with an arbitrarily chosen hostname, created as follows $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj \"/CN=nginxsvc/O=nginxsvc\" Generating a 2048 bit RSA private key ................+++ ................+++ writing new private key to 'tls.key' ----- $ kubectl create secret tls tls-secret --key tls.key --cert tls.crt secret \"tls-secret\" created Note: If using CA Authentication, described below, you will need to sign the server certificate with the CA. Client Certificate Authentication \u00b6 CA Authentication also known as Mutual Authentication allows both the server and client to verify each others identity via a common CA. We have a CA Certificate which we usually obtain from a Certificate Authority and use that to sign both our server certificate and client certificate. Then every time we want to access our backend, we must pass the client certificate. These instructions are based on the following blog Generate the CA Key and Certificate: openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=My Cert Authority' Generate the Server Key, and Certificate and Sign with the CA Certificate: openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=mydomain.com' openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt Generate the Client Key, and Certificate and Sign with the CA Certificate: openssl req -new -newkey rsa:4096 -keyout client.key -out client.csr -nodes -subj '/CN=My Client' openssl x509 -req -sha256 -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 02 -out client.crt Once this is complete you can continue to follow the instructions here Test HTTP Service \u00b6 All examples that require a test HTTP Service use the standard http-svc pod, which you can deploy as follows $ kubectl create -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/http-svc.yaml service \"http-svc\" created replicationcontroller \"http-svc\" created $ kubectl get po NAME READY STATUS RESTARTS AGE http-svc-p1t3t 1/1 Running 0 1d $ kubectl get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE http-svc 10.0.122.116 <pending> 80:30301/TCP 1d You can test that the HTTP Service works by exposing it temporarily $ kubectl patch svc http-svc -p '{\"spec\":{\"type\": \"LoadBalancer\"}}' \"http-svc\" patched $ kubectl get svc http-svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE http-svc 10.0.122.116 <pending> 80:30301/TCP 1d $ kubectl describe svc http-svc Name: http-svc Namespace: default Labels: app=http-svc Selector: app=http-svc Type: LoadBalancer IP: 10.0.122.116 LoadBalancer Ingress: 108.59.87.136 Port: http 80/TCP NodePort: http 30301/TCP Endpoints: 10.180.1.6:8080 Session Affinity: None Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 {service-controller } Normal Type ClusterIP -> LoadBalancer 1m 1m 1 {service-controller } Normal CreatingLoadBalancer Creating load balancer 16s 16s 1 {service-controller } Normal CreatedLoadBalancer Created load balancer $ curl 108 .59.87.136 CLIENT VALUES: client_address=10.240.0.3 command=GET real path=/ query=nil request_version=1.1 request_uri=http://108.59.87.136:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* host=108.59.87.136 user-agent=curl/7.46.0 BODY: -no body in request- $ kubectl patch svc http-svc -p '{\"spec\":{\"type\": \"NodePort\"}}' \"http-svc\" patched","title":"Prerequisites"},{"location":"examples/PREREQUISITES/#prerequisites","text":"Many of the examples in this directory have common prerequisites.","title":"Prerequisites"},{"location":"examples/PREREQUISITES/#tls-certificates","text":"Unless otherwise mentioned, the TLS secret used in examples is a 2048 bit RSA key/cert pair with an arbitrarily chosen hostname, created as follows $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj \"/CN=nginxsvc/O=nginxsvc\" Generating a 2048 bit RSA private key ................+++ ................+++ writing new private key to 'tls.key' ----- $ kubectl create secret tls tls-secret --key tls.key --cert tls.crt secret \"tls-secret\" created Note: If using CA Authentication, described below, you will need to sign the server certificate with the CA.","title":"TLS certificates"},{"location":"examples/PREREQUISITES/#client-certificate-authentication","text":"CA Authentication also known as Mutual Authentication allows both the server and client to verify each others identity via a common CA. We have a CA Certificate which we usually obtain from a Certificate Authority and use that to sign both our server certificate and client certificate. Then every time we want to access our backend, we must pass the client certificate. These instructions are based on the following blog Generate the CA Key and Certificate: openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=My Cert Authority' Generate the Server Key, and Certificate and Sign with the CA Certificate: openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=mydomain.com' openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt Generate the Client Key, and Certificate and Sign with the CA Certificate: openssl req -new -newkey rsa:4096 -keyout client.key -out client.csr -nodes -subj '/CN=My Client' openssl x509 -req -sha256 -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 02 -out client.crt Once this is complete you can continue to follow the instructions here","title":"Client Certificate Authentication"},{"location":"examples/PREREQUISITES/#test-http-service","text":"All examples that require a test HTTP Service use the standard http-svc pod, which you can deploy as follows $ kubectl create -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/http-svc.yaml service \"http-svc\" created replicationcontroller \"http-svc\" created $ kubectl get po NAME READY STATUS RESTARTS AGE http-svc-p1t3t 1/1 Running 0 1d $ kubectl get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE http-svc 10.0.122.116 <pending> 80:30301/TCP 1d You can test that the HTTP Service works by exposing it temporarily $ kubectl patch svc http-svc -p '{\"spec\":{\"type\": \"LoadBalancer\"}}' \"http-svc\" patched $ kubectl get svc http-svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE http-svc 10.0.122.116 <pending> 80:30301/TCP 1d $ kubectl describe svc http-svc Name: http-svc Namespace: default Labels: app=http-svc Selector: app=http-svc Type: LoadBalancer IP: 10.0.122.116 LoadBalancer Ingress: 108.59.87.136 Port: http 80/TCP NodePort: http 30301/TCP Endpoints: 10.180.1.6:8080 Session Affinity: None Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 {service-controller } Normal Type ClusterIP -> LoadBalancer 1m 1m 1 {service-controller } Normal CreatingLoadBalancer Creating load balancer 16s 16s 1 {service-controller } Normal CreatedLoadBalancer Created load balancer $ curl 108 .59.87.136 CLIENT VALUES: client_address=10.240.0.3 command=GET real path=/ query=nil request_version=1.1 request_uri=http://108.59.87.136:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* host=108.59.87.136 user-agent=curl/7.46.0 BODY: -no body in request- $ kubectl patch svc http-svc -p '{\"spec\":{\"type\": \"NodePort\"}}' \"http-svc\" patched","title":"Test HTTP Service"},{"location":"examples/affinity/cookie/","text":"Sticky sessions \u00b6 This example demonstrates how to achieve session affinity using cookies. Deployment \u00b6 Session affinity can be configured using the following annotations: Name Description Value nginx.ingress.kubernetes.io/affinity Type of the affinity, set this to cookie to enable session affinity string (NGINX only supports cookie ) nginx.ingress.kubernetes.io/affinity-mode The affinity mode defines how sticky a session is. Use balanced to redistribute some sessions when scaling pods or persistent for maximum stickiness. balanced (default) or persistent nginx.ingress.kubernetes.io/affinity-canary-behavior Defines session affinity behavior of canaries. By default the behavior is sticky , and canaries respect session affinity configuration. Set this to legacy to restore original canary behavior, when session affinity parameters were not respected. sticky (default) or legacy nginx.ingress.kubernetes.io/session-cookie-name Name of the cookie that will be created string (defaults to INGRESSCOOKIE ) nginx.ingress.kubernetes.io/session-cookie-secure Set the cookie as secure regardless the protocol of the incoming request \"true\" or \"false\" nginx.ingress.kubernetes.io/session-cookie-path Path that will be set on the cookie (required if your Ingress paths use regular expressions) string (defaults to the currently matched path ) nginx.ingress.kubernetes.io/session-cookie-domain Domain that will be set on the cookie string nginx.ingress.kubernetes.io/session-cookie-samesite SameSite attribute to apply to the cookie Browser accepted values are None , Lax , and Strict nginx.ingress.kubernetes.io/session-cookie-conditional-samesite-none Will omit SameSite=None attribute for older browsers which reject the more-recently defined SameSite=None value \"true\" or \"false\" nginx.ingress.kubernetes.io/session-cookie-max-age Time until the cookie expires, corresponds to the Max-Age cookie directive number of seconds nginx.ingress.kubernetes.io/session-cookie-expires Legacy version of the previous annotation for compatibility with older browsers, generates an Expires cookie directive by adding the seconds to the current date number of seconds nginx.ingress.kubernetes.io/session-cookie-change-on-failure When set to false nginx ingress will send request to upstream pointed by sticky cookie even if previous attempt failed. When set to true and previous attempt failed, sticky cookie will be changed to point to another upstream. true or false (defaults to false ) You can create the session affinity example Ingress to test this: kubectl create -f ingress.yaml Validation \u00b6 You can confirm that the Ingress works: $ kubectl describe ing nginx-test Name: nginx-test Namespace: default Address: Default backend: default-http-backend:80 (10.180.0.4:8080,10.240.0.2:8080) Rules: Host Path Backends ---- ---- -------- stickyingress.example.com / nginx-service:80 (<none>) Annotations: affinity: cookie session-cookie-name: INGRESSCOOKIE session-cookie-expires: 172800 session-cookie-max-age: 172800 Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 7s 7s 1 {ingress-nginx-controller } Normal CREATE default/nginx-test $ curl -I http://stickyingress.example.com HTTP/1.1 200 OK Server: nginx/1.11.9 Date: Fri, 10 Feb 2017 14:11:12 GMT Content-Type: text/html Content-Length: 612 Connection: keep-alive Set-Cookie: INGRESSCOOKIE=a9907b79b248140b56bb13723f72b67697baac3d; Expires=Sun, 12-Feb-17 14:11:12 GMT; Max-Age=172800; Path=/; HttpOnly Last-Modified: Tue, 24 Jan 2017 14:02:19 GMT ETag: \"58875e6b-264\" Accept-Ranges: bytes In the example above, you can see that the response contains a Set-Cookie header with the settings we have defined. This cookie is created by the NGINX Ingress Controller, it contains a randomly generated key corresponding to the upstream used for that request (selected using consistent hashing ) and has an Expires directive. If a client sends a cookie that doesn't correspond to an upstream, NGINX selects an upstream and creates a corresponding cookie. If the backend pool grows NGINX will keep sending the requests through the same server of the first request, even if it's overloaded. When the backend server is removed, the requests are re-routed to another upstream server. This does not require the cookie to be updated because the key's consistent hash will change. Caveats \u00b6 When you have a Service pointing to more than one Ingress, with only one containing affinity configuration, the first created Ingress will be used. This means that you can face the situation that you've configured session affinity on one Ingress and it doesn't work because the Service is pointing to another Ingress that doesn't configure this.","title":"Sticky Sessions"},{"location":"examples/affinity/cookie/#sticky-sessions","text":"This example demonstrates how to achieve session affinity using cookies.","title":"Sticky sessions"},{"location":"examples/affinity/cookie/#deployment","text":"Session affinity can be configured using the following annotations: Name Description Value nginx.ingress.kubernetes.io/affinity Type of the affinity, set this to cookie to enable session affinity string (NGINX only supports cookie ) nginx.ingress.kubernetes.io/affinity-mode The affinity mode defines how sticky a session is. Use balanced to redistribute some sessions when scaling pods or persistent for maximum stickiness. balanced (default) or persistent nginx.ingress.kubernetes.io/affinity-canary-behavior Defines session affinity behavior of canaries. By default the behavior is sticky , and canaries respect session affinity configuration. Set this to legacy to restore original canary behavior, when session affinity parameters were not respected. sticky (default) or legacy nginx.ingress.kubernetes.io/session-cookie-name Name of the cookie that will be created string (defaults to INGRESSCOOKIE ) nginx.ingress.kubernetes.io/session-cookie-secure Set the cookie as secure regardless the protocol of the incoming request \"true\" or \"false\" nginx.ingress.kubernetes.io/session-cookie-path Path that will be set on the cookie (required if your Ingress paths use regular expressions) string (defaults to the currently matched path ) nginx.ingress.kubernetes.io/session-cookie-domain Domain that will be set on the cookie string nginx.ingress.kubernetes.io/session-cookie-samesite SameSite attribute to apply to the cookie Browser accepted values are None , Lax , and Strict nginx.ingress.kubernetes.io/session-cookie-conditional-samesite-none Will omit SameSite=None attribute for older browsers which reject the more-recently defined SameSite=None value \"true\" or \"false\" nginx.ingress.kubernetes.io/session-cookie-max-age Time until the cookie expires, corresponds to the Max-Age cookie directive number of seconds nginx.ingress.kubernetes.io/session-cookie-expires Legacy version of the previous annotation for compatibility with older browsers, generates an Expires cookie directive by adding the seconds to the current date number of seconds nginx.ingress.kubernetes.io/session-cookie-change-on-failure When set to false nginx ingress will send request to upstream pointed by sticky cookie even if previous attempt failed. When set to true and previous attempt failed, sticky cookie will be changed to point to another upstream. true or false (defaults to false ) You can create the session affinity example Ingress to test this: kubectl create -f ingress.yaml","title":"Deployment"},{"location":"examples/affinity/cookie/#validation","text":"You can confirm that the Ingress works: $ kubectl describe ing nginx-test Name: nginx-test Namespace: default Address: Default backend: default-http-backend:80 (10.180.0.4:8080,10.240.0.2:8080) Rules: Host Path Backends ---- ---- -------- stickyingress.example.com / nginx-service:80 (<none>) Annotations: affinity: cookie session-cookie-name: INGRESSCOOKIE session-cookie-expires: 172800 session-cookie-max-age: 172800 Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 7s 7s 1 {ingress-nginx-controller } Normal CREATE default/nginx-test $ curl -I http://stickyingress.example.com HTTP/1.1 200 OK Server: nginx/1.11.9 Date: Fri, 10 Feb 2017 14:11:12 GMT Content-Type: text/html Content-Length: 612 Connection: keep-alive Set-Cookie: INGRESSCOOKIE=a9907b79b248140b56bb13723f72b67697baac3d; Expires=Sun, 12-Feb-17 14:11:12 GMT; Max-Age=172800; Path=/; HttpOnly Last-Modified: Tue, 24 Jan 2017 14:02:19 GMT ETag: \"58875e6b-264\" Accept-Ranges: bytes In the example above, you can see that the response contains a Set-Cookie header with the settings we have defined. This cookie is created by the NGINX Ingress Controller, it contains a randomly generated key corresponding to the upstream used for that request (selected using consistent hashing ) and has an Expires directive. If a client sends a cookie that doesn't correspond to an upstream, NGINX selects an upstream and creates a corresponding cookie. If the backend pool grows NGINX will keep sending the requests through the same server of the first request, even if it's overloaded. When the backend server is removed, the requests are re-routed to another upstream server. This does not require the cookie to be updated because the key's consistent hash will change.","title":"Validation"},{"location":"examples/affinity/cookie/#caveats","text":"When you have a Service pointing to more than one Ingress, with only one containing affinity configuration, the first created Ingress will be used. This means that you can face the situation that you've configured session affinity on one Ingress and it doesn't work because the Service is pointing to another Ingress that doesn't configure this.","title":"Caveats"},{"location":"examples/auth/basic/","text":"Basic Authentication \u00b6 This example shows how to add authentication in a Ingress rule using a secret that contains a file generated with htpasswd . It's important the file generated is named auth (actually - that the secret has a key data.auth ), otherwise the ingress-controller returns a 503. Create htpasswd file \u00b6 $ htpasswd -c auth foo New password: <bar> New password: Re-type new password: Adding password for user foo Convert htpasswd into a secret \u00b6 $ kubectl create secret generic basic-auth --from-file = auth secret \"basic-auth\" created Examine secret \u00b6 $ kubectl get secret basic-auth -o yaml apiVersion: v1 data: auth: Zm9vOiRhcHIxJE9GRzNYeWJwJGNrTDBGSERBa29YWUlsSDkuY3lzVDAK kind: Secret metadata: name: basic-auth namespace: default type: Opaque Using kubectl, create an ingress tied to the basic-auth secret \u00b6 $ echo \" apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-with-auth annotations: # type of authentication nginx.ingress.kubernetes.io/auth-type: basic # name of the secret that contains the user/password definitions nginx.ingress.kubernetes.io/auth-secret: basic-auth # message to display with an appropriate context why the authentication is required nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - foo' spec: ingressClassName: nginx rules: - host: foo.bar.com http: paths: - path: / pathType: Prefix backend: service: name: http-svc port: number: 80 \" | kubectl create -f - Use curl to confirm authorization is required by the ingress \u00b6 $ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' * Trying 10.2.29.4... * Connected to 10.2.29.4 (10.2.29.4) port 80 (#0) > GET / HTTP/1.1 > Host: foo.bar.com > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 401 Unauthorized < Server: nginx/1.10.0 < Date: Wed, 11 May 2016 05:27:23 GMT < Content-Type: text/html < Content-Length: 195 < Connection: keep-alive < WWW-Authenticate: Basic realm=\"Authentication Required - foo\" < <html> <head><title>401 Authorization Required</title></head> <body bgcolor=\"white\"> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx/1.10.0</center> </body> </html> * Connection #0 to host 10.2.29.4 left intact Use curl with the correct credentials to connect to the ingress \u00b6 $ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' -u 'foo:bar' * Trying 10.2.29.4... * Connected to 10.2.29.4 (10.2.29.4) port 80 (#0) * Server auth using Basic with user 'foo' > GET / HTTP/1.1 > Host: foo.bar.com > Authorization: Basic Zm9vOmJhcg== > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.10.0 < Date: Wed, 11 May 2016 06:05:26 GMT < Content-Type: text/plain < Transfer-Encoding: chunked < Connection: keep-alive < Vary: Accept-Encoding < CLIENT VALUES: client_address=10.2.29.4 command=GET real path=/ query=nil request_version=1.1 request_uri=http://foo.bar.com:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=close host=foo.bar.com user-agent=curl/7.43.0 x-request-id=e426c7829ef9f3b18d40730857c3eddb x-forwarded-for=10.2.29.1 x-forwarded-host=foo.bar.com x-forwarded-port=80 x-forwarded-proto=http x-real-ip=10.2.29.1 x-scheme=http BODY: * Connection #0 to host 10.2.29.4 left intact -no body in request-","title":"Basic Authentication"},{"location":"examples/auth/basic/#basic-authentication","text":"This example shows how to add authentication in a Ingress rule using a secret that contains a file generated with htpasswd . It's important the file generated is named auth (actually - that the secret has a key data.auth ), otherwise the ingress-controller returns a 503.","title":"Basic Authentication"},{"location":"examples/auth/basic/#create-htpasswd-file","text":"$ htpasswd -c auth foo New password: <bar> New password: Re-type new password: Adding password for user foo","title":"Create htpasswd file"},{"location":"examples/auth/basic/#convert-htpasswd-into-a-secret","text":"$ kubectl create secret generic basic-auth --from-file = auth secret \"basic-auth\" created","title":"Convert htpasswd into a secret"},{"location":"examples/auth/basic/#examine-secret","text":"$ kubectl get secret basic-auth -o yaml apiVersion: v1 data: auth: Zm9vOiRhcHIxJE9GRzNYeWJwJGNrTDBGSERBa29YWUlsSDkuY3lzVDAK kind: Secret metadata: name: basic-auth namespace: default type: Opaque","title":"Examine secret"},{"location":"examples/auth/basic/#using-kubectl-create-an-ingress-tied-to-the-basic-auth-secret","text":"$ echo \" apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-with-auth annotations: # type of authentication nginx.ingress.kubernetes.io/auth-type: basic # name of the secret that contains the user/password definitions nginx.ingress.kubernetes.io/auth-secret: basic-auth # message to display with an appropriate context why the authentication is required nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - foo' spec: ingressClassName: nginx rules: - host: foo.bar.com http: paths: - path: / pathType: Prefix backend: service: name: http-svc port: number: 80 \" | kubectl create -f -","title":"Using kubectl, create an ingress tied to the basic-auth secret"},{"location":"examples/auth/basic/#use-curl-to-confirm-authorization-is-required-by-the-ingress","text":"$ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' * Trying 10.2.29.4... * Connected to 10.2.29.4 (10.2.29.4) port 80 (#0) > GET / HTTP/1.1 > Host: foo.bar.com > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 401 Unauthorized < Server: nginx/1.10.0 < Date: Wed, 11 May 2016 05:27:23 GMT < Content-Type: text/html < Content-Length: 195 < Connection: keep-alive < WWW-Authenticate: Basic realm=\"Authentication Required - foo\" < <html> <head><title>401 Authorization Required</title></head> <body bgcolor=\"white\"> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx/1.10.0</center> </body> </html> * Connection #0 to host 10.2.29.4 left intact","title":"Use curl to confirm authorization is required by the ingress"},{"location":"examples/auth/basic/#use-curl-with-the-correct-credentials-to-connect-to-the-ingress","text":"$ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' -u 'foo:bar' * Trying 10.2.29.4... * Connected to 10.2.29.4 (10.2.29.4) port 80 (#0) * Server auth using Basic with user 'foo' > GET / HTTP/1.1 > Host: foo.bar.com > Authorization: Basic Zm9vOmJhcg== > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.10.0 < Date: Wed, 11 May 2016 06:05:26 GMT < Content-Type: text/plain < Transfer-Encoding: chunked < Connection: keep-alive < Vary: Accept-Encoding < CLIENT VALUES: client_address=10.2.29.4 command=GET real path=/ query=nil request_version=1.1 request_uri=http://foo.bar.com:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=close host=foo.bar.com user-agent=curl/7.43.0 x-request-id=e426c7829ef9f3b18d40730857c3eddb x-forwarded-for=10.2.29.1 x-forwarded-host=foo.bar.com x-forwarded-port=80 x-forwarded-proto=http x-real-ip=10.2.29.1 x-scheme=http BODY: * Connection #0 to host 10.2.29.4 left intact -no body in request-","title":"Use curl with the correct credentials to connect to the ingress"},{"location":"examples/auth/client-certs/","text":"Client Certificate Authentication \u00b6 It is possible to enable Client-Certificate Authentication by adding additional annotations to your Ingress Resource. Before getting started you must have the following Certificates configured: CA certificate and Key (Intermediate Certs need to be in CA) Server Certificate (Signed by CA) and Key (CN should be equal the hostname you will use) Client Certificate (Signed by CA) and Key For more details on the generation process, checkout the Prerequisite docs . You can have as many certificates as you want. If they're in the binary DER format, you can convert them as the following: openssl x509 -in certificate.der -inform der -out certificate.crt -outform pem Then, you can concatenate them all into one file, named 'ca.crt' with the following: cat certificate1.crt certificate2.crt certificate3.crt >> ca.crt Note: Make sure that the Key Size is greater than 1024 and Hashing Algorithm (Digest) is something better than md5 for each certificate generated. Otherwise you will receive an error. Creating Certificate Secrets \u00b6 There are many different ways of configuring your secrets to enable Client-Certificate Authentication to work properly. You can create a secret containing just the CA certificate and another Secret containing the Server Certificate which is Signed by the CA. kubectl create secret generic ca-secret --from-file = ca.crt = ca.crt kubectl create secret generic tls-secret --from-file = tls.crt = server.crt --from-file = tls.key = server.key You can create a secret containing CA certificate along with the Server Certificate that can be used for both TLS and Client Auth. kubectl create secret generic ca-secret --from-file = tls.crt = server.crt --from-file = tls.key = server.key --from-file = ca.crt = ca.crt If you want to also enable Certificate Revocation List verification you can create the secret also containing the CRL file in PEM format: kubectl create secret generic ca-secret --from-file = ca.crt = ca.crt --from-file = ca.crl = ca.crl Note: The CA Certificate must contain the trusted certificate authority chain to verify client certificates. Setup Instructions \u00b6 Add the annotations as provided in the ingress.yaml example to your own ingress resources as required. Test by performing a curl against the Ingress Path without the Client Cert and expect a Status Code 400. Test by performing a curl against the Ingress Path with the Client Cert and expect a Status Code 200.","title":"Client Certificate Authentication"},{"location":"examples/auth/client-certs/#client-certificate-authentication","text":"It is possible to enable Client-Certificate Authentication by adding additional annotations to your Ingress Resource. Before getting started you must have the following Certificates configured: CA certificate and Key (Intermediate Certs need to be in CA) Server Certificate (Signed by CA) and Key (CN should be equal the hostname you will use) Client Certificate (Signed by CA) and Key For more details on the generation process, checkout the Prerequisite docs . You can have as many certificates as you want. If they're in the binary DER format, you can convert them as the following: openssl x509 -in certificate.der -inform der -out certificate.crt -outform pem Then, you can concatenate them all into one file, named 'ca.crt' with the following: cat certificate1.crt certificate2.crt certificate3.crt >> ca.crt Note: Make sure that the Key Size is greater than 1024 and Hashing Algorithm (Digest) is something better than md5 for each certificate generated. Otherwise you will receive an error.","title":"Client Certificate Authentication"},{"location":"examples/auth/client-certs/#creating-certificate-secrets","text":"There are many different ways of configuring your secrets to enable Client-Certificate Authentication to work properly. You can create a secret containing just the CA certificate and another Secret containing the Server Certificate which is Signed by the CA. kubectl create secret generic ca-secret --from-file = ca.crt = ca.crt kubectl create secret generic tls-secret --from-file = tls.crt = server.crt --from-file = tls.key = server.key You can create a secret containing CA certificate along with the Server Certificate that can be used for both TLS and Client Auth. kubectl create secret generic ca-secret --from-file = tls.crt = server.crt --from-file = tls.key = server.key --from-file = ca.crt = ca.crt If you want to also enable Certificate Revocation List verification you can create the secret also containing the CRL file in PEM format: kubectl create secret generic ca-secret --from-file = ca.crt = ca.crt --from-file = ca.crl = ca.crl Note: The CA Certificate must contain the trusted certificate authority chain to verify client certificates.","title":"Creating Certificate Secrets"},{"location":"examples/auth/client-certs/#setup-instructions","text":"Add the annotations as provided in the ingress.yaml example to your own ingress resources as required. Test by performing a curl against the Ingress Path without the Client Cert and expect a Status Code 400. Test by performing a curl against the Ingress Path with the Client Cert and expect a Status Code 200.","title":"Setup Instructions"},{"location":"examples/auth/external-auth/","text":"External Basic Authentication \u00b6 Example 1 \u00b6 Use an external service (Basic Auth) located in https://httpbin.org $ kubectl create -f ingress.yaml ingress \"external-auth\" created $ kubectl get ing external-auth NAME HOSTS ADDRESS PORTS AGE external-auth external-auth-01.sample.com 172.17.4.99 80 13s $ kubectl get ing external-auth -o yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/auth-url: https://httpbin.org/basic-auth/user/passwd creationTimestamp: 2016-10-03T13:50:35Z generation: 1 name: external-auth namespace: default resourceVersion: \"2068378\" selfLink: /apis/networking/v1/namespaces/default/ingresses/external-auth uid: 5c388f1d-8970-11e6-9004-080027d2dc94 spec: rules: - host: external-auth-01.sample.com http: paths: - path: / pathType: Prefix backend: service: name: http-svc port: number: 80 status: loadBalancer: ingress: - ip: 172.17.4.99 $ Test 1: no username/password (expect code 401) \u00b6 $ curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) > GET / HTTP/1.1 > Host: external-auth-01.sample.com > User-Agent: curl/7.50.1 > Accept: */* > < HTTP/1.1 401 Unauthorized < Server: nginx/1.11.3 < Date: Mon, 03 Oct 2016 14:52:08 GMT < Content-Type: text/html < Content-Length: 195 < Connection: keep-alive < WWW-Authenticate: Basic realm=\"Fake Realm\" < <html> <head><title>401 Authorization Required</title></head> <body bgcolor=\"white\"> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx/1.11.3</center> </body> </html> * Connection #0 to host 172.17.4.99 left intact Test 2: valid username/password (expect code 200) \u00b6 $ curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' -u 'user:passwd' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) * Server auth using Basic with user 'user' > GET / HTTP/1.1 > Host: external-auth-01.sample.com > Authorization: Basic dXNlcjpwYXNzd2Q= > User-Agent: curl/7.50.1 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.11.3 < Date: Mon, 03 Oct 2016 14:52:50 GMT < Content-Type: text/plain < Transfer-Encoding: chunked < Connection: keep-alive < CLIENT VALUES: client_address=10.2.60.2 command=GET real path=/ query=nil request_version=1.1 request_uri=http://external-auth-01.sample.com:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* authorization=Basic dXNlcjpwYXNzd2Q= connection=close host=external-auth-01.sample.com user-agent=curl/7.50.1 x-forwarded-for=10.2.60.1 x-forwarded-host=external-auth-01.sample.com x-forwarded-port=80 x-forwarded-proto=http x-real-ip=10.2.60.1 BODY: * Connection #0 to host 172.17.4.99 left intact -no body in request- Test 3: invalid username/password (expect code 401) \u00b6 curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' -u 'user:user' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) * Server auth using Basic with user 'user' > GET / HTTP/1.1 > Host: external-auth-01.sample.com > Authorization: Basic dXNlcjp1c2Vy > User-Agent: curl/7.50.1 > Accept: */* > < HTTP/1.1 401 Unauthorized < Server: nginx/1.11.3 < Date: Mon, 03 Oct 2016 14:53:04 GMT < Content-Type: text/html < Content-Length: 195 < Connection: keep-alive * Authentication problem. Ignoring this. < WWW-Authenticate: Basic realm=\"Fake Realm\" < <html> <head><title>401 Authorization Required</title></head> <body bgcolor=\"white\"> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx/1.11.3</center> </body> </html> * Connection #0 to host 172.17.4.99 left intact","title":"External Basic Authentication"},{"location":"examples/auth/external-auth/#external-basic-authentication","text":"","title":"External Basic Authentication"},{"location":"examples/auth/external-auth/#example-1","text":"Use an external service (Basic Auth) located in https://httpbin.org $ kubectl create -f ingress.yaml ingress \"external-auth\" created $ kubectl get ing external-auth NAME HOSTS ADDRESS PORTS AGE external-auth external-auth-01.sample.com 172.17.4.99 80 13s $ kubectl get ing external-auth -o yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/auth-url: https://httpbin.org/basic-auth/user/passwd creationTimestamp: 2016-10-03T13:50:35Z generation: 1 name: external-auth namespace: default resourceVersion: \"2068378\" selfLink: /apis/networking/v1/namespaces/default/ingresses/external-auth uid: 5c388f1d-8970-11e6-9004-080027d2dc94 spec: rules: - host: external-auth-01.sample.com http: paths: - path: / pathType: Prefix backend: service: name: http-svc port: number: 80 status: loadBalancer: ingress: - ip: 172.17.4.99 $","title":"Example 1"},{"location":"examples/auth/external-auth/#test-1-no-usernamepassword-expect-code-401","text":"$ curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) > GET / HTTP/1.1 > Host: external-auth-01.sample.com > User-Agent: curl/7.50.1 > Accept: */* > < HTTP/1.1 401 Unauthorized < Server: nginx/1.11.3 < Date: Mon, 03 Oct 2016 14:52:08 GMT < Content-Type: text/html < Content-Length: 195 < Connection: keep-alive < WWW-Authenticate: Basic realm=\"Fake Realm\" < <html> <head><title>401 Authorization Required</title></head> <body bgcolor=\"white\"> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx/1.11.3</center> </body> </html> * Connection #0 to host 172.17.4.99 left intact","title":"Test 1: no username/password (expect code 401)"},{"location":"examples/auth/external-auth/#test-2-valid-usernamepassword-expect-code-200","text":"$ curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' -u 'user:passwd' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) * Server auth using Basic with user 'user' > GET / HTTP/1.1 > Host: external-auth-01.sample.com > Authorization: Basic dXNlcjpwYXNzd2Q= > User-Agent: curl/7.50.1 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.11.3 < Date: Mon, 03 Oct 2016 14:52:50 GMT < Content-Type: text/plain < Transfer-Encoding: chunked < Connection: keep-alive < CLIENT VALUES: client_address=10.2.60.2 command=GET real path=/ query=nil request_version=1.1 request_uri=http://external-auth-01.sample.com:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* authorization=Basic dXNlcjpwYXNzd2Q= connection=close host=external-auth-01.sample.com user-agent=curl/7.50.1 x-forwarded-for=10.2.60.1 x-forwarded-host=external-auth-01.sample.com x-forwarded-port=80 x-forwarded-proto=http x-real-ip=10.2.60.1 BODY: * Connection #0 to host 172.17.4.99 left intact -no body in request-","title":"Test 2: valid username/password (expect code 200)"},{"location":"examples/auth/external-auth/#test-3-invalid-usernamepassword-expect-code-401","text":"curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' -u 'user:user' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) * Server auth using Basic with user 'user' > GET / HTTP/1.1 > Host: external-auth-01.sample.com > Authorization: Basic dXNlcjp1c2Vy > User-Agent: curl/7.50.1 > Accept: */* > < HTTP/1.1 401 Unauthorized < Server: nginx/1.11.3 < Date: Mon, 03 Oct 2016 14:53:04 GMT < Content-Type: text/html < Content-Length: 195 < Connection: keep-alive * Authentication problem. Ignoring this. < WWW-Authenticate: Basic realm=\"Fake Realm\" < <html> <head><title>401 Authorization Required</title></head> <body bgcolor=\"white\"> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx/1.11.3</center> </body> </html> * Connection #0 to host 172.17.4.99 left intact","title":"Test 3: invalid username/password (expect code 401)"},{"location":"examples/auth/oauth-external-auth/","text":"External OAUTH Authentication \u00b6 Overview \u00b6 The auth-url and auth-signin annotations allow you to use an external authentication provider to protect your Ingress resources. Important This annotation requires ingress-nginx-controller v0.9.0 or greater. Key Detail \u00b6 This functionality is enabled by deploying multiple Ingress objects for a single host. One Ingress object has no special annotations and handles authentication. Other Ingress objects can then be annotated in such a way that require the user to authenticate against the first Ingress's endpoint, and can redirect 401 s to the same endpoint. Sample: ... metadata : name : application annotations : nginx.ingress.kubernetes.io/auth-url : \"https://$host/oauth2/auth\" nginx.ingress.kubernetes.io/auth-signin : \"https://$host/oauth2/start?rd=$escaped_request_uri\" ... Example: OAuth2 Proxy + Kubernetes-Dashboard \u00b6 This example will show you how to deploy oauth2_proxy into a Kubernetes cluster and use it to protect the Kubernetes Dashboard using GitHub as the OAuth2 provider. Prepare \u00b6 Install the kubernetes dashboard kubectl create -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/kubernetes-dashboard/v1.10.1.yaml Create a custom GitHub OAuth application Homepage URL is the FQDN in the Ingress rule, like https://foo.bar.com Authorization callback URL is the same as the base FQDN plus /oauth2/callback , like https://foo.bar.com/oauth2/callback Configure oauth2_proxy values in the file oauth2-proxy.yaml with the values: OAUTH2_PROXY_CLIENT_ID with the github <Client ID> OAUTH2_PROXY_CLIENT_SECRET with the github <Client Secret> OAUTH2_PROXY_COOKIE_SECRET with value of python -c 'import os,base64; print(base64.b64encode(os.urandom(16)).decode(\"ascii\"))' Customize the contents of the file dashboard-ingress.yaml : Replace __INGRESS_HOST__ with a valid FQDN and __INGRESS_SECRET__ with a Secret with a valid SSL certificate. Deploy the oauth2 proxy and the ingress rules running: $ kubectl create -f oauth2-proxy.yaml,dashboard-ingress.yaml Test \u00b6 Test the oauth integration accessing the configured URL, e.g. https://foo.bar.com","title":"External OAUTH Authentication"},{"location":"examples/auth/oauth-external-auth/#external-oauth-authentication","text":"","title":"External OAUTH Authentication"},{"location":"examples/auth/oauth-external-auth/#overview","text":"The auth-url and auth-signin annotations allow you to use an external authentication provider to protect your Ingress resources. Important This annotation requires ingress-nginx-controller v0.9.0 or greater.","title":"Overview"},{"location":"examples/auth/oauth-external-auth/#key-detail","text":"This functionality is enabled by deploying multiple Ingress objects for a single host. One Ingress object has no special annotations and handles authentication. Other Ingress objects can then be annotated in such a way that require the user to authenticate against the first Ingress's endpoint, and can redirect 401 s to the same endpoint. Sample: ... metadata : name : application annotations : nginx.ingress.kubernetes.io/auth-url : \"https://$host/oauth2/auth\" nginx.ingress.kubernetes.io/auth-signin : \"https://$host/oauth2/start?rd=$escaped_request_uri\" ...","title":"Key Detail"},{"location":"examples/auth/oauth-external-auth/#example-oauth2-proxy-kubernetes-dashboard","text":"This example will show you how to deploy oauth2_proxy into a Kubernetes cluster and use it to protect the Kubernetes Dashboard using GitHub as the OAuth2 provider.","title":"Example: OAuth2 Proxy + Kubernetes-Dashboard"},{"location":"examples/auth/oauth-external-auth/#prepare","text":"Install the kubernetes dashboard kubectl create -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/kubernetes-dashboard/v1.10.1.yaml Create a custom GitHub OAuth application Homepage URL is the FQDN in the Ingress rule, like https://foo.bar.com Authorization callback URL is the same as the base FQDN plus /oauth2/callback , like https://foo.bar.com/oauth2/callback Configure oauth2_proxy values in the file oauth2-proxy.yaml with the values: OAUTH2_PROXY_CLIENT_ID with the github <Client ID> OAUTH2_PROXY_CLIENT_SECRET with the github <Client Secret> OAUTH2_PROXY_COOKIE_SECRET with value of python -c 'import os,base64; print(base64.b64encode(os.urandom(16)).decode(\"ascii\"))' Customize the contents of the file dashboard-ingress.yaml : Replace __INGRESS_HOST__ with a valid FQDN and __INGRESS_SECRET__ with a Secret with a valid SSL certificate. Deploy the oauth2 proxy and the ingress rules running: $ kubectl create -f oauth2-proxy.yaml,dashboard-ingress.yaml","title":"Prepare"},{"location":"examples/auth/oauth-external-auth/#test","text":"Test the oauth integration accessing the configured URL, e.g. https://foo.bar.com","title":"Test"},{"location":"examples/customization/configuration-snippets/","text":"Configuration Snippets \u00b6 Ingress \u00b6 The Ingress in this example adds a custom header to Nginx configuration that only applies to that specific Ingress. If you want to add headers that apply globally to all Ingresses, please have a look at an example of specifying custom headers . kubectl apply -f ingress.yaml Test \u00b6 Check if the contents of the annotation are present in the nginx.conf file using: kubectl exec ingress-nginx-controller-873061567-4n3k2 -n kube-system -- cat /etc/nginx/nginx.conf","title":"Configuration Snippets"},{"location":"examples/customization/configuration-snippets/#configuration-snippets","text":"","title":"Configuration Snippets"},{"location":"examples/customization/configuration-snippets/#ingress","text":"The Ingress in this example adds a custom header to Nginx configuration that only applies to that specific Ingress. If you want to add headers that apply globally to all Ingresses, please have a look at an example of specifying custom headers . kubectl apply -f ingress.yaml","title":"Ingress"},{"location":"examples/customization/configuration-snippets/#test","text":"Check if the contents of the annotation are present in the nginx.conf file using: kubectl exec ingress-nginx-controller-873061567-4n3k2 -n kube-system -- cat /etc/nginx/nginx.conf","title":"Test"},{"location":"examples/customization/custom-configuration/","text":"Custom Configuration \u00b6 Using a ConfigMap is possible to customize the NGINX configuration For example, if we want to change the timeouts we need to create a ConfigMap: $ cat configmap.yaml apiVersion: v1 data: proxy-connect-timeout: \"10\" proxy-read-timeout: \"120\" proxy-send-timeout: \"120\" kind: ConfigMap metadata: name: ingress-nginx-controller curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/custom-configuration/configmap.yaml \\ | kubectl apply -f - If the Configmap is updated, NGINX will be reloaded with the new configuration.","title":"Custom Configuration"},{"location":"examples/customization/custom-configuration/#custom-configuration","text":"Using a ConfigMap is possible to customize the NGINX configuration For example, if we want to change the timeouts we need to create a ConfigMap: $ cat configmap.yaml apiVersion: v1 data: proxy-connect-timeout: \"10\" proxy-read-timeout: \"120\" proxy-send-timeout: \"120\" kind: ConfigMap metadata: name: ingress-nginx-controller curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/custom-configuration/configmap.yaml \\ | kubectl apply -f - If the Configmap is updated, NGINX will be reloaded with the new configuration.","title":"Custom Configuration"},{"location":"examples/customization/custom-errors/","text":"Custom Errors \u00b6 This example demonstrates how to use a custom backend to render custom error pages. If you are using Helm Chart, look at example values and don't forget to add configMap to your deployment, otherwise continue with Customized default backend manual deployment. Customized default backend \u00b6 First, create the custom default-backend . It will be used by the Ingress controller later on. To do that, you can take a look at the example manifest in this project's GitHub repository. $ kubectl create -f custom-default-backend.yaml service \"nginx-errors\" created deployment.apps \"nginx-errors\" created This should have created a Deployment and a Service with the name nginx-errors . $ kubectl get deploy,svc NAME DESIRED CURRENT READY AGE deployment.apps/nginx-errors 1 1 1 10s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nginx-errors ClusterIP 10.0.0.12 <none> 80/TCP 10s Ingress controller configuration \u00b6 If you do not already have an instance of the NGINX Ingress controller running, deploy it according to the deployment guide , then follow these steps: Edit the ingress-nginx-controller Deployment and set the value of the --default-backend-service flag to the name of the newly created error backend. Edit the ingress-nginx-controller ConfigMap and create the key custom-http-errors with a value of 404,503 . Take note of the IP address assigned to the NGINX Ingress controller Service. $ kubectl get svc ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx ClusterIP 10.0.0.13 <none> 80/TCP,443/TCP 10m Note The ingress-nginx Service is of type ClusterIP in this example. This may vary depending on your environment. Make sure you can use the Service to reach NGINX before proceeding with the rest of this example. Testing error pages \u00b6 Let us send a couple of HTTP requests using cURL and validate everything is working as expected. A request to the default backend returns a 404 error with a custom message: $ curl -D- http://10.0.0.13/ HTTP/1.1 404 Not Found Server: nginx/1.13.12 Date: Tue, 12 Jun 2018 19:11:24 GMT Content-Type: */* Transfer-Encoding: chunked Connection: keep-alive <span>The page you're looking for could not be found.</span> A request with a custom Accept header returns the corresponding document type (JSON): $ curl -D- -H 'Accept: application/json' http://10.0.0.13/ HTTP/1.1 404 Not Found Server: nginx/1.13.12 Date: Tue, 12 Jun 2018 19:12:36 GMT Content-Type: application/json Transfer-Encoding: chunked Connection: keep-alive Vary: Accept-Encoding { \"message\": \"The page you're looking for could not be found\" } To go further with this example, feel free to deploy your own applications and Ingress objects, and validate that the responses are still in the correct format when a backend returns 503 (eg. if you scale a Deployment down to 0 replica).","title":"Custom Errors"},{"location":"examples/customization/custom-errors/#custom-errors","text":"This example demonstrates how to use a custom backend to render custom error pages. If you are using Helm Chart, look at example values and don't forget to add configMap to your deployment, otherwise continue with Customized default backend manual deployment.","title":"Custom Errors"},{"location":"examples/customization/custom-errors/#customized-default-backend","text":"First, create the custom default-backend . It will be used by the Ingress controller later on. To do that, you can take a look at the example manifest in this project's GitHub repository. $ kubectl create -f custom-default-backend.yaml service \"nginx-errors\" created deployment.apps \"nginx-errors\" created This should have created a Deployment and a Service with the name nginx-errors . $ kubectl get deploy,svc NAME DESIRED CURRENT READY AGE deployment.apps/nginx-errors 1 1 1 10s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nginx-errors ClusterIP 10.0.0.12 <none> 80/TCP 10s","title":"Customized default backend"},{"location":"examples/customization/custom-errors/#ingress-controller-configuration","text":"If you do not already have an instance of the NGINX Ingress controller running, deploy it according to the deployment guide , then follow these steps: Edit the ingress-nginx-controller Deployment and set the value of the --default-backend-service flag to the name of the newly created error backend. Edit the ingress-nginx-controller ConfigMap and create the key custom-http-errors with a value of 404,503 . Take note of the IP address assigned to the NGINX Ingress controller Service. $ kubectl get svc ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx ClusterIP 10.0.0.13 <none> 80/TCP,443/TCP 10m Note The ingress-nginx Service is of type ClusterIP in this example. This may vary depending on your environment. Make sure you can use the Service to reach NGINX before proceeding with the rest of this example.","title":"Ingress controller configuration"},{"location":"examples/customization/custom-errors/#testing-error-pages","text":"Let us send a couple of HTTP requests using cURL and validate everything is working as expected. A request to the default backend returns a 404 error with a custom message: $ curl -D- http://10.0.0.13/ HTTP/1.1 404 Not Found Server: nginx/1.13.12 Date: Tue, 12 Jun 2018 19:11:24 GMT Content-Type: */* Transfer-Encoding: chunked Connection: keep-alive <span>The page you're looking for could not be found.</span> A request with a custom Accept header returns the corresponding document type (JSON): $ curl -D- -H 'Accept: application/json' http://10.0.0.13/ HTTP/1.1 404 Not Found Server: nginx/1.13.12 Date: Tue, 12 Jun 2018 19:12:36 GMT Content-Type: application/json Transfer-Encoding: chunked Connection: keep-alive Vary: Accept-Encoding { \"message\": \"The page you're looking for could not be found\" } To go further with this example, feel free to deploy your own applications and Ingress objects, and validate that the responses are still in the correct format when a backend returns 503 (eg. if you scale a Deployment down to 0 replica).","title":"Testing error pages"},{"location":"examples/customization/custom-headers/","text":"Custom Headers \u00b6 Caveats \u00b6 Changes to the custom header config maps do not force a reload of the ingress-nginx-controllers. Workaround \u00b6 To work around this limitation, perform a rolling restart of the deployment. Example \u00b6 This example demonstrates configuration of the nginx ingress controller via a ConfigMap to pass a custom list of headers to the upstream server. custom-headers.yaml defines a ConfigMap in the ingress-nginx namespace named custom-headers , holding several custom X-prefixed HTTP headers. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/custom-headers/custom-headers.yaml configmap.yaml defines a ConfigMap in the ingress-nginx namespace named ingress-nginx-controller . This controls the global configuration of the ingress controller, and already exists in a standard installation. The key proxy-set-headers is set to cite the previously-created ingress-nginx/custom-headers ConfigMap. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/custom-headers/configmap.yaml The nginx ingress controller will read the ingress-nginx/ingress-nginx-controller ConfigMap, find the proxy-set-headers key, read HTTP headers from the ingress-nginx/custom-headers ConfigMap, and include those HTTP headers in all requests flowing from nginx to the backends. The above example was for passing a custom list of headers to the upstream server. To pass the custom headers before sending response traffic to the client, use the add-headers key: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/custom-headers/configmap-client-response.yaml Test \u00b6 Check the contents of the ConfigMaps are present in the nginx.conf file using: kubectl exec ingress-nginx-controller-873061567-4n3k2 -n ingress-nginx -- cat /etc/nginx/nginx.conf","title":"Custom Headers"},{"location":"examples/customization/custom-headers/#custom-headers","text":"","title":"Custom Headers"},{"location":"examples/customization/custom-headers/#caveats","text":"Changes to the custom header config maps do not force a reload of the ingress-nginx-controllers.","title":"Caveats"},{"location":"examples/customization/custom-headers/#workaround","text":"To work around this limitation, perform a rolling restart of the deployment.","title":"Workaround"},{"location":"examples/customization/custom-headers/#example","text":"This example demonstrates configuration of the nginx ingress controller via a ConfigMap to pass a custom list of headers to the upstream server. custom-headers.yaml defines a ConfigMap in the ingress-nginx namespace named custom-headers , holding several custom X-prefixed HTTP headers. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/custom-headers/custom-headers.yaml configmap.yaml defines a ConfigMap in the ingress-nginx namespace named ingress-nginx-controller . This controls the global configuration of the ingress controller, and already exists in a standard installation. The key proxy-set-headers is set to cite the previously-created ingress-nginx/custom-headers ConfigMap. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/custom-headers/configmap.yaml The nginx ingress controller will read the ingress-nginx/ingress-nginx-controller ConfigMap, find the proxy-set-headers key, read HTTP headers from the ingress-nginx/custom-headers ConfigMap, and include those HTTP headers in all requests flowing from nginx to the backends. The above example was for passing a custom list of headers to the upstream server. To pass the custom headers before sending response traffic to the client, use the add-headers key: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/custom-headers/configmap-client-response.yaml","title":"Example"},{"location":"examples/customization/custom-headers/#test","text":"Check the contents of the ConfigMaps are present in the nginx.conf file using: kubectl exec ingress-nginx-controller-873061567-4n3k2 -n ingress-nginx -- cat /etc/nginx/nginx.conf","title":"Test"},{"location":"examples/customization/external-auth-headers/","text":"External authentication, authentication service response headers propagation \u00b6 This example demonstrates propagation of selected authentication service response headers to a backend service. Sample configuration includes: Sample authentication service producing several response headers Authentication logic is based on HTTP header: requests with header User containing string internal are considered authenticated After successful authentication service generates response headers UserID and UserRole Sample echo service displaying header information Two ingress objects pointing to echo service Public, which allows access from unauthenticated users Private, which allows access from authenticated users only You can deploy the controller as follows: $ kubectl create -f deploy/ deployment \"demo-auth-service\" created service \"demo-auth-service\" created ingress \"demo-auth-service\" created deployment \"demo-echo-service\" created service \"demo-echo-service\" created ingress \"public-demo-echo-service\" created ingress \"secure-demo-echo-service\" created $ kubectl get po NAME READY STATUS RESTARTS AGE demo-auth-service-2769076528-7g9mh 1/1 Running 0 30s demo-echo-service-3636052215-3vw8c 1/1 Running 0 29s kubectl get ing NAME HOSTS ADDRESS PORTS AGE public-demo-echo-service public-demo-echo-service.kube.local 80 1m secure-demo-echo-service secure-demo-echo-service.kube.local 80 1m Test 1: public service with no auth header \u00b6 $ curl -H 'Host: public-demo-echo-service.kube.local' -v 192 .168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) > GET / HTTP/1.1 > Host: public-demo-echo-service.kube.local > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.11.10 < Date: Mon, 13 Mar 2017 20:19:21 GMT < Content-Type: text/plain; charset=utf-8 < Content-Length: 20 < Connection: keep-alive < * Connection #0 to host 192.168.99.100 left intact UserID: , UserRole: Test 2: secure service with no auth header \u00b6 $ curl -H 'Host: secure-demo-echo-service.kube.local' -v 192 .168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) > GET / HTTP/1.1 > Host: secure-demo-echo-service.kube.local > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 403 Forbidden < Server: nginx/1.11.10 < Date: Mon, 13 Mar 2017 20:18:48 GMT < Content-Type: text/html < Content-Length: 170 < Connection: keep-alive < <html> <head><title>403 Forbidden</title></head> <body bgcolor=\"white\"> <center><h1>403 Forbidden</h1></center> <hr><center>nginx/1.11.10</center> </body> </html> * Connection #0 to host 192.168.99.100 left intact Test 3: public service with valid auth header \u00b6 $ curl -H 'Host: public-demo-echo-service.kube.local' -H 'User:internal' -v 192 .168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) > GET / HTTP/1.1 > Host: public-demo-echo-service.kube.local > User-Agent: curl/7.43.0 > Accept: */* > User:internal > < HTTP/1.1 200 OK < Server: nginx/1.11.10 < Date: Mon, 13 Mar 2017 20:19:59 GMT < Content-Type: text/plain; charset=utf-8 < Content-Length: 44 < Connection: keep-alive < * Connection #0 to host 192.168.99.100 left intact UserID: 1443635317331776148, UserRole: admin Test 4: secure service with valid auth header \u00b6 $ curl -H 'Host: secure-demo-echo-service.kube.local' -H 'User:internal' -v 192 .168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) > GET / HTTP/1.1 > Host: secure-demo-echo-service.kube.local > User-Agent: curl/7.43.0 > Accept: */* > User:internal > < HTTP/1.1 200 OK < Server: nginx/1.11.10 < Date: Mon, 13 Mar 2017 20:17:23 GMT < Content-Type: text/plain; charset=utf-8 < Content-Length: 43 < Connection: keep-alive < * Connection #0 to host 192.168.99.100 left intact UserID: 605394647632969758, UserRole: admin","title":"External authentication"},{"location":"examples/customization/external-auth-headers/#external-authentication-authentication-service-response-headers-propagation","text":"This example demonstrates propagation of selected authentication service response headers to a backend service. Sample configuration includes: Sample authentication service producing several response headers Authentication logic is based on HTTP header: requests with header User containing string internal are considered authenticated After successful authentication service generates response headers UserID and UserRole Sample echo service displaying header information Two ingress objects pointing to echo service Public, which allows access from unauthenticated users Private, which allows access from authenticated users only You can deploy the controller as follows: $ kubectl create -f deploy/ deployment \"demo-auth-service\" created service \"demo-auth-service\" created ingress \"demo-auth-service\" created deployment \"demo-echo-service\" created service \"demo-echo-service\" created ingress \"public-demo-echo-service\" created ingress \"secure-demo-echo-service\" created $ kubectl get po NAME READY STATUS RESTARTS AGE demo-auth-service-2769076528-7g9mh 1/1 Running 0 30s demo-echo-service-3636052215-3vw8c 1/1 Running 0 29s kubectl get ing NAME HOSTS ADDRESS PORTS AGE public-demo-echo-service public-demo-echo-service.kube.local 80 1m secure-demo-echo-service secure-demo-echo-service.kube.local 80 1m","title":"External authentication, authentication service response headers propagation"},{"location":"examples/customization/external-auth-headers/#test-1-public-service-with-no-auth-header","text":"$ curl -H 'Host: public-demo-echo-service.kube.local' -v 192 .168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) > GET / HTTP/1.1 > Host: public-demo-echo-service.kube.local > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.11.10 < Date: Mon, 13 Mar 2017 20:19:21 GMT < Content-Type: text/plain; charset=utf-8 < Content-Length: 20 < Connection: keep-alive < * Connection #0 to host 192.168.99.100 left intact UserID: , UserRole:","title":"Test 1: public service with no auth header"},{"location":"examples/customization/external-auth-headers/#test-2-secure-service-with-no-auth-header","text":"$ curl -H 'Host: secure-demo-echo-service.kube.local' -v 192 .168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) > GET / HTTP/1.1 > Host: secure-demo-echo-service.kube.local > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 403 Forbidden < Server: nginx/1.11.10 < Date: Mon, 13 Mar 2017 20:18:48 GMT < Content-Type: text/html < Content-Length: 170 < Connection: keep-alive < <html> <head><title>403 Forbidden</title></head> <body bgcolor=\"white\"> <center><h1>403 Forbidden</h1></center> <hr><center>nginx/1.11.10</center> </body> </html> * Connection #0 to host 192.168.99.100 left intact","title":"Test 2: secure service with no auth header"},{"location":"examples/customization/external-auth-headers/#test-3-public-service-with-valid-auth-header","text":"$ curl -H 'Host: public-demo-echo-service.kube.local' -H 'User:internal' -v 192 .168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) > GET / HTTP/1.1 > Host: public-demo-echo-service.kube.local > User-Agent: curl/7.43.0 > Accept: */* > User:internal > < HTTP/1.1 200 OK < Server: nginx/1.11.10 < Date: Mon, 13 Mar 2017 20:19:59 GMT < Content-Type: text/plain; charset=utf-8 < Content-Length: 44 < Connection: keep-alive < * Connection #0 to host 192.168.99.100 left intact UserID: 1443635317331776148, UserRole: admin","title":"Test 3: public service with valid auth header"},{"location":"examples/customization/external-auth-headers/#test-4-secure-service-with-valid-auth-header","text":"$ curl -H 'Host: secure-demo-echo-service.kube.local' -H 'User:internal' -v 192 .168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) > GET / HTTP/1.1 > Host: secure-demo-echo-service.kube.local > User-Agent: curl/7.43.0 > Accept: */* > User:internal > < HTTP/1.1 200 OK < Server: nginx/1.11.10 < Date: Mon, 13 Mar 2017 20:17:23 GMT < Content-Type: text/plain; charset=utf-8 < Content-Length: 43 < Connection: keep-alive < * Connection #0 to host 192.168.99.100 left intact UserID: 605394647632969758, UserRole: admin","title":"Test 4: secure service with valid auth header"},{"location":"examples/customization/jwt/","text":"Accommodation for JWT \u00b6 JWT (short for Json Web Token) is an authentication method widely used. Basically an authentication server generates a JWT and you then use this token in every request you make to a backend service. The JWT can be quite big and is present in every http headers. This means you may have to adapt the max-header size of your nginx-ingress in order to support it. Symptoms \u00b6 If you use JWT and you get http 502 error from your ingress, it may be a sign that the buffer size is not big enough. To be 100% sure look at the logs of the ingress-nginx-controller pod, you should see something like this: upstream sent too big header while reading response header from upstream... Increase buffer size for headers \u00b6 In nginx, we want to modify the property proxy-buffer-size . The size is arbitrary. It depends on your needs. Be aware that a high value can lower the performance of your ingress proxy. In general a value of 16k should get you covered. Using helm \u00b6 If you're using helm you can simply use the config properties . # -- Will add custom configuration options to Nginx https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/ config : proxy-buffer-size : 16k Manually in kubernetes config files \u00b6 If you use an already generated config from for a provider, you will have to change the controller-configmap.yaml --- # Source: ingress-nginx/templates/controller-configmap.yaml apiVersion : v1 kind : ConfigMap # ... data : #... proxy-buffer-size : \"16k\" References: * Custom Configuration","title":"Accommodation for JWT"},{"location":"examples/customization/jwt/#accommodation-for-jwt","text":"JWT (short for Json Web Token) is an authentication method widely used. Basically an authentication server generates a JWT and you then use this token in every request you make to a backend service. The JWT can be quite big and is present in every http headers. This means you may have to adapt the max-header size of your nginx-ingress in order to support it.","title":"Accommodation for JWT"},{"location":"examples/customization/jwt/#symptoms","text":"If you use JWT and you get http 502 error from your ingress, it may be a sign that the buffer size is not big enough. To be 100% sure look at the logs of the ingress-nginx-controller pod, you should see something like this: upstream sent too big header while reading response header from upstream...","title":"Symptoms"},{"location":"examples/customization/jwt/#increase-buffer-size-for-headers","text":"In nginx, we want to modify the property proxy-buffer-size . The size is arbitrary. It depends on your needs. Be aware that a high value can lower the performance of your ingress proxy. In general a value of 16k should get you covered.","title":"Increase buffer size for headers"},{"location":"examples/customization/jwt/#using-helm","text":"If you're using helm you can simply use the config properties . # -- Will add custom configuration options to Nginx https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/ config : proxy-buffer-size : 16k","title":"Using helm"},{"location":"examples/customization/jwt/#manually-in-kubernetes-config-files","text":"If you use an already generated config from for a provider, you will have to change the controller-configmap.yaml --- # Source: ingress-nginx/templates/controller-configmap.yaml apiVersion : v1 kind : ConfigMap # ... data : #... proxy-buffer-size : \"16k\" References: * Custom Configuration","title":"Manually in kubernetes config files"},{"location":"examples/customization/ssl-dh-param/","text":"Custom DH parameters for perfect forward secrecy \u00b6 This example aims to demonstrate the deployment of an nginx ingress controller and use a ConfigMap to configure a custom Diffie-Hellman parameters file to help with \"Perfect Forward Secrecy\". Custom configuration \u00b6 $ cat configmap.yaml apiVersion: v1 data: ssl-dh-param: \"ingress-nginx/lb-dhparam\" kind: ConfigMap metadata: name: ingress-nginx-controller namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx $ kubectl create -f configmap.yaml Custom DH parameters secret \u00b6 $ openssl dhparam 4096 2 > /dev/null | base64 LS0tLS1CRUdJTiBESCBQQVJBTUVURVJ... $ cat ssl-dh-param.yaml apiVersion: v1 data: dhparam.pem: \"LS0tLS1CRUdJTiBESCBQQVJBTUVURVJ...\" kind: Secret metadata: name: lb-dhparam namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx $ kubectl create -f ssl-dh-param.yaml Test \u00b6 Check the contents of the configmap is present in the nginx.conf file using: $ kubectl exec ingress-nginx-controller-873061567-4n3k2 -n kube-system -- cat /etc/nginx/nginx.conf","title":"Custom DH parameters for perfect forward secrecy"},{"location":"examples/customization/ssl-dh-param/#custom-dh-parameters-for-perfect-forward-secrecy","text":"This example aims to demonstrate the deployment of an nginx ingress controller and use a ConfigMap to configure a custom Diffie-Hellman parameters file to help with \"Perfect Forward Secrecy\".","title":"Custom DH parameters for perfect forward secrecy"},{"location":"examples/customization/ssl-dh-param/#custom-configuration","text":"$ cat configmap.yaml apiVersion: v1 data: ssl-dh-param: \"ingress-nginx/lb-dhparam\" kind: ConfigMap metadata: name: ingress-nginx-controller namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx $ kubectl create -f configmap.yaml","title":"Custom configuration"},{"location":"examples/customization/ssl-dh-param/#custom-dh-parameters-secret","text":"$ openssl dhparam 4096 2 > /dev/null | base64 LS0tLS1CRUdJTiBESCBQQVJBTUVURVJ... $ cat ssl-dh-param.yaml apiVersion: v1 data: dhparam.pem: \"LS0tLS1CRUdJTiBESCBQQVJBTUVURVJ...\" kind: Secret metadata: name: lb-dhparam namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx $ kubectl create -f ssl-dh-param.yaml","title":"Custom DH parameters secret"},{"location":"examples/customization/ssl-dh-param/#test","text":"Check the contents of the configmap is present in the nginx.conf file using: $ kubectl exec ingress-nginx-controller-873061567-4n3k2 -n kube-system -- cat /etc/nginx/nginx.conf","title":"Test"},{"location":"examples/customization/sysctl/","text":"Sysctl tuning \u00b6 This example aims to demonstrate the use of an Init Container to adjust sysctl default values using kubectl patch . kubectl patch deployment -n ingress-nginx ingress-nginx-controller \\ --patch=\"$(curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/sysctl/patch.json)\" Changes: Backlog Queue setting net.core.somaxconn from 128 to 32768 Ephemeral Ports setting net.ipv4.ip_local_port_range from 32768 60999 to 1024 65000 In a post from the NGINX blog , it is possible to see an explanation for the changes.","title":"Sysctl tuning"},{"location":"examples/customization/sysctl/#sysctl-tuning","text":"This example aims to demonstrate the use of an Init Container to adjust sysctl default values using kubectl patch . kubectl patch deployment -n ingress-nginx ingress-nginx-controller \\ --patch=\"$(curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/sysctl/patch.json)\" Changes: Backlog Queue setting net.core.somaxconn from 128 to 32768 Ephemeral Ports setting net.ipv4.ip_local_port_range from 32768 60999 to 1024 65000 In a post from the NGINX blog , it is possible to see an explanation for the changes.","title":"Sysctl tuning"},{"location":"examples/docker-registry/","text":"Docker registry \u00b6 This example demonstrates how to deploy a docker registry in the cluster and configure Ingress to enable access from the Internet. Deployment \u00b6 First we deploy the docker registry in the cluster: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/docker-registry/deployment.yaml Important DO NOT RUN THIS IN PRODUCTION This deployment uses emptyDir in the volumeMount which means the contents of the registry will be deleted when the pod dies. The next required step is creation of the ingress rules. To do this we have two options: with and without TLS Without TLS \u00b6 Download and edit the yaml deployment replacing registry.<your domain> with a valid DNS name pointing to the ingress controller: wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/docker-registry/ingress-without-tls.yaml Important Running a docker registry without TLS requires we configure our local docker daemon with the insecure registry flag. Please check deploy a plain http registry With TLS \u00b6 Download and edit the yaml deployment replacing registry.<your domain> with a valid DNS name pointing to the ingress controller: wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/docker-registry/ingress-with-tls.yaml Deploy kube lego use Let's Encrypt certificates or edit the ingress rule to use a secret with an existing SSL certificate. Testing \u00b6 To test the registry is working correctly we download a known image from docker hub , create a tag pointing to the new registry and upload the image: docker pull ubuntu:16.04 docker tag ubuntu:16.04 `registry.<your domain>/ubuntu:16.04` docker push `registry.<your domain>/ubuntu:16.04` Please replace registry.<your domain> with your domain.","title":"Docker registry"},{"location":"examples/docker-registry/#docker-registry","text":"This example demonstrates how to deploy a docker registry in the cluster and configure Ingress to enable access from the Internet.","title":"Docker registry"},{"location":"examples/docker-registry/#deployment","text":"First we deploy the docker registry in the cluster: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/docker-registry/deployment.yaml Important DO NOT RUN THIS IN PRODUCTION This deployment uses emptyDir in the volumeMount which means the contents of the registry will be deleted when the pod dies. The next required step is creation of the ingress rules. To do this we have two options: with and without TLS","title":"Deployment"},{"location":"examples/docker-registry/#without-tls","text":"Download and edit the yaml deployment replacing registry.<your domain> with a valid DNS name pointing to the ingress controller: wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/docker-registry/ingress-without-tls.yaml Important Running a docker registry without TLS requires we configure our local docker daemon with the insecure registry flag. Please check deploy a plain http registry","title":"Without TLS"},{"location":"examples/docker-registry/#with-tls","text":"Download and edit the yaml deployment replacing registry.<your domain> with a valid DNS name pointing to the ingress controller: wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/docker-registry/ingress-with-tls.yaml Deploy kube lego use Let's Encrypt certificates or edit the ingress rule to use a secret with an existing SSL certificate.","title":"With TLS"},{"location":"examples/docker-registry/#testing","text":"To test the registry is working correctly we download a known image from docker hub , create a tag pointing to the new registry and upload the image: docker pull ubuntu:16.04 docker tag ubuntu:16.04 `registry.<your domain>/ubuntu:16.04` docker push `registry.<your domain>/ubuntu:16.04` Please replace registry.<your domain> with your domain.","title":"Testing"},{"location":"examples/grpc/","text":"gRPC \u00b6 This example demonstrates how to route traffic to a gRPC service through the Ingress-NGINX controller. Prerequisites \u00b6 You have a kubernetes cluster running. You have a domain name such as example.com that is configured to route traffic to the Ingress-NGINX controller. You have the ingress-nginx-controller installed as per docs. You have a backend application running a gRPC server listening for TCP traffic. If you want, you can use https://github.com/grpc/grpc-go/blob/91e0aeb192456225adf27966d04ada4cf8599915/examples/features/reflection/server/main.go as an example. You're also responsible for provisioning an SSL certificate for the ingress. So you need to have a valid SSL certificate, deployed as a Kubernetes secret of type tls , in the same namespace as the gRPC application. Step 1: Create a Kubernetes Deployment for gRPC app \u00b6 Make sure your gRPC application pod is running and listening for connections. For example you can try a kubectl command like this below: $ kubectl get po -A -o wide | grep go-grpc-greeter-server If you have a gRPC app deployed in your cluster, then skip further notes in this Step 1, and continue from Step 2 below. As an example gRPC application, we can use this app https://github.com/grpc/grpc-go/blob/91e0aeb192456225adf27966d04ada4cf8599915/examples/features/reflection/server/main.go . To create a container image for this app, you can use this Dockerfile . If you use the Dockerfile mentioned above, to create a image, then you can use the following example Kubernetes manifest to create a deployment resource that uses that image. If necessary edit this manifest to suit your needs. cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: labels: app: go-grpc-greeter-server name: go-grpc-greeter-server spec: replicas: 1 selector: matchLabels: app: go-grpc-greeter-server template: metadata: labels: app: go-grpc-greeter-server spec: containers: - image: <reponame>/go-grpc-greeter-server # Edit this for your reponame resources: limits: cpu: 100m memory: 100Mi requests: cpu: 50m memory: 50Mi name: go-grpc-greeter-server ports: - containerPort: 50051 EOF Step 2: Create the Kubernetes Service for the gRPC app \u00b6 You can use the following example manifest to create a service of type ClusterIP. Edit the name/namespace/label/port to match your deployment/pod. cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Service metadata: labels: app: go-grpc-greeter-server name: go-grpc-greeter-server spec: ports: - port: 80 protocol: TCP targetPort: 50051 selector: app: go-grpc-greeter-server type: ClusterIP EOF You can save the above example manifest to a file with name service.go-grpc-greeter-server.yaml and edit it to match your deployment/pod, if required. You can create the service resource with a kubectl command like this: $ kubectl create -f service.go-grpc-greeter-server.yaml Step 3: Create the Kubernetes Ingress resource for the gRPC app \u00b6 Use the following example manifest of a ingress resource to create a ingress for your grpc app. If required, edit it to match your app's details like name, namespace, service, secret etc. Make sure you have the required SSL-Certificate, existing in your Kubernetes cluster in the same namespace where the gRPC app is. The certificate must be available as a kubernetes secret resource, of type \"kubernetes.io/tls\" https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets. This is because we are terminating TLS on the ingress. cat <<EOF | kubectl apply -f - apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/ssl-redirect: \"true\" nginx.ingress.kubernetes.io/backend-protocol: \"GRPC\" name: fortune-ingress namespace: default spec: ingressClassName: nginx rules: - host: grpctest.dev.mydomain.com http: paths: - path: / pathType: Prefix backend: service: name: go-grpc-greeter-server port: number: 80 tls: # This secret must exist beforehand # The cert must also contain the subj-name grpctest.dev.mydomain.com # https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/PREREQUISITES.md#tls-certificates - secretName: wildcard.dev.mydomain.com hosts: - grpctest.dev.mydomain.com EOF If you save the above example manifest as a file named ingress.go-grpc-greeter-server.yaml and edit it to match your deployment and service, you can create the ingress like this: $ kubectl create -f ingress.go-grpc-greeter-server.yaml The takeaway is that we are not doing any TLS configuration on the server (as we are terminating TLS at the ingress level, gRPC traffic will travel unencrypted inside the cluster and arrive \"insecure\"). For your own application you may or may not want to do this. If you prefer to forward encrypted traffic to your POD and terminate TLS at the gRPC server itself, add the ingress annotation nginx.ingress.kubernetes.io/backend-protocol: \"GRPCS\" . A few more things to note: We've tagged the ingress with the annotation nginx.ingress.kubernetes.io/backend-protocol: \"GRPC\" . This is the magic ingredient that sets up the appropriate nginx configuration to route http/2 traffic to our service. We're terminating TLS at the ingress and have configured an SSL certificate wildcard.dev.mydomain.com . The ingress matches traffic arriving as https://grpctest.dev.mydomain.com:443 and routes unencrypted messages to the backend Kubernetes service. Step 4: test the connection \u00b6 Once we've applied our configuration to Kubernetes, it's time to test that we can actually talk to the backend. To do this, we'll use the grpcurl utility: $ grpcurl grpctest.dev.mydomain.com:443 helloworld.Greeter/SayHello { \"message\": \"Hello \" } Debugging Hints \u00b6 Obviously, watch the logs on your app. Watch the logs for the ingress-nginx-controller (increasing verbosity as needed). Double-check your address and ports. Set the GODEBUG=http2debug=2 environment variable to get detailed http/2 logging on the client and/or server. Study RFC 7540 (http/2) https://tools.ietf.org/html/rfc7540 . If you are developing public gRPC endpoints, check out https://proto.stack.build, a protocol buffer / gRPC build service that can use to help make it easier for your users to consume your API. See also the specific gRPC settings of NGINX: https://nginx.org/en/docs/http/ngx_http_grpc_module.html Notes on using response/request streams \u00b6 If your server only does response streaming and you expect a stream to be open longer than 60 seconds, you will have to change the grpc_read_timeout to accommodate this. If your service only does request streaming and you expect a stream to be open longer than 60 seconds, you have to change the grpc_send_timeout and the client_body_timeout . If you do both response and request streaming with an open stream longer than 60 seconds, you have to change all three timeouts: grpc_read_timeout , grpc_send_timeout and client_body_timeout . Values for the timeouts must be specified as e.g. \"1200s\" . On the most recent versions of ingress-nginx, changing these timeouts requires using the nginx.ingress.kubernetes.io/server-snippet annotation. There are plans for future releases to allow using the Kubernetes annotations to define each timeout separately.","title":"gRPC"},{"location":"examples/grpc/#grpc","text":"This example demonstrates how to route traffic to a gRPC service through the Ingress-NGINX controller.","title":"gRPC"},{"location":"examples/grpc/#prerequisites","text":"You have a kubernetes cluster running. You have a domain name such as example.com that is configured to route traffic to the Ingress-NGINX controller. You have the ingress-nginx-controller installed as per docs. You have a backend application running a gRPC server listening for TCP traffic. If you want, you can use https://github.com/grpc/grpc-go/blob/91e0aeb192456225adf27966d04ada4cf8599915/examples/features/reflection/server/main.go as an example. You're also responsible for provisioning an SSL certificate for the ingress. So you need to have a valid SSL certificate, deployed as a Kubernetes secret of type tls , in the same namespace as the gRPC application.","title":"Prerequisites"},{"location":"examples/grpc/#step-1-create-a-kubernetes-deployment-for-grpc-app","text":"Make sure your gRPC application pod is running and listening for connections. For example you can try a kubectl command like this below: $ kubectl get po -A -o wide | grep go-grpc-greeter-server If you have a gRPC app deployed in your cluster, then skip further notes in this Step 1, and continue from Step 2 below. As an example gRPC application, we can use this app https://github.com/grpc/grpc-go/blob/91e0aeb192456225adf27966d04ada4cf8599915/examples/features/reflection/server/main.go . To create a container image for this app, you can use this Dockerfile . If you use the Dockerfile mentioned above, to create a image, then you can use the following example Kubernetes manifest to create a deployment resource that uses that image. If necessary edit this manifest to suit your needs. cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: labels: app: go-grpc-greeter-server name: go-grpc-greeter-server spec: replicas: 1 selector: matchLabels: app: go-grpc-greeter-server template: metadata: labels: app: go-grpc-greeter-server spec: containers: - image: <reponame>/go-grpc-greeter-server # Edit this for your reponame resources: limits: cpu: 100m memory: 100Mi requests: cpu: 50m memory: 50Mi name: go-grpc-greeter-server ports: - containerPort: 50051 EOF","title":"Step 1: Create a Kubernetes Deployment for gRPC app"},{"location":"examples/grpc/#step-2-create-the-kubernetes-service-for-the-grpc-app","text":"You can use the following example manifest to create a service of type ClusterIP. Edit the name/namespace/label/port to match your deployment/pod. cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Service metadata: labels: app: go-grpc-greeter-server name: go-grpc-greeter-server spec: ports: - port: 80 protocol: TCP targetPort: 50051 selector: app: go-grpc-greeter-server type: ClusterIP EOF You can save the above example manifest to a file with name service.go-grpc-greeter-server.yaml and edit it to match your deployment/pod, if required. You can create the service resource with a kubectl command like this: $ kubectl create -f service.go-grpc-greeter-server.yaml","title":"Step 2: Create the Kubernetes Service for the gRPC app"},{"location":"examples/grpc/#step-3-create-the-kubernetes-ingress-resource-for-the-grpc-app","text":"Use the following example manifest of a ingress resource to create a ingress for your grpc app. If required, edit it to match your app's details like name, namespace, service, secret etc. Make sure you have the required SSL-Certificate, existing in your Kubernetes cluster in the same namespace where the gRPC app is. The certificate must be available as a kubernetes secret resource, of type \"kubernetes.io/tls\" https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets. This is because we are terminating TLS on the ingress. cat <<EOF | kubectl apply -f - apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/ssl-redirect: \"true\" nginx.ingress.kubernetes.io/backend-protocol: \"GRPC\" name: fortune-ingress namespace: default spec: ingressClassName: nginx rules: - host: grpctest.dev.mydomain.com http: paths: - path: / pathType: Prefix backend: service: name: go-grpc-greeter-server port: number: 80 tls: # This secret must exist beforehand # The cert must also contain the subj-name grpctest.dev.mydomain.com # https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/PREREQUISITES.md#tls-certificates - secretName: wildcard.dev.mydomain.com hosts: - grpctest.dev.mydomain.com EOF If you save the above example manifest as a file named ingress.go-grpc-greeter-server.yaml and edit it to match your deployment and service, you can create the ingress like this: $ kubectl create -f ingress.go-grpc-greeter-server.yaml The takeaway is that we are not doing any TLS configuration on the server (as we are terminating TLS at the ingress level, gRPC traffic will travel unencrypted inside the cluster and arrive \"insecure\"). For your own application you may or may not want to do this. If you prefer to forward encrypted traffic to your POD and terminate TLS at the gRPC server itself, add the ingress annotation nginx.ingress.kubernetes.io/backend-protocol: \"GRPCS\" . A few more things to note: We've tagged the ingress with the annotation nginx.ingress.kubernetes.io/backend-protocol: \"GRPC\" . This is the magic ingredient that sets up the appropriate nginx configuration to route http/2 traffic to our service. We're terminating TLS at the ingress and have configured an SSL certificate wildcard.dev.mydomain.com . The ingress matches traffic arriving as https://grpctest.dev.mydomain.com:443 and routes unencrypted messages to the backend Kubernetes service.","title":"Step 3: Create the Kubernetes Ingress resource for the gRPC app"},{"location":"examples/grpc/#step-4-test-the-connection","text":"Once we've applied our configuration to Kubernetes, it's time to test that we can actually talk to the backend. To do this, we'll use the grpcurl utility: $ grpcurl grpctest.dev.mydomain.com:443 helloworld.Greeter/SayHello { \"message\": \"Hello \" }","title":"Step 4: test the connection"},{"location":"examples/grpc/#debugging-hints","text":"Obviously, watch the logs on your app. Watch the logs for the ingress-nginx-controller (increasing verbosity as needed). Double-check your address and ports. Set the GODEBUG=http2debug=2 environment variable to get detailed http/2 logging on the client and/or server. Study RFC 7540 (http/2) https://tools.ietf.org/html/rfc7540 . If you are developing public gRPC endpoints, check out https://proto.stack.build, a protocol buffer / gRPC build service that can use to help make it easier for your users to consume your API. See also the specific gRPC settings of NGINX: https://nginx.org/en/docs/http/ngx_http_grpc_module.html","title":"Debugging Hints"},{"location":"examples/grpc/#notes-on-using-responserequest-streams","text":"If your server only does response streaming and you expect a stream to be open longer than 60 seconds, you will have to change the grpc_read_timeout to accommodate this. If your service only does request streaming and you expect a stream to be open longer than 60 seconds, you have to change the grpc_send_timeout and the client_body_timeout . If you do both response and request streaming with an open stream longer than 60 seconds, you have to change all three timeouts: grpc_read_timeout , grpc_send_timeout and client_body_timeout . Values for the timeouts must be specified as e.g. \"1200s\" . On the most recent versions of ingress-nginx, changing these timeouts requires using the nginx.ingress.kubernetes.io/server-snippet annotation. There are plans for future releases to allow using the Kubernetes annotations to define each timeout separately.","title":"Notes on using response/request streams"},{"location":"examples/multi-tls/","text":"Multi TLS certificate termination \u00b6 This example uses 2 different certificates to terminate SSL for 2 hostnames. Create tls secrets for foo.bar.com and bar.baz.com as indicated in the yaml Create multi-tls.yaml This should generate a segment like: $ kubectl exec -it ingress-nginx-controller-6vwd1 -- cat /etc/nginx/nginx.conf | grep \"foo.bar.com\" -B 7 -A 35 server { listen 80; listen 443 ssl http2; ssl_certificate /etc/nginx-ssl/default-foobar.pem; ssl_certificate_key /etc/nginx-ssl/default-foobar.pem; server_name foo.bar.com; if ($scheme = http) { return 301 https://$host$request_uri; } location / { proxy_set_header Host $host; # Pass Real IP proxy_set_header X-Real-IP $remote_addr; # Allow websocket connections proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Proto $pass_access_scheme; proxy_connect_timeout 5s; proxy_send_timeout 60s; proxy_read_timeout 60s; proxy_redirect off; proxy_buffering off; proxy_http_version 1.1; proxy_pass http://default-http-svc-80; } And you should be able to reach your nginx service or http-svc service using a hostname switch: $ kubectl get ing NAME RULE BACKEND ADDRESS AGE foo-tls - 104.154.30.67 13m foo.bar.com / http-svc:80 bar.baz.com / nginx:80 $ curl https://104.154.30.67 -H 'Host:foo.bar.com' -k CLIENT VALUES: client_address=10.245.0.6 command=GET real path=/ query=nil request_version=1.1 request_uri=http://foo.bar.com:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=close host=foo.bar.com user-agent=curl/7.35.0 x-forwarded-for=10.245.0.1 x-forwarded-host=foo.bar.com x-forwarded-proto=https $ curl https://104.154.30.67 -H 'Host:bar.baz.com' -k <!DOCTYPE html> <html> <head> <title>Welcome to nginx on Debian!</title> $ curl 104 .154.30.67 default backend - 404","title":"Multi TLS certificate termination"},{"location":"examples/multi-tls/#multi-tls-certificate-termination","text":"This example uses 2 different certificates to terminate SSL for 2 hostnames. Create tls secrets for foo.bar.com and bar.baz.com as indicated in the yaml Create multi-tls.yaml This should generate a segment like: $ kubectl exec -it ingress-nginx-controller-6vwd1 -- cat /etc/nginx/nginx.conf | grep \"foo.bar.com\" -B 7 -A 35 server { listen 80; listen 443 ssl http2; ssl_certificate /etc/nginx-ssl/default-foobar.pem; ssl_certificate_key /etc/nginx-ssl/default-foobar.pem; server_name foo.bar.com; if ($scheme = http) { return 301 https://$host$request_uri; } location / { proxy_set_header Host $host; # Pass Real IP proxy_set_header X-Real-IP $remote_addr; # Allow websocket connections proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Proto $pass_access_scheme; proxy_connect_timeout 5s; proxy_send_timeout 60s; proxy_read_timeout 60s; proxy_redirect off; proxy_buffering off; proxy_http_version 1.1; proxy_pass http://default-http-svc-80; } And you should be able to reach your nginx service or http-svc service using a hostname switch: $ kubectl get ing NAME RULE BACKEND ADDRESS AGE foo-tls - 104.154.30.67 13m foo.bar.com / http-svc:80 bar.baz.com / nginx:80 $ curl https://104.154.30.67 -H 'Host:foo.bar.com' -k CLIENT VALUES: client_address=10.245.0.6 command=GET real path=/ query=nil request_version=1.1 request_uri=http://foo.bar.com:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=close host=foo.bar.com user-agent=curl/7.35.0 x-forwarded-for=10.245.0.1 x-forwarded-host=foo.bar.com x-forwarded-proto=https $ curl https://104.154.30.67 -H 'Host:bar.baz.com' -k <!DOCTYPE html> <html> <head> <title>Welcome to nginx on Debian!</title> $ curl 104 .154.30.67 default backend - 404","title":"Multi TLS certificate termination"},{"location":"examples/psp/","text":"Pod Security Policy (PSP) \u00b6 In most clusters today, by default, all resources (e.g. Deployments and ReplicatSets ) have permissions to create pods. Kubernetes however provides a more fine-grained authorization policy called Pod Security Policy (PSP) . PSP allows the cluster owner to define the permission of each object, for example creating a pod. If you have PSP enabled on the cluster, and you deploy ingress-nginx, you will need to provide the Deployment with the permissions to create pods. Before applying any objects, first apply the PSP permissions by running: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/psp/psp.yaml Note: PSP permissions must be granted before the creation of the Deployment and the ReplicaSet .","title":"Pod Security Policy (PSP)"},{"location":"examples/psp/#pod-security-policy-psp","text":"In most clusters today, by default, all resources (e.g. Deployments and ReplicatSets ) have permissions to create pods. Kubernetes however provides a more fine-grained authorization policy called Pod Security Policy (PSP) . PSP allows the cluster owner to define the permission of each object, for example creating a pod. If you have PSP enabled on the cluster, and you deploy ingress-nginx, you will need to provide the Deployment with the permissions to create pods. Before applying any objects, first apply the PSP permissions by running: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/psp/psp.yaml Note: PSP permissions must be granted before the creation of the Deployment and the ReplicaSet .","title":"Pod Security Policy (PSP)"},{"location":"examples/rewrite/","text":"Rewrite \u00b6 This example demonstrates how to use Rewrite annotations. Prerequisites \u00b6 You will need to make sure your Ingress targets exactly one Ingress controller by specifying the ingress.class annotation , and that you have an ingress controller running in your cluster. Deployment \u00b6 Rewriting can be controlled using the following annotations: Name Description Values nginx.ingress.kubernetes.io/rewrite-target Target URI where the traffic must be redirected string nginx.ingress.kubernetes.io/ssl-redirect Indicates if the location section is only accessible via SSL (defaults to True when Ingress contains a Certificate) bool nginx.ingress.kubernetes.io/force-ssl-redirect Forces the redirection to HTTPS even if the Ingress is not TLS Enabled bool nginx.ingress.kubernetes.io/app-root Defines the Application Root that the Controller must redirect if it's in / context string nginx.ingress.kubernetes.io/use-regex Indicates if the paths defined on an Ingress use regular expressions bool Examples \u00b6 Rewrite Target \u00b6 Attention Starting in Version 0.22.0, ingress definitions using the annotation nginx.ingress.kubernetes.io/rewrite-target are not backwards compatible with previous versions. In Version 0.22.0 and beyond, any substrings within the request URI that need to be passed to the rewritten path must explicitly be defined in a capture group . Note Captured groups are saved in numbered placeholders, chronologically, in the form $1 , $2 ... $n . These placeholders can be used as parameters in the rewrite-target annotation. Create an Ingress rule with a rewrite annotation: $ echo ' apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/rewrite-target: /$2 name: rewrite namespace: default spec: ingressClassName: nginx rules: - host: rewrite.bar.com http: paths: - path: /something(/|$)(.*) pathType: Prefix backend: service: name: http-svc port: number: 80 ' | kubectl create -f - In this ingress definition, any characters captured by (.*) will be assigned to the placeholder $2 , which is then used as a parameter in the rewrite-target annotation. For example, the ingress definition above will result in the following rewrites: rewrite.bar.com/something rewrites to rewrite.bar.com/ rewrite.bar.com/something/ rewrites to rewrite.bar.com/ rewrite.bar.com/something/new rewrites to rewrite.bar.com/new App Root \u00b6 Create an Ingress rule with an app-root annotation: $ echo \" apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/app-root: /app1 name: approot namespace: default spec: ingressClassName: nginx rules: - host: approot.bar.com http: paths: - path: / pathType: Prefix backend: service: name: http-svc port: number: 80 \" | kubectl create -f - Check the rewrite is working $ curl -I -k http://approot.bar.com/ HTTP/1.1 302 Moved Temporarily Server: nginx/1.11.10 Date: Mon, 13 Mar 2017 14:57:15 GMT Content-Type: text/html Content-Length: 162 Location: http://approot.bar.com/app1 Connection: keep-alive","title":"Rewrite"},{"location":"examples/rewrite/#rewrite","text":"This example demonstrates how to use Rewrite annotations.","title":"Rewrite"},{"location":"examples/rewrite/#prerequisites","text":"You will need to make sure your Ingress targets exactly one Ingress controller by specifying the ingress.class annotation , and that you have an ingress controller running in your cluster.","title":"Prerequisites"},{"location":"examples/rewrite/#deployment","text":"Rewriting can be controlled using the following annotations: Name Description Values nginx.ingress.kubernetes.io/rewrite-target Target URI where the traffic must be redirected string nginx.ingress.kubernetes.io/ssl-redirect Indicates if the location section is only accessible via SSL (defaults to True when Ingress contains a Certificate) bool nginx.ingress.kubernetes.io/force-ssl-redirect Forces the redirection to HTTPS even if the Ingress is not TLS Enabled bool nginx.ingress.kubernetes.io/app-root Defines the Application Root that the Controller must redirect if it's in / context string nginx.ingress.kubernetes.io/use-regex Indicates if the paths defined on an Ingress use regular expressions bool","title":"Deployment"},{"location":"examples/rewrite/#examples","text":"","title":"Examples"},{"location":"examples/rewrite/#rewrite-target","text":"Attention Starting in Version 0.22.0, ingress definitions using the annotation nginx.ingress.kubernetes.io/rewrite-target are not backwards compatible with previous versions. In Version 0.22.0 and beyond, any substrings within the request URI that need to be passed to the rewritten path must explicitly be defined in a capture group . Note Captured groups are saved in numbered placeholders, chronologically, in the form $1 , $2 ... $n . These placeholders can be used as parameters in the rewrite-target annotation. Create an Ingress rule with a rewrite annotation: $ echo ' apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/rewrite-target: /$2 name: rewrite namespace: default spec: ingressClassName: nginx rules: - host: rewrite.bar.com http: paths: - path: /something(/|$)(.*) pathType: Prefix backend: service: name: http-svc port: number: 80 ' | kubectl create -f - In this ingress definition, any characters captured by (.*) will be assigned to the placeholder $2 , which is then used as a parameter in the rewrite-target annotation. For example, the ingress definition above will result in the following rewrites: rewrite.bar.com/something rewrites to rewrite.bar.com/ rewrite.bar.com/something/ rewrites to rewrite.bar.com/ rewrite.bar.com/something/new rewrites to rewrite.bar.com/new","title":"Rewrite Target"},{"location":"examples/rewrite/#app-root","text":"Create an Ingress rule with an app-root annotation: $ echo \" apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/app-root: /app1 name: approot namespace: default spec: ingressClassName: nginx rules: - host: approot.bar.com http: paths: - path: / pathType: Prefix backend: service: name: http-svc port: number: 80 \" | kubectl create -f - Check the rewrite is working $ curl -I -k http://approot.bar.com/ HTTP/1.1 302 Moved Temporarily Server: nginx/1.11.10 Date: Mon, 13 Mar 2017 14:57:15 GMT Content-Type: text/html Content-Length: 162 Location: http://approot.bar.com/app1 Connection: keep-alive","title":"App Root"},{"location":"examples/static-ip/","text":"Static IPs \u00b6 This example demonstrates how to assign a static-ip to an Ingress on through the Ingress-NGINX controller. Prerequisites \u00b6 You need a TLS cert and a test HTTP service for this example. You will also need to make sure your Ingress targets exactly one Ingress controller by specifying the ingress.class annotation , and that you have an ingress controller running in your cluster. Acquiring an IP \u00b6 Since instances of the ingress nginx controller actually run on nodes in your cluster, by default nginx Ingresses will only get static IPs if your cloudprovider supports static IP assignments to nodes. On GKE/GCE for example, even though nodes get static IPs, the IPs are not retained across upgrades. To acquire a static IP for the ingress-nginx-controller, simply put it behind a Service of Type=LoadBalancer . First, create a loadbalancer Service and wait for it to acquire an IP: $ kubectl create -f static-ip-svc.yaml service \"ingress-nginx-lb\" created $ kubectl get svc ingress-nginx-lb NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-lb 10.0.138.113 104.154.109.191 80:31457/TCP,443:32240/TCP 15m Then, update the ingress controller so it adopts the static IP of the Service by passing the --publish-service flag (the example yaml used in the next step already has it set to \"ingress-nginx-lb\"). $ kubectl create -f ingress-nginx-controller.yaml deployment \"ingress-nginx-controller\" created Assigning the IP to an Ingress \u00b6 From here on every Ingress created with the ingress.class annotation set to nginx will get the IP allocated in the previous step. $ kubectl create -f ingress-nginx.yaml ingress \"ingress-nginx\" created $ kubectl get ing ingress-nginx NAME HOSTS ADDRESS PORTS AGE ingress-nginx * 104.154.109.191 80, 443 13m $ curl 104 .154.109.191 -kL CLIENT VALUES: client_address=10.180.1.25 command=GET real path=/ query=nil request_version=1.1 request_uri=http://104.154.109.191:8080/ ... Retaining the IP \u00b6 You can test retention by deleting the Ingress: $ kubectl delete ing ingress-nginx ingress \"ingress-nginx\" deleted $ kubectl create -f ingress-nginx.yaml ingress \"ingress-nginx\" created $ kubectl get ing ingress-nginx NAME HOSTS ADDRESS PORTS AGE ingress-nginx * 104.154.109.191 80, 443 13m Note that unlike the GCE Ingress, the same loadbalancer IP is shared amongst all Ingresses, because all requests are proxied through the same set of nginx controllers. Promote ephemeral to static IP \u00b6 To promote the allocated IP to static, you can update the Service manifest: $ kubectl patch svc ingress-nginx-lb -p '{\"spec\": {\"loadBalancerIP\": \"104.154.109.191\"}}' \"ingress-nginx-lb\" patched ... and promote the IP to static (promotion works differently for cloudproviders, provided example is for GKE/GCE): $ gcloud compute addresses create ingress-nginx-lb --addresses 104 .154.109.191 --region us-central1 Created [https://www.googleapis.com/compute/v1/projects/kubernetesdev/regions/us-central1/addresses/ingress-nginx-lb]. --- address: 104.154.109.191 creationTimestamp: '2017-01-31T16:34:50.089-08:00' description: '' id: '5208037144487826373' kind: compute#address name: ingress-nginx-lb region: us-central1 selfLink: https://www.googleapis.com/compute/v1/projects/kubernetesdev/regions/us-central1/addresses/ingress-nginx-lb status: IN_USE users: - us-central1/forwardingRules/a09f6913ae80e11e6a8c542010af0000 Now even if the Service is deleted, the IP will persist, so you can recreate the Service with spec.loadBalancerIP set to 104.154.109.191 .","title":"Static IPs"},{"location":"examples/static-ip/#static-ips","text":"This example demonstrates how to assign a static-ip to an Ingress on through the Ingress-NGINX controller.","title":"Static IPs"},{"location":"examples/static-ip/#prerequisites","text":"You need a TLS cert and a test HTTP service for this example. You will also need to make sure your Ingress targets exactly one Ingress controller by specifying the ingress.class annotation , and that you have an ingress controller running in your cluster.","title":"Prerequisites"},{"location":"examples/static-ip/#acquiring-an-ip","text":"Since instances of the ingress nginx controller actually run on nodes in your cluster, by default nginx Ingresses will only get static IPs if your cloudprovider supports static IP assignments to nodes. On GKE/GCE for example, even though nodes get static IPs, the IPs are not retained across upgrades. To acquire a static IP for the ingress-nginx-controller, simply put it behind a Service of Type=LoadBalancer . First, create a loadbalancer Service and wait for it to acquire an IP: $ kubectl create -f static-ip-svc.yaml service \"ingress-nginx-lb\" created $ kubectl get svc ingress-nginx-lb NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-lb 10.0.138.113 104.154.109.191 80:31457/TCP,443:32240/TCP 15m Then, update the ingress controller so it adopts the static IP of the Service by passing the --publish-service flag (the example yaml used in the next step already has it set to \"ingress-nginx-lb\"). $ kubectl create -f ingress-nginx-controller.yaml deployment \"ingress-nginx-controller\" created","title":"Acquiring an IP"},{"location":"examples/static-ip/#assigning-the-ip-to-an-ingress","text":"From here on every Ingress created with the ingress.class annotation set to nginx will get the IP allocated in the previous step. $ kubectl create -f ingress-nginx.yaml ingress \"ingress-nginx\" created $ kubectl get ing ingress-nginx NAME HOSTS ADDRESS PORTS AGE ingress-nginx * 104.154.109.191 80, 443 13m $ curl 104 .154.109.191 -kL CLIENT VALUES: client_address=10.180.1.25 command=GET real path=/ query=nil request_version=1.1 request_uri=http://104.154.109.191:8080/ ...","title":"Assigning the IP to an Ingress"},{"location":"examples/static-ip/#retaining-the-ip","text":"You can test retention by deleting the Ingress: $ kubectl delete ing ingress-nginx ingress \"ingress-nginx\" deleted $ kubectl create -f ingress-nginx.yaml ingress \"ingress-nginx\" created $ kubectl get ing ingress-nginx NAME HOSTS ADDRESS PORTS AGE ingress-nginx * 104.154.109.191 80, 443 13m Note that unlike the GCE Ingress, the same loadbalancer IP is shared amongst all Ingresses, because all requests are proxied through the same set of nginx controllers.","title":"Retaining the IP"},{"location":"examples/static-ip/#promote-ephemeral-to-static-ip","text":"To promote the allocated IP to static, you can update the Service manifest: $ kubectl patch svc ingress-nginx-lb -p '{\"spec\": {\"loadBalancerIP\": \"104.154.109.191\"}}' \"ingress-nginx-lb\" patched ... and promote the IP to static (promotion works differently for cloudproviders, provided example is for GKE/GCE): $ gcloud compute addresses create ingress-nginx-lb --addresses 104 .154.109.191 --region us-central1 Created [https://www.googleapis.com/compute/v1/projects/kubernetesdev/regions/us-central1/addresses/ingress-nginx-lb]. --- address: 104.154.109.191 creationTimestamp: '2017-01-31T16:34:50.089-08:00' description: '' id: '5208037144487826373' kind: compute#address name: ingress-nginx-lb region: us-central1 selfLink: https://www.googleapis.com/compute/v1/projects/kubernetesdev/regions/us-central1/addresses/ingress-nginx-lb status: IN_USE users: - us-central1/forwardingRules/a09f6913ae80e11e6a8c542010af0000 Now even if the Service is deleted, the IP will persist, so you can recreate the Service with spec.loadBalancerIP set to 104.154.109.191 .","title":"Promote ephemeral to static IP"},{"location":"examples/tls-termination/","text":"TLS termination \u00b6 This example demonstrates how to terminate TLS through the nginx Ingress controller. Prerequisites \u00b6 You need a TLS cert and a test HTTP service for this example. Deployment \u00b6 Create a ingress.yaml file. apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : nginx-test spec : tls : - hosts : - foo.bar.com # This assumes tls-secret exists and the SSL # certificate contains a CN for foo.bar.com secretName : tls-secret ingressClassName : nginx rules : - host : foo.bar.com http : paths : - path : / pathType : Prefix backend : # This assumes http-svc exists and routes to healthy endpoints service : name : http-svc port : number : 80 The following command instructs the controller to terminate traffic using the provided TLS cert, and forward un-encrypted HTTP traffic to the test HTTP service. kubectl apply -f ingress.yaml Validation \u00b6 You can confirm that the Ingress works. $ kubectl describe ing nginx-test Name: nginx-test Namespace: default Address: 104.198.183.6 Default backend: default-http-backend:80 (10.180.0.4:8080,10.240.0.2:8080) TLS: tls-secret terminates Rules: Host Path Backends ---- ---- -------- * http-svc:80 (<none>) Annotations: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 7s 7s 1 {ingress-nginx-controller } Normal CREATE default/nginx-test 7s 7s 1 {ingress-nginx-controller } Normal UPDATE default/nginx-test 7s 7s 1 {ingress-nginx-controller } Normal CREATE ip: 104.198.183.6 7s 7s 1 {ingress-nginx-controller } Warning MAPPING Ingress rule 'default/nginx-test' contains no path definition. Assuming / $ curl 104 .198.183.6 -L curl: (60) SSL certificate problem: self signed certificate More details here: http://curl.haxx.se/docs/sslcerts.html $ curl 104 .198.183.6 -Lk CLIENT VALUES: client_address=10.240.0.4 command=GET real path=/ query=nil request_version=1.1 request_uri=http://35.186.221.137:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=Keep-Alive host=35.186.221.137 user-agent=curl/7.46.0 via=1.1 google x-cloud-trace-context=f708ea7e369d4514fc90d51d7e27e91d/13322322294276298106 x-forwarded-for=104.132.0.80, 35.186.221.137 x-forwarded-proto=https BODY:","title":"TLS termination"},{"location":"examples/tls-termination/#tls-termination","text":"This example demonstrates how to terminate TLS through the nginx Ingress controller.","title":"TLS termination"},{"location":"examples/tls-termination/#prerequisites","text":"You need a TLS cert and a test HTTP service for this example.","title":"Prerequisites"},{"location":"examples/tls-termination/#deployment","text":"Create a ingress.yaml file. apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : nginx-test spec : tls : - hosts : - foo.bar.com # This assumes tls-secret exists and the SSL # certificate contains a CN for foo.bar.com secretName : tls-secret ingressClassName : nginx rules : - host : foo.bar.com http : paths : - path : / pathType : Prefix backend : # This assumes http-svc exists and routes to healthy endpoints service : name : http-svc port : number : 80 The following command instructs the controller to terminate traffic using the provided TLS cert, and forward un-encrypted HTTP traffic to the test HTTP service. kubectl apply -f ingress.yaml","title":"Deployment"},{"location":"examples/tls-termination/#validation","text":"You can confirm that the Ingress works. $ kubectl describe ing nginx-test Name: nginx-test Namespace: default Address: 104.198.183.6 Default backend: default-http-backend:80 (10.180.0.4:8080,10.240.0.2:8080) TLS: tls-secret terminates Rules: Host Path Backends ---- ---- -------- * http-svc:80 (<none>) Annotations: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 7s 7s 1 {ingress-nginx-controller } Normal CREATE default/nginx-test 7s 7s 1 {ingress-nginx-controller } Normal UPDATE default/nginx-test 7s 7s 1 {ingress-nginx-controller } Normal CREATE ip: 104.198.183.6 7s 7s 1 {ingress-nginx-controller } Warning MAPPING Ingress rule 'default/nginx-test' contains no path definition. Assuming / $ curl 104 .198.183.6 -L curl: (60) SSL certificate problem: self signed certificate More details here: http://curl.haxx.se/docs/sslcerts.html $ curl 104 .198.183.6 -Lk CLIENT VALUES: client_address=10.240.0.4 command=GET real path=/ query=nil request_version=1.1 request_uri=http://35.186.221.137:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=Keep-Alive host=35.186.221.137 user-agent=curl/7.46.0 via=1.1 google x-cloud-trace-context=f708ea7e369d4514fc90d51d7e27e91d/13322322294276298106 x-forwarded-for=104.132.0.80, 35.186.221.137 x-forwarded-proto=https BODY:","title":"Validation"},{"location":"user-guide/basic-usage/","text":"Basic usage - host based routing \u00b6 ingress-nginx can be used for many use cases, inside various cloud provider and supports a lot of configurations. In this section you can find a common usage scenario where a single load balancer powered by ingress-nginx will route traffic to 2 different HTTP backend services based on the host name. First of all follow the instructions to install ingress-nginx. Then imagine that you need to expose 2 HTTP services already installed, myServiceA , myServiceB , and configured as type: ClusterIP . Let's say that you want to expose the first at myServiceA.foo.org and the second at myServiceB.foo.org . If the cluster version is < 1.19, you can create two ingress resources like this: apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-myservicea spec: ingressClassName: nginx rules: - host: myservicea.foo.org http: paths: - path: / backend: serviceName: myservicea servicePort: 80 --- apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-myserviceb annotations: # use the shared ingress-nginx kubernetes.io/ingress.class: \"nginx\" spec: rules: - host: myserviceb.foo.org http: paths: - path: / backend: serviceName: myserviceb servicePort: 80 If the cluster uses Kubernetes version >= 1.19.x, then its suggested to create 2 ingress resources, using yaml examples shown below. These examples are in conformity with the networking.kubernetes.io/v1 api. apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-myservicea spec: rules: - host: myservicea.foo.org http: paths: - path: / pathType: Prefix backend: service: name: myservicea port: number: 80 ingressClassName: nginx --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-myserviceb spec: rules: - host: myserviceb.foo.org http: paths: - path: / pathType: Prefix backend: service: name: myserviceb port: number: 80 ingressClassName: nginx When you apply this yaml, 2 ingress resources will be created managed by the ingress-nginx instance. Nginx is configured to automatically discover all ingress with the kubernetes.io/ingress.class: \"nginx\" annotation or where ingressClassName: nginx is present. Please note that the ingress resource should be placed inside the same namespace of the backend resource. On many cloud providers ingress-nginx will also create the corresponding Load Balancer resource. All you have to do is get the external IP and add a DNS A record inside your DNS provider that point myservicea.foo.org and myserviceb.foo.org to the nginx external IP. Get the external IP by running: kubectl get services -n ingress-nginx To test inside minikube refer to this documentation: Set up Ingress on Minikube with the NGINX Ingress Controller","title":"Basic usage"},{"location":"user-guide/basic-usage/#basic-usage-host-based-routing","text":"ingress-nginx can be used for many use cases, inside various cloud provider and supports a lot of configurations. In this section you can find a common usage scenario where a single load balancer powered by ingress-nginx will route traffic to 2 different HTTP backend services based on the host name. First of all follow the instructions to install ingress-nginx. Then imagine that you need to expose 2 HTTP services already installed, myServiceA , myServiceB , and configured as type: ClusterIP . Let's say that you want to expose the first at myServiceA.foo.org and the second at myServiceB.foo.org . If the cluster version is < 1.19, you can create two ingress resources like this: apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-myservicea spec: ingressClassName: nginx rules: - host: myservicea.foo.org http: paths: - path: / backend: serviceName: myservicea servicePort: 80 --- apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-myserviceb annotations: # use the shared ingress-nginx kubernetes.io/ingress.class: \"nginx\" spec: rules: - host: myserviceb.foo.org http: paths: - path: / backend: serviceName: myserviceb servicePort: 80 If the cluster uses Kubernetes version >= 1.19.x, then its suggested to create 2 ingress resources, using yaml examples shown below. These examples are in conformity with the networking.kubernetes.io/v1 api. apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-myservicea spec: rules: - host: myservicea.foo.org http: paths: - path: / pathType: Prefix backend: service: name: myservicea port: number: 80 ingressClassName: nginx --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-myserviceb spec: rules: - host: myserviceb.foo.org http: paths: - path: / pathType: Prefix backend: service: name: myserviceb port: number: 80 ingressClassName: nginx When you apply this yaml, 2 ingress resources will be created managed by the ingress-nginx instance. Nginx is configured to automatically discover all ingress with the kubernetes.io/ingress.class: \"nginx\" annotation or where ingressClassName: nginx is present. Please note that the ingress resource should be placed inside the same namespace of the backend resource. On many cloud providers ingress-nginx will also create the corresponding Load Balancer resource. All you have to do is get the external IP and add a DNS A record inside your DNS provider that point myservicea.foo.org and myserviceb.foo.org to the nginx external IP. Get the external IP by running: kubectl get services -n ingress-nginx To test inside minikube refer to this documentation: Set up Ingress on Minikube with the NGINX Ingress Controller","title":"Basic usage - host based routing"},{"location":"user-guide/cli-arguments/","text":"Command line arguments \u00b6 The following command line arguments are accepted by the Ingress controller executable. They are set in the container spec of the ingress-nginx-controller Deployment manifest Argument Description --add_dir_header If true, adds the file directory to the header --alsologtostderr log to standard error as well as files --annotations-prefix Prefix of the Ingress annotations specific to the NGINX controller. (default \"nginx.ingress.kubernetes.io\") --apiserver-host Address of the Kubernetes API server. Takes the form \"protocol://address:port\". If not specified, it is assumed the program runs inside a Kubernetes cluster and local discovery is attempted. --certificate-authority Path to a cert file for the certificate authority. This certificate is used only when the flag --apiserver-host is specified. --configmap Name of the ConfigMap containing custom global configurations for the controller. --deep-inspect Enables ingress object security deep inspector. (default true) --default-backend-service Service used to serve HTTP requests not matching any known server name (catch-all). Takes the form \"namespace/name\". The controller configures NGINX to forward requests to the first port of this Service. --default-server-port Port to use for exposing the default server (catch-all). (default 8181) --default-ssl-certificate Secret containing a SSL certificate to be used by the default HTTPS server (catch-all). Takes the form \"namespace/name\". --disable-catch-all Disable support for catch-all Ingresses --disable-full-test Disable full test of all merged ingresses at the admission stage and tests the template of the ingress being created or updated (full test of all ingresses is enabled by default) --election-id Election id to use for Ingress status updates. (default \"ingress-controller-leader\") --enable-metrics Enables the collection of NGINX metrics (default true) --enable-ssl-chain-completion Autocomplete SSL certificate chains with missing intermediate CA certificates. Certificates uploaded to Kubernetes must have the \"Authority Information Access\" X.509 v3 extension for this to succeed. --enable-ssl-passthrough Enable SSL Passthrough. --health-check-path URL path of the health check endpoint. Configured inside the NGINX status server. All requests received on the port defined by the healthz-port parameter are forwarded internally to this path. (default \"/healthz\") --health-check-timeout Time limit, in seconds, for a probe to health-check-path to succeed. (default 10) --healthz-port Port to use for the healthz endpoint. (default 10254) --healthz-host Address to bind the healthz endpoint. --http-port Port to use for servicing HTTP traffic. (default 80) --https-port Port to use for servicing HTTPS traffic. (default 443) --ingress-class Name of the ingress class this controller satisfies. The class of an Ingress object is set using the field IngressClassName in Kubernetes clusters version v1.18.0 or higher or the annotation \"kubernetes.io/ingress.class\" (deprecated). If this parameter is not set, or set to the default value of \"nginx\", it will handle ingresses with either an empty or \"nginx\" class name. --ingress-class-by-name Define if Ingress Controller should watch for Ingress Class by Name together with Controller Class. (default false) --internal-logger-address Define the internal logger address to use when chroot images is used. (default 127.0.0.1:11514) --kubeconfig Path to a kubeconfig file containing authorization and API server information. --length-buckets Set of buckets which will be used for prometheus histogram metrics such as RequestLength, ResponseLength. (default [10, 20, 30, 40, 50, 60, 70, 80, 90, 100] ) --log_backtrace_at when logging hits line file:N, emit a stack trace (default :0) --log_dir If non-empty, write log files in this directory --log_file If non-empty, use this log file --log_file_max_size Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800) --logtostderr log to standard error instead of files (default true) --maxmind-edition-ids Maxmind edition ids to download GeoLite2 Databases. (default \"GeoLite2-City,GeoLite2-ASN\") --maxmind-retries-timeout Maxmind downloading delay between 1 st and 2 nd attempt, 0s - do not retry to download if something went wrong. (default 0s) --maxmind-retries-count Number of attempts to download the GeoIP DB. (default 1) --maxmind-license-key Maxmind license key to download GeoLite2 Databases. https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases --metrics-per-host Export metrics per-host (default true) --post-shutdown-grace-period Additional delay in seconds before controller container exits. (default 10) --profiler-port Port to use for expose the ingress controller Go profiler when it is enabled. (default 10245) --profiling Enable profiling via web interface host:port/debug/pprof/ (default true) --publish-service Service fronting the Ingress controller. Takes the form \"namespace/name\". When used together with update-status, the controller mirrors the address of this service's endpoints to the load-balancer status of all Ingress objects it satisfies. --publish-status-address Customized address (or addresses, separated by comma) to set as the load-balancer status of Ingress objects this controller satisfies. Requires the update-status parameter. --report-node-internal-ip-address Set the load-balancer status of Ingress objects to internal Node addresses instead of external. Requires the update-status parameter. --report-status-classes If true, report status classes in metrics (2xx, 3xx, 4xx and 5xx) instead of full status codes. (default false) --skip_headers If true, avoid header prefixes in the log messages --skip_log_headers If true, avoid headers when opening log files --ssl-passthrough-proxy-port Port to use internally for SSL Passthrough. (default 442) --size-buckets Set of buckets which will be used for prometheus histogram metrics such as BytesSent. (default [10, 100, 1000, 10000, 100000, 1e+06, 1e+07] ) --status-port Port to use for the lua HTTP endpoint configuration. (default 10246) --status-update-interval Time interval in seconds in which the status should check if an update is required. Default is 60 seconds (default 60) --stderrthreshold logs at or above this threshold go to stderr (default 2) --stream-port Port to use for the lua TCP/UDP endpoint configuration. (default 10247) --sync-period Period at which the controller forces the repopulation of its local object stores. Disabled by default. --sync-rate-limit Define the sync frequency upper limit (default 0.3) --tcp-services-configmap Name of the ConfigMap containing the definition of the TCP services to expose. The key in the map indicates the external port to be used. The value is a reference to a Service in the form \"namespace/name:port\", where \"port\" can either be a port number or name. TCP ports 80 and 443 are reserved by the controller for servicing HTTP traffic. --time-buckets Set of buckets which will be used for prometheus histogram metrics such as RequestTime, ResponseTime. (default [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] ) --udp-services-configmap Name of the ConfigMap containing the definition of the UDP services to expose. The key in the map indicates the external port to be used. The value is a reference to a Service in the form \"namespace/name:port\", where \"port\" can either be a port name or number. --update-status Update the load-balancer status of Ingress objects this controller satisfies. Requires setting the publish-service parameter to a valid Service reference. (default true) --update-status-on-shutdown Update the load-balancer status of Ingress objects when the controller shuts down. Requires the update-status parameter. (default true) --shutdown-grace-period Seconds to wait after receiving the shutdown signal, before stopping the nginx process. -v, --v Level number for the log level verbosity --validating-webhook The address to start an admission controller on to validate incoming ingresses. Takes the form \" :port\". If not provided, no admission controller is started. --validating-webhook-certificate The path of the validating webhook certificate PEM. --validating-webhook-key The path of the validating webhook key PEM. --version Show release information about the NGINX Ingress controller and exit. --vmodule comma-separated list of pattern=N settings for file-filtered logging --watch-namespace Namespace the controller watches for updates to Kubernetes objects. This includes Ingresses, Services and all configuration resources. All namespaces are watched if this parameter is left empty. --watch-namespace-selector The controller will watch namespaces whose labels match the given selector. This flag only takes effective when --watch-namespace is empty.","title":"Command line arguments"},{"location":"user-guide/cli-arguments/#command-line-arguments","text":"The following command line arguments are accepted by the Ingress controller executable. They are set in the container spec of the ingress-nginx-controller Deployment manifest Argument Description --add_dir_header If true, adds the file directory to the header --alsologtostderr log to standard error as well as files --annotations-prefix Prefix of the Ingress annotations specific to the NGINX controller. (default \"nginx.ingress.kubernetes.io\") --apiserver-host Address of the Kubernetes API server. Takes the form \"protocol://address:port\". If not specified, it is assumed the program runs inside a Kubernetes cluster and local discovery is attempted. --certificate-authority Path to a cert file for the certificate authority. This certificate is used only when the flag --apiserver-host is specified. --configmap Name of the ConfigMap containing custom global configurations for the controller. --deep-inspect Enables ingress object security deep inspector. (default true) --default-backend-service Service used to serve HTTP requests not matching any known server name (catch-all). Takes the form \"namespace/name\". The controller configures NGINX to forward requests to the first port of this Service. --default-server-port Port to use for exposing the default server (catch-all). (default 8181) --default-ssl-certificate Secret containing a SSL certificate to be used by the default HTTPS server (catch-all). Takes the form \"namespace/name\". --disable-catch-all Disable support for catch-all Ingresses --disable-full-test Disable full test of all merged ingresses at the admission stage and tests the template of the ingress being created or updated (full test of all ingresses is enabled by default) --election-id Election id to use for Ingress status updates. (default \"ingress-controller-leader\") --enable-metrics Enables the collection of NGINX metrics (default true) --enable-ssl-chain-completion Autocomplete SSL certificate chains with missing intermediate CA certificates. Certificates uploaded to Kubernetes must have the \"Authority Information Access\" X.509 v3 extension for this to succeed. --enable-ssl-passthrough Enable SSL Passthrough. --health-check-path URL path of the health check endpoint. Configured inside the NGINX status server. All requests received on the port defined by the healthz-port parameter are forwarded internally to this path. (default \"/healthz\") --health-check-timeout Time limit, in seconds, for a probe to health-check-path to succeed. (default 10) --healthz-port Port to use for the healthz endpoint. (default 10254) --healthz-host Address to bind the healthz endpoint. --http-port Port to use for servicing HTTP traffic. (default 80) --https-port Port to use for servicing HTTPS traffic. (default 443) --ingress-class Name of the ingress class this controller satisfies. The class of an Ingress object is set using the field IngressClassName in Kubernetes clusters version v1.18.0 or higher or the annotation \"kubernetes.io/ingress.class\" (deprecated). If this parameter is not set, or set to the default value of \"nginx\", it will handle ingresses with either an empty or \"nginx\" class name. --ingress-class-by-name Define if Ingress Controller should watch for Ingress Class by Name together with Controller Class. (default false) --internal-logger-address Define the internal logger address to use when chroot images is used. (default 127.0.0.1:11514) --kubeconfig Path to a kubeconfig file containing authorization and API server information. --length-buckets Set of buckets which will be used for prometheus histogram metrics such as RequestLength, ResponseLength. (default [10, 20, 30, 40, 50, 60, 70, 80, 90, 100] ) --log_backtrace_at when logging hits line file:N, emit a stack trace (default :0) --log_dir If non-empty, write log files in this directory --log_file If non-empty, use this log file --log_file_max_size Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800) --logtostderr log to standard error instead of files (default true) --maxmind-edition-ids Maxmind edition ids to download GeoLite2 Databases. (default \"GeoLite2-City,GeoLite2-ASN\") --maxmind-retries-timeout Maxmind downloading delay between 1 st and 2 nd attempt, 0s - do not retry to download if something went wrong. (default 0s) --maxmind-retries-count Number of attempts to download the GeoIP DB. (default 1) --maxmind-license-key Maxmind license key to download GeoLite2 Databases. https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases --metrics-per-host Export metrics per-host (default true) --post-shutdown-grace-period Additional delay in seconds before controller container exits. (default 10) --profiler-port Port to use for expose the ingress controller Go profiler when it is enabled. (default 10245) --profiling Enable profiling via web interface host:port/debug/pprof/ (default true) --publish-service Service fronting the Ingress controller. Takes the form \"namespace/name\". When used together with update-status, the controller mirrors the address of this service's endpoints to the load-balancer status of all Ingress objects it satisfies. --publish-status-address Customized address (or addresses, separated by comma) to set as the load-balancer status of Ingress objects this controller satisfies. Requires the update-status parameter. --report-node-internal-ip-address Set the load-balancer status of Ingress objects to internal Node addresses instead of external. Requires the update-status parameter. --report-status-classes If true, report status classes in metrics (2xx, 3xx, 4xx and 5xx) instead of full status codes. (default false) --skip_headers If true, avoid header prefixes in the log messages --skip_log_headers If true, avoid headers when opening log files --ssl-passthrough-proxy-port Port to use internally for SSL Passthrough. (default 442) --size-buckets Set of buckets which will be used for prometheus histogram metrics such as BytesSent. (default [10, 100, 1000, 10000, 100000, 1e+06, 1e+07] ) --status-port Port to use for the lua HTTP endpoint configuration. (default 10246) --status-update-interval Time interval in seconds in which the status should check if an update is required. Default is 60 seconds (default 60) --stderrthreshold logs at or above this threshold go to stderr (default 2) --stream-port Port to use for the lua TCP/UDP endpoint configuration. (default 10247) --sync-period Period at which the controller forces the repopulation of its local object stores. Disabled by default. --sync-rate-limit Define the sync frequency upper limit (default 0.3) --tcp-services-configmap Name of the ConfigMap containing the definition of the TCP services to expose. The key in the map indicates the external port to be used. The value is a reference to a Service in the form \"namespace/name:port\", where \"port\" can either be a port number or name. TCP ports 80 and 443 are reserved by the controller for servicing HTTP traffic. --time-buckets Set of buckets which will be used for prometheus histogram metrics such as RequestTime, ResponseTime. (default [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] ) --udp-services-configmap Name of the ConfigMap containing the definition of the UDP services to expose. The key in the map indicates the external port to be used. The value is a reference to a Service in the form \"namespace/name:port\", where \"port\" can either be a port name or number. --update-status Update the load-balancer status of Ingress objects this controller satisfies. Requires setting the publish-service parameter to a valid Service reference. (default true) --update-status-on-shutdown Update the load-balancer status of Ingress objects when the controller shuts down. Requires the update-status parameter. (default true) --shutdown-grace-period Seconds to wait after receiving the shutdown signal, before stopping the nginx process. -v, --v Level number for the log level verbosity --validating-webhook The address to start an admission controller on to validate incoming ingresses. Takes the form \" :port\". If not provided, no admission controller is started. --validating-webhook-certificate The path of the validating webhook certificate PEM. --validating-webhook-key The path of the validating webhook key PEM. --version Show release information about the NGINX Ingress controller and exit. --vmodule comma-separated list of pattern=N settings for file-filtered logging --watch-namespace Namespace the controller watches for updates to Kubernetes objects. This includes Ingresses, Services and all configuration resources. All namespaces are watched if this parameter is left empty. --watch-namespace-selector The controller will watch namespaces whose labels match the given selector. This flag only takes effective when --watch-namespace is empty.","title":"Command line arguments"},{"location":"user-guide/custom-errors/","text":"Custom errors \u00b6 When the custom-http-errors option is enabled, the Ingress controller configures NGINX so that it passes several HTTP headers down to its default-backend in case of error: Header Value X-Code HTTP status code returned by the request X-Format Value of the Accept header sent by the client X-Original-URI URI that caused the error X-Namespace Namespace where the backend Service is located X-Ingress-Name Name of the Ingress where the backend is defined X-Service-Name Name of the Service backing the backend X-Service-Port Port number of the Service backing the backend X-Request-ID Unique ID that identifies the request - same as for backend service A custom error backend can use this information to return the best possible representation of an error page. For example, if the value of the Accept header send by the client was application/json , a carefully crafted backend could decide to return the error payload as a JSON document instead of HTML. Important The custom backend is expected to return the correct HTTP status code instead of 200 . NGINX does not change the response from the custom default backend. An example of such custom backend is available inside the source repository at images/custom-error-pages . See also the Custom errors example.","title":"Custom errors"},{"location":"user-guide/custom-errors/#custom-errors","text":"When the custom-http-errors option is enabled, the Ingress controller configures NGINX so that it passes several HTTP headers down to its default-backend in case of error: Header Value X-Code HTTP status code returned by the request X-Format Value of the Accept header sent by the client X-Original-URI URI that caused the error X-Namespace Namespace where the backend Service is located X-Ingress-Name Name of the Ingress where the backend is defined X-Service-Name Name of the Service backing the backend X-Service-Port Port number of the Service backing the backend X-Request-ID Unique ID that identifies the request - same as for backend service A custom error backend can use this information to return the best possible representation of an error page. For example, if the value of the Accept header send by the client was application/json , a carefully crafted backend could decide to return the error payload as a JSON document instead of HTML. Important The custom backend is expected to return the correct HTTP status code instead of 200 . NGINX does not change the response from the custom default backend. An example of such custom backend is available inside the source repository at images/custom-error-pages . See also the Custom errors example.","title":"Custom errors"},{"location":"user-guide/default-backend/","text":"Default backend \u00b6 The default backend is a service which handles all URL paths and hosts the Ingress-NGINX controller doesn't understand (i.e., all the requests that are not mapped with an Ingress). Basically a default backend exposes two URLs: /healthz that returns 200 / that returns 404 Example The sub-directory /images/custom-error-pages provides an additional service for the purpose of customizing the error pages served via the default backend.","title":"Default backend"},{"location":"user-guide/default-backend/#default-backend","text":"The default backend is a service which handles all URL paths and hosts the Ingress-NGINX controller doesn't understand (i.e., all the requests that are not mapped with an Ingress). Basically a default backend exposes two URLs: /healthz that returns 200 / that returns 404 Example The sub-directory /images/custom-error-pages provides an additional service for the purpose of customizing the error pages served via the default backend.","title":"Default backend"},{"location":"user-guide/exposing-tcp-udp-services/","text":"Exposing TCP and UDP services \u00b6 Ingress does not support TCP or UDP services. For this reason this Ingress controller uses the flags --tcp-services-configmap and --udp-services-configmap to point to an existing config map where the key is the external port to use and the value indicates the service to expose using the format: <namespace/service name>:<service port>:[PROXY]:[PROXY] It is also possible to use a number or the name of the port. The two last fields are optional. Adding PROXY in either or both of the two last fields we can use Proxy Protocol decoding (listen) and/or encoding (proxy_pass) in a TCP service. The first PROXY controls the decode of the proxy protocol and the second PROXY controls the encoding using proxy protocol. This allows an incoming connection to be decoded or an outgoing connection to be encoded. It is also possible to arbitrate between two different proxies by turning on the decode and encode on a TCP service. The next example shows how to expose the service example-go running in the namespace default in the port 8080 using the port 9000 apiVersion : v1 kind : ConfigMap metadata : name : tcp-services namespace : ingress-nginx data : 9000 : \"default/example-go:8080\" Since 1.9.13 NGINX provides UDP Load Balancing . The next example shows how to expose the service kube-dns running in the namespace kube-system in the port 53 using the port 53 apiVersion : v1 kind : ConfigMap metadata : name : udp-services namespace : ingress-nginx data : 53 : \"kube-system/kube-dns:53\" If TCP/UDP proxy support is used, then those ports need to be exposed in the Service defined for the Ingress. apiVersion : v1 kind : Service metadata : name : ingress-nginx namespace : ingress-nginx labels : app.kubernetes.io/name : ingress-nginx app.kubernetes.io/part-of : ingress-nginx spec : type : LoadBalancer ports : - name : http port : 80 targetPort : 80 protocol : TCP - name : https port : 443 targetPort : 443 protocol : TCP - name : proxied-tcp-9000 port : 9000 targetPort : 9000 protocol : TCP selector : app.kubernetes.io/name : ingress-nginx app.kubernetes.io/part-of : ingress-nginx","title":"Exposing TCP and UDP services"},{"location":"user-guide/exposing-tcp-udp-services/#exposing-tcp-and-udp-services","text":"Ingress does not support TCP or UDP services. For this reason this Ingress controller uses the flags --tcp-services-configmap and --udp-services-configmap to point to an existing config map where the key is the external port to use and the value indicates the service to expose using the format: <namespace/service name>:<service port>:[PROXY]:[PROXY] It is also possible to use a number or the name of the port. The two last fields are optional. Adding PROXY in either or both of the two last fields we can use Proxy Protocol decoding (listen) and/or encoding (proxy_pass) in a TCP service. The first PROXY controls the decode of the proxy protocol and the second PROXY controls the encoding using proxy protocol. This allows an incoming connection to be decoded or an outgoing connection to be encoded. It is also possible to arbitrate between two different proxies by turning on the decode and encode on a TCP service. The next example shows how to expose the service example-go running in the namespace default in the port 8080 using the port 9000 apiVersion : v1 kind : ConfigMap metadata : name : tcp-services namespace : ingress-nginx data : 9000 : \"default/example-go:8080\" Since 1.9.13 NGINX provides UDP Load Balancing . The next example shows how to expose the service kube-dns running in the namespace kube-system in the port 53 using the port 53 apiVersion : v1 kind : ConfigMap metadata : name : udp-services namespace : ingress-nginx data : 53 : \"kube-system/kube-dns:53\" If TCP/UDP proxy support is used, then those ports need to be exposed in the Service defined for the Ingress. apiVersion : v1 kind : Service metadata : name : ingress-nginx namespace : ingress-nginx labels : app.kubernetes.io/name : ingress-nginx app.kubernetes.io/part-of : ingress-nginx spec : type : LoadBalancer ports : - name : http port : 80 targetPort : 80 protocol : TCP - name : https port : 443 targetPort : 443 protocol : TCP - name : proxied-tcp-9000 port : 9000 targetPort : 9000 protocol : TCP selector : app.kubernetes.io/name : ingress-nginx app.kubernetes.io/part-of : ingress-nginx","title":"Exposing TCP and UDP services"},{"location":"user-guide/external-articles/","text":"External Articles \u00b6 Pain(less) NGINX Ingress Accessing Kubernetes Pods from Outside of the Cluster Kubernetes - Redirect HTTP to HTTPS with ELB and the nginx ingress controller Configure Nginx Ingress Controller for TLS termination on Kubernetes on Azure","title":"External Articles"},{"location":"user-guide/external-articles/#external-articles","text":"Pain(less) NGINX Ingress Accessing Kubernetes Pods from Outside of the Cluster Kubernetes - Redirect HTTP to HTTPS with ELB and the nginx ingress controller Configure Nginx Ingress Controller for TLS termination on Kubernetes on Azure","title":"External Articles"},{"location":"user-guide/fcgi-services/","text":"Exposing FastCGI Servers \u00b6 FastCGI is a binary protocol for interfacing interactive programs with a web server . [...] (It's) aim is to reduce the overhead related to interfacing between web server and CGI programs, allowing a server to handle more web page requests per unit of time. \u2014 Wikipedia The ingress-nginx ingress controller can be used to directly expose FastCGI servers. Enabling FastCGI in your Ingress only requires setting the backend-protocol annotation to FCGI , and with a couple more annotations you can customize the way ingress-nginx handles the communication with your FastCGI server . Example Objects to Expose a FastCGI Pod \u00b6 The Pod example object below exposes port 9000 , which is the conventional FastCGI port. apiVersion : v1 kind : Pod metadata : name : example-app labels : app : example-app spec : containers : - name : example-app image : example-app:1.0 ports : - containerPort : 9000 name : fastcgi The Service object example below matches port 9000 from the Pod object above. apiVersion : v1 kind : Service metadata : name : example-service spec : selector : app : example-app ports : - port : 9000 targetPort : 9000 name : fastcgi And the Ingress and ConfigMap objects below demonstrates the supported FastCGI specific annotations (NGINX actually has 50 FastCGI directives, all of which have not been exposed in the ingress yet), and matches the service example-service , and the port named fastcgi from above. The ConfigMap must be created first for the Ingress Controller to be able to find it when the Ingress object is created, otherwise you will need to restart the Ingress Controller pods. # The ConfigMap MUST be created first for the ingress controller to be able to # find it when the Ingress object is created. apiVersion : v1 kind : ConfigMap metadata : name : example-cm data : SCRIPT_FILENAME : \"/example/index.php\" --- apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : nginx.ingress.kubernetes.io/backend-protocol : \"FCGI\" nginx.ingress.kubernetes.io/fastcgi-index : \"index.php\" nginx.ingress.kubernetes.io/fastcgi-params-configmap : \"example-cm\" name : example-app spec : ingressClassName : nginx rules : - host : app.example.com http : paths : - path : / pathType : Prefix backend : service : name : example-service port : name : fastcgi FastCGI Ingress Annotations \u00b6 To enable FastCGI, the nginx.ingress.kubernetes.io/backend-protocol annotation needs to be set to FCGI , which overrides the default HTTP value. nginx.ingress.kubernetes.io/backend-protocol: \"FCGI\" This enables the FastCGI mode for all paths defined in the Ingress object The nginx.ingress.kubernetes.io/fastcgi-index Annotation \u00b6 To specify an index file, the fastcgi-index annotation value can optionally be set. In the example below, the value is set to index.php . This annotation corresponds to the NGINX fastcgi_index directive . nginx.ingress.kubernetes.io/fastcgi-index: \"index.php\" The nginx.ingress.kubernetes.io/fastcgi-params-configmap Annotation \u00b6 To specify NGINX fastcgi_param directives , the fastcgi-params-configmap annotation is used, which in turn must lead to a ConfigMap object containing the NGINX fastcgi_param directives as key/values. nginx.ingress.kubernetes.io/fastcgi-params-configmap: \"example-configmap\" And the ConfigMap object to specify the SCRIPT_FILENAME and HTTP_PROXY NGINX's fastcgi_param directives will look like the following: apiVersion : v1 kind : ConfigMap metadata : name : example-configmap data : SCRIPT_FILENAME : \"/example/index.php\" HTTP_PROXY : \"\" Using the namespace/ prefix is also supported, for example: nginx.ingress.kubernetes.io/fastcgi-params-configmap: \"example-namespace/example-configmap\"","title":"Exposing FCGI services"},{"location":"user-guide/fcgi-services/#exposing-fastcgi-servers","text":"FastCGI is a binary protocol for interfacing interactive programs with a web server . [...] (It's) aim is to reduce the overhead related to interfacing between web server and CGI programs, allowing a server to handle more web page requests per unit of time. \u2014 Wikipedia The ingress-nginx ingress controller can be used to directly expose FastCGI servers. Enabling FastCGI in your Ingress only requires setting the backend-protocol annotation to FCGI , and with a couple more annotations you can customize the way ingress-nginx handles the communication with your FastCGI server .","title":"Exposing FastCGI Servers"},{"location":"user-guide/fcgi-services/#example-objects-to-expose-a-fastcgi-pod","text":"The Pod example object below exposes port 9000 , which is the conventional FastCGI port. apiVersion : v1 kind : Pod metadata : name : example-app labels : app : example-app spec : containers : - name : example-app image : example-app:1.0 ports : - containerPort : 9000 name : fastcgi The Service object example below matches port 9000 from the Pod object above. apiVersion : v1 kind : Service metadata : name : example-service spec : selector : app : example-app ports : - port : 9000 targetPort : 9000 name : fastcgi And the Ingress and ConfigMap objects below demonstrates the supported FastCGI specific annotations (NGINX actually has 50 FastCGI directives, all of which have not been exposed in the ingress yet), and matches the service example-service , and the port named fastcgi from above. The ConfigMap must be created first for the Ingress Controller to be able to find it when the Ingress object is created, otherwise you will need to restart the Ingress Controller pods. # The ConfigMap MUST be created first for the ingress controller to be able to # find it when the Ingress object is created. apiVersion : v1 kind : ConfigMap metadata : name : example-cm data : SCRIPT_FILENAME : \"/example/index.php\" --- apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : nginx.ingress.kubernetes.io/backend-protocol : \"FCGI\" nginx.ingress.kubernetes.io/fastcgi-index : \"index.php\" nginx.ingress.kubernetes.io/fastcgi-params-configmap : \"example-cm\" name : example-app spec : ingressClassName : nginx rules : - host : app.example.com http : paths : - path : / pathType : Prefix backend : service : name : example-service port : name : fastcgi","title":"Example Objects to Expose a FastCGI Pod"},{"location":"user-guide/fcgi-services/#fastcgi-ingress-annotations","text":"To enable FastCGI, the nginx.ingress.kubernetes.io/backend-protocol annotation needs to be set to FCGI , which overrides the default HTTP value. nginx.ingress.kubernetes.io/backend-protocol: \"FCGI\" This enables the FastCGI mode for all paths defined in the Ingress object","title":"FastCGI Ingress Annotations"},{"location":"user-guide/fcgi-services/#the-nginxingresskubernetesiofastcgi-index-annotation","text":"To specify an index file, the fastcgi-index annotation value can optionally be set. In the example below, the value is set to index.php . This annotation corresponds to the NGINX fastcgi_index directive . nginx.ingress.kubernetes.io/fastcgi-index: \"index.php\"","title":"The nginx.ingress.kubernetes.io/fastcgi-index Annotation"},{"location":"user-guide/fcgi-services/#the-nginxingresskubernetesiofastcgi-params-configmap-annotation","text":"To specify NGINX fastcgi_param directives , the fastcgi-params-configmap annotation is used, which in turn must lead to a ConfigMap object containing the NGINX fastcgi_param directives as key/values. nginx.ingress.kubernetes.io/fastcgi-params-configmap: \"example-configmap\" And the ConfigMap object to specify the SCRIPT_FILENAME and HTTP_PROXY NGINX's fastcgi_param directives will look like the following: apiVersion : v1 kind : ConfigMap metadata : name : example-configmap data : SCRIPT_FILENAME : \"/example/index.php\" HTTP_PROXY : \"\" Using the namespace/ prefix is also supported, for example: nginx.ingress.kubernetes.io/fastcgi-params-configmap: \"example-namespace/example-configmap\"","title":"The nginx.ingress.kubernetes.io/fastcgi-params-configmap Annotation"},{"location":"user-guide/ingress-path-matching/","text":"Ingress Path Matching \u00b6 Regular Expression Support \u00b6 Important Regular expressions and wild cards are not supported in the spec.rules.host field. Full hostnames must be used. The ingress controller supports case insensitive regular expressions in the spec.rules.http.paths.path field. This can be enabled by setting the nginx.ingress.kubernetes.io/use-regex annotation to true (the default is false). Hint Kubernetes only accept expressions that comply with the RE2 engine syntax. It is possible that valid expressions accepted by NGINX cannot be used with ingress-nginx, because the PCRE library (used in NGINX) supports a wider syntax than RE2. See the RE2 Syntax documentation for differences. See the description of the use-regex annotation for more details. apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : test-ingress annotations : nginx.ingress.kubernetes.io/use-regex : \"true\" spec : ingressClassName : nginx rules : - host : test.com http : paths : - path : /foo/.* pathType : Prefix backend : service : name : test port : number : 80 The preceding ingress definition would translate to the following location block within the NGINX configuration for the test.com server: location ~* \"^/foo/.*\" { ... } Path Priority \u00b6 In NGINX, regular expressions follow a first match policy. In order to enable more accurate path matching, ingress-nginx first orders the paths by descending length before writing them to the NGINX template as location blocks. Please read the warning before using regular expressions in your ingress definitions. Example \u00b6 Let the following two ingress definitions be created: apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : test-ingress-1 spec : ingressClassName : nginx rules : - host : test.com http : paths : - path : /foo/bar pathType : Prefix backend : service : name : service1 port : number : 80 - path : /foo/bar/ pathType : Prefix backend : service : name : service2 port : number : 80 apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : test-ingress-2 annotations : nginx.ingress.kubernetes.io/rewrite-target : /$1 spec : ingressClassName : nginx rules : - host : test.com http : paths : - path : /foo/bar/(.+) pathType : Prefix backend : service : name : service3 port : number : 80 The ingress controller would define the following location blocks, in order of descending length, within the NGINX template for the test.com server: location ~* ^/foo/bar/.+ { ... } location ~* \"^/foo/bar/\" { ... } location ~* \"^/foo/bar\" { ... } The following request URI's would match the corresponding location blocks: test.com/foo/bar/1 matches ~* ^/foo/bar/.+ and will go to service 3. test.com/foo/bar/ matches ~* ^/foo/bar/ and will go to service 2. test.com/foo/bar matches ~* ^/foo/bar and will go to service 1. IMPORTANT NOTES : If the use-regex OR rewrite-target annotation is used on any Ingress for a given host, then the case insensitive regular expression location modifier will be enforced on ALL paths for a given host regardless of what Ingress they are defined on. Warning \u00b6 The following example describes a case that may inflict unwanted path matching behavior. This case is expected and a result of NGINX's a first match policy for paths that use the regular expression location modifier . For more information about how a path is chosen, please read the following article: \"Understanding Nginx Server and Location Block Selection Algorithms\" . Example \u00b6 Let the following ingress be defined: apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : test-ingress-3 annotations : nginx.ingress.kubernetes.io/use-regex : \"true\" spec : ingressClassName : nginx rules : - host : test.com http : paths : - path : /foo/bar/bar pathType : Prefix backend : service : name : test port : number : 80 - path : /foo/bar/[A-Z0-9]{3} pathType : Prefix backend : service : name : test port : number : 80 The ingress controller would define the following location blocks (in this order) within the NGINX template for the test.com server: location ~* \"^/foo/bar/[A-Z0-9]{3}\" { ... } location ~* \"^/foo/bar/bar\" { ... } A request to test.com/foo/bar/bar would match the ^/foo/bar/[A-Z0-9]{3} location block instead of the longest EXACT matching path.","title":"Regular expressions in paths"},{"location":"user-guide/ingress-path-matching/#ingress-path-matching","text":"","title":"Ingress Path Matching"},{"location":"user-guide/ingress-path-matching/#regular-expression-support","text":"Important Regular expressions and wild cards are not supported in the spec.rules.host field. Full hostnames must be used. The ingress controller supports case insensitive regular expressions in the spec.rules.http.paths.path field. This can be enabled by setting the nginx.ingress.kubernetes.io/use-regex annotation to true (the default is false). Hint Kubernetes only accept expressions that comply with the RE2 engine syntax. It is possible that valid expressions accepted by NGINX cannot be used with ingress-nginx, because the PCRE library (used in NGINX) supports a wider syntax than RE2. See the RE2 Syntax documentation for differences. See the description of the use-regex annotation for more details. apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : test-ingress annotations : nginx.ingress.kubernetes.io/use-regex : \"true\" spec : ingressClassName : nginx rules : - host : test.com http : paths : - path : /foo/.* pathType : Prefix backend : service : name : test port : number : 80 The preceding ingress definition would translate to the following location block within the NGINX configuration for the test.com server: location ~* \"^/foo/.*\" { ... }","title":"Regular Expression Support"},{"location":"user-guide/ingress-path-matching/#path-priority","text":"In NGINX, regular expressions follow a first match policy. In order to enable more accurate path matching, ingress-nginx first orders the paths by descending length before writing them to the NGINX template as location blocks. Please read the warning before using regular expressions in your ingress definitions.","title":"Path Priority"},{"location":"user-guide/ingress-path-matching/#example","text":"Let the following two ingress definitions be created: apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : test-ingress-1 spec : ingressClassName : nginx rules : - host : test.com http : paths : - path : /foo/bar pathType : Prefix backend : service : name : service1 port : number : 80 - path : /foo/bar/ pathType : Prefix backend : service : name : service2 port : number : 80 apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : test-ingress-2 annotations : nginx.ingress.kubernetes.io/rewrite-target : /$1 spec : ingressClassName : nginx rules : - host : test.com http : paths : - path : /foo/bar/(.+) pathType : Prefix backend : service : name : service3 port : number : 80 The ingress controller would define the following location blocks, in order of descending length, within the NGINX template for the test.com server: location ~* ^/foo/bar/.+ { ... } location ~* \"^/foo/bar/\" { ... } location ~* \"^/foo/bar\" { ... } The following request URI's would match the corresponding location blocks: test.com/foo/bar/1 matches ~* ^/foo/bar/.+ and will go to service 3. test.com/foo/bar/ matches ~* ^/foo/bar/ and will go to service 2. test.com/foo/bar matches ~* ^/foo/bar and will go to service 1. IMPORTANT NOTES : If the use-regex OR rewrite-target annotation is used on any Ingress for a given host, then the case insensitive regular expression location modifier will be enforced on ALL paths for a given host regardless of what Ingress they are defined on.","title":"Example"},{"location":"user-guide/ingress-path-matching/#warning","text":"The following example describes a case that may inflict unwanted path matching behavior. This case is expected and a result of NGINX's a first match policy for paths that use the regular expression location modifier . For more information about how a path is chosen, please read the following article: \"Understanding Nginx Server and Location Block Selection Algorithms\" .","title":"Warning"},{"location":"user-guide/ingress-path-matching/#example_1","text":"Let the following ingress be defined: apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : test-ingress-3 annotations : nginx.ingress.kubernetes.io/use-regex : \"true\" spec : ingressClassName : nginx rules : - host : test.com http : paths : - path : /foo/bar/bar pathType : Prefix backend : service : name : test port : number : 80 - path : /foo/bar/[A-Z0-9]{3} pathType : Prefix backend : service : name : test port : number : 80 The ingress controller would define the following location blocks (in this order) within the NGINX template for the test.com server: location ~* \"^/foo/bar/[A-Z0-9]{3}\" { ... } location ~* \"^/foo/bar/bar\" { ... } A request to test.com/foo/bar/bar would match the ^/foo/bar/[A-Z0-9]{3} location block instead of the longest EXACT matching path.","title":"Example"},{"location":"user-guide/miscellaneous/","text":"Miscellaneous \u00b6 Source IP address \u00b6 By default NGINX uses the content of the header X-Forwarded-For as the source of truth to get information about the client IP address. This works without issues in L7 if we configure the setting proxy-real-ip-cidr with the correct information of the IP/network address of trusted external load balancer. If the ingress controller is running in AWS we need to use the VPC IPv4 CIDR. Another option is to enable proxy protocol using use-proxy-protocol: \"true\" . In this mode NGINX does not use the content of the header to get the source IP address of the connection. Path types \u00b6 Each path in an Ingress is required to have a corresponding path type. Paths that do not include an explicit pathType will fail validation. By default NGINX path type is Prefix to not break existing definitions Proxy Protocol \u00b6 If you are using a L4 proxy to forward the traffic to the NGINX pods and terminate HTTP/HTTPS there, you will lose the remote endpoint's IP address. To prevent this you could use the Proxy Protocol for forwarding traffic, this will send the connection details before forwarding the actual TCP connection itself. Amongst others ELBs in AWS and HAProxy support Proxy Protocol. Websockets \u00b6 Support for websockets is provided by NGINX out of the box. No special configuration required. The only requirement to avoid the close of connections is the increase of the values of proxy-read-timeout and proxy-send-timeout . The default value of this settings is 60 seconds . A more adequate value to support websockets is a value higher than one hour ( 3600 ). Important If the NGINX ingress controller is exposed with a service type=LoadBalancer make sure the protocol between the loadbalancer and NGINX is TCP. Optimizing TLS Time To First Byte (TTTFB) \u00b6 NGINX provides the configuration option ssl_buffer_size to allow the optimization of the TLS record size. This improves the TLS Time To First Byte (TTTFB). The default value in the Ingress controller is 4k (NGINX default is 16k ). Retries in non-idempotent methods \u00b6 Since 1.9.13 NGINX will not retry non-idempotent requests (POST, LOCK, PATCH) in case of an error. The previous behavior can be restored using retry-non-idempotent=true in the configuration ConfigMap. Limitations \u00b6 Ingress rules for TLS require the definition of the field host Why endpoints and not services \u00b6 The NGINX ingress controller does not use Services to route traffic to the pods. Instead it uses the Endpoints API in order to bypass kube-proxy to allow NGINX features like session affinity and custom load balancing algorithms. It also removes some overhead, such as conntrack entries for iptables DNAT.","title":"Miscellaneous"},{"location":"user-guide/miscellaneous/#miscellaneous","text":"","title":"Miscellaneous"},{"location":"user-guide/miscellaneous/#source-ip-address","text":"By default NGINX uses the content of the header X-Forwarded-For as the source of truth to get information about the client IP address. This works without issues in L7 if we configure the setting proxy-real-ip-cidr with the correct information of the IP/network address of trusted external load balancer. If the ingress controller is running in AWS we need to use the VPC IPv4 CIDR. Another option is to enable proxy protocol using use-proxy-protocol: \"true\" . In this mode NGINX does not use the content of the header to get the source IP address of the connection.","title":"Source IP address"},{"location":"user-guide/miscellaneous/#path-types","text":"Each path in an Ingress is required to have a corresponding path type. Paths that do not include an explicit pathType will fail validation. By default NGINX path type is Prefix to not break existing definitions","title":"Path types"},{"location":"user-guide/miscellaneous/#proxy-protocol","text":"If you are using a L4 proxy to forward the traffic to the NGINX pods and terminate HTTP/HTTPS there, you will lose the remote endpoint's IP address. To prevent this you could use the Proxy Protocol for forwarding traffic, this will send the connection details before forwarding the actual TCP connection itself. Amongst others ELBs in AWS and HAProxy support Proxy Protocol.","title":"Proxy Protocol"},{"location":"user-guide/miscellaneous/#websockets","text":"Support for websockets is provided by NGINX out of the box. No special configuration required. The only requirement to avoid the close of connections is the increase of the values of proxy-read-timeout and proxy-send-timeout . The default value of this settings is 60 seconds . A more adequate value to support websockets is a value higher than one hour ( 3600 ). Important If the NGINX ingress controller is exposed with a service type=LoadBalancer make sure the protocol between the loadbalancer and NGINX is TCP.","title":"Websockets"},{"location":"user-guide/miscellaneous/#optimizing-tls-time-to-first-byte-tttfb","text":"NGINX provides the configuration option ssl_buffer_size to allow the optimization of the TLS record size. This improves the TLS Time To First Byte (TTTFB). The default value in the Ingress controller is 4k (NGINX default is 16k ).","title":"Optimizing TLS Time To First Byte (TTTFB)"},{"location":"user-guide/miscellaneous/#retries-in-non-idempotent-methods","text":"Since 1.9.13 NGINX will not retry non-idempotent requests (POST, LOCK, PATCH) in case of an error. The previous behavior can be restored using retry-non-idempotent=true in the configuration ConfigMap.","title":"Retries in non-idempotent methods"},{"location":"user-guide/miscellaneous/#limitations","text":"Ingress rules for TLS require the definition of the field host","title":"Limitations"},{"location":"user-guide/miscellaneous/#why-endpoints-and-not-services","text":"The NGINX ingress controller does not use Services to route traffic to the pods. Instead it uses the Endpoints API in order to bypass kube-proxy to allow NGINX features like session affinity and custom load balancing algorithms. It also removes some overhead, such as conntrack entries for iptables DNAT.","title":"Why endpoints and not services"},{"location":"user-guide/monitoring/","text":"Monitoring \u00b6 Two different methods to install and configure Prometheus and Grafana are described in this doc. * Prometheus and Grafana installation using Pod Annotations. This installs Prometheus and Grafana in the same namespace as NGINX Ingress * Prometheus and Grafana installation using Service Monitors. This installs Prometheus and Grafana in two different namespaces. This is the preferred method, and helm charts supports this by default. Prometheus and Grafana installation using Pod Annotations \u00b6 This tutorial will show you how to install Prometheus and Grafana for scraping the metrics of the NGINX Ingress controller. Important This example uses emptyDir volumes for Prometheus and Grafana. This means once the pod gets terminated you will lose all the data. Before You Begin \u00b6 The NGINX Ingress controller should already be deployed according to the deployment instructions here . The controller should be configured for exporting metrics. This requires 3 configurations to the controller. These configurations are : controller.metrics.enabled=true controller.podAnnotations.\"prometheus.io/scrape\"=\"true\" controller.podAnnotations.\"prometheus.io/port\"=\"10254\" The easiest way to configure the controller for metrics is via helm upgrade. Assuming you have installed the ingress-nginx controller as a helm release named ingress-nginx, then you can simply type the command shown below : helm upgrade ingress-nginx ingress-nginx \\ --repo https://kubernetes.github.io/ingress-nginx \\ --namespace ingress-nginx \\ --set controller.metrics.enabled=true \\ --set-string controller.podAnnotations.\"prometheus\\.io/scrape\"=\"true\" \\ --set-string controller.podAnnotations.\"prometheus\\.io/port\"=\"10254\" You can validate that the controller is configured for metrics by looking at the values of the installed release, like this: helm get values ingress-nginx --namespace ingress-nginx You should be able to see the values shown below: .. controller: metrics: enabled: true service: annotations: prometheus.io/port: \"10254\" prometheus.io/scrape: \"true\" .. If you are not using helm , you will have to edit your manifests like this: Service manifest: apiVersion: v1 kind: Service metadata: annotations: prometheus.io/scrape: \"true\" prometheus.io/port: \"10254\" .. spec: ports: - name: prometheus port: 10254 targetPort: prometheus .. Deployment manifest: apiVersion: v1 kind: Deployment metadata: annotations: prometheus.io/scrape: \"true\" prometheus.io/port: \"10254\" .. spec: ports: - name: prometheus containerPort: 10254 .. Deploy and configure Prometheus Server \u00b6 Note that the kustomize bases used in this tutorial are stored in the deploy folder of the GitHub repository kubernetes/ingress-nginx . The Prometheus server must be configured so that it can discover endpoints of services. If a Prometheus server is already running in the cluster and if it is configured in a way that it can find the ingress controller pods, no extra configuration is needed. If there is no existing Prometheus server running, the rest of this tutorial will guide you through the steps needed to deploy a properly configured Prometheus server. Running the following command deploys prometheus in Kubernetes: kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/prometheus/ Prometheus Dashboard \u00b6 Open Prometheus dashboard in a web browser: kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-http-backend ClusterIP 10.103.59.201 <none> 80/TCP 3d ingress-nginx NodePort 10.97.44.72 <none> 80:30100/TCP,443:30154/TCP,10254:32049/TCP 5h prometheus-server NodePort 10.98.233.86 <none> 9090:32630/TCP 1m Obtain the IP address of the nodes in the running cluster: kubectl get nodes -o wide In some cases where the node only have internal IP addresses we need to execute: kubectl get nodes --selector=kubernetes.io/role!=master -o jsonpath={.items[*].status.addresses[?\\(@.type==\\\"InternalIP\\\"\\)].address} 10.192.0.2 10.192.0.3 10.192.0.4 Open your browser and visit the following URL: http://{node IP address}:{prometheus-svc-nodeport} to load the Prometheus Dashboard. According to the above example, this URL will be http://10.192.0.3:32630 Grafana \u00b6 Install grafana using the below command kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/grafana/ Look at the services kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-http-backend ClusterIP 10.103.59.201 <none> 80/TCP 3d ingress-nginx NodePort 10.97.44.72 <none> 80:30100/TCP,443:30154/TCP,10254:32049/TCP 5h prometheus-server NodePort 10.98.233.86 <none> 9090:32630/TCP 10m grafana NodePort 10.98.233.87 <none> 3000:31086/TCP 10m Open your browser and visit the following URL: http://{node IP address}:{grafana-svc-nodeport} to load the Grafana Dashboard. According to the above example, this URL will be http://10.192.0.3:31086 The username and password is admin After the login you can import the Grafana dashboard from official dashboards , by following steps given below : Navigate to lefthand panel of grafana Hover on the gearwheel icon for Configuration and click \"Data Sources\" Click \"Add data source\" Select \"Prometheus\" Enter the details (note: I used http://CLUSTER_IP_PROMETHEUS_SVC:9090) Left menu (hover over +) -> Dashboard Click \"Import\" Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/grafana/dashboards/nginx.json Click Import JSON Select the Prometheus data source Click \"Import\" Caveats \u00b6 Wildcard ingresses \u00b6 By default request metrics are labeled with the hostname. When you have a wildcard domain ingress, then there will be no metrics for that ingress (to prevent the metrics from exploding in cardinality). To get metrics in this case you need to run the ingress controller with --metrics-per-host=false (you will lose labeling by hostname, but still have labeling by ingress). Grafana dashboard using ingress resource \u00b6 If you want to expose the dashboard for grafana using a ingress resource, then you can : change the service type of the prometheus-server service and the grafana service to \"ClusterIP\" like this : kubectl -n ingress-nginx edit svc grafana This will open the currently deployed service grafana in the default editor configured in your shell (vi/nvim/nano/other) scroll down to line 34 that looks like \"type: NodePort\" change it to look like \"type: ClusterIP\". Save and exit. create a ingress resource with backend as \"grafana\" and port as \"3000\" Similarly, you can edit the service \"prometheus-server\" and add a ingress resource. Prometheus and Grafana installation using Service Monitors \u00b6 This document assumes you're using helm and using the kube-prometheus-stack package to install Prometheus and Grafana. Verify NGINX Ingress controller is installed \u00b6 The NGINX Ingress controller should already be deployed according to the deployment instructions here . To check if Ingress controller is deployed, kubectl get pods -n ingress-nginx The result should look something like: NAME READY STATUS RESTARTS AGE ingress-nginx-controller-7c489dc7b7-ccrf6 1/1 Running 0 19h Verify Prometheus is installed \u00b6 To check if Prometheus is already deployed, run the following command: helm ls -A NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION ingress-nginx ingress-nginx 10 2022-01-20 18:08:55.267373 -0800 PST deployed ingress-nginx-4.0.16 1.1.1 prometheus prometheus 1 2022-01-20 16:07:25.086828 -0800 PST deployed kube-prometheus-stack-30.1.0 0.53.1 - Notice that prometheus is installed in a differenet namespace than ingress-nginx If prometheus is not installed, then you can install from here Re-configure NGINX Ingress controller \u00b6 The Ingress NGINX controller needs to be reconfigured for exporting metrics. This requires 3 additional configurations to the controller. These configurations are : controller.metrics.enabled=true controller.metrics.serviceMonitor.enabled=true controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" The easiest way of doing this is to helm upgrade helm upgrade ingress-nginx ingress-nginx/ingress-nginx \\ --namespace ingress-nginx \\ --set controller.metrics.enabled=true \\ --set controller.metrics.serviceMonitor.enabled=true \\ --set controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" Here controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" should match the name of the helm release of the kube-prometheus-stack You can validate that the controller has been successfully reconfigured to export metrics by looking at the values of the installed release, like this: helm get values ingress-nginx --namespace ingress-nginx controller: metrics: enabled: true serviceMonitor: additionalLabels: release: prometheus enabled: true Configure Prometheus \u00b6 Since Prometheus is running in a different namespace and not in the ingress-nginx namespace, it would not be able to discover ServiceMonitors in other namespaces when installed. Reconfigure your kube-prometheus-stack Helm installation to set serviceMonitorSelectorNilUsesHelmValues flag to false. By default, Prometheus only discovers PodMonitors within its own namespace. This should be disabled by setting podMonitorSelectorNilUsesHelmValues to false The configurations required are: prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false The easiest way of doing this is to use helm upgrade ... helm upgrade prometheus prometheus-community/kube-prometheus-stack \\ --namespace prometheus \\ --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \\ --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false You can validate that Prometheus has been reconfigured by looking at the values of the installed release, like this: helm get values prometheus --namespace prometheus You should be able to see the values shown below: prometheus: prometheusSpec: podMonitorSelectorNilUsesHelmValues: false serviceMonitorSelectorNilUsesHelmValues: false Connect and view Prometheus dashboard \u00b6 Port forward to Prometheus service. Find out the name of the prometheus service by using the following command: kubectl get svc -n prometheus The result of this command would look like: NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 7h46m prometheus-grafana ClusterIP 10.106.28.162 <none> 80/TCP 7h46m prometheus-kube-prometheus-alertmanager ClusterIP 10.108.125.245 <none> 9093/TCP 7h46m prometheus-kube-prometheus-operator ClusterIP 10.110.220.1 <none> 443/TCP 7h46m prometheus-kube-prometheus-prometheus ClusterIP 10.102.72.134 <none> 9090/TCP 7h46m prometheus-kube-state-metrics ClusterIP 10.104.231.181 <none> 8080/TCP 7h46m prometheus-operated ClusterIP None <none> 9090/TCP 7h46m prometheus-prometheus-node-exporter ClusterIP 10.96.247.128 <none> 9100/TCP 7h46m prometheus-kube-prometheus-prometheus is the service we want to port forward to. We can do so using the following command: kubectl port-forward svc/prometheus-kube-prometheus-prometheus -n prometheus 9090:9090 When you run the above command, you should see something like: Forwarding from 127.0.0.1:9090 -> 9090 Forwarding from [::1]:9090 -> 9090 - Open your browser and visit the following URL http://localhost:{port-forwarded-port} according to the above example it would be, http://localhost:9090 Connect and view Grafana dashboard \u00b6 Port forward to Grafana service. Find out the name of the Grafana service by using the following command: kubectl get svc -n prometheus The result of this command would look like: NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 7h46m prometheus-grafana ClusterIP 10.106.28.162 <none> 80/TCP 7h46m prometheus-kube-prometheus-alertmanager ClusterIP 10.108.125.245 <none> 9093/TCP 7h46m prometheus-kube-prometheus-operator ClusterIP 10.110.220.1 <none> 443/TCP 7h46m prometheus-kube-prometheus-prometheus ClusterIP 10.102.72.134 <none> 9090/TCP 7h46m prometheus-kube-state-metrics ClusterIP 10.104.231.181 <none> 8080/TCP 7h46m prometheus-operated ClusterIP None <none> 9090/TCP 7h46m prometheus-prometheus-node-exporter ClusterIP 10.96.247.128 <none> 9100/TCP 7h46m prometheus-grafana is the service we want to port forward to. We can do so using the following command: kubectl port-forward svc/prometheus-grafana 3000:80 -n prometheus When you run the above command, you should see something like: Forwarding from 127.0.0.1:3000 -> 3000 Forwarding from [::1]:3000 -> 3000 - Open your browser and visit the following URL http://localhost:{port-forwarded-port} according to the above example it would be, http://localhost:3000 The default username/ password is admin/prom-operator - After the login you can import the Grafana dashboard from official dashboards , by following steps given below : Navigate to lefthand panel of grafana Hover on the gearwheel icon for Configuration and click \"Data Sources\" Click \"Add data source\" Select \"Prometheus\" Enter the details (note: I used http://10.102.72.134:9090 which is the CLUSTER-IP for Prometheus service) Left menu (hover over +) -> Dashboard Click \"Import\" Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/grafana/dashboards/nginx.json Click Import JSON Select the Prometheus data source Click \"Import\" Exposed metrics \u00b6 Prometheus metrics are exposed on port 10254. Request metrics \u00b6 nginx_ingress_controller_request_duration_seconds Histogram The request processing time in milliseconds (affected by client speed) nginx var: request_time nginx_ingress_controller_response_duration_seconds Histogram The time spent on receiving the response from the upstream server (affected by client speed) nginx var: upstream_response_time nginx_ingress_controller_header_duration_seconds Histogram The time spent on receiving first header from the upstream server nginx var: upstream_header_time nginx_ingress_controller_connect_duration_seconds Histogram The time spent on establishing a connection with the upstream server nginx var: upstream_connect_time nginx_ingress_controller_response_size Histogram The response length (including request line, header, and request body) nginx var: bytes_sent nginx_ingress_controller_request_size Histogram The request length (including request line, header, and request body) nginx var: request_length nginx_ingress_controller_requests Counter The total number of client requests nginx_ingress_controller_bytes_sent Histogram The number of bytes sent to a client. Deprecated , use nginx_ingress_controller_response_size nginx var: bytes_sent nginx_ingress_controller_ingress_upstream_latency_seconds Summary Upstream service latency per Ingress. Deprecated , use nginx_ingress_controller_connect_duration_seconds nginx var: upstream_connect_time # HELP nginx_ingress_controller_bytes_sent The number of bytes sent to a client. DEPRECATED! Use nginx_ingress_controller_response_size # TYPE nginx_ingress_controller_bytes_sent histogram # HELP nginx_ingress_controller_connect_duration_seconds The time spent on establishing a connection with the upstream server # TYPE nginx_ingress_controller_connect_duration_seconds nginx_ingress_controller_connect_duration_seconds * HELP nginx_ingress_controller_header_duration_seconds The time spent on receiving first header from the upstream server # TYPE nginx_ingress_controller_header_duration_seconds histogram # HELP nginx_ingress_controller_ingress_upstream_latency_seconds Upstream service latency per Ingress DEPRECATED! Use nginx_ingress_controller_connect_duration_seconds # TYPE nginx_ingress_controller_ingress_upstream_latency_seconds summary # HELP nginx_ingress_controller_request_duration_seconds The request processing time in milliseconds # TYPE nginx_ingress_controller_request_duration_seconds histogram # HELP nginx_ingress_controller_request_size The request length (including request line, header, and request body) # TYPE nginx_ingress_controller_request_size histogram # HELP nginx_ingress_controller_requests The total number of client requests. # TYPE nginx_ingress_controller_requests counter # HELP nginx_ingress_controller_response_duration_seconds The time spent on receiving the response from the upstream server # TYPE nginx_ingress_controller_response_duration_seconds histogram # HELP nginx_ingress_controller_response_size The response length (including request line, header, and request body) # TYPE nginx_ingress_controller_response_size histogram Nginx process metrics \u00b6 # HELP nginx_ingress_controller_nginx_process_connections current number of client connections with state {active, reading, writing, waiting} # TYPE nginx_ingress_controller_nginx_process_connections gauge # HELP nginx_ingress_controller_nginx_process_connections_total total number of connections with state {accepted, handled} # TYPE nginx_ingress_controller_nginx_process_connections_total counter # HELP nginx_ingress_controller_nginx_process_cpu_seconds_total Cpu usage in seconds # TYPE nginx_ingress_controller_nginx_process_cpu_seconds_total counter # HELP nginx_ingress_controller_nginx_process_num_procs number of processes # TYPE nginx_ingress_controller_nginx_process_num_procs gauge # HELP nginx_ingress_controller_nginx_process_oldest_start_time_seconds start time in seconds since 1970/01/01 # TYPE nginx_ingress_controller_nginx_process_oldest_start_time_seconds gauge # HELP nginx_ingress_controller_nginx_process_read_bytes_total number of bytes read # TYPE nginx_ingress_controller_nginx_process_read_bytes_total counter # HELP nginx_ingress_controller_nginx_process_requests_total total number of client requests # TYPE nginx_ingress_controller_nginx_process_requests_total counter # HELP nginx_ingress_controller_nginx_process_resident_memory_bytes number of bytes of memory in use # TYPE nginx_ingress_controller_nginx_process_resident_memory_bytes gauge # HELP nginx_ingress_controller_nginx_process_virtual_memory_bytes number of bytes of memory in use # TYPE nginx_ingress_controller_nginx_process_virtual_memory_bytes gauge # HELP nginx_ingress_controller_nginx_process_write_bytes_total number of bytes written # TYPE nginx_ingress_controller_nginx_process_write_bytes_total counter Controller metrics \u00b6 # HELP nginx_ingress_controller_build_info A metric with a constant '1' labeled with information about the build. # TYPE nginx_ingress_controller_build_info gauge # HELP nginx_ingress_controller_check_success Cumulative number of Ingress controller syntax check operations # TYPE nginx_ingress_controller_check_success counter # HELP nginx_ingress_controller_config_hash Running configuration hash actually running # TYPE nginx_ingress_controller_config_hash gauge # HELP nginx_ingress_controller_config_last_reload_successful Whether the last configuration reload attempt was successful # TYPE nginx_ingress_controller_config_last_reload_successful gauge # HELP nginx_ingress_controller_config_last_reload_successful_timestamp_seconds Timestamp of the last successful configuration reload. # TYPE nginx_ingress_controller_config_last_reload_successful_timestamp_seconds gauge # HELP nginx_ingress_controller_ssl_certificate_info Hold all labels associated to a certificate # TYPE nginx_ingress_controller_ssl_certificate_info gauge # HELP nginx_ingress_controller_success Cumulative number of Ingress controller reload operations # TYPE nginx_ingress_controller_success counter Admission metrics \u00b6 # HELP nginx_ingress_controller_admission_config_size The size of the tested configuration # TYPE nginx_ingress_controller_admission_config_size gauge # HELP nginx_ingress_controller_admission_render_duration The processing duration of ingresses rendering by the admission controller (float seconds) # TYPE nginx_ingress_controller_admission_render_duration gauge # HELP nginx_ingress_controller_admission_render_ingresses The length of ingresses rendered by the admission controller # TYPE nginx_ingress_controller_admission_render_ingresses gauge # HELP nginx_ingress_controller_admission_roundtrip_duration The complete duration of the admission controller at the time to process a new event (float seconds) # TYPE nginx_ingress_controller_admission_roundtrip_duration gauge # HELP nginx_ingress_controller_admission_tested_duration The processing duration of the admission controller tests (float seconds) # TYPE nginx_ingress_controller_admission_tested_duration gauge # HELP nginx_ingress_controller_admission_tested_ingresses The length of ingresses processed by the admission controller # TYPE nginx_ingress_controller_admission_tested_ingresses gauge Histogram buckets \u00b6 You can configure buckets for histogram metrics using these command line options (here are their default values): * --time-buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] * --length-buckets=[10, 20, 30, 40, 50, 60, 70, 80, 90, 100] * --size-buckets=[10, 100, 1000, 10000, 100000, 1e+06, 1e+07]","title":"Prometheus and Grafana installation"},{"location":"user-guide/monitoring/#monitoring","text":"Two different methods to install and configure Prometheus and Grafana are described in this doc. * Prometheus and Grafana installation using Pod Annotations. This installs Prometheus and Grafana in the same namespace as NGINX Ingress * Prometheus and Grafana installation using Service Monitors. This installs Prometheus and Grafana in two different namespaces. This is the preferred method, and helm charts supports this by default.","title":"Monitoring"},{"location":"user-guide/monitoring/#prometheus-and-grafana-installation-using-pod-annotations","text":"This tutorial will show you how to install Prometheus and Grafana for scraping the metrics of the NGINX Ingress controller. Important This example uses emptyDir volumes for Prometheus and Grafana. This means once the pod gets terminated you will lose all the data.","title":"Prometheus and Grafana installation using Pod Annotations"},{"location":"user-guide/monitoring/#before-you-begin","text":"The NGINX Ingress controller should already be deployed according to the deployment instructions here . The controller should be configured for exporting metrics. This requires 3 configurations to the controller. These configurations are : controller.metrics.enabled=true controller.podAnnotations.\"prometheus.io/scrape\"=\"true\" controller.podAnnotations.\"prometheus.io/port\"=\"10254\" The easiest way to configure the controller for metrics is via helm upgrade. Assuming you have installed the ingress-nginx controller as a helm release named ingress-nginx, then you can simply type the command shown below : helm upgrade ingress-nginx ingress-nginx \\ --repo https://kubernetes.github.io/ingress-nginx \\ --namespace ingress-nginx \\ --set controller.metrics.enabled=true \\ --set-string controller.podAnnotations.\"prometheus\\.io/scrape\"=\"true\" \\ --set-string controller.podAnnotations.\"prometheus\\.io/port\"=\"10254\" You can validate that the controller is configured for metrics by looking at the values of the installed release, like this: helm get values ingress-nginx --namespace ingress-nginx You should be able to see the values shown below: .. controller: metrics: enabled: true service: annotations: prometheus.io/port: \"10254\" prometheus.io/scrape: \"true\" .. If you are not using helm , you will have to edit your manifests like this: Service manifest: apiVersion: v1 kind: Service metadata: annotations: prometheus.io/scrape: \"true\" prometheus.io/port: \"10254\" .. spec: ports: - name: prometheus port: 10254 targetPort: prometheus .. Deployment manifest: apiVersion: v1 kind: Deployment metadata: annotations: prometheus.io/scrape: \"true\" prometheus.io/port: \"10254\" .. spec: ports: - name: prometheus containerPort: 10254 ..","title":"Before You Begin"},{"location":"user-guide/monitoring/#deploy-and-configure-prometheus-server","text":"Note that the kustomize bases used in this tutorial are stored in the deploy folder of the GitHub repository kubernetes/ingress-nginx . The Prometheus server must be configured so that it can discover endpoints of services. If a Prometheus server is already running in the cluster and if it is configured in a way that it can find the ingress controller pods, no extra configuration is needed. If there is no existing Prometheus server running, the rest of this tutorial will guide you through the steps needed to deploy a properly configured Prometheus server. Running the following command deploys prometheus in Kubernetes: kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/prometheus/","title":"Deploy and configure Prometheus Server"},{"location":"user-guide/monitoring/#prometheus-dashboard","text":"Open Prometheus dashboard in a web browser: kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-http-backend ClusterIP 10.103.59.201 <none> 80/TCP 3d ingress-nginx NodePort 10.97.44.72 <none> 80:30100/TCP,443:30154/TCP,10254:32049/TCP 5h prometheus-server NodePort 10.98.233.86 <none> 9090:32630/TCP 1m Obtain the IP address of the nodes in the running cluster: kubectl get nodes -o wide In some cases where the node only have internal IP addresses we need to execute: kubectl get nodes --selector=kubernetes.io/role!=master -o jsonpath={.items[*].status.addresses[?\\(@.type==\\\"InternalIP\\\"\\)].address} 10.192.0.2 10.192.0.3 10.192.0.4 Open your browser and visit the following URL: http://{node IP address}:{prometheus-svc-nodeport} to load the Prometheus Dashboard. According to the above example, this URL will be http://10.192.0.3:32630","title":"Prometheus Dashboard"},{"location":"user-guide/monitoring/#grafana","text":"Install grafana using the below command kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/grafana/ Look at the services kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-http-backend ClusterIP 10.103.59.201 <none> 80/TCP 3d ingress-nginx NodePort 10.97.44.72 <none> 80:30100/TCP,443:30154/TCP,10254:32049/TCP 5h prometheus-server NodePort 10.98.233.86 <none> 9090:32630/TCP 10m grafana NodePort 10.98.233.87 <none> 3000:31086/TCP 10m Open your browser and visit the following URL: http://{node IP address}:{grafana-svc-nodeport} to load the Grafana Dashboard. According to the above example, this URL will be http://10.192.0.3:31086 The username and password is admin After the login you can import the Grafana dashboard from official dashboards , by following steps given below : Navigate to lefthand panel of grafana Hover on the gearwheel icon for Configuration and click \"Data Sources\" Click \"Add data source\" Select \"Prometheus\" Enter the details (note: I used http://CLUSTER_IP_PROMETHEUS_SVC:9090) Left menu (hover over +) -> Dashboard Click \"Import\" Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/grafana/dashboards/nginx.json Click Import JSON Select the Prometheus data source Click \"Import\"","title":"Grafana"},{"location":"user-guide/monitoring/#caveats","text":"","title":"Caveats"},{"location":"user-guide/monitoring/#wildcard-ingresses","text":"By default request metrics are labeled with the hostname. When you have a wildcard domain ingress, then there will be no metrics for that ingress (to prevent the metrics from exploding in cardinality). To get metrics in this case you need to run the ingress controller with --metrics-per-host=false (you will lose labeling by hostname, but still have labeling by ingress).","title":"Wildcard ingresses"},{"location":"user-guide/monitoring/#grafana-dashboard-using-ingress-resource","text":"If you want to expose the dashboard for grafana using a ingress resource, then you can : change the service type of the prometheus-server service and the grafana service to \"ClusterIP\" like this : kubectl -n ingress-nginx edit svc grafana This will open the currently deployed service grafana in the default editor configured in your shell (vi/nvim/nano/other) scroll down to line 34 that looks like \"type: NodePort\" change it to look like \"type: ClusterIP\". Save and exit. create a ingress resource with backend as \"grafana\" and port as \"3000\" Similarly, you can edit the service \"prometheus-server\" and add a ingress resource.","title":"Grafana dashboard using ingress resource"},{"location":"user-guide/monitoring/#prometheus-and-grafana-installation-using-service-monitors","text":"This document assumes you're using helm and using the kube-prometheus-stack package to install Prometheus and Grafana.","title":"Prometheus and Grafana installation using Service Monitors"},{"location":"user-guide/monitoring/#verify-nginx-ingress-controller-is-installed","text":"The NGINX Ingress controller should already be deployed according to the deployment instructions here . To check if Ingress controller is deployed, kubectl get pods -n ingress-nginx The result should look something like: NAME READY STATUS RESTARTS AGE ingress-nginx-controller-7c489dc7b7-ccrf6 1/1 Running 0 19h","title":"Verify NGINX Ingress controller is installed"},{"location":"user-guide/monitoring/#verify-prometheus-is-installed","text":"To check if Prometheus is already deployed, run the following command: helm ls -A NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION ingress-nginx ingress-nginx 10 2022-01-20 18:08:55.267373 -0800 PST deployed ingress-nginx-4.0.16 1.1.1 prometheus prometheus 1 2022-01-20 16:07:25.086828 -0800 PST deployed kube-prometheus-stack-30.1.0 0.53.1 - Notice that prometheus is installed in a differenet namespace than ingress-nginx If prometheus is not installed, then you can install from here","title":"Verify Prometheus is installed"},{"location":"user-guide/monitoring/#re-configure-nginx-ingress-controller","text":"The Ingress NGINX controller needs to be reconfigured for exporting metrics. This requires 3 additional configurations to the controller. These configurations are : controller.metrics.enabled=true controller.metrics.serviceMonitor.enabled=true controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" The easiest way of doing this is to helm upgrade helm upgrade ingress-nginx ingress-nginx/ingress-nginx \\ --namespace ingress-nginx \\ --set controller.metrics.enabled=true \\ --set controller.metrics.serviceMonitor.enabled=true \\ --set controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" Here controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" should match the name of the helm release of the kube-prometheus-stack You can validate that the controller has been successfully reconfigured to export metrics by looking at the values of the installed release, like this: helm get values ingress-nginx --namespace ingress-nginx controller: metrics: enabled: true serviceMonitor: additionalLabels: release: prometheus enabled: true","title":"Re-configure NGINX Ingress controller"},{"location":"user-guide/monitoring/#configure-prometheus","text":"Since Prometheus is running in a different namespace and not in the ingress-nginx namespace, it would not be able to discover ServiceMonitors in other namespaces when installed. Reconfigure your kube-prometheus-stack Helm installation to set serviceMonitorSelectorNilUsesHelmValues flag to false. By default, Prometheus only discovers PodMonitors within its own namespace. This should be disabled by setting podMonitorSelectorNilUsesHelmValues to false The configurations required are: prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false The easiest way of doing this is to use helm upgrade ... helm upgrade prometheus prometheus-community/kube-prometheus-stack \\ --namespace prometheus \\ --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \\ --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false You can validate that Prometheus has been reconfigured by looking at the values of the installed release, like this: helm get values prometheus --namespace prometheus You should be able to see the values shown below: prometheus: prometheusSpec: podMonitorSelectorNilUsesHelmValues: false serviceMonitorSelectorNilUsesHelmValues: false","title":"Configure Prometheus"},{"location":"user-guide/monitoring/#connect-and-view-prometheus-dashboard","text":"Port forward to Prometheus service. Find out the name of the prometheus service by using the following command: kubectl get svc -n prometheus The result of this command would look like: NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 7h46m prometheus-grafana ClusterIP 10.106.28.162 <none> 80/TCP 7h46m prometheus-kube-prometheus-alertmanager ClusterIP 10.108.125.245 <none> 9093/TCP 7h46m prometheus-kube-prometheus-operator ClusterIP 10.110.220.1 <none> 443/TCP 7h46m prometheus-kube-prometheus-prometheus ClusterIP 10.102.72.134 <none> 9090/TCP 7h46m prometheus-kube-state-metrics ClusterIP 10.104.231.181 <none> 8080/TCP 7h46m prometheus-operated ClusterIP None <none> 9090/TCP 7h46m prometheus-prometheus-node-exporter ClusterIP 10.96.247.128 <none> 9100/TCP 7h46m prometheus-kube-prometheus-prometheus is the service we want to port forward to. We can do so using the following command: kubectl port-forward svc/prometheus-kube-prometheus-prometheus -n prometheus 9090:9090 When you run the above command, you should see something like: Forwarding from 127.0.0.1:9090 -> 9090 Forwarding from [::1]:9090 -> 9090 - Open your browser and visit the following URL http://localhost:{port-forwarded-port} according to the above example it would be, http://localhost:9090","title":"Connect and view Prometheus dashboard"},{"location":"user-guide/monitoring/#connect-and-view-grafana-dashboard","text":"Port forward to Grafana service. Find out the name of the Grafana service by using the following command: kubectl get svc -n prometheus The result of this command would look like: NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 7h46m prometheus-grafana ClusterIP 10.106.28.162 <none> 80/TCP 7h46m prometheus-kube-prometheus-alertmanager ClusterIP 10.108.125.245 <none> 9093/TCP 7h46m prometheus-kube-prometheus-operator ClusterIP 10.110.220.1 <none> 443/TCP 7h46m prometheus-kube-prometheus-prometheus ClusterIP 10.102.72.134 <none> 9090/TCP 7h46m prometheus-kube-state-metrics ClusterIP 10.104.231.181 <none> 8080/TCP 7h46m prometheus-operated ClusterIP None <none> 9090/TCP 7h46m prometheus-prometheus-node-exporter ClusterIP 10.96.247.128 <none> 9100/TCP 7h46m prometheus-grafana is the service we want to port forward to. We can do so using the following command: kubectl port-forward svc/prometheus-grafana 3000:80 -n prometheus When you run the above command, you should see something like: Forwarding from 127.0.0.1:3000 -> 3000 Forwarding from [::1]:3000 -> 3000 - Open your browser and visit the following URL http://localhost:{port-forwarded-port} according to the above example it would be, http://localhost:3000 The default username/ password is admin/prom-operator - After the login you can import the Grafana dashboard from official dashboards , by following steps given below : Navigate to lefthand panel of grafana Hover on the gearwheel icon for Configuration and click \"Data Sources\" Click \"Add data source\" Select \"Prometheus\" Enter the details (note: I used http://10.102.72.134:9090 which is the CLUSTER-IP for Prometheus service) Left menu (hover over +) -> Dashboard Click \"Import\" Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/grafana/dashboards/nginx.json Click Import JSON Select the Prometheus data source Click \"Import\"","title":"Connect and view Grafana dashboard"},{"location":"user-guide/monitoring/#exposed-metrics","text":"Prometheus metrics are exposed on port 10254.","title":"Exposed metrics"},{"location":"user-guide/monitoring/#request-metrics","text":"nginx_ingress_controller_request_duration_seconds Histogram The request processing time in milliseconds (affected by client speed) nginx var: request_time nginx_ingress_controller_response_duration_seconds Histogram The time spent on receiving the response from the upstream server (affected by client speed) nginx var: upstream_response_time nginx_ingress_controller_header_duration_seconds Histogram The time spent on receiving first header from the upstream server nginx var: upstream_header_time nginx_ingress_controller_connect_duration_seconds Histogram The time spent on establishing a connection with the upstream server nginx var: upstream_connect_time nginx_ingress_controller_response_size Histogram The response length (including request line, header, and request body) nginx var: bytes_sent nginx_ingress_controller_request_size Histogram The request length (including request line, header, and request body) nginx var: request_length nginx_ingress_controller_requests Counter The total number of client requests nginx_ingress_controller_bytes_sent Histogram The number of bytes sent to a client. Deprecated , use nginx_ingress_controller_response_size nginx var: bytes_sent nginx_ingress_controller_ingress_upstream_latency_seconds Summary Upstream service latency per Ingress. Deprecated , use nginx_ingress_controller_connect_duration_seconds nginx var: upstream_connect_time # HELP nginx_ingress_controller_bytes_sent The number of bytes sent to a client. DEPRECATED! Use nginx_ingress_controller_response_size # TYPE nginx_ingress_controller_bytes_sent histogram # HELP nginx_ingress_controller_connect_duration_seconds The time spent on establishing a connection with the upstream server # TYPE nginx_ingress_controller_connect_duration_seconds nginx_ingress_controller_connect_duration_seconds * HELP nginx_ingress_controller_header_duration_seconds The time spent on receiving first header from the upstream server # TYPE nginx_ingress_controller_header_duration_seconds histogram # HELP nginx_ingress_controller_ingress_upstream_latency_seconds Upstream service latency per Ingress DEPRECATED! Use nginx_ingress_controller_connect_duration_seconds # TYPE nginx_ingress_controller_ingress_upstream_latency_seconds summary # HELP nginx_ingress_controller_request_duration_seconds The request processing time in milliseconds # TYPE nginx_ingress_controller_request_duration_seconds histogram # HELP nginx_ingress_controller_request_size The request length (including request line, header, and request body) # TYPE nginx_ingress_controller_request_size histogram # HELP nginx_ingress_controller_requests The total number of client requests. # TYPE nginx_ingress_controller_requests counter # HELP nginx_ingress_controller_response_duration_seconds The time spent on receiving the response from the upstream server # TYPE nginx_ingress_controller_response_duration_seconds histogram # HELP nginx_ingress_controller_response_size The response length (including request line, header, and request body) # TYPE nginx_ingress_controller_response_size histogram","title":"Request metrics"},{"location":"user-guide/monitoring/#nginx-process-metrics","text":"# HELP nginx_ingress_controller_nginx_process_connections current number of client connections with state {active, reading, writing, waiting} # TYPE nginx_ingress_controller_nginx_process_connections gauge # HELP nginx_ingress_controller_nginx_process_connections_total total number of connections with state {accepted, handled} # TYPE nginx_ingress_controller_nginx_process_connections_total counter # HELP nginx_ingress_controller_nginx_process_cpu_seconds_total Cpu usage in seconds # TYPE nginx_ingress_controller_nginx_process_cpu_seconds_total counter # HELP nginx_ingress_controller_nginx_process_num_procs number of processes # TYPE nginx_ingress_controller_nginx_process_num_procs gauge # HELP nginx_ingress_controller_nginx_process_oldest_start_time_seconds start time in seconds since 1970/01/01 # TYPE nginx_ingress_controller_nginx_process_oldest_start_time_seconds gauge # HELP nginx_ingress_controller_nginx_process_read_bytes_total number of bytes read # TYPE nginx_ingress_controller_nginx_process_read_bytes_total counter # HELP nginx_ingress_controller_nginx_process_requests_total total number of client requests # TYPE nginx_ingress_controller_nginx_process_requests_total counter # HELP nginx_ingress_controller_nginx_process_resident_memory_bytes number of bytes of memory in use # TYPE nginx_ingress_controller_nginx_process_resident_memory_bytes gauge # HELP nginx_ingress_controller_nginx_process_virtual_memory_bytes number of bytes of memory in use # TYPE nginx_ingress_controller_nginx_process_virtual_memory_bytes gauge # HELP nginx_ingress_controller_nginx_process_write_bytes_total number of bytes written # TYPE nginx_ingress_controller_nginx_process_write_bytes_total counter","title":"Nginx process metrics"},{"location":"user-guide/monitoring/#controller-metrics","text":"# HELP nginx_ingress_controller_build_info A metric with a constant '1' labeled with information about the build. # TYPE nginx_ingress_controller_build_info gauge # HELP nginx_ingress_controller_check_success Cumulative number of Ingress controller syntax check operations # TYPE nginx_ingress_controller_check_success counter # HELP nginx_ingress_controller_config_hash Running configuration hash actually running # TYPE nginx_ingress_controller_config_hash gauge # HELP nginx_ingress_controller_config_last_reload_successful Whether the last configuration reload attempt was successful # TYPE nginx_ingress_controller_config_last_reload_successful gauge # HELP nginx_ingress_controller_config_last_reload_successful_timestamp_seconds Timestamp of the last successful configuration reload. # TYPE nginx_ingress_controller_config_last_reload_successful_timestamp_seconds gauge # HELP nginx_ingress_controller_ssl_certificate_info Hold all labels associated to a certificate # TYPE nginx_ingress_controller_ssl_certificate_info gauge # HELP nginx_ingress_controller_success Cumulative number of Ingress controller reload operations # TYPE nginx_ingress_controller_success counter","title":"Controller metrics"},{"location":"user-guide/monitoring/#admission-metrics","text":"# HELP nginx_ingress_controller_admission_config_size The size of the tested configuration # TYPE nginx_ingress_controller_admission_config_size gauge # HELP nginx_ingress_controller_admission_render_duration The processing duration of ingresses rendering by the admission controller (float seconds) # TYPE nginx_ingress_controller_admission_render_duration gauge # HELP nginx_ingress_controller_admission_render_ingresses The length of ingresses rendered by the admission controller # TYPE nginx_ingress_controller_admission_render_ingresses gauge # HELP nginx_ingress_controller_admission_roundtrip_duration The complete duration of the admission controller at the time to process a new event (float seconds) # TYPE nginx_ingress_controller_admission_roundtrip_duration gauge # HELP nginx_ingress_controller_admission_tested_duration The processing duration of the admission controller tests (float seconds) # TYPE nginx_ingress_controller_admission_tested_duration gauge # HELP nginx_ingress_controller_admission_tested_ingresses The length of ingresses processed by the admission controller # TYPE nginx_ingress_controller_admission_tested_ingresses gauge","title":"Admission metrics"},{"location":"user-guide/monitoring/#histogram-buckets","text":"You can configure buckets for histogram metrics using these command line options (here are their default values): * --time-buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] * --length-buckets=[10, 20, 30, 40, 50, 60, 70, 80, 90, 100] * --size-buckets=[10, 100, 1000, 10000, 100000, 1e+06, 1e+07]","title":"Histogram buckets"},{"location":"user-guide/multiple-ingress/","text":"Multiple Ingress controllers \u00b6 By default, deploying multiple Ingress controllers (e.g., ingress-nginx & gce ) will result in all controllers simultaneously racing to update Ingress status fields in confusing ways. To fix this problem, use IngressClasses . The kubernetes.io/ingress.class annotation is deprecated from kubernetes v1.22+. Using IngressClasses \u00b6 If all ingress controllers respect IngressClasses (e.g. multiple instances of ingress-nginx v1.0), you can deploy two Ingress controllers by granting them control over two different IngressClasses, then selecting one of the two IngressClasses with ingressClassName . First, ensure the --controller-class= and --ingress-class are set to something different on each ingress controller, If your additional ingress controller is to be installed in a namespace, where there is/are one/more-than-one ingress-nginx-controller(s) already installed, then you need to specify a different unique --election-id for the new instance of the controller. # ingress-nginx Deployment/Statefulset spec : template : spec : containers : - name : ingress-nginx-internal-controller args : - /nginx-ingress-controller - '--election-id=ingress-controller-leader' - '--controller-class=k8s.io/internal-ingress-nginx' - '--ingress-class=k8s.io/internal-nginx' ... Then use the same value in the IngressClass: # ingress-nginx IngressClass apiVersion : networking.k8s.io/v1 kind : IngressClass metadata : name : internal-nginx spec : controller : k8s.io/internal-ingress-nginx ... And refer to that IngressClass in your Ingress: apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : my-ingress spec : ingressClassName : internal-nginx ... or if installing with Helm: controller : electionID : ingress-controller-leader ingressClassResource : name : internal-nginx # default: nginx enabled : true default : false controllerValue : \"k8s.io/internal-ingress-nginx\" # default: k8s.io/ingress-nginx Important When running multiple ingress-nginx controllers, it will only process an unset class annotation if one of the controllers uses the default --controller-class value (see IsValid method in internal/ingress/annotations/class/main.go ), otherwise the class annotation becomes required. If --controller-class is set to the default value of k8s.io/ingress-nginx , the controller will monitor Ingresses with no class annotation and Ingresses with annotation class set to nginx . Use a non-default value for --controller-class , to ensure that the controller only satisfied the specific class of Ingresses. Using the kubernetes.io/ingress.class annotation (in deprecation) \u00b6 If you're running multiple ingress controllers where one or more do not support IngressClasses, you must specify the annotation kubernetes.io/ingress.class: \"nginx\" in all ingresses that you would like ingress-nginx to claim. For instance, metadata : name : foo annotations : kubernetes.io/ingress.class : \"gce\" will target the GCE controller, forcing the Ingress-NGINX controller to ignore it, while an annotation like: metadata : name : foo annotations : kubernetes.io/ingress.class : \"nginx\" will target the Ingress-NGINX controller, forcing the GCE controller to ignore it. You can change the value \"nginx\" to something else by setting the --ingress-class flag: spec : template : spec : containers : - name : ingress-nginx-internal-controller args : - /nginx-ingress-controller - --ingress-class=internal-nginx then setting the corresponding kubernetes.io/ingress.class: \"internal-nginx\" annotation on your Ingresses. To reiterate, setting the annotation to any value which does not match a valid ingress class will force the NGINX Ingress controller to ignore your Ingress. If you are only running a single NGINX ingress controller, this can be achieved by setting the annotation to any value except \"nginx\" or an empty string. Do this if you wish to use one of the other Ingress controllers at the same time as the NGINX controller.","title":"Multiple Ingress controllers"},{"location":"user-guide/multiple-ingress/#multiple-ingress-controllers","text":"By default, deploying multiple Ingress controllers (e.g., ingress-nginx & gce ) will result in all controllers simultaneously racing to update Ingress status fields in confusing ways. To fix this problem, use IngressClasses . The kubernetes.io/ingress.class annotation is deprecated from kubernetes v1.22+.","title":"Multiple Ingress controllers"},{"location":"user-guide/multiple-ingress/#using-ingressclasses","text":"If all ingress controllers respect IngressClasses (e.g. multiple instances of ingress-nginx v1.0), you can deploy two Ingress controllers by granting them control over two different IngressClasses, then selecting one of the two IngressClasses with ingressClassName . First, ensure the --controller-class= and --ingress-class are set to something different on each ingress controller, If your additional ingress controller is to be installed in a namespace, where there is/are one/more-than-one ingress-nginx-controller(s) already installed, then you need to specify a different unique --election-id for the new instance of the controller. # ingress-nginx Deployment/Statefulset spec : template : spec : containers : - name : ingress-nginx-internal-controller args : - /nginx-ingress-controller - '--election-id=ingress-controller-leader' - '--controller-class=k8s.io/internal-ingress-nginx' - '--ingress-class=k8s.io/internal-nginx' ... Then use the same value in the IngressClass: # ingress-nginx IngressClass apiVersion : networking.k8s.io/v1 kind : IngressClass metadata : name : internal-nginx spec : controller : k8s.io/internal-ingress-nginx ... And refer to that IngressClass in your Ingress: apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : my-ingress spec : ingressClassName : internal-nginx ... or if installing with Helm: controller : electionID : ingress-controller-leader ingressClassResource : name : internal-nginx # default: nginx enabled : true default : false controllerValue : \"k8s.io/internal-ingress-nginx\" # default: k8s.io/ingress-nginx Important When running multiple ingress-nginx controllers, it will only process an unset class annotation if one of the controllers uses the default --controller-class value (see IsValid method in internal/ingress/annotations/class/main.go ), otherwise the class annotation becomes required. If --controller-class is set to the default value of k8s.io/ingress-nginx , the controller will monitor Ingresses with no class annotation and Ingresses with annotation class set to nginx . Use a non-default value for --controller-class , to ensure that the controller only satisfied the specific class of Ingresses.","title":"Using IngressClasses"},{"location":"user-guide/multiple-ingress/#using-the-kubernetesioingressclass-annotation-in-deprecation","text":"If you're running multiple ingress controllers where one or more do not support IngressClasses, you must specify the annotation kubernetes.io/ingress.class: \"nginx\" in all ingresses that you would like ingress-nginx to claim. For instance, metadata : name : foo annotations : kubernetes.io/ingress.class : \"gce\" will target the GCE controller, forcing the Ingress-NGINX controller to ignore it, while an annotation like: metadata : name : foo annotations : kubernetes.io/ingress.class : \"nginx\" will target the Ingress-NGINX controller, forcing the GCE controller to ignore it. You can change the value \"nginx\" to something else by setting the --ingress-class flag: spec : template : spec : containers : - name : ingress-nginx-internal-controller args : - /nginx-ingress-controller - --ingress-class=internal-nginx then setting the corresponding kubernetes.io/ingress.class: \"internal-nginx\" annotation on your Ingresses. To reiterate, setting the annotation to any value which does not match a valid ingress class will force the NGINX Ingress controller to ignore your Ingress. If you are only running a single NGINX ingress controller, this can be achieved by setting the annotation to any value except \"nginx\" or an empty string. Do this if you wish to use one of the other Ingress controllers at the same time as the NGINX controller.","title":"Using the kubernetes.io/ingress.class annotation (in deprecation)"},{"location":"user-guide/tls/","text":"TLS/HTTPS \u00b6 TLS Secrets \u00b6 Anytime we reference a TLS secret, we mean a PEM-encoded X.509, RSA (2048) secret. Warning Ensure that the certificate order is leaf->intermediate->root, otherwise the controller will not be able to import the certificate, and you'll see this error in the logs W1012 09:15:45.920000 6 backend_ssl.go:46] Error obtaining X.509 certificate: unexpected error creating SSL Cert: certificate and private key does not have a matching public key: tls: private key does not match public key You can generate a self-signed certificate and private key with: $ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ${ KEY_FILE } -out ${ CERT_FILE } -subj \"/CN= ${ HOST } /O= ${ HOST } \" Then create the secret in the cluster via: kubectl create secret tls ${ CERT_NAME } --key ${ KEY_FILE } --cert ${ CERT_FILE } The resulting secret will be of type kubernetes.io/tls . Host names \u00b6 Ensure that the relevant ingress rules specify a matching host name . Default SSL Certificate \u00b6 NGINX provides the option to configure a server as a catch-all with server_name for requests that do not match any of the configured server names. This configuration works out-of-the-box for HTTP traffic. For HTTPS, a certificate is naturally required. For this reason the Ingress controller provides the flag --default-ssl-certificate . The secret referred to by this flag contains the default certificate to be used when accessing the catch-all server. If this flag is not provided NGINX will use a self-signed certificate. For instance, if you have a TLS secret foo-tls in the default namespace, add --default-ssl-certificate=default/foo-tls in the nginx-controller deployment. The default certificate will also be used for ingress tls: sections that do not have a secretName option. To force redirects for Ingresses that do not specify a TLS-block at all, take a look at force-ssl-redirect in ConfigMap . SSL Passthrough \u00b6 The --enable-ssl-passthrough flag enables the SSL Passthrough feature, which is disabled by default. This is required to enable passthrough backends in Ingress objects. Warning This feature is implemented by intercepting all traffic on the configured HTTPS port (default: 443) and handing it over to a local TCP proxy. This bypasses NGINX completely and introduces a non-negligible performance penalty. SSL Passthrough leverages SNI and reads the virtual domain from the TLS negotiation, which requires compatible clients. After a connection has been accepted by the TLS listener, it is handled by the controller itself and piped back and forth between the backend and the client. If there is no hostname matching the requested host name, the request is handed over to NGINX on the configured passthrough proxy port (default: 442), which proxies the request to the default backend. Note Unlike HTTP backends, traffic to Passthrough backends is sent to the clusterIP of the backing Service instead of individual Endpoints. HTTP Strict Transport Security \u00b6 HTTP Strict Transport Security (HSTS) is an opt-in security enhancement specified through the use of a special response header. Once a supported browser receives this header that browser will prevent any communications from being sent over HTTP to the specified domain and will instead send all communications over HTTPS. HSTS is enabled by default. To disable this behavior use hsts: \"false\" in the configuration ConfigMap . Server-side HTTPS enforcement through redirect \u00b6 By default the controller redirects HTTP clients to the HTTPS port 443 using a 308 Permanent Redirect response if TLS is enabled for that Ingress. This can be disabled globally using ssl-redirect: \"false\" in the NGINX config map , or per-Ingress with the nginx.ingress.kubernetes.io/ssl-redirect: \"false\" annotation in the particular resource. Tip When using SSL offloading outside of cluster (e.g. AWS ELB) it may be useful to enforce a redirect to HTTPS even when there is no TLS certificate available. This can be achieved by using the nginx.ingress.kubernetes.io/force-ssl-redirect: \"true\" annotation in the particular resource. Automated Certificate Management with cert-manager \u00b6 cert-manager automatically requests missing or expired certificates from a range of supported issuers (including Let's Encrypt ) by monitoring ingress resources. To set up cert-manager you should take a look at this full example . To enable it for an ingress resource you have to deploy cert-manager, configure a certificate issuer update the manifest: apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : ingress-demo annotations : cert-manager.io/issuer : \"letsencrypt-staging\" # Replace this with a production issuer once you've tested it [ .. ] spec : tls : - hosts : - ingress-demo.example.com secretName : ingress-demo-tls [ ... ] Default TLS Version and Ciphers \u00b6 To provide the most secure baseline configuration possible, ingress-nginx defaults to using TLS 1.2 and 1.3 only, with a secure set of TLS ciphers . Legacy TLS \u00b6 The default configuration, though secure, does not support some older browsers and operating systems. For instance, TLS 1.1+ is only enabled by default from Android 5.0 on. At the time of writing, May 2018, approximately 15% of Android devices are not compatible with ingress-nginx's default configuration. To change this default behavior, use a ConfigMap . A sample ConfigMap fragment to allow these older clients to connect could look something like the following (generated using the Mozilla SSL Configuration Generator) mozilla-ssl-config-old : kind: ConfigMap apiVersion: v1 metadata: name: nginx-config data: ssl-ciphers: \"ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA\" ssl-protocols: \"TLSv1 TLSv1.1 TLSv1.2 TLSv1.3\"","title":"TLS/HTTPS"},{"location":"user-guide/tls/#tlshttps","text":"","title":"TLS/HTTPS"},{"location":"user-guide/tls/#tls-secrets","text":"Anytime we reference a TLS secret, we mean a PEM-encoded X.509, RSA (2048) secret. Warning Ensure that the certificate order is leaf->intermediate->root, otherwise the controller will not be able to import the certificate, and you'll see this error in the logs W1012 09:15:45.920000 6 backend_ssl.go:46] Error obtaining X.509 certificate: unexpected error creating SSL Cert: certificate and private key does not have a matching public key: tls: private key does not match public key You can generate a self-signed certificate and private key with: $ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ${ KEY_FILE } -out ${ CERT_FILE } -subj \"/CN= ${ HOST } /O= ${ HOST } \" Then create the secret in the cluster via: kubectl create secret tls ${ CERT_NAME } --key ${ KEY_FILE } --cert ${ CERT_FILE } The resulting secret will be of type kubernetes.io/tls .","title":"TLS Secrets"},{"location":"user-guide/tls/#host-names","text":"Ensure that the relevant ingress rules specify a matching host name .","title":"Host names"},{"location":"user-guide/tls/#default-ssl-certificate","text":"NGINX provides the option to configure a server as a catch-all with server_name for requests that do not match any of the configured server names. This configuration works out-of-the-box for HTTP traffic. For HTTPS, a certificate is naturally required. For this reason the Ingress controller provides the flag --default-ssl-certificate . The secret referred to by this flag contains the default certificate to be used when accessing the catch-all server. If this flag is not provided NGINX will use a self-signed certificate. For instance, if you have a TLS secret foo-tls in the default namespace, add --default-ssl-certificate=default/foo-tls in the nginx-controller deployment. The default certificate will also be used for ingress tls: sections that do not have a secretName option. To force redirects for Ingresses that do not specify a TLS-block at all, take a look at force-ssl-redirect in ConfigMap .","title":"Default SSL Certificate"},{"location":"user-guide/tls/#ssl-passthrough","text":"The --enable-ssl-passthrough flag enables the SSL Passthrough feature, which is disabled by default. This is required to enable passthrough backends in Ingress objects. Warning This feature is implemented by intercepting all traffic on the configured HTTPS port (default: 443) and handing it over to a local TCP proxy. This bypasses NGINX completely and introduces a non-negligible performance penalty. SSL Passthrough leverages SNI and reads the virtual domain from the TLS negotiation, which requires compatible clients. After a connection has been accepted by the TLS listener, it is handled by the controller itself and piped back and forth between the backend and the client. If there is no hostname matching the requested host name, the request is handed over to NGINX on the configured passthrough proxy port (default: 442), which proxies the request to the default backend. Note Unlike HTTP backends, traffic to Passthrough backends is sent to the clusterIP of the backing Service instead of individual Endpoints.","title":"SSL Passthrough"},{"location":"user-guide/tls/#http-strict-transport-security","text":"HTTP Strict Transport Security (HSTS) is an opt-in security enhancement specified through the use of a special response header. Once a supported browser receives this header that browser will prevent any communications from being sent over HTTP to the specified domain and will instead send all communications over HTTPS. HSTS is enabled by default. To disable this behavior use hsts: \"false\" in the configuration ConfigMap .","title":"HTTP Strict Transport Security"},{"location":"user-guide/tls/#server-side-https-enforcement-through-redirect","text":"By default the controller redirects HTTP clients to the HTTPS port 443 using a 308 Permanent Redirect response if TLS is enabled for that Ingress. This can be disabled globally using ssl-redirect: \"false\" in the NGINX config map , or per-Ingress with the nginx.ingress.kubernetes.io/ssl-redirect: \"false\" annotation in the particular resource. Tip When using SSL offloading outside of cluster (e.g. AWS ELB) it may be useful to enforce a redirect to HTTPS even when there is no TLS certificate available. This can be achieved by using the nginx.ingress.kubernetes.io/force-ssl-redirect: \"true\" annotation in the particular resource.","title":"Server-side HTTPS enforcement through redirect"},{"location":"user-guide/tls/#automated-certificate-management-with-cert-manager","text":"cert-manager automatically requests missing or expired certificates from a range of supported issuers (including Let's Encrypt ) by monitoring ingress resources. To set up cert-manager you should take a look at this full example . To enable it for an ingress resource you have to deploy cert-manager, configure a certificate issuer update the manifest: apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : ingress-demo annotations : cert-manager.io/issuer : \"letsencrypt-staging\" # Replace this with a production issuer once you've tested it [ .. ] spec : tls : - hosts : - ingress-demo.example.com secretName : ingress-demo-tls [ ... ]","title":"Automated Certificate Management with cert-manager"},{"location":"user-guide/tls/#default-tls-version-and-ciphers","text":"To provide the most secure baseline configuration possible, ingress-nginx defaults to using TLS 1.2 and 1.3 only, with a secure set of TLS ciphers .","title":"Default TLS Version and Ciphers"},{"location":"user-guide/tls/#legacy-tls","text":"The default configuration, though secure, does not support some older browsers and operating systems. For instance, TLS 1.1+ is only enabled by default from Android 5.0 on. At the time of writing, May 2018, approximately 15% of Android devices are not compatible with ingress-nginx's default configuration. To change this default behavior, use a ConfigMap . A sample ConfigMap fragment to allow these older clients to connect could look something like the following (generated using the Mozilla SSL Configuration Generator) mozilla-ssl-config-old : kind: ConfigMap apiVersion: v1 metadata: name: nginx-config data: ssl-ciphers: \"ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA\" ssl-protocols: \"TLSv1 TLSv1.1 TLSv1.2 TLSv1.3\"","title":"Legacy TLS"},{"location":"user-guide/nginx-configuration/","text":"NGINX Configuration \u00b6 There are three ways to customize NGINX: ConfigMap : using a Configmap to set global configurations in NGINX. Annotations : use this if you want a specific configuration for a particular Ingress rule. Custom template : when more specific settings are required, like open_file_cache , adjust listen options as rcvbuf or when is not possible to change the configuration through the ConfigMap.","title":"Introduction"},{"location":"user-guide/nginx-configuration/#nginx-configuration","text":"There are three ways to customize NGINX: ConfigMap : using a Configmap to set global configurations in NGINX. Annotations : use this if you want a specific configuration for a particular Ingress rule. Custom template : when more specific settings are required, like open_file_cache , adjust listen options as rcvbuf or when is not possible to change the configuration through the ConfigMap.","title":"NGINX Configuration"},{"location":"user-guide/nginx-configuration/annotations/","text":"Annotations \u00b6 You can add these Kubernetes annotations to specific Ingress objects to customize their behavior. Tip Annotation keys and values can only be strings. Other types, such as boolean or numeric values must be quoted, i.e. \"true\" , \"false\" , \"100\" . Note The annotation prefix can be changed using the --annotations-prefix command line argument , but the default is nginx.ingress.kubernetes.io , as described in the table below. Name type nginx.ingress.kubernetes.io/app-root string nginx.ingress.kubernetes.io/affinity cookie nginx.ingress.kubernetes.io/affinity-mode \"balanced\" or \"persistent\" nginx.ingress.kubernetes.io/affinity-canary-behavior \"sticky\" or \"legacy\" nginx.ingress.kubernetes.io/auth-realm string nginx.ingress.kubernetes.io/auth-secret string nginx.ingress.kubernetes.io/auth-secret-type string nginx.ingress.kubernetes.io/auth-type basic or digest nginx.ingress.kubernetes.io/auth-tls-secret string nginx.ingress.kubernetes.io/auth-tls-verify-depth number nginx.ingress.kubernetes.io/auth-tls-verify-client string nginx.ingress.kubernetes.io/auth-tls-error-page string nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream \"true\" or \"false\" nginx.ingress.kubernetes.io/auth-tls-match-cn string nginx.ingress.kubernetes.io/auth-url string nginx.ingress.kubernetes.io/auth-cache-key string nginx.ingress.kubernetes.io/auth-cache-duration string nginx.ingress.kubernetes.io/auth-keepalive number nginx.ingress.kubernetes.io/auth-keepalive-requests number nginx.ingress.kubernetes.io/auth-keepalive-timeout number nginx.ingress.kubernetes.io/auth-proxy-set-headers string nginx.ingress.kubernetes.io/auth-snippet string nginx.ingress.kubernetes.io/enable-global-auth \"true\" or \"false\" nginx.ingress.kubernetes.io/backend-protocol string nginx.ingress.kubernetes.io/canary \"true\" or \"false\" nginx.ingress.kubernetes.io/canary-by-header string nginx.ingress.kubernetes.io/canary-by-header-value string nginx.ingress.kubernetes.io/canary-by-header-pattern string nginx.ingress.kubernetes.io/canary-by-cookie string nginx.ingress.kubernetes.io/canary-weight number nginx.ingress.kubernetes.io/canary-weight-total number nginx.ingress.kubernetes.io/client-body-buffer-size string nginx.ingress.kubernetes.io/configuration-snippet string nginx.ingress.kubernetes.io/custom-http-errors []int nginx.ingress.kubernetes.io/default-backend string nginx.ingress.kubernetes.io/enable-cors \"true\" or \"false\" nginx.ingress.kubernetes.io/cors-allow-origin string nginx.ingress.kubernetes.io/cors-allow-methods string nginx.ingress.kubernetes.io/cors-allow-headers string nginx.ingress.kubernetes.io/cors-expose-headers string nginx.ingress.kubernetes.io/cors-allow-credentials \"true\" or \"false\" nginx.ingress.kubernetes.io/cors-max-age number nginx.ingress.kubernetes.io/force-ssl-redirect \"true\" or \"false\" nginx.ingress.kubernetes.io/from-to-www-redirect \"true\" or \"false\" nginx.ingress.kubernetes.io/http2-push-preload \"true\" or \"false\" nginx.ingress.kubernetes.io/limit-connections number nginx.ingress.kubernetes.io/limit-rps number nginx.ingress.kubernetes.io/global-rate-limit number nginx.ingress.kubernetes.io/global-rate-limit-window duration nginx.ingress.kubernetes.io/global-rate-limit-key string nginx.ingress.kubernetes.io/global-rate-limit-ignored-cidrs string nginx.ingress.kubernetes.io/permanent-redirect string nginx.ingress.kubernetes.io/permanent-redirect-code number nginx.ingress.kubernetes.io/temporal-redirect string nginx.ingress.kubernetes.io/preserve-trailing-slash \"true\" or \"false\" nginx.ingress.kubernetes.io/proxy-body-size string nginx.ingress.kubernetes.io/proxy-cookie-domain string nginx.ingress.kubernetes.io/proxy-cookie-path string nginx.ingress.kubernetes.io/proxy-connect-timeout number nginx.ingress.kubernetes.io/proxy-send-timeout number nginx.ingress.kubernetes.io/proxy-read-timeout number nginx.ingress.kubernetes.io/proxy-next-upstream string nginx.ingress.kubernetes.io/proxy-next-upstream-timeout number nginx.ingress.kubernetes.io/proxy-next-upstream-tries number nginx.ingress.kubernetes.io/proxy-request-buffering string nginx.ingress.kubernetes.io/proxy-redirect-from string nginx.ingress.kubernetes.io/proxy-redirect-to string nginx.ingress.kubernetes.io/proxy-http-version \"1.0\" or \"1.1\" nginx.ingress.kubernetes.io/proxy-ssl-secret string nginx.ingress.kubernetes.io/proxy-ssl-ciphers string nginx.ingress.kubernetes.io/proxy-ssl-name string nginx.ingress.kubernetes.io/proxy-ssl-protocols string nginx.ingress.kubernetes.io/proxy-ssl-verify string nginx.ingress.kubernetes.io/proxy-ssl-verify-depth number nginx.ingress.kubernetes.io/proxy-ssl-server-name string nginx.ingress.kubernetes.io/enable-rewrite-log \"true\" or \"false\" nginx.ingress.kubernetes.io/rewrite-target URI nginx.ingress.kubernetes.io/satisfy string nginx.ingress.kubernetes.io/server-alias string nginx.ingress.kubernetes.io/server-snippet string nginx.ingress.kubernetes.io/service-upstream \"true\" or \"false\" nginx.ingress.kubernetes.io/session-cookie-name string nginx.ingress.kubernetes.io/session-cookie-path string nginx.ingress.kubernetes.io/session-cookie-domain string nginx.ingress.kubernetes.io/session-cookie-change-on-failure \"true\" or \"false\" nginx.ingress.kubernetes.io/session-cookie-samesite string nginx.ingress.kubernetes.io/session-cookie-conditional-samesite-none \"true\" or \"false\" nginx.ingress.kubernetes.io/ssl-redirect \"true\" or \"false\" nginx.ingress.kubernetes.io/ssl-passthrough \"true\" or \"false\" nginx.ingress.kubernetes.io/stream-snippet string nginx.ingress.kubernetes.io/upstream-hash-by string nginx.ingress.kubernetes.io/x-forwarded-prefix string nginx.ingress.kubernetes.io/load-balance string nginx.ingress.kubernetes.io/upstream-vhost string nginx.ingress.kubernetes.io/whitelist-source-range CIDR nginx.ingress.kubernetes.io/proxy-buffering string nginx.ingress.kubernetes.io/proxy-buffers-number number nginx.ingress.kubernetes.io/proxy-buffer-size string nginx.ingress.kubernetes.io/proxy-max-temp-file-size string nginx.ingress.kubernetes.io/ssl-ciphers string nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers \"true\" or \"false\" nginx.ingress.kubernetes.io/connection-proxy-header string nginx.ingress.kubernetes.io/enable-access-log \"true\" or \"false\" nginx.ingress.kubernetes.io/enable-opentracing \"true\" or \"false\" nginx.ingress.kubernetes.io/opentracing-trust-incoming-span \"true\" or \"false\" nginx.ingress.kubernetes.io/enable-influxdb \"true\" or \"false\" nginx.ingress.kubernetes.io/influxdb-measurement string nginx.ingress.kubernetes.io/influxdb-port string nginx.ingress.kubernetes.io/influxdb-host string nginx.ingress.kubernetes.io/influxdb-server-name string nginx.ingress.kubernetes.io/use-regex bool nginx.ingress.kubernetes.io/enable-modsecurity bool nginx.ingress.kubernetes.io/enable-owasp-core-rules bool nginx.ingress.kubernetes.io/modsecurity-transaction-id string nginx.ingress.kubernetes.io/modsecurity-snippet string nginx.ingress.kubernetes.io/mirror-request-body string nginx.ingress.kubernetes.io/mirror-target string nginx.ingress.kubernetes.io/mirror-host string Canary \u00b6 In some cases, you may want to \"canary\" a new set of changes by sending a small number of requests to a different service than the production service. The canary annotation enables the Ingress spec to act as an alternative service for requests to route to depending on the rules applied. The following annotations to configure canary can be enabled after nginx.ingress.kubernetes.io/canary: \"true\" is set: nginx.ingress.kubernetes.io/canary-by-header : The header to use for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the request header is set to always , it will be routed to the canary. When the header is set to never , it will never be routed to the canary. For any other value, the header will be ignored and the request compared against the other canary rules by precedence. nginx.ingress.kubernetes.io/canary-by-header-value : The header value to match for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the request header is set to this value, it will be routed to the canary. For any other header value, the header will be ignored and the request compared against the other canary rules by precedence. This annotation has to be used together with nginx.ingress.kubernetes.io/canary-by-header . The annotation is an extension of the nginx.ingress.kubernetes.io/canary-by-header to allow customizing the header value instead of using hardcoded values. It doesn't have any effect if the nginx.ingress.kubernetes.io/canary-by-header annotation is not defined. nginx.ingress.kubernetes.io/canary-by-header-pattern : This works the same way as canary-by-header-value except it does PCRE Regex matching. Note that when canary-by-header-value is set this annotation will be ignored. When the given Regex causes error during request processing, the request will be considered as not matching. nginx.ingress.kubernetes.io/canary-by-cookie : The cookie to use for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the cookie value is set to always , it will be routed to the canary. When the cookie is set to never , it will never be routed to the canary. For any other value, the cookie will be ignored and the request compared against the other canary rules by precedence. nginx.ingress.kubernetes.io/canary-weight : The integer based (0 - ) percent of random requests that should be routed to the service specified in the canary Ingress. A weight of 0 implies that no requests will be sent to the service in the Canary ingress by this canary rule. A weight of means implies all requests will be sent to the alternative service specified in the Ingress. <weight-total> defaults to 100, and can be increased via nginx.ingress.kubernetes.io/canary-weight-total . nginx.ingress.kubernetes.io/canary-weight-total : The total weight of traffic. If unspecified, it defaults to 100. Canary rules are evaluated in order of precedence. Precedence is as follows: canary-by-header -> canary-by-cookie -> canary-weight Note that when you mark an ingress as canary, then all the other non-canary annotations will be ignored (inherited from the corresponding main ingress) except nginx.ingress.kubernetes.io/load-balance , nginx.ingress.kubernetes.io/upstream-hash-by , and annotations related to session affinity . If you want to restore the original behavior of canaries when session affinity was ignored, set nginx.ingress.kubernetes.io/affinity-canary-behavior annotation with value legacy on the canary ingress definition. Known Limitations Currently a maximum of one canary ingress can be applied per Ingress rule. Rewrite \u00b6 In some scenarios the exposed URL in the backend service differs from the specified path in the Ingress rule. Without a rewrite any request will return 404. Set the annotation nginx.ingress.kubernetes.io/rewrite-target to the path expected by the service. If the Application Root is exposed in a different path and needs to be redirected, set the annotation nginx.ingress.kubernetes.io/app-root to redirect requests for / . Example Please check the rewrite example. Session Affinity \u00b6 The annotation nginx.ingress.kubernetes.io/affinity enables and sets the affinity type in all Upstreams of an Ingress. This way, a request will always be directed to the same upstream server. The only affinity type available for NGINX is cookie . The annotation nginx.ingress.kubernetes.io/affinity-mode defines the stickiness of a session. Setting this to balanced (default) will redistribute some sessions if a deployment gets scaled up, therefore rebalancing the load on the servers. Setting this to persistent will not rebalance sessions to new servers, therefore providing maximum stickiness. The annotation nginx.ingress.kubernetes.io/affinity-canary-behavior defines the behavior of canaries when session affinity is enabled. Setting this to sticky (default) will ensure that users that were served by canaries, will continue to be served by canaries. Setting this to legacy will restore original canary behavior, when session affinity was ignored. Attention If more than one Ingress is defined for a host and at least one Ingress uses nginx.ingress.kubernetes.io/affinity: cookie , then only paths on the Ingress using nginx.ingress.kubernetes.io/affinity will use session cookie affinity. All paths defined on other Ingresses for the host will be load balanced through the random selection of a backend server. Example Please check the affinity example. Cookie affinity \u00b6 If you use the cookie affinity type you can also specify the name of the cookie that will be used to route the requests with the annotation nginx.ingress.kubernetes.io/session-cookie-name . The default is to create a cookie named 'INGRESSCOOKIE'. The NGINX annotation nginx.ingress.kubernetes.io/session-cookie-path defines the path that will be set on the cookie. This is optional unless the annotation nginx.ingress.kubernetes.io/use-regex is set to true; Session cookie paths do not support regex. Use nginx.ingress.kubernetes.io/session-cookie-domain to set the Domain attribute of the sticky cookie. Use nginx.ingress.kubernetes.io/session-cookie-samesite to apply a SameSite attribute to the sticky cookie. Browser accepted values are None , Lax , and Strict . Some browsers reject cookies with SameSite=None , including those created before the SameSite=None specification (e.g. Chrome 5X). Other browsers mistakenly treat SameSite=None cookies as SameSite=Strict (e.g. Safari running on OSX 14). To omit SameSite=None from browsers with these incompatibilities, add the annotation nginx.ingress.kubernetes.io/session-cookie-conditional-samesite-none: \"true\" . Authentication \u00b6 It is possible to add authentication by adding additional annotations in the Ingress rule. The source of the authentication is a secret that contains usernames and passwords. The annotations are: nginx.ingress.kubernetes.io/auth-type: [basic|digest] Indicates the HTTP Authentication Type: Basic or Digest Access Authentication . nginx.ingress.kubernetes.io/auth-secret: secretName The name of the Secret that contains the usernames and passwords which are granted access to the path s defined in the Ingress rules. This annotation also accepts the alternative form \"namespace/secretName\", in which case the Secret lookup is performed in the referenced namespace instead of the Ingress namespace. nginx.ingress.kubernetes.io/auth-secret-type: [auth-file|auth-map] The auth-secret can have two forms: auth-file - default, an htpasswd file in the key auth within the secret auth-map - the keys of the secret are the usernames, and the values are the hashed passwords nginx.ingress.kubernetes.io/auth-realm: \"realm string\" Example Please check the auth example. Custom NGINX upstream hashing \u00b6 NGINX supports load balancing by client-server mapping based on consistent hashing for a given key. The key can contain text, variables or any combination thereof. This feature allows for request stickiness other than client IP or cookies. The ketama consistent hashing method will be used which ensures only a few keys would be remapped to different servers on upstream group changes. There is a special mode of upstream hashing called subset. In this mode, upstream servers are grouped into subsets, and stickiness works by mapping keys to a subset instead of individual upstream servers. Specific server is chosen uniformly at random from the selected sticky subset. It provides a balance between stickiness and load distribution. To enable consistent hashing for a backend: nginx.ingress.kubernetes.io/upstream-hash-by : the nginx variable, text value or any combination thereof to use for consistent hashing. For example: nginx.ingress.kubernetes.io/upstream-hash-by: \"$request_uri\" or nginx.ingress.kubernetes.io/upstream-hash-by: \"$request_uri$host\" or nginx.ingress.kubernetes.io/upstream-hash-by: \"${request_uri}-text-value\" to consistently hash upstream requests by the current request URI. \"subset\" hashing can be enabled setting nginx.ingress.kubernetes.io/upstream-hash-by-subset : \"true\". This maps requests to subset of nodes instead of a single one. upstream-hash-by-subset-size determines the size of each subset (default 3). Please check the chashsubset example. Custom NGINX load balancing \u00b6 This is similar to load-balance in ConfigMap , but configures load balancing algorithm per ingress. Note that nginx.ingress.kubernetes.io/upstream-hash-by takes preference over this. If this and nginx.ingress.kubernetes.io/upstream-hash-by are not set then we fallback to using globally configured load balancing algorithm. Custom NGINX upstream vhost \u00b6 This configuration setting allows you to control the value for host in the following statement: proxy_set_header Host $host , which forms part of the location block. This is useful if you need to call the upstream server by something other than $host . Client Certificate Authentication \u00b6 It is possible to enable Client Certificate Authentication using additional annotations in Ingress Rule. Client Certificate Authentication is applied per host and it is not possible to specify rules that differ for individual paths. To enable, add the annotation nginx.ingress.kubernetes.io/auth-tls-secret: namespace/secretName . This secret must have a file named ca.crt containing the full Certificate Authority chain ca.crt that is enabled to authenticate against this Ingress. You can further customize client certificate authentication and behavior with these annotations: nginx.ingress.kubernetes.io/auth-tls-verify-depth : The validation depth between the provided client certificate and the Certification Authority chain. (default: 1) nginx.ingress.kubernetes.io/auth-tls-verify-client : Enables verification of client certificates. Possible values are: on : Request a client certificate that must be signed by a certificate that is included in the secret key ca.crt of the secret specified by nginx.ingress.kubernetes.io/auth-tls-secret: namespace/secretName . Failed certificate verification will result in a status code 400 (Bad Request) (default) off : Don't request client certificates and don't do client certificate verification. optional : Do optional client certificate validation against the CAs from auth-tls-secret . The request fails with status code 400 (Bad Request) when a certificate is provided that is not signed by the CA. When no or an otherwise invalid certificate is provided, the request does not fail, but instead the verification result is sent to the upstream service. optional_no_ca : Do optional client certificate validation, but do not fail the request when the client certificate is not signed by the CAs from auth-tls-secret . Certificate verification result is sent to the upstream service. nginx.ingress.kubernetes.io/auth-tls-error-page : The URL/Page that user should be redirected in case of a Certificate Authentication Error nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream : Indicates if the received certificates should be passed or not to the upstream server in the header ssl-client-cert . Possible values are \"true\" or \"false\" (default). nginx.ingress.kubernetes.io/auth-tls-match-cn : Adds a sanity check for the CN of the client certificate that is sent over using a string / regex starting with \"CN=\", example: \"CN=myvalidclient\" . If the certificate CN sent during mTLS does not match your string / regex it will fail with status code 403. Another way of using this is by adding multiple options in your regex, example: \"CN=(option1|option2|myvalidclient)\" . In this case, as long as one of the options in the brackets matches the certificate CN then you will receive a 200 status code. The following headers are sent to the upstream service according to the auth-tls-* annotations: ssl-client-issuer-dn : The issuer information of the client certificate. Example: \"CN=My CA\" ssl-client-subject-dn : The subject information of the client certificate. Example: \"CN=My Client\" ssl-client-verify : The result of the client verification. Possible values: \"SUCCESS\", \"FAILED: \" ssl-client-cert : The full client certificate in PEM format. Will only be sent when nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream is set to \"true\". Example: -----BEGIN%20CERTIFICATE-----%0A...---END%20CERTIFICATE-----%0A Example Please check the client-certs example. Attention TLS with Client Authentication is not possible in Cloudflare and might result in unexpected behavior. Cloudflare only allows Authenticated Origin Pulls and is required to use their own certificate: https://blog.cloudflare.com/protecting-the-origin-with-tls-authenticated-origin-pulls/ Only Authenticated Origin Pulls are allowed and can be configured by following their tutorial: https://support.cloudflare.com/hc/en-us/articles/204494148-Setting-up-NGINX-to-use-TLS-Authenticated-Origin-Pulls Backend Certificate Authentication \u00b6 It is possible to authenticate to a proxied HTTPS backend with certificate using additional annotations in Ingress Rule. nginx.ingress.kubernetes.io/proxy-ssl-secret: secretName : Specifies a Secret with the certificate tls.crt , key tls.key in PEM format used for authentication to a proxied HTTPS server. It should also contain trusted CA certificates ca.crt in PEM format used to verify the certificate of the proxied HTTPS server. This annotation expects the Secret name in the form \"namespace/secretName\". nginx.ingress.kubernetes.io/proxy-ssl-verify : Enables or disables verification of the proxied HTTPS server certificate. (default: off) nginx.ingress.kubernetes.io/proxy-ssl-verify-depth : Sets the verification depth in the proxied HTTPS server certificates chain. (default: 1) nginx.ingress.kubernetes.io/proxy-ssl-ciphers : Specifies the enabled ciphers for requests to a proxied HTTPS server. The ciphers are specified in the format understood by the OpenSSL library. nginx.ingress.kubernetes.io/proxy-ssl-name : Allows to set proxy_ssl_name . This allows overriding the server name used to verify the certificate of the proxied HTTPS server. This value is also passed through SNI when a connection is established to the proxied HTTPS server. nginx.ingress.kubernetes.io/proxy-ssl-protocols : Enables the specified protocols for requests to a proxied HTTPS server. nginx.ingress.kubernetes.io/proxy-ssl-server-name : Enables passing of the server name through TLS Server Name Indication extension (SNI, RFC 6066) when establishing a connection with the proxied HTTPS server. Configuration snippet \u00b6 Using this annotation you can add additional configuration to the NGINX location. For example: nginx.ingress.kubernetes.io/configuration-snippet : | more_set_headers \"Request-Id: $req_id\"; Be aware this can be dangerous in multi-tenant clusters, as it can lead to people with otherwise limited permissions being able to retrieve all secrets on the cluster. The recommended mitigation for this threat is to disable this feature, so it may not work for you. See CVE-2021-25742 and the related issue on github for more information. Custom HTTP Errors \u00b6 Like the custom-http-errors value in the ConfigMap, this annotation will set NGINX proxy-intercept-errors , but only for the NGINX location associated with this ingress. If a default backend annotation is specified on the ingress, the errors will be routed to that annotation's default backend service (instead of the global default backend). Different ingresses can specify different sets of error codes. Even if multiple ingress objects share the same hostname, this annotation can be used to intercept different error codes for each ingress (for example, different error codes to be intercepted for different paths on the same hostname, if each path is on a different ingress). If custom-http-errors is also specified globally, the error values specified in this annotation will override the global value for the given ingress' hostname and path. Example usage: nginx.ingress.kubernetes.io/custom-http-errors: \"404,415\" Default Backend \u00b6 This annotation is of the form nginx.ingress.kubernetes.io/default-backend: <svc name> to specify a custom default backend. This <svc name> is a reference to a service inside of the same namespace in which you are applying this annotation. This annotation overrides the global default backend. In case the service has multiple ports , the first one is the one which will receive the backend traffic. This service will be used to handle the response when the configured service in the Ingress rule does not have any active endpoints. It will also be used to handle the error responses if both this annotation and the custom-http-errors annotation are set. Enable CORS \u00b6 To enable Cross-Origin Resource Sharing (CORS) in an Ingress rule, add the annotation nginx.ingress.kubernetes.io/enable-cors: \"true\" . This will add a section in the server location enabling this functionality. CORS can be controlled with the following annotations: nginx.ingress.kubernetes.io/cors-allow-methods : Controls which methods are accepted. This is a multi-valued field, separated by ',' and accepts only letters (upper and lower case). Default: GET, PUT, POST, DELETE, PATCH, OPTIONS Example: nginx.ingress.kubernetes.io/cors-allow-methods: \"PUT, GET, POST, OPTIONS\" nginx.ingress.kubernetes.io/cors-allow-headers : Controls which headers are accepted. This is a multi-valued field, separated by ',' and accepts letters, numbers, _ and -. Default: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization Example: nginx.ingress.kubernetes.io/cors-allow-headers: \"X-Forwarded-For, X-app123-XPTO\" nginx.ingress.kubernetes.io/cors-expose-headers : Controls which headers are exposed to response. This is a multi-valued field, separated by ',' and accepts letters, numbers, _, - and *. Default: empty Example: nginx.ingress.kubernetes.io/cors-expose-headers: \"*, X-CustomResponseHeader\" nginx.ingress.kubernetes.io/cors-allow-origin : Controls what's the accepted Origin for CORS. This is a multi-valued field, separated by ','. It must follow this format: http(s)://origin-site.com or http(s)://origin-site.com:port Default: * Example: nginx.ingress.kubernetes.io/cors-allow-origin: \"https://origin-site.com:4443, http://origin-site.com, https://example.org:1199\" It also supports single level wildcard subdomains and follows this format: http(s)://*.foo.bar , http(s)://*.bar.foo:8080 or http(s)://*.abc.bar.foo:9000 - Example: nginx.ingress.kubernetes.io/cors-allow-origin: \"https://*.origin-site.com:4443, http://*.origin-site.com, https://example.org:1199\" nginx.ingress.kubernetes.io/cors-allow-credentials : Controls if credentials can be passed during CORS operations. Default: true Example: nginx.ingress.kubernetes.io/cors-allow-credentials: \"false\" nginx.ingress.kubernetes.io/cors-max-age : Controls how long preflight requests can be cached. Default: 1728000 Example: nginx.ingress.kubernetes.io/cors-max-age: 600 Note For more information please see https://enable-cors.org HTTP2 Push Preload. \u00b6 Enables automatic conversion of preload links specified in the \u201cLink\u201d response header fields into push requests. Example nginx.ingress.kubernetes.io/http2-push-preload: \"true\" Server Alias \u00b6 Allows the definition of one or more aliases in the server definition of the NGINX configuration using the annotation nginx.ingress.kubernetes.io/server-alias: \"<alias 1>,<alias 2>\" . This will create a server with the same configuration, but adding new values to the server_name directive. Note A server-alias name cannot conflict with the hostname of an existing server. If it does, the server-alias annotation will be ignored. If a server-alias is created and later a new server with the same hostname is created, the new server configuration will take place over the alias configuration. For more information please see the server_name documentation . Server snippet \u00b6 Using the annotation nginx.ingress.kubernetes.io/server-snippet it is possible to add custom configuration in the server configuration block. apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : nginx.ingress.kubernetes.io/server-snippet : | set $agentflag 0; if ($http_user_agent ~* \"(Mobile)\" ){ set $agentflag 1; } if ( $agentflag = 1 ) { return 301 https://m.example.com; } Attention This annotation can be used only once per host. Client Body Buffer Size \u00b6 Sets buffer size for reading client request body per location. In case the request body is larger than the buffer, the whole body or only its part is written to a temporary file. By default, buffer size is equal to two memory pages. This is 8K on x86, other 32-bit platforms, and x86-64. It is usually 16K on other 64-bit platforms. This annotation is applied to each location provided in the ingress rule. Note The annotation value must be given in a format understood by Nginx. Example nginx.ingress.kubernetes.io/client-body-buffer-size: \"1000\" # 1000 bytes nginx.ingress.kubernetes.io/client-body-buffer-size: 1k # 1 kilobyte nginx.ingress.kubernetes.io/client-body-buffer-size: 1K # 1 kilobyte nginx.ingress.kubernetes.io/client-body-buffer-size: 1m # 1 megabyte nginx.ingress.kubernetes.io/client-body-buffer-size: 1M # 1 megabyte For more information please see https://nginx.org External Authentication \u00b6 To use an existing service that provides authentication the Ingress rule can be annotated with nginx.ingress.kubernetes.io/auth-url to indicate the URL where the HTTP request should be sent. nginx.ingress.kubernetes.io/auth-url : \"URL to the authentication service\" Additionally it is possible to set: nginx.ingress.kubernetes.io/auth-keepalive : <Connections> to specify the maximum number of keepalive connections to auth-url . Only takes effect when no variables are used in the host part of the URL. Defaults to 0 (keepalive disabled). Note: does not work with HTTP/2 listener because of a limitation in Lua subrequests . UseHTTP2 configuration should be disabled! nginx.ingress.kubernetes.io/auth-keepalive-requests : <Requests> to specify the maximum number of requests that can be served through one keepalive connection. Defaults to 1000 and only applied if auth-keepalive is set to higher than 0 . nginx.ingress.kubernetes.io/auth-keepalive-timeout : <Timeout> to specify a duration in seconds which an idle keepalive connection to an upstream server will stay open. Defaults to 60 and only applied if auth-keepalive is set to higher than 0 . nginx.ingress.kubernetes.io/auth-method : <Method> to specify the HTTP method to use. nginx.ingress.kubernetes.io/auth-signin : <SignIn_URL> to specify the location of the error page. nginx.ingress.kubernetes.io/auth-signin-redirect-param : <SignIn_URL> to specify the URL parameter in the error page which should contain the original URL for a failed signin request. nginx.ingress.kubernetes.io/auth-response-headers : <Response_Header_1, ..., Response_Header_n> to specify headers to pass to backend once authentication request completes. nginx.ingress.kubernetes.io/auth-proxy-set-headers : <ConfigMap> the name of a ConfigMap that specifies headers to pass to the authentication service nginx.ingress.kubernetes.io/auth-request-redirect : <Request_Redirect_URL> to specify the X-Auth-Request-Redirect header value. nginx.ingress.kubernetes.io/auth-cache-key : <Cache_Key> this enables caching for auth requests. specify a lookup key for auth responses. e.g. $remote_user$http_authorization . Each server and location has it's own keyspace. Hence a cached response is only valid on a per-server and per-location basis. nginx.ingress.kubernetes.io/auth-cache-duration : <Cache_duration> to specify a caching time for auth responses based on their response codes, e.g. 200 202 30m . See proxy_cache_valid for details. You may specify multiple, comma-separated values: 200 202 10m, 401 5m . defaults to 200 202 401 5m . nginx.ingress.kubernetes.io/auth-always-set-cookie : <Boolean_Flag> to set a cookie returned by auth request. By default, the cookie will be set only if an upstream reports with the code 200, 201, 204, 206, 301, 302, 303, 304, 307, or 308. nginx.ingress.kubernetes.io/auth-snippet : <Auth_Snippet> to specify a custom snippet to use with external authentication, e.g. nginx.ingress.kubernetes.io/auth-url : http://foo.com/external-auth nginx.ingress.kubernetes.io/auth-snippet : | proxy_set_header Foo-Header 42; Note: nginx.ingress.kubernetes.io/auth-snippet is an optional annotation. However, it may only be used in conjunction with nginx.ingress.kubernetes.io/auth-url and will be ignored if nginx.ingress.kubernetes.io/auth-url is not set Example Please check the external-auth example. Global External Authentication \u00b6 By default the controller redirects all requests to an existing service that provides authentication if global-auth-url is set in the NGINX ConfigMap. If you want to disable this behavior for that ingress, you can use enable-global-auth: \"false\" in the NGINX ConfigMap. nginx.ingress.kubernetes.io/enable-global-auth : indicates if GlobalExternalAuth configuration should be applied or not to this Ingress rule. Default values is set to \"true\" . Note For more information please see global-auth-url . Rate Limiting \u00b6 These annotations define limits on connections and transmission rates. These can be used to mitigate DDoS Attacks . nginx.ingress.kubernetes.io/limit-connections : number of concurrent connections allowed from a single IP address. A 503 error is returned when exceeding this limit. nginx.ingress.kubernetes.io/limit-rps : number of requests accepted from a given IP each second. The burst limit is set to this limit multiplied by the burst multiplier, the default multiplier is 5. When clients exceed this limit, limit-req-status-code default: 503 is returned. nginx.ingress.kubernetes.io/limit-rpm : number of requests accepted from a given IP each minute. The burst limit is set to this limit multiplied by the burst multiplier, the default multiplier is 5. When clients exceed this limit, limit-req-status-code default: 503 is returned. nginx.ingress.kubernetes.io/limit-burst-multiplier : multiplier of the limit rate for burst size. The default burst multiplier is 5, this annotation override the default multiplier. When clients exceed this limit, limit-req-status-code default: 503 is returned. nginx.ingress.kubernetes.io/limit-rate-after : initial number of kilobytes after which the further transmission of a response to a given connection will be rate limited. This feature must be used with proxy-buffering enabled. nginx.ingress.kubernetes.io/limit-rate : number of kilobytes per second allowed to send to a given connection. The zero value disables rate limiting. This feature must be used with proxy-buffering enabled. nginx.ingress.kubernetes.io/limit-whitelist : client IP source ranges to be excluded from rate-limiting. The value is a comma separated list of CIDRs. If you specify multiple annotations in a single Ingress rule, limits are applied in the order limit-connections , limit-rpm , limit-rps . To configure settings globally for all Ingress rules, the limit-rate-after and limit-rate values may be set in the NGINX ConfigMap . The value set in an Ingress annotation will override the global setting. The client IP address will be set based on the use of PROXY protocol or from the X-Forwarded-For header value when use-forwarded-headers is enabled. Global Rate Limiting \u00b6 Note: Be careful when configuring both (Local) Rate Limiting and Global Rate Limiting at the same time. They are two completely different rate limiting implementations. Whichever limit exceeds first will reject the requests. It might be a good idea to configure both of them to ease load on Global Rate Limiting backend in cases of spike in traffic. The stock NGINX rate limiting does not share its counters among different NGINX instances. Given that most ingress-nginx deployments are elastic and number of replicas can change any day it is impossible to configure a proper rate limit using stock NGINX functionalities. Global Rate Limiting overcome this by using lua-resty-global-throttle . lua-resty-global-throttle shares its counters via a central store such as memcached . The obvious shortcoming of this is users have to deploy and operate a memcached instance in order to benefit from this functionality. Configure the memcached using these configmap settings . Here are a few remarks for ingress-nginx integration of lua-resty-global-throttle : We minimize memcached access by caching exceeding limit decisions. The expiry of cache entry is the desired delay lua-resty-global-throttle calculates for us. The Lua Shared Dictionary used for that is global_throttle_cache . Currently its size defaults to 10M. Customize it as per your needs using lua-shared-dicts . When we fail to cache the exceeding limit decision then we log an NGINX error. You can monitor for that error to decide if you need to bump the cache size. Without cache the cost of processing a request is two memcached commands: GET , and INCR . With the cache it is only INCR . Log NGINX variable $global_rate_limit_exceeding 's value to have some visibility into what portion of requests are rejected (value y ), whether they are rejected using cached decision (value c ), or if they are not rejected (default value n ). You can use log-format-upstream to include that in access logs. In case of an error it will log the error message and fail open . The annotations below creates Global Rate Limiting instance per ingress. That means if there are multiple paths configured under the same ingress, the Global Rate Limiting will count requests to all the paths under the same counter. Extract a path out into its own ingress if you need to isolate a certain path. nginx.ingress.kubernetes.io/global-rate-limit : Configures maximum allowed number of requests per window. Required. nginx.ingress.kubernetes.io/global-rate-limit-window : Configures a time window (i.e 1m ) that the limit is applied. Required. nginx.ingress.kubernetes.io/global-rate-limit-key : Configures a key for counting the samples. Defaults to $remote_addr . You can also combine multiple NGINX variables here, like ${remote_addr}-${http_x_api_client} which would mean the limit will be applied to requests coming from the same API client (indicated by X-API-Client HTTP request header) with the same source IP address. nginx.ingress.kubernetes.io/global-rate-limit-ignored-cidrs : comma separated list of IPs and CIDRs to match client IP against. When there's a match request is not considered for rate limiting. Permanent Redirect \u00b6 This annotation allows to return a permanent redirect (Return Code 301) instead of sending data to the upstream. For example nginx.ingress.kubernetes.io/permanent-redirect: https://www.google.com would redirect everything to Google. Permanent Redirect Code \u00b6 This annotation allows you to modify the status code used for permanent redirects. For example nginx.ingress.kubernetes.io/permanent-redirect-code: '308' would return your permanent-redirect with a 308. Temporal Redirect \u00b6 This annotation allows you to return a temporal redirect (Return Code 302) instead of sending data to the upstream. For example nginx.ingress.kubernetes.io/temporal-redirect: https://www.google.com would redirect everything to Google with a Return Code of 302 (Moved Temporarily) SSL Passthrough \u00b6 The annotation nginx.ingress.kubernetes.io/ssl-passthrough instructs the controller to send TLS connections directly to the backend instead of letting NGINX decrypt the communication. See also TLS/HTTPS in the User guide. Note SSL Passthrough is disabled by default and requires starting the controller with the --enable-ssl-passthrough flag. Attention Because SSL Passthrough works on layer 4 of the OSI model (TCP) and not on the layer 7 (HTTP), using SSL Passthrough invalidates all the other annotations set on an Ingress object. Service Upstream \u00b6 By default the NGINX ingress controller uses a list of all endpoints (Pod IP/port) in the NGINX upstream configuration. The nginx.ingress.kubernetes.io/service-upstream annotation disables that behavior and instead uses a single upstream in NGINX, the service's Cluster IP and port. This can be desirable for things like zero-downtime deployments . See issue #257 . Known Issues \u00b6 If the service-upstream annotation is specified the following things should be taken into consideration: Sticky Sessions will not work as only round-robin load balancing is supported. The proxy_next_upstream directive will not have any effect meaning on error the request will not be dispatched to another upstream. Server-side HTTPS enforcement through redirect \u00b6 By default the controller redirects (308) to HTTPS if TLS is enabled for that ingress. If you want to disable this behavior globally, you can use ssl-redirect: \"false\" in the NGINX ConfigMap . To configure this feature for specific ingress resources, you can use the nginx.ingress.kubernetes.io/ssl-redirect: \"false\" annotation in the particular resource. When using SSL offloading outside of cluster (e.g. AWS ELB) it may be useful to enforce a redirect to HTTPS even when there is no TLS certificate available. This can be achieved by using the nginx.ingress.kubernetes.io/force-ssl-redirect: \"true\" annotation in the particular resource. To preserve the trailing slash in the URI with ssl-redirect , set nginx.ingress.kubernetes.io/preserve-trailing-slash: \"true\" annotation for that particular resource. Redirect from/to www \u00b6 In some scenarios is required to redirect from www.domain.com to domain.com or vice versa. To enable this feature use the annotation nginx.ingress.kubernetes.io/from-to-www-redirect: \"true\" Attention If at some point a new Ingress is created with a host equal to one of the options (like domain.com ) the annotation will be omitted. Attention For HTTPS to HTTPS redirects is mandatory the SSL Certificate defined in the Secret, located in the TLS section of Ingress, contains both FQDN in the common name of the certificate. Whitelist source range \u00b6 You can specify allowed client IP source ranges through the nginx.ingress.kubernetes.io/whitelist-source-range annotation. The value is a comma separated list of CIDRs , e.g. 10.0.0.0/24,172.10.0.1 . To configure this setting globally for all Ingress rules, the whitelist-source-range value may be set in the NGINX ConfigMap . Note Adding an annotation to an Ingress rule overrides any global restriction. Custom timeouts \u00b6 Using the configuration configmap it is possible to set the default global timeout for connections to the upstream servers. In some scenarios is required to have different values. To allow this we provide annotations that allows this customization: nginx.ingress.kubernetes.io/proxy-connect-timeout nginx.ingress.kubernetes.io/proxy-send-timeout nginx.ingress.kubernetes.io/proxy-read-timeout nginx.ingress.kubernetes.io/proxy-next-upstream nginx.ingress.kubernetes.io/proxy-next-upstream-timeout nginx.ingress.kubernetes.io/proxy-next-upstream-tries nginx.ingress.kubernetes.io/proxy-request-buffering Note: All timeout values are unitless and in seconds e.g. nginx.ingress.kubernetes.io/proxy-read-timeout: \"120\" sets a valid 120 seconds proxy read timeout. Proxy redirect \u00b6 The annotations nginx.ingress.kubernetes.io/proxy-redirect-from and nginx.ingress.kubernetes.io/proxy-redirect-to will set the first and second parameters of NGINX's proxy_redirect directive respectively. It is possible to set the text that should be changed in the Location and Refresh header fields of a proxied server response Setting \"off\" or \"default\" in the annotation nginx.ingress.kubernetes.io/proxy-redirect-from disables nginx.ingress.kubernetes.io/proxy-redirect-to , otherwise, both annotations must be used in unison. Note that each annotation must be a string without spaces. By default the value of each annotation is \"off\". Custom max body size \u00b6 For NGINX, an 413 error will be returned to the client when the size in a request exceeds the maximum allowed size of the client request body. This size can be configured by the parameter client_max_body_size . To configure this setting globally for all Ingress rules, the proxy-body-size value may be set in the NGINX ConfigMap . To use custom values in an Ingress rule define these annotation: nginx.ingress.kubernetes.io/proxy-body-size : 8m Proxy cookie domain \u00b6 Sets a text that should be changed in the domain attribute of the \"Set-Cookie\" header fields of a proxied server response. To configure this setting globally for all Ingress rules, the proxy-cookie-domain value may be set in the NGINX ConfigMap . Proxy cookie path \u00b6 Sets a text that should be changed in the path attribute of the \"Set-Cookie\" header fields of a proxied server response. To configure this setting globally for all Ingress rules, the proxy-cookie-path value may be set in the NGINX ConfigMap . Proxy buffering \u00b6 Enable or disable proxy buffering proxy_buffering . By default proxy buffering is disabled in the NGINX config. To configure this setting globally for all Ingress rules, the proxy-buffering value may be set in the NGINX ConfigMap . To use custom values in an Ingress rule define these annotation: nginx.ingress.kubernetes.io/proxy-buffering : \"on\" Proxy buffers Number \u00b6 Sets the number of the buffers in proxy_buffers used for reading the first part of the response received from the proxied server. By default proxy buffers number is set as 4 To configure this setting globally, set proxy-buffers-number in NGINX ConfigMap . To use custom values in an Ingress rule, define this annotation: nginx.ingress.kubernetes.io/proxy-buffers-number : \"4\" Proxy buffer size \u00b6 Sets the size of the buffer proxy_buffer_size used for reading the first part of the response received from the proxied server. By default proxy buffer size is set as \"4k\" To configure this setting globally, set proxy-buffer-size in NGINX ConfigMap . To use custom values in an Ingress rule, define this annotation: nginx.ingress.kubernetes.io/proxy-buffer-size : \"8k\" Proxy max temp file size \u00b6 When buffering of responses from the proxied server is enabled, and the whole response does not fit into the buffers set by the proxy_buffer_size and proxy_buffers directives, a part of the response can be saved to a temporary file. This directive sets the maximum size of the temporary file setting the proxy_max_temp_file_size . The size of data written to the temporary file at a time is set by the proxy_temp_file_write_size directive. The zero value disables buffering of responses to temporary files. To use custom values in an Ingress rule, define this annotation: nginx.ingress.kubernetes.io/proxy-max-temp-file-size : \"1024m\" Proxy HTTP version \u00b6 Using this annotation sets the proxy_http_version that the Nginx reverse proxy will use to communicate with the backend. By default this is set to \"1.1\". nginx.ingress.kubernetes.io/proxy-http-version : \"1.0\" SSL ciphers \u00b6 Specifies the enabled ciphers . Using this annotation will set the ssl_ciphers directive at the server level. This configuration is active for all the paths in the host. nginx.ingress.kubernetes.io/ssl-ciphers : \"ALL:!aNULL:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP\" The following annotation will set the ssl_prefer_server_ciphers directive at the server level. This configuration specifies that server ciphers should be preferred over client ciphers when using the SSLv3 and TLS protocols. nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers : \"true\" Connection proxy header \u00b6 Using this annotation will override the default connection header set by NGINX. To use custom values in an Ingress rule, define the annotation: nginx.ingress.kubernetes.io/connection-proxy-header : \"keep-alive\" Enable Access Log \u00b6 Access logs are enabled by default, but in some scenarios access logs might be required to be disabled for a given ingress. To do this, use the annotation: nginx.ingress.kubernetes.io/enable-access-log : \"false\" Enable Rewrite Log \u00b6 Rewrite logs are not enabled by default. In some scenarios it could be required to enable NGINX rewrite logs. Note that rewrite logs are sent to the error_log file at the notice level. To enable this feature use the annotation: nginx.ingress.kubernetes.io/enable-rewrite-log : \"true\" Enable Opentracing \u00b6 Opentracing can be enabled or disabled globally through the ConfigMap but this will sometimes need to be overridden to enable it or disable it for a specific ingress (e.g. to turn off tracing of external health check endpoints) nginx.ingress.kubernetes.io/enable-opentracing : \"true\" Opentracing Trust Incoming Span \u00b6 The option to trust incoming trace spans can be enabled or disabled globally through the ConfigMap but this will sometimes need to be overridden to enable it or disable it for a specific ingress (e.g. only enable on a private endpoint) nginx.ingress.kubernetes.io/opentracing-trust-incoming-span : \"true\" X-Forwarded-Prefix Header \u00b6 To add the non-standard X-Forwarded-Prefix header to the upstream request with a string value, the following annotation can be used: nginx.ingress.kubernetes.io/x-forwarded-prefix : \"/path\" ModSecurity \u00b6 ModSecurity is an OpenSource Web Application firewall. It can be enabled for a particular set of ingress locations. The ModSecurity module must first be enabled by enabling ModSecurity in the ConfigMap . Note this will enable ModSecurity for all paths, and each path must be disabled manually. It can be enabled using the following annotation: nginx.ingress.kubernetes.io/enable-modsecurity : \"true\" ModSecurity will run in \"Detection-Only\" mode using the recommended configuration . You can enable the OWASP Core Rule Set by setting the following annotation: nginx.ingress.kubernetes.io/enable-owasp-core-rules : \"true\" You can pass transactionIDs from nginx by setting up the following: nginx.ingress.kubernetes.io/modsecurity-transaction-id : \"$request_id\" You can also add your own set of modsecurity rules via a snippet: nginx.ingress.kubernetes.io/modsecurity-snippet : | SecRuleEngine On SecDebugLog /tmp/modsec_debug.log Note: If you use both enable-owasp-core-rules and modsecurity-snippet annotations together, only the modsecurity-snippet will take effect. If you wish to include the OWASP Core Rule Set or recommended configuration simply use the include statement: nginx 0.24.1 and below nginx.ingress.kubernetes.io/modsecurity-snippet : | Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf Include /etc/nginx/modsecurity/modsecurity.conf nginx 0.25.0 and above nginx.ingress.kubernetes.io/modsecurity-snippet : | Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf InfluxDB \u00b6 Using influxdb-* annotations we can monitor requests passing through a Location by sending them to an InfluxDB backend exposing the UDP socket using the nginx-influxdb-module . nginx.ingress.kubernetes.io/enable-influxdb : \"true\" nginx.ingress.kubernetes.io/influxdb-measurement : \"nginx-reqs\" nginx.ingress.kubernetes.io/influxdb-port : \"8089\" nginx.ingress.kubernetes.io/influxdb-host : \"127.0.0.1\" nginx.ingress.kubernetes.io/influxdb-server-name : \"nginx-ingress\" For the influxdb-host parameter you have two options: Use an InfluxDB server configured with the UDP protocol enabled. Deploy Telegraf as a sidecar proxy to the Ingress controller configured to listen UDP with the socket listener input and to write using anyone of the outputs plugins like InfluxDB, Apache Kafka, Prometheus, etc.. (recommended) It's important to remember that there's no DNS resolver at this stage so you will have to configure an ip address to nginx.ingress.kubernetes.io/influxdb-host . If you deploy Influx or Telegraf as sidecar (another container in the same pod) this becomes straightforward since you can directly use 127.0.0.1 . Backend Protocol \u00b6 Using backend-protocol annotations is possible to indicate how NGINX should communicate with the backend service. (Replaces secure-backends in older versions) Valid Values: HTTP, HTTPS, GRPC, GRPCS, AJP and FCGI By default NGINX uses HTTP . Example: nginx.ingress.kubernetes.io/backend-protocol : \"HTTPS\" Use Regex \u00b6 Attention When using this annotation with the NGINX annotation nginx.ingress.kubernetes.io/affinity of type cookie , nginx.ingress.kubernetes.io/session-cookie-path must be also set; Session cookie paths do not support regex. Using the nginx.ingress.kubernetes.io/use-regex annotation will indicate whether or not the paths defined on an Ingress use regular expressions. The default value is false . The following will indicate that regular expression paths are being used: nginx.ingress.kubernetes.io/use-regex : \"true\" The following will indicate that regular expression paths are not being used: nginx.ingress.kubernetes.io/use-regex : \"false\" When this annotation is set to true , the case insensitive regular expression location modifier will be enforced on ALL paths for a given host regardless of what Ingress they are defined on. Additionally, if the rewrite-target annotation is used on any Ingress for a given host, then the case insensitive regular expression location modifier will be enforced on ALL paths for a given host regardless of what Ingress they are defined on. Please read about ingress path matching before using this modifier. Satisfy \u00b6 By default, a request would need to satisfy all authentication requirements in order to be allowed. By using this annotation, requests that satisfy either any or all authentication requirements are allowed, based on the configuration value. nginx.ingress.kubernetes.io/satisfy : \"any\" Mirror \u00b6 Enables a request to be mirrored to a mirror backend. Responses by mirror backends are ignored. This feature is useful, to see how requests will react in \"test\" backends. The mirror backend can be set by applying: nginx.ingress.kubernetes.io/mirror-target : https://test.env.com/$request_uri By default the request-body is sent to the mirror backend, but can be turned off by applying: nginx.ingress.kubernetes.io/mirror-request-body : \"off\" Also by default header Host for mirrored requests will be set the same as a host part of uri in the \"mirror-target\" annotation. You can override it by \"mirror-host\" annotation: nginx.ingress.kubernetes.io/mirror-target : https://1.2.3.4/$request_uri nginx.ingress.kubernetes.io/mirror-host : \"test.env.com\" Note: The mirror directive will be applied to all paths within the ingress resource. The request sent to the mirror is linked to the original request. If you have a slow mirror backend, then the original request will throttle. For more information on the mirror module see ngx_http_mirror_module Stream snippet \u00b6 Using the annotation nginx.ingress.kubernetes.io/stream-snippet it is possible to add custom stream configuration. apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : nginx.ingress.kubernetes.io/stream-snippet : | server { listen 8000; proxy_pass 127.0.0.1:80; }","title":"Annotations"},{"location":"user-guide/nginx-configuration/annotations/#annotations","text":"You can add these Kubernetes annotations to specific Ingress objects to customize their behavior. Tip Annotation keys and values can only be strings. Other types, such as boolean or numeric values must be quoted, i.e. \"true\" , \"false\" , \"100\" . Note The annotation prefix can be changed using the --annotations-prefix command line argument , but the default is nginx.ingress.kubernetes.io , as described in the table below. Name type nginx.ingress.kubernetes.io/app-root string nginx.ingress.kubernetes.io/affinity cookie nginx.ingress.kubernetes.io/affinity-mode \"balanced\" or \"persistent\" nginx.ingress.kubernetes.io/affinity-canary-behavior \"sticky\" or \"legacy\" nginx.ingress.kubernetes.io/auth-realm string nginx.ingress.kubernetes.io/auth-secret string nginx.ingress.kubernetes.io/auth-secret-type string nginx.ingress.kubernetes.io/auth-type basic or digest nginx.ingress.kubernetes.io/auth-tls-secret string nginx.ingress.kubernetes.io/auth-tls-verify-depth number nginx.ingress.kubernetes.io/auth-tls-verify-client string nginx.ingress.kubernetes.io/auth-tls-error-page string nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream \"true\" or \"false\" nginx.ingress.kubernetes.io/auth-tls-match-cn string nginx.ingress.kubernetes.io/auth-url string nginx.ingress.kubernetes.io/auth-cache-key string nginx.ingress.kubernetes.io/auth-cache-duration string nginx.ingress.kubernetes.io/auth-keepalive number nginx.ingress.kubernetes.io/auth-keepalive-requests number nginx.ingress.kubernetes.io/auth-keepalive-timeout number nginx.ingress.kubernetes.io/auth-proxy-set-headers string nginx.ingress.kubernetes.io/auth-snippet string nginx.ingress.kubernetes.io/enable-global-auth \"true\" or \"false\" nginx.ingress.kubernetes.io/backend-protocol string nginx.ingress.kubernetes.io/canary \"true\" or \"false\" nginx.ingress.kubernetes.io/canary-by-header string nginx.ingress.kubernetes.io/canary-by-header-value string nginx.ingress.kubernetes.io/canary-by-header-pattern string nginx.ingress.kubernetes.io/canary-by-cookie string nginx.ingress.kubernetes.io/canary-weight number nginx.ingress.kubernetes.io/canary-weight-total number nginx.ingress.kubernetes.io/client-body-buffer-size string nginx.ingress.kubernetes.io/configuration-snippet string nginx.ingress.kubernetes.io/custom-http-errors []int nginx.ingress.kubernetes.io/default-backend string nginx.ingress.kubernetes.io/enable-cors \"true\" or \"false\" nginx.ingress.kubernetes.io/cors-allow-origin string nginx.ingress.kubernetes.io/cors-allow-methods string nginx.ingress.kubernetes.io/cors-allow-headers string nginx.ingress.kubernetes.io/cors-expose-headers string nginx.ingress.kubernetes.io/cors-allow-credentials \"true\" or \"false\" nginx.ingress.kubernetes.io/cors-max-age number nginx.ingress.kubernetes.io/force-ssl-redirect \"true\" or \"false\" nginx.ingress.kubernetes.io/from-to-www-redirect \"true\" or \"false\" nginx.ingress.kubernetes.io/http2-push-preload \"true\" or \"false\" nginx.ingress.kubernetes.io/limit-connections number nginx.ingress.kubernetes.io/limit-rps number nginx.ingress.kubernetes.io/global-rate-limit number nginx.ingress.kubernetes.io/global-rate-limit-window duration nginx.ingress.kubernetes.io/global-rate-limit-key string nginx.ingress.kubernetes.io/global-rate-limit-ignored-cidrs string nginx.ingress.kubernetes.io/permanent-redirect string nginx.ingress.kubernetes.io/permanent-redirect-code number nginx.ingress.kubernetes.io/temporal-redirect string nginx.ingress.kubernetes.io/preserve-trailing-slash \"true\" or \"false\" nginx.ingress.kubernetes.io/proxy-body-size string nginx.ingress.kubernetes.io/proxy-cookie-domain string nginx.ingress.kubernetes.io/proxy-cookie-path string nginx.ingress.kubernetes.io/proxy-connect-timeout number nginx.ingress.kubernetes.io/proxy-send-timeout number nginx.ingress.kubernetes.io/proxy-read-timeout number nginx.ingress.kubernetes.io/proxy-next-upstream string nginx.ingress.kubernetes.io/proxy-next-upstream-timeout number nginx.ingress.kubernetes.io/proxy-next-upstream-tries number nginx.ingress.kubernetes.io/proxy-request-buffering string nginx.ingress.kubernetes.io/proxy-redirect-from string nginx.ingress.kubernetes.io/proxy-redirect-to string nginx.ingress.kubernetes.io/proxy-http-version \"1.0\" or \"1.1\" nginx.ingress.kubernetes.io/proxy-ssl-secret string nginx.ingress.kubernetes.io/proxy-ssl-ciphers string nginx.ingress.kubernetes.io/proxy-ssl-name string nginx.ingress.kubernetes.io/proxy-ssl-protocols string nginx.ingress.kubernetes.io/proxy-ssl-verify string nginx.ingress.kubernetes.io/proxy-ssl-verify-depth number nginx.ingress.kubernetes.io/proxy-ssl-server-name string nginx.ingress.kubernetes.io/enable-rewrite-log \"true\" or \"false\" nginx.ingress.kubernetes.io/rewrite-target URI nginx.ingress.kubernetes.io/satisfy string nginx.ingress.kubernetes.io/server-alias string nginx.ingress.kubernetes.io/server-snippet string nginx.ingress.kubernetes.io/service-upstream \"true\" or \"false\" nginx.ingress.kubernetes.io/session-cookie-name string nginx.ingress.kubernetes.io/session-cookie-path string nginx.ingress.kubernetes.io/session-cookie-domain string nginx.ingress.kubernetes.io/session-cookie-change-on-failure \"true\" or \"false\" nginx.ingress.kubernetes.io/session-cookie-samesite string nginx.ingress.kubernetes.io/session-cookie-conditional-samesite-none \"true\" or \"false\" nginx.ingress.kubernetes.io/ssl-redirect \"true\" or \"false\" nginx.ingress.kubernetes.io/ssl-passthrough \"true\" or \"false\" nginx.ingress.kubernetes.io/stream-snippet string nginx.ingress.kubernetes.io/upstream-hash-by string nginx.ingress.kubernetes.io/x-forwarded-prefix string nginx.ingress.kubernetes.io/load-balance string nginx.ingress.kubernetes.io/upstream-vhost string nginx.ingress.kubernetes.io/whitelist-source-range CIDR nginx.ingress.kubernetes.io/proxy-buffering string nginx.ingress.kubernetes.io/proxy-buffers-number number nginx.ingress.kubernetes.io/proxy-buffer-size string nginx.ingress.kubernetes.io/proxy-max-temp-file-size string nginx.ingress.kubernetes.io/ssl-ciphers string nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers \"true\" or \"false\" nginx.ingress.kubernetes.io/connection-proxy-header string nginx.ingress.kubernetes.io/enable-access-log \"true\" or \"false\" nginx.ingress.kubernetes.io/enable-opentracing \"true\" or \"false\" nginx.ingress.kubernetes.io/opentracing-trust-incoming-span \"true\" or \"false\" nginx.ingress.kubernetes.io/enable-influxdb \"true\" or \"false\" nginx.ingress.kubernetes.io/influxdb-measurement string nginx.ingress.kubernetes.io/influxdb-port string nginx.ingress.kubernetes.io/influxdb-host string nginx.ingress.kubernetes.io/influxdb-server-name string nginx.ingress.kubernetes.io/use-regex bool nginx.ingress.kubernetes.io/enable-modsecurity bool nginx.ingress.kubernetes.io/enable-owasp-core-rules bool nginx.ingress.kubernetes.io/modsecurity-transaction-id string nginx.ingress.kubernetes.io/modsecurity-snippet string nginx.ingress.kubernetes.io/mirror-request-body string nginx.ingress.kubernetes.io/mirror-target string nginx.ingress.kubernetes.io/mirror-host string","title":"Annotations"},{"location":"user-guide/nginx-configuration/annotations/#canary","text":"In some cases, you may want to \"canary\" a new set of changes by sending a small number of requests to a different service than the production service. The canary annotation enables the Ingress spec to act as an alternative service for requests to route to depending on the rules applied. The following annotations to configure canary can be enabled after nginx.ingress.kubernetes.io/canary: \"true\" is set: nginx.ingress.kubernetes.io/canary-by-header : The header to use for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the request header is set to always , it will be routed to the canary. When the header is set to never , it will never be routed to the canary. For any other value, the header will be ignored and the request compared against the other canary rules by precedence. nginx.ingress.kubernetes.io/canary-by-header-value : The header value to match for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the request header is set to this value, it will be routed to the canary. For any other header value, the header will be ignored and the request compared against the other canary rules by precedence. This annotation has to be used together with nginx.ingress.kubernetes.io/canary-by-header . The annotation is an extension of the nginx.ingress.kubernetes.io/canary-by-header to allow customizing the header value instead of using hardcoded values. It doesn't have any effect if the nginx.ingress.kubernetes.io/canary-by-header annotation is not defined. nginx.ingress.kubernetes.io/canary-by-header-pattern : This works the same way as canary-by-header-value except it does PCRE Regex matching. Note that when canary-by-header-value is set this annotation will be ignored. When the given Regex causes error during request processing, the request will be considered as not matching. nginx.ingress.kubernetes.io/canary-by-cookie : The cookie to use for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the cookie value is set to always , it will be routed to the canary. When the cookie is set to never , it will never be routed to the canary. For any other value, the cookie will be ignored and the request compared against the other canary rules by precedence. nginx.ingress.kubernetes.io/canary-weight : The integer based (0 - ) percent of random requests that should be routed to the service specified in the canary Ingress. A weight of 0 implies that no requests will be sent to the service in the Canary ingress by this canary rule. A weight of means implies all requests will be sent to the alternative service specified in the Ingress. <weight-total> defaults to 100, and can be increased via nginx.ingress.kubernetes.io/canary-weight-total . nginx.ingress.kubernetes.io/canary-weight-total : The total weight of traffic. If unspecified, it defaults to 100. Canary rules are evaluated in order of precedence. Precedence is as follows: canary-by-header -> canary-by-cookie -> canary-weight Note that when you mark an ingress as canary, then all the other non-canary annotations will be ignored (inherited from the corresponding main ingress) except nginx.ingress.kubernetes.io/load-balance , nginx.ingress.kubernetes.io/upstream-hash-by , and annotations related to session affinity . If you want to restore the original behavior of canaries when session affinity was ignored, set nginx.ingress.kubernetes.io/affinity-canary-behavior annotation with value legacy on the canary ingress definition. Known Limitations Currently a maximum of one canary ingress can be applied per Ingress rule.","title":"Canary"},{"location":"user-guide/nginx-configuration/annotations/#rewrite","text":"In some scenarios the exposed URL in the backend service differs from the specified path in the Ingress rule. Without a rewrite any request will return 404. Set the annotation nginx.ingress.kubernetes.io/rewrite-target to the path expected by the service. If the Application Root is exposed in a different path and needs to be redirected, set the annotation nginx.ingress.kubernetes.io/app-root to redirect requests for / . Example Please check the rewrite example.","title":"Rewrite"},{"location":"user-guide/nginx-configuration/annotations/#session-affinity","text":"The annotation nginx.ingress.kubernetes.io/affinity enables and sets the affinity type in all Upstreams of an Ingress. This way, a request will always be directed to the same upstream server. The only affinity type available for NGINX is cookie . The annotation nginx.ingress.kubernetes.io/affinity-mode defines the stickiness of a session. Setting this to balanced (default) will redistribute some sessions if a deployment gets scaled up, therefore rebalancing the load on the servers. Setting this to persistent will not rebalance sessions to new servers, therefore providing maximum stickiness. The annotation nginx.ingress.kubernetes.io/affinity-canary-behavior defines the behavior of canaries when session affinity is enabled. Setting this to sticky (default) will ensure that users that were served by canaries, will continue to be served by canaries. Setting this to legacy will restore original canary behavior, when session affinity was ignored. Attention If more than one Ingress is defined for a host and at least one Ingress uses nginx.ingress.kubernetes.io/affinity: cookie , then only paths on the Ingress using nginx.ingress.kubernetes.io/affinity will use session cookie affinity. All paths defined on other Ingresses for the host will be load balanced through the random selection of a backend server. Example Please check the affinity example.","title":"Session Affinity"},{"location":"user-guide/nginx-configuration/annotations/#cookie-affinity","text":"If you use the cookie affinity type you can also specify the name of the cookie that will be used to route the requests with the annotation nginx.ingress.kubernetes.io/session-cookie-name . The default is to create a cookie named 'INGRESSCOOKIE'. The NGINX annotation nginx.ingress.kubernetes.io/session-cookie-path defines the path that will be set on the cookie. This is optional unless the annotation nginx.ingress.kubernetes.io/use-regex is set to true; Session cookie paths do not support regex. Use nginx.ingress.kubernetes.io/session-cookie-domain to set the Domain attribute of the sticky cookie. Use nginx.ingress.kubernetes.io/session-cookie-samesite to apply a SameSite attribute to the sticky cookie. Browser accepted values are None , Lax , and Strict . Some browsers reject cookies with SameSite=None , including those created before the SameSite=None specification (e.g. Chrome 5X). Other browsers mistakenly treat SameSite=None cookies as SameSite=Strict (e.g. Safari running on OSX 14). To omit SameSite=None from browsers with these incompatibilities, add the annotation nginx.ingress.kubernetes.io/session-cookie-conditional-samesite-none: \"true\" .","title":"Cookie affinity"},{"location":"user-guide/nginx-configuration/annotations/#authentication","text":"It is possible to add authentication by adding additional annotations in the Ingress rule. The source of the authentication is a secret that contains usernames and passwords. The annotations are: nginx.ingress.kubernetes.io/auth-type: [basic|digest] Indicates the HTTP Authentication Type: Basic or Digest Access Authentication . nginx.ingress.kubernetes.io/auth-secret: secretName The name of the Secret that contains the usernames and passwords which are granted access to the path s defined in the Ingress rules. This annotation also accepts the alternative form \"namespace/secretName\", in which case the Secret lookup is performed in the referenced namespace instead of the Ingress namespace. nginx.ingress.kubernetes.io/auth-secret-type: [auth-file|auth-map] The auth-secret can have two forms: auth-file - default, an htpasswd file in the key auth within the secret auth-map - the keys of the secret are the usernames, and the values are the hashed passwords nginx.ingress.kubernetes.io/auth-realm: \"realm string\" Example Please check the auth example.","title":"Authentication"},{"location":"user-guide/nginx-configuration/annotations/#custom-nginx-upstream-hashing","text":"NGINX supports load balancing by client-server mapping based on consistent hashing for a given key. The key can contain text, variables or any combination thereof. This feature allows for request stickiness other than client IP or cookies. The ketama consistent hashing method will be used which ensures only a few keys would be remapped to different servers on upstream group changes. There is a special mode of upstream hashing called subset. In this mode, upstream servers are grouped into subsets, and stickiness works by mapping keys to a subset instead of individual upstream servers. Specific server is chosen uniformly at random from the selected sticky subset. It provides a balance between stickiness and load distribution. To enable consistent hashing for a backend: nginx.ingress.kubernetes.io/upstream-hash-by : the nginx variable, text value or any combination thereof to use for consistent hashing. For example: nginx.ingress.kubernetes.io/upstream-hash-by: \"$request_uri\" or nginx.ingress.kubernetes.io/upstream-hash-by: \"$request_uri$host\" or nginx.ingress.kubernetes.io/upstream-hash-by: \"${request_uri}-text-value\" to consistently hash upstream requests by the current request URI. \"subset\" hashing can be enabled setting nginx.ingress.kubernetes.io/upstream-hash-by-subset : \"true\". This maps requests to subset of nodes instead of a single one. upstream-hash-by-subset-size determines the size of each subset (default 3). Please check the chashsubset example.","title":"Custom NGINX upstream hashing"},{"location":"user-guide/nginx-configuration/annotations/#custom-nginx-load-balancing","text":"This is similar to load-balance in ConfigMap , but configures load balancing algorithm per ingress. Note that nginx.ingress.kubernetes.io/upstream-hash-by takes preference over this. If this and nginx.ingress.kubernetes.io/upstream-hash-by are not set then we fallback to using globally configured load balancing algorithm.","title":"Custom NGINX load balancing"},{"location":"user-guide/nginx-configuration/annotations/#custom-nginx-upstream-vhost","text":"This configuration setting allows you to control the value for host in the following statement: proxy_set_header Host $host , which forms part of the location block. This is useful if you need to call the upstream server by something other than $host .","title":"Custom NGINX upstream vhost"},{"location":"user-guide/nginx-configuration/annotations/#client-certificate-authentication","text":"It is possible to enable Client Certificate Authentication using additional annotations in Ingress Rule. Client Certificate Authentication is applied per host and it is not possible to specify rules that differ for individual paths. To enable, add the annotation nginx.ingress.kubernetes.io/auth-tls-secret: namespace/secretName . This secret must have a file named ca.crt containing the full Certificate Authority chain ca.crt that is enabled to authenticate against this Ingress. You can further customize client certificate authentication and behavior with these annotations: nginx.ingress.kubernetes.io/auth-tls-verify-depth : The validation depth between the provided client certificate and the Certification Authority chain. (default: 1) nginx.ingress.kubernetes.io/auth-tls-verify-client : Enables verification of client certificates. Possible values are: on : Request a client certificate that must be signed by a certificate that is included in the secret key ca.crt of the secret specified by nginx.ingress.kubernetes.io/auth-tls-secret: namespace/secretName . Failed certificate verification will result in a status code 400 (Bad Request) (default) off : Don't request client certificates and don't do client certificate verification. optional : Do optional client certificate validation against the CAs from auth-tls-secret . The request fails with status code 400 (Bad Request) when a certificate is provided that is not signed by the CA. When no or an otherwise invalid certificate is provided, the request does not fail, but instead the verification result is sent to the upstream service. optional_no_ca : Do optional client certificate validation, but do not fail the request when the client certificate is not signed by the CAs from auth-tls-secret . Certificate verification result is sent to the upstream service. nginx.ingress.kubernetes.io/auth-tls-error-page : The URL/Page that user should be redirected in case of a Certificate Authentication Error nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream : Indicates if the received certificates should be passed or not to the upstream server in the header ssl-client-cert . Possible values are \"true\" or \"false\" (default). nginx.ingress.kubernetes.io/auth-tls-match-cn : Adds a sanity check for the CN of the client certificate that is sent over using a string / regex starting with \"CN=\", example: \"CN=myvalidclient\" . If the certificate CN sent during mTLS does not match your string / regex it will fail with status code 403. Another way of using this is by adding multiple options in your regex, example: \"CN=(option1|option2|myvalidclient)\" . In this case, as long as one of the options in the brackets matches the certificate CN then you will receive a 200 status code. The following headers are sent to the upstream service according to the auth-tls-* annotations: ssl-client-issuer-dn : The issuer information of the client certificate. Example: \"CN=My CA\" ssl-client-subject-dn : The subject information of the client certificate. Example: \"CN=My Client\" ssl-client-verify : The result of the client verification. Possible values: \"SUCCESS\", \"FAILED: \" ssl-client-cert : The full client certificate in PEM format. Will only be sent when nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream is set to \"true\". Example: -----BEGIN%20CERTIFICATE-----%0A...---END%20CERTIFICATE-----%0A Example Please check the client-certs example. Attention TLS with Client Authentication is not possible in Cloudflare and might result in unexpected behavior. Cloudflare only allows Authenticated Origin Pulls and is required to use their own certificate: https://blog.cloudflare.com/protecting-the-origin-with-tls-authenticated-origin-pulls/ Only Authenticated Origin Pulls are allowed and can be configured by following their tutorial: https://support.cloudflare.com/hc/en-us/articles/204494148-Setting-up-NGINX-to-use-TLS-Authenticated-Origin-Pulls","title":"Client Certificate Authentication"},{"location":"user-guide/nginx-configuration/annotations/#backend-certificate-authentication","text":"It is possible to authenticate to a proxied HTTPS backend with certificate using additional annotations in Ingress Rule. nginx.ingress.kubernetes.io/proxy-ssl-secret: secretName : Specifies a Secret with the certificate tls.crt , key tls.key in PEM format used for authentication to a proxied HTTPS server. It should also contain trusted CA certificates ca.crt in PEM format used to verify the certificate of the proxied HTTPS server. This annotation expects the Secret name in the form \"namespace/secretName\". nginx.ingress.kubernetes.io/proxy-ssl-verify : Enables or disables verification of the proxied HTTPS server certificate. (default: off) nginx.ingress.kubernetes.io/proxy-ssl-verify-depth : Sets the verification depth in the proxied HTTPS server certificates chain. (default: 1) nginx.ingress.kubernetes.io/proxy-ssl-ciphers : Specifies the enabled ciphers for requests to a proxied HTTPS server. The ciphers are specified in the format understood by the OpenSSL library. nginx.ingress.kubernetes.io/proxy-ssl-name : Allows to set proxy_ssl_name . This allows overriding the server name used to verify the certificate of the proxied HTTPS server. This value is also passed through SNI when a connection is established to the proxied HTTPS server. nginx.ingress.kubernetes.io/proxy-ssl-protocols : Enables the specified protocols for requests to a proxied HTTPS server. nginx.ingress.kubernetes.io/proxy-ssl-server-name : Enables passing of the server name through TLS Server Name Indication extension (SNI, RFC 6066) when establishing a connection with the proxied HTTPS server.","title":"Backend Certificate Authentication"},{"location":"user-guide/nginx-configuration/annotations/#configuration-snippet","text":"Using this annotation you can add additional configuration to the NGINX location. For example: nginx.ingress.kubernetes.io/configuration-snippet : | more_set_headers \"Request-Id: $req_id\"; Be aware this can be dangerous in multi-tenant clusters, as it can lead to people with otherwise limited permissions being able to retrieve all secrets on the cluster. The recommended mitigation for this threat is to disable this feature, so it may not work for you. See CVE-2021-25742 and the related issue on github for more information.","title":"Configuration snippet"},{"location":"user-guide/nginx-configuration/annotations/#custom-http-errors","text":"Like the custom-http-errors value in the ConfigMap, this annotation will set NGINX proxy-intercept-errors , but only for the NGINX location associated with this ingress. If a default backend annotation is specified on the ingress, the errors will be routed to that annotation's default backend service (instead of the global default backend). Different ingresses can specify different sets of error codes. Even if multiple ingress objects share the same hostname, this annotation can be used to intercept different error codes for each ingress (for example, different error codes to be intercepted for different paths on the same hostname, if each path is on a different ingress). If custom-http-errors is also specified globally, the error values specified in this annotation will override the global value for the given ingress' hostname and path. Example usage: nginx.ingress.kubernetes.io/custom-http-errors: \"404,415\"","title":"Custom HTTP Errors"},{"location":"user-guide/nginx-configuration/annotations/#default-backend","text":"This annotation is of the form nginx.ingress.kubernetes.io/default-backend: <svc name> to specify a custom default backend. This <svc name> is a reference to a service inside of the same namespace in which you are applying this annotation. This annotation overrides the global default backend. In case the service has multiple ports , the first one is the one which will receive the backend traffic. This service will be used to handle the response when the configured service in the Ingress rule does not have any active endpoints. It will also be used to handle the error responses if both this annotation and the custom-http-errors annotation are set.","title":"Default Backend"},{"location":"user-guide/nginx-configuration/annotations/#enable-cors","text":"To enable Cross-Origin Resource Sharing (CORS) in an Ingress rule, add the annotation nginx.ingress.kubernetes.io/enable-cors: \"true\" . This will add a section in the server location enabling this functionality. CORS can be controlled with the following annotations: nginx.ingress.kubernetes.io/cors-allow-methods : Controls which methods are accepted. This is a multi-valued field, separated by ',' and accepts only letters (upper and lower case). Default: GET, PUT, POST, DELETE, PATCH, OPTIONS Example: nginx.ingress.kubernetes.io/cors-allow-methods: \"PUT, GET, POST, OPTIONS\" nginx.ingress.kubernetes.io/cors-allow-headers : Controls which headers are accepted. This is a multi-valued field, separated by ',' and accepts letters, numbers, _ and -. Default: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization Example: nginx.ingress.kubernetes.io/cors-allow-headers: \"X-Forwarded-For, X-app123-XPTO\" nginx.ingress.kubernetes.io/cors-expose-headers : Controls which headers are exposed to response. This is a multi-valued field, separated by ',' and accepts letters, numbers, _, - and *. Default: empty Example: nginx.ingress.kubernetes.io/cors-expose-headers: \"*, X-CustomResponseHeader\" nginx.ingress.kubernetes.io/cors-allow-origin : Controls what's the accepted Origin for CORS. This is a multi-valued field, separated by ','. It must follow this format: http(s)://origin-site.com or http(s)://origin-site.com:port Default: * Example: nginx.ingress.kubernetes.io/cors-allow-origin: \"https://origin-site.com:4443, http://origin-site.com, https://example.org:1199\" It also supports single level wildcard subdomains and follows this format: http(s)://*.foo.bar , http(s)://*.bar.foo:8080 or http(s)://*.abc.bar.foo:9000 - Example: nginx.ingress.kubernetes.io/cors-allow-origin: \"https://*.origin-site.com:4443, http://*.origin-site.com, https://example.org:1199\" nginx.ingress.kubernetes.io/cors-allow-credentials : Controls if credentials can be passed during CORS operations. Default: true Example: nginx.ingress.kubernetes.io/cors-allow-credentials: \"false\" nginx.ingress.kubernetes.io/cors-max-age : Controls how long preflight requests can be cached. Default: 1728000 Example: nginx.ingress.kubernetes.io/cors-max-age: 600 Note For more information please see https://enable-cors.org","title":"Enable CORS"},{"location":"user-guide/nginx-configuration/annotations/#http2-push-preload","text":"Enables automatic conversion of preload links specified in the \u201cLink\u201d response header fields into push requests. Example nginx.ingress.kubernetes.io/http2-push-preload: \"true\"","title":"HTTP2 Push Preload."},{"location":"user-guide/nginx-configuration/annotations/#server-alias","text":"Allows the definition of one or more aliases in the server definition of the NGINX configuration using the annotation nginx.ingress.kubernetes.io/server-alias: \"<alias 1>,<alias 2>\" . This will create a server with the same configuration, but adding new values to the server_name directive. Note A server-alias name cannot conflict with the hostname of an existing server. If it does, the server-alias annotation will be ignored. If a server-alias is created and later a new server with the same hostname is created, the new server configuration will take place over the alias configuration. For more information please see the server_name documentation .","title":"Server Alias"},{"location":"user-guide/nginx-configuration/annotations/#server-snippet","text":"Using the annotation nginx.ingress.kubernetes.io/server-snippet it is possible to add custom configuration in the server configuration block. apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : nginx.ingress.kubernetes.io/server-snippet : | set $agentflag 0; if ($http_user_agent ~* \"(Mobile)\" ){ set $agentflag 1; } if ( $agentflag = 1 ) { return 301 https://m.example.com; } Attention This annotation can be used only once per host.","title":"Server snippet"},{"location":"user-guide/nginx-configuration/annotations/#client-body-buffer-size","text":"Sets buffer size for reading client request body per location. In case the request body is larger than the buffer, the whole body or only its part is written to a temporary file. By default, buffer size is equal to two memory pages. This is 8K on x86, other 32-bit platforms, and x86-64. It is usually 16K on other 64-bit platforms. This annotation is applied to each location provided in the ingress rule. Note The annotation value must be given in a format understood by Nginx. Example nginx.ingress.kubernetes.io/client-body-buffer-size: \"1000\" # 1000 bytes nginx.ingress.kubernetes.io/client-body-buffer-size: 1k # 1 kilobyte nginx.ingress.kubernetes.io/client-body-buffer-size: 1K # 1 kilobyte nginx.ingress.kubernetes.io/client-body-buffer-size: 1m # 1 megabyte nginx.ingress.kubernetes.io/client-body-buffer-size: 1M # 1 megabyte For more information please see https://nginx.org","title":"Client Body Buffer Size"},{"location":"user-guide/nginx-configuration/annotations/#external-authentication","text":"To use an existing service that provides authentication the Ingress rule can be annotated with nginx.ingress.kubernetes.io/auth-url to indicate the URL where the HTTP request should be sent. nginx.ingress.kubernetes.io/auth-url : \"URL to the authentication service\" Additionally it is possible to set: nginx.ingress.kubernetes.io/auth-keepalive : <Connections> to specify the maximum number of keepalive connections to auth-url . Only takes effect when no variables are used in the host part of the URL. Defaults to 0 (keepalive disabled). Note: does not work with HTTP/2 listener because of a limitation in Lua subrequests . UseHTTP2 configuration should be disabled! nginx.ingress.kubernetes.io/auth-keepalive-requests : <Requests> to specify the maximum number of requests that can be served through one keepalive connection. Defaults to 1000 and only applied if auth-keepalive is set to higher than 0 . nginx.ingress.kubernetes.io/auth-keepalive-timeout : <Timeout> to specify a duration in seconds which an idle keepalive connection to an upstream server will stay open. Defaults to 60 and only applied if auth-keepalive is set to higher than 0 . nginx.ingress.kubernetes.io/auth-method : <Method> to specify the HTTP method to use. nginx.ingress.kubernetes.io/auth-signin : <SignIn_URL> to specify the location of the error page. nginx.ingress.kubernetes.io/auth-signin-redirect-param : <SignIn_URL> to specify the URL parameter in the error page which should contain the original URL for a failed signin request. nginx.ingress.kubernetes.io/auth-response-headers : <Response_Header_1, ..., Response_Header_n> to specify headers to pass to backend once authentication request completes. nginx.ingress.kubernetes.io/auth-proxy-set-headers : <ConfigMap> the name of a ConfigMap that specifies headers to pass to the authentication service nginx.ingress.kubernetes.io/auth-request-redirect : <Request_Redirect_URL> to specify the X-Auth-Request-Redirect header value. nginx.ingress.kubernetes.io/auth-cache-key : <Cache_Key> this enables caching for auth requests. specify a lookup key for auth responses. e.g. $remote_user$http_authorization . Each server and location has it's own keyspace. Hence a cached response is only valid on a per-server and per-location basis. nginx.ingress.kubernetes.io/auth-cache-duration : <Cache_duration> to specify a caching time for auth responses based on their response codes, e.g. 200 202 30m . See proxy_cache_valid for details. You may specify multiple, comma-separated values: 200 202 10m, 401 5m . defaults to 200 202 401 5m . nginx.ingress.kubernetes.io/auth-always-set-cookie : <Boolean_Flag> to set a cookie returned by auth request. By default, the cookie will be set only if an upstream reports with the code 200, 201, 204, 206, 301, 302, 303, 304, 307, or 308. nginx.ingress.kubernetes.io/auth-snippet : <Auth_Snippet> to specify a custom snippet to use with external authentication, e.g. nginx.ingress.kubernetes.io/auth-url : http://foo.com/external-auth nginx.ingress.kubernetes.io/auth-snippet : | proxy_set_header Foo-Header 42; Note: nginx.ingress.kubernetes.io/auth-snippet is an optional annotation. However, it may only be used in conjunction with nginx.ingress.kubernetes.io/auth-url and will be ignored if nginx.ingress.kubernetes.io/auth-url is not set Example Please check the external-auth example.","title":"External Authentication"},{"location":"user-guide/nginx-configuration/annotations/#global-external-authentication","text":"By default the controller redirects all requests to an existing service that provides authentication if global-auth-url is set in the NGINX ConfigMap. If you want to disable this behavior for that ingress, you can use enable-global-auth: \"false\" in the NGINX ConfigMap. nginx.ingress.kubernetes.io/enable-global-auth : indicates if GlobalExternalAuth configuration should be applied or not to this Ingress rule. Default values is set to \"true\" . Note For more information please see global-auth-url .","title":"Global External Authentication"},{"location":"user-guide/nginx-configuration/annotations/#rate-limiting","text":"These annotations define limits on connections and transmission rates. These can be used to mitigate DDoS Attacks . nginx.ingress.kubernetes.io/limit-connections : number of concurrent connections allowed from a single IP address. A 503 error is returned when exceeding this limit. nginx.ingress.kubernetes.io/limit-rps : number of requests accepted from a given IP each second. The burst limit is set to this limit multiplied by the burst multiplier, the default multiplier is 5. When clients exceed this limit, limit-req-status-code default: 503 is returned. nginx.ingress.kubernetes.io/limit-rpm : number of requests accepted from a given IP each minute. The burst limit is set to this limit multiplied by the burst multiplier, the default multiplier is 5. When clients exceed this limit, limit-req-status-code default: 503 is returned. nginx.ingress.kubernetes.io/limit-burst-multiplier : multiplier of the limit rate for burst size. The default burst multiplier is 5, this annotation override the default multiplier. When clients exceed this limit, limit-req-status-code default: 503 is returned. nginx.ingress.kubernetes.io/limit-rate-after : initial number of kilobytes after which the further transmission of a response to a given connection will be rate limited. This feature must be used with proxy-buffering enabled. nginx.ingress.kubernetes.io/limit-rate : number of kilobytes per second allowed to send to a given connection. The zero value disables rate limiting. This feature must be used with proxy-buffering enabled. nginx.ingress.kubernetes.io/limit-whitelist : client IP source ranges to be excluded from rate-limiting. The value is a comma separated list of CIDRs. If you specify multiple annotations in a single Ingress rule, limits are applied in the order limit-connections , limit-rpm , limit-rps . To configure settings globally for all Ingress rules, the limit-rate-after and limit-rate values may be set in the NGINX ConfigMap . The value set in an Ingress annotation will override the global setting. The client IP address will be set based on the use of PROXY protocol or from the X-Forwarded-For header value when use-forwarded-headers is enabled.","title":"Rate Limiting"},{"location":"user-guide/nginx-configuration/annotations/#global-rate-limiting","text":"Note: Be careful when configuring both (Local) Rate Limiting and Global Rate Limiting at the same time. They are two completely different rate limiting implementations. Whichever limit exceeds first will reject the requests. It might be a good idea to configure both of them to ease load on Global Rate Limiting backend in cases of spike in traffic. The stock NGINX rate limiting does not share its counters among different NGINX instances. Given that most ingress-nginx deployments are elastic and number of replicas can change any day it is impossible to configure a proper rate limit using stock NGINX functionalities. Global Rate Limiting overcome this by using lua-resty-global-throttle . lua-resty-global-throttle shares its counters via a central store such as memcached . The obvious shortcoming of this is users have to deploy and operate a memcached instance in order to benefit from this functionality. Configure the memcached using these configmap settings . Here are a few remarks for ingress-nginx integration of lua-resty-global-throttle : We minimize memcached access by caching exceeding limit decisions. The expiry of cache entry is the desired delay lua-resty-global-throttle calculates for us. The Lua Shared Dictionary used for that is global_throttle_cache . Currently its size defaults to 10M. Customize it as per your needs using lua-shared-dicts . When we fail to cache the exceeding limit decision then we log an NGINX error. You can monitor for that error to decide if you need to bump the cache size. Without cache the cost of processing a request is two memcached commands: GET , and INCR . With the cache it is only INCR . Log NGINX variable $global_rate_limit_exceeding 's value to have some visibility into what portion of requests are rejected (value y ), whether they are rejected using cached decision (value c ), or if they are not rejected (default value n ). You can use log-format-upstream to include that in access logs. In case of an error it will log the error message and fail open . The annotations below creates Global Rate Limiting instance per ingress. That means if there are multiple paths configured under the same ingress, the Global Rate Limiting will count requests to all the paths under the same counter. Extract a path out into its own ingress if you need to isolate a certain path. nginx.ingress.kubernetes.io/global-rate-limit : Configures maximum allowed number of requests per window. Required. nginx.ingress.kubernetes.io/global-rate-limit-window : Configures a time window (i.e 1m ) that the limit is applied. Required. nginx.ingress.kubernetes.io/global-rate-limit-key : Configures a key for counting the samples. Defaults to $remote_addr . You can also combine multiple NGINX variables here, like ${remote_addr}-${http_x_api_client} which would mean the limit will be applied to requests coming from the same API client (indicated by X-API-Client HTTP request header) with the same source IP address. nginx.ingress.kubernetes.io/global-rate-limit-ignored-cidrs : comma separated list of IPs and CIDRs to match client IP against. When there's a match request is not considered for rate limiting.","title":"Global Rate Limiting"},{"location":"user-guide/nginx-configuration/annotations/#permanent-redirect","text":"This annotation allows to return a permanent redirect (Return Code 301) instead of sending data to the upstream. For example nginx.ingress.kubernetes.io/permanent-redirect: https://www.google.com would redirect everything to Google.","title":"Permanent Redirect"},{"location":"user-guide/nginx-configuration/annotations/#permanent-redirect-code","text":"This annotation allows you to modify the status code used for permanent redirects. For example nginx.ingress.kubernetes.io/permanent-redirect-code: '308' would return your permanent-redirect with a 308.","title":"Permanent Redirect Code"},{"location":"user-guide/nginx-configuration/annotations/#temporal-redirect","text":"This annotation allows you to return a temporal redirect (Return Code 302) instead of sending data to the upstream. For example nginx.ingress.kubernetes.io/temporal-redirect: https://www.google.com would redirect everything to Google with a Return Code of 302 (Moved Temporarily)","title":"Temporal Redirect"},{"location":"user-guide/nginx-configuration/annotations/#ssl-passthrough","text":"The annotation nginx.ingress.kubernetes.io/ssl-passthrough instructs the controller to send TLS connections directly to the backend instead of letting NGINX decrypt the communication. See also TLS/HTTPS in the User guide. Note SSL Passthrough is disabled by default and requires starting the controller with the --enable-ssl-passthrough flag. Attention Because SSL Passthrough works on layer 4 of the OSI model (TCP) and not on the layer 7 (HTTP), using SSL Passthrough invalidates all the other annotations set on an Ingress object.","title":"SSL Passthrough"},{"location":"user-guide/nginx-configuration/annotations/#service-upstream","text":"By default the NGINX ingress controller uses a list of all endpoints (Pod IP/port) in the NGINX upstream configuration. The nginx.ingress.kubernetes.io/service-upstream annotation disables that behavior and instead uses a single upstream in NGINX, the service's Cluster IP and port. This can be desirable for things like zero-downtime deployments . See issue #257 .","title":"Service Upstream"},{"location":"user-guide/nginx-configuration/annotations/#known-issues","text":"If the service-upstream annotation is specified the following things should be taken into consideration: Sticky Sessions will not work as only round-robin load balancing is supported. The proxy_next_upstream directive will not have any effect meaning on error the request will not be dispatched to another upstream.","title":"Known Issues"},{"location":"user-guide/nginx-configuration/annotations/#server-side-https-enforcement-through-redirect","text":"By default the controller redirects (308) to HTTPS if TLS is enabled for that ingress. If you want to disable this behavior globally, you can use ssl-redirect: \"false\" in the NGINX ConfigMap . To configure this feature for specific ingress resources, you can use the nginx.ingress.kubernetes.io/ssl-redirect: \"false\" annotation in the particular resource. When using SSL offloading outside of cluster (e.g. AWS ELB) it may be useful to enforce a redirect to HTTPS even when there is no TLS certificate available. This can be achieved by using the nginx.ingress.kubernetes.io/force-ssl-redirect: \"true\" annotation in the particular resource. To preserve the trailing slash in the URI with ssl-redirect , set nginx.ingress.kubernetes.io/preserve-trailing-slash: \"true\" annotation for that particular resource.","title":"Server-side HTTPS enforcement through redirect"},{"location":"user-guide/nginx-configuration/annotations/#redirect-fromto-www","text":"In some scenarios is required to redirect from www.domain.com to domain.com or vice versa. To enable this feature use the annotation nginx.ingress.kubernetes.io/from-to-www-redirect: \"true\" Attention If at some point a new Ingress is created with a host equal to one of the options (like domain.com ) the annotation will be omitted. Attention For HTTPS to HTTPS redirects is mandatory the SSL Certificate defined in the Secret, located in the TLS section of Ingress, contains both FQDN in the common name of the certificate.","title":"Redirect from/to www"},{"location":"user-guide/nginx-configuration/annotations/#whitelist-source-range","text":"You can specify allowed client IP source ranges through the nginx.ingress.kubernetes.io/whitelist-source-range annotation. The value is a comma separated list of CIDRs , e.g. 10.0.0.0/24,172.10.0.1 . To configure this setting globally for all Ingress rules, the whitelist-source-range value may be set in the NGINX ConfigMap . Note Adding an annotation to an Ingress rule overrides any global restriction.","title":"Whitelist source range"},{"location":"user-guide/nginx-configuration/annotations/#custom-timeouts","text":"Using the configuration configmap it is possible to set the default global timeout for connections to the upstream servers. In some scenarios is required to have different values. To allow this we provide annotations that allows this customization: nginx.ingress.kubernetes.io/proxy-connect-timeout nginx.ingress.kubernetes.io/proxy-send-timeout nginx.ingress.kubernetes.io/proxy-read-timeout nginx.ingress.kubernetes.io/proxy-next-upstream nginx.ingress.kubernetes.io/proxy-next-upstream-timeout nginx.ingress.kubernetes.io/proxy-next-upstream-tries nginx.ingress.kubernetes.io/proxy-request-buffering Note: All timeout values are unitless and in seconds e.g. nginx.ingress.kubernetes.io/proxy-read-timeout: \"120\" sets a valid 120 seconds proxy read timeout.","title":"Custom timeouts"},{"location":"user-guide/nginx-configuration/annotations/#proxy-redirect","text":"The annotations nginx.ingress.kubernetes.io/proxy-redirect-from and nginx.ingress.kubernetes.io/proxy-redirect-to will set the first and second parameters of NGINX's proxy_redirect directive respectively. It is possible to set the text that should be changed in the Location and Refresh header fields of a proxied server response Setting \"off\" or \"default\" in the annotation nginx.ingress.kubernetes.io/proxy-redirect-from disables nginx.ingress.kubernetes.io/proxy-redirect-to , otherwise, both annotations must be used in unison. Note that each annotation must be a string without spaces. By default the value of each annotation is \"off\".","title":"Proxy redirect"},{"location":"user-guide/nginx-configuration/annotations/#custom-max-body-size","text":"For NGINX, an 413 error will be returned to the client when the size in a request exceeds the maximum allowed size of the client request body. This size can be configured by the parameter client_max_body_size . To configure this setting globally for all Ingress rules, the proxy-body-size value may be set in the NGINX ConfigMap . To use custom values in an Ingress rule define these annotation: nginx.ingress.kubernetes.io/proxy-body-size : 8m","title":"Custom max body size"},{"location":"user-guide/nginx-configuration/annotations/#proxy-cookie-domain","text":"Sets a text that should be changed in the domain attribute of the \"Set-Cookie\" header fields of a proxied server response. To configure this setting globally for all Ingress rules, the proxy-cookie-domain value may be set in the NGINX ConfigMap .","title":"Proxy cookie domain"},{"location":"user-guide/nginx-configuration/annotations/#proxy-cookie-path","text":"Sets a text that should be changed in the path attribute of the \"Set-Cookie\" header fields of a proxied server response. To configure this setting globally for all Ingress rules, the proxy-cookie-path value may be set in the NGINX ConfigMap .","title":"Proxy cookie path"},{"location":"user-guide/nginx-configuration/annotations/#proxy-buffering","text":"Enable or disable proxy buffering proxy_buffering . By default proxy buffering is disabled in the NGINX config. To configure this setting globally for all Ingress rules, the proxy-buffering value may be set in the NGINX ConfigMap . To use custom values in an Ingress rule define these annotation: nginx.ingress.kubernetes.io/proxy-buffering : \"on\"","title":"Proxy buffering"},{"location":"user-guide/nginx-configuration/annotations/#proxy-buffers-number","text":"Sets the number of the buffers in proxy_buffers used for reading the first part of the response received from the proxied server. By default proxy buffers number is set as 4 To configure this setting globally, set proxy-buffers-number in NGINX ConfigMap . To use custom values in an Ingress rule, define this annotation: nginx.ingress.kubernetes.io/proxy-buffers-number : \"4\"","title":"Proxy buffers Number"},{"location":"user-guide/nginx-configuration/annotations/#proxy-buffer-size","text":"Sets the size of the buffer proxy_buffer_size used for reading the first part of the response received from the proxied server. By default proxy buffer size is set as \"4k\" To configure this setting globally, set proxy-buffer-size in NGINX ConfigMap . To use custom values in an Ingress rule, define this annotation: nginx.ingress.kubernetes.io/proxy-buffer-size : \"8k\"","title":"Proxy buffer size"},{"location":"user-guide/nginx-configuration/annotations/#proxy-max-temp-file-size","text":"When buffering of responses from the proxied server is enabled, and the whole response does not fit into the buffers set by the proxy_buffer_size and proxy_buffers directives, a part of the response can be saved to a temporary file. This directive sets the maximum size of the temporary file setting the proxy_max_temp_file_size . The size of data written to the temporary file at a time is set by the proxy_temp_file_write_size directive. The zero value disables buffering of responses to temporary files. To use custom values in an Ingress rule, define this annotation: nginx.ingress.kubernetes.io/proxy-max-temp-file-size : \"1024m\"","title":"Proxy max temp file size"},{"location":"user-guide/nginx-configuration/annotations/#proxy-http-version","text":"Using this annotation sets the proxy_http_version that the Nginx reverse proxy will use to communicate with the backend. By default this is set to \"1.1\". nginx.ingress.kubernetes.io/proxy-http-version : \"1.0\"","title":"Proxy HTTP version"},{"location":"user-guide/nginx-configuration/annotations/#ssl-ciphers","text":"Specifies the enabled ciphers . Using this annotation will set the ssl_ciphers directive at the server level. This configuration is active for all the paths in the host. nginx.ingress.kubernetes.io/ssl-ciphers : \"ALL:!aNULL:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP\" The following annotation will set the ssl_prefer_server_ciphers directive at the server level. This configuration specifies that server ciphers should be preferred over client ciphers when using the SSLv3 and TLS protocols. nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers : \"true\"","title":"SSL ciphers"},{"location":"user-guide/nginx-configuration/annotations/#connection-proxy-header","text":"Using this annotation will override the default connection header set by NGINX. To use custom values in an Ingress rule, define the annotation: nginx.ingress.kubernetes.io/connection-proxy-header : \"keep-alive\"","title":"Connection proxy header"},{"location":"user-guide/nginx-configuration/annotations/#enable-access-log","text":"Access logs are enabled by default, but in some scenarios access logs might be required to be disabled for a given ingress. To do this, use the annotation: nginx.ingress.kubernetes.io/enable-access-log : \"false\"","title":"Enable Access Log"},{"location":"user-guide/nginx-configuration/annotations/#enable-rewrite-log","text":"Rewrite logs are not enabled by default. In some scenarios it could be required to enable NGINX rewrite logs. Note that rewrite logs are sent to the error_log file at the notice level. To enable this feature use the annotation: nginx.ingress.kubernetes.io/enable-rewrite-log : \"true\"","title":"Enable Rewrite Log"},{"location":"user-guide/nginx-configuration/annotations/#enable-opentracing","text":"Opentracing can be enabled or disabled globally through the ConfigMap but this will sometimes need to be overridden to enable it or disable it for a specific ingress (e.g. to turn off tracing of external health check endpoints) nginx.ingress.kubernetes.io/enable-opentracing : \"true\"","title":"Enable Opentracing"},{"location":"user-guide/nginx-configuration/annotations/#opentracing-trust-incoming-span","text":"The option to trust incoming trace spans can be enabled or disabled globally through the ConfigMap but this will sometimes need to be overridden to enable it or disable it for a specific ingress (e.g. only enable on a private endpoint) nginx.ingress.kubernetes.io/opentracing-trust-incoming-span : \"true\"","title":"Opentracing Trust Incoming Span"},{"location":"user-guide/nginx-configuration/annotations/#x-forwarded-prefix-header","text":"To add the non-standard X-Forwarded-Prefix header to the upstream request with a string value, the following annotation can be used: nginx.ingress.kubernetes.io/x-forwarded-prefix : \"/path\"","title":"X-Forwarded-Prefix Header"},{"location":"user-guide/nginx-configuration/annotations/#modsecurity","text":"ModSecurity is an OpenSource Web Application firewall. It can be enabled for a particular set of ingress locations. The ModSecurity module must first be enabled by enabling ModSecurity in the ConfigMap . Note this will enable ModSecurity for all paths, and each path must be disabled manually. It can be enabled using the following annotation: nginx.ingress.kubernetes.io/enable-modsecurity : \"true\" ModSecurity will run in \"Detection-Only\" mode using the recommended configuration . You can enable the OWASP Core Rule Set by setting the following annotation: nginx.ingress.kubernetes.io/enable-owasp-core-rules : \"true\" You can pass transactionIDs from nginx by setting up the following: nginx.ingress.kubernetes.io/modsecurity-transaction-id : \"$request_id\" You can also add your own set of modsecurity rules via a snippet: nginx.ingress.kubernetes.io/modsecurity-snippet : | SecRuleEngine On SecDebugLog /tmp/modsec_debug.log Note: If you use both enable-owasp-core-rules and modsecurity-snippet annotations together, only the modsecurity-snippet will take effect. If you wish to include the OWASP Core Rule Set or recommended configuration simply use the include statement: nginx 0.24.1 and below nginx.ingress.kubernetes.io/modsecurity-snippet : | Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf Include /etc/nginx/modsecurity/modsecurity.conf nginx 0.25.0 and above nginx.ingress.kubernetes.io/modsecurity-snippet : | Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf","title":"ModSecurity"},{"location":"user-guide/nginx-configuration/annotations/#influxdb","text":"Using influxdb-* annotations we can monitor requests passing through a Location by sending them to an InfluxDB backend exposing the UDP socket using the nginx-influxdb-module . nginx.ingress.kubernetes.io/enable-influxdb : \"true\" nginx.ingress.kubernetes.io/influxdb-measurement : \"nginx-reqs\" nginx.ingress.kubernetes.io/influxdb-port : \"8089\" nginx.ingress.kubernetes.io/influxdb-host : \"127.0.0.1\" nginx.ingress.kubernetes.io/influxdb-server-name : \"nginx-ingress\" For the influxdb-host parameter you have two options: Use an InfluxDB server configured with the UDP protocol enabled. Deploy Telegraf as a sidecar proxy to the Ingress controller configured to listen UDP with the socket listener input and to write using anyone of the outputs plugins like InfluxDB, Apache Kafka, Prometheus, etc.. (recommended) It's important to remember that there's no DNS resolver at this stage so you will have to configure an ip address to nginx.ingress.kubernetes.io/influxdb-host . If you deploy Influx or Telegraf as sidecar (another container in the same pod) this becomes straightforward since you can directly use 127.0.0.1 .","title":"InfluxDB"},{"location":"user-guide/nginx-configuration/annotations/#backend-protocol","text":"Using backend-protocol annotations is possible to indicate how NGINX should communicate with the backend service. (Replaces secure-backends in older versions) Valid Values: HTTP, HTTPS, GRPC, GRPCS, AJP and FCGI By default NGINX uses HTTP . Example: nginx.ingress.kubernetes.io/backend-protocol : \"HTTPS\"","title":"Backend Protocol"},{"location":"user-guide/nginx-configuration/annotations/#use-regex","text":"Attention When using this annotation with the NGINX annotation nginx.ingress.kubernetes.io/affinity of type cookie , nginx.ingress.kubernetes.io/session-cookie-path must be also set; Session cookie paths do not support regex. Using the nginx.ingress.kubernetes.io/use-regex annotation will indicate whether or not the paths defined on an Ingress use regular expressions. The default value is false . The following will indicate that regular expression paths are being used: nginx.ingress.kubernetes.io/use-regex : \"true\" The following will indicate that regular expression paths are not being used: nginx.ingress.kubernetes.io/use-regex : \"false\" When this annotation is set to true , the case insensitive regular expression location modifier will be enforced on ALL paths for a given host regardless of what Ingress they are defined on. Additionally, if the rewrite-target annotation is used on any Ingress for a given host, then the case insensitive regular expression location modifier will be enforced on ALL paths for a given host regardless of what Ingress they are defined on. Please read about ingress path matching before using this modifier.","title":"Use Regex"},{"location":"user-guide/nginx-configuration/annotations/#satisfy","text":"By default, a request would need to satisfy all authentication requirements in order to be allowed. By using this annotation, requests that satisfy either any or all authentication requirements are allowed, based on the configuration value. nginx.ingress.kubernetes.io/satisfy : \"any\"","title":"Satisfy"},{"location":"user-guide/nginx-configuration/annotations/#mirror","text":"Enables a request to be mirrored to a mirror backend. Responses by mirror backends are ignored. This feature is useful, to see how requests will react in \"test\" backends. The mirror backend can be set by applying: nginx.ingress.kubernetes.io/mirror-target : https://test.env.com/$request_uri By default the request-body is sent to the mirror backend, but can be turned off by applying: nginx.ingress.kubernetes.io/mirror-request-body : \"off\" Also by default header Host for mirrored requests will be set the same as a host part of uri in the \"mirror-target\" annotation. You can override it by \"mirror-host\" annotation: nginx.ingress.kubernetes.io/mirror-target : https://1.2.3.4/$request_uri nginx.ingress.kubernetes.io/mirror-host : \"test.env.com\" Note: The mirror directive will be applied to all paths within the ingress resource. The request sent to the mirror is linked to the original request. If you have a slow mirror backend, then the original request will throttle. For more information on the mirror module see ngx_http_mirror_module","title":"Mirror"},{"location":"user-guide/nginx-configuration/annotations/#stream-snippet","text":"Using the annotation nginx.ingress.kubernetes.io/stream-snippet it is possible to add custom stream configuration. apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : nginx.ingress.kubernetes.io/stream-snippet : | server { listen 8000; proxy_pass 127.0.0.1:80; }","title":"Stream snippet"},{"location":"user-guide/nginx-configuration/configmap/","text":"ConfigMaps \u00b6 ConfigMaps allow you to decouple configuration artifacts from image content to keep containerized applications portable. The ConfigMap API resource stores configuration data as key-value pairs. The data provides the configurations for system components for the nginx-controller. In order to overwrite nginx-controller configuration values as seen in config.go , you can add key-value pairs to the data section of the config-map. For Example: data : map-hash-bucket-size : \"128\" ssl-protocols : SSLv2 Important The key and values in a ConfigMap can only be strings. This means that we want a value with boolean values we need to quote the values, like \"true\" or \"false\". Same for numbers, like \"100\". \"Slice\" types (defined below as []string or []int ) can be provided as a comma-delimited string. Configuration options \u00b6 The following table shows a configuration option's name, type, and the default value: name type default add-headers string \"\" allow-backend-server-header bool \"false\" allow-snippet-annotations bool true annotation-value-word-blocklist string array \"\" hide-headers string array empty access-log-params string \"\" access-log-path string \"/var/log/nginx/access.log\" http-access-log-path string \"\" stream-access-log-path string \"\" enable-access-log-for-default-backend bool \"false\" error-log-path string \"/var/log/nginx/error.log\" enable-modsecurity bool \"false\" modsecurity-snippet string \"\" enable-owasp-modsecurity-crs bool \"false\" client-header-buffer-size string \"1k\" client-header-timeout int 60 client-body-buffer-size string \"8k\" client-body-timeout int 60 disable-access-log bool false disable-ipv6 bool false disable-ipv6-dns bool false enable-underscores-in-headers bool false enable-ocsp bool false ignore-invalid-headers bool true retry-non-idempotent bool \"false\" error-log-level string \"notice\" http2-max-field-size string \"4k\" http2-max-header-size string \"16k\" http2-max-requests int 1000 http2-max-concurrent-streams int 128 hsts bool \"true\" hsts-include-subdomains bool \"true\" hsts-max-age string \"15724800\" hsts-preload bool \"false\" keep-alive int 75 keep-alive-requests int 100 large-client-header-buffers string \"4 8k\" log-format-escape-none bool \"false\" log-format-escape-json bool \"false\" log-format-upstream string $remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id log-format-stream string [$remote_addr] [$time_local] $protocol $status $bytes_sent $bytes_received $session_time enable-multi-accept bool \"true\" max-worker-connections int 16384 max-worker-open-files int 0 map-hash-bucket-size int 64 nginx-status-ipv4-whitelist []string \"127.0.0.1\" nginx-status-ipv6-whitelist []string \"::1\" proxy-real-ip-cidr []string \"0.0.0.0/0\" proxy-set-headers string \"\" server-name-hash-max-size int 1024 server-name-hash-bucket-size int <size of the processor\u2019s cache line> proxy-headers-hash-max-size int 512 proxy-headers-hash-bucket-size int 64 plugins []string reuse-port bool \"true\" server-tokens bool \"false\" ssl-ciphers string \"ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384\" ssl-ecdh-curve string \"auto\" ssl-dh-param string \"\" ssl-protocols string \"TLSv1.2 TLSv1.3\" ssl-session-cache bool \"true\" ssl-session-cache-size string \"10m\" ssl-session-tickets bool \"false\" ssl-session-ticket-key string <Randomly Generated> ssl-session-timeout string \"10m\" ssl-buffer-size string \"4k\" use-proxy-protocol bool \"false\" proxy-protocol-header-timeout string \"5s\" use-gzip bool \"false\" use-geoip bool \"true\" use-geoip2 bool \"false\" enable-brotli bool \"false\" brotli-level int 4 brotli-min-length int 20 brotli-types string \"application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/javascript text/plain text/x-component\" use-http2 bool \"true\" gzip-level int 1 gzip-types string \"application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/javascript text/plain text/x-component\" worker-processes string <Number of CPUs> worker-cpu-affinity string \"\" worker-shutdown-timeout string \"240s\" load-balance string \"round_robin\" variables-hash-bucket-size int 128 variables-hash-max-size int 2048 upstream-keepalive-connections int 320 upstream-keepalive-time string \"1h\" upstream-keepalive-timeout int 60 upstream-keepalive-requests int 10000 limit-conn-zone-variable string \"$binary_remote_addr\" proxy-stream-timeout string \"600s\" proxy-stream-next-upstream bool \"true\" proxy-stream-next-upstream-timeout string \"600s\" proxy-stream-next-upstream-tries int 3 proxy-stream-responses int 1 bind-address []string \"\" use-forwarded-headers bool \"false\" enable-real-ip bool \"false\" forwarded-for-header string \"X-Forwarded-For\" compute-full-forwarded-for bool \"false\" proxy-add-original-uri-header bool \"false\" generate-request-id bool \"true\" enable-opentracing bool \"false\" opentracing-operation-name string \"\" opentracing-location-operation-name string \"\" zipkin-collector-host string \"\" zipkin-collector-port int 9411 zipkin-service-name string \"nginx\" zipkin-sample-rate float 1.0 jaeger-collector-host string \"\" jaeger-collector-port int 6831 jaeger-endpoint string \"\" jaeger-service-name string \"nginx\" jaeger-propagation-format string \"jaeger\" jaeger-sampler-type string \"const\" jaeger-sampler-param string \"1\" jaeger-sampler-host string \"http://127.0.0.1\" jaeger-sampler-port int 5778 jaeger-trace-context-header-name string uber-trace-id jaeger-debug-header string uber-debug-id jaeger-baggage-header string jaeger-baggage jaeger-trace-baggage-header-prefix string uberctx- datadog-collector-host string \"\" datadog-collector-port int 8126 datadog-service-name string \"nginx\" datadog-environment string \"prod\" datadog-operation-name-override string \"nginx.handle\" datadog-priority-sampling bool \"true\" datadog-sample-rate float 1.0 main-snippet string \"\" http-snippet string \"\" server-snippet string \"\" stream-snippet string \"\" location-snippet string \"\" custom-http-errors []int []int{} proxy-body-size string \"1m\" proxy-connect-timeout int 5 proxy-read-timeout int 60 proxy-send-timeout int 60 proxy-buffers-number int 4 proxy-buffer-size string \"4k\" proxy-cookie-path string \"off\" proxy-cookie-domain string \"off\" proxy-next-upstream string \"error timeout\" proxy-next-upstream-timeout int 0 proxy-next-upstream-tries int 3 proxy-redirect-from string \"off\" proxy-request-buffering string \"on\" ssl-redirect bool \"true\" force-ssl-redirect bool \"false\" whitelist-source-range []string []string{} skip-access-log-urls []string []string{} limit-rate int 0 limit-rate-after int 0 lua-shared-dicts string \"\" http-redirect-code int 308 proxy-buffering string \"off\" limit-req-status-code int 503 limit-conn-status-code int 503 enable-syslog bool false syslog-host string \"\" syslog-port int 514 no-tls-redirect-locations string \"/.well-known/acme-challenge\" global-auth-url string \"\" global-auth-method string \"\" global-auth-signin string \"\" global-auth-signin-redirect-param string \"rd\" global-auth-response-headers string \"\" global-auth-request-redirect string \"\" global-auth-snippet string \"\" global-auth-cache-key string \"\" global-auth-cache-duration string \"200 202 401 5m\" no-auth-locations string \"/.well-known/acme-challenge\" block-cidrs []string \"\" block-user-agents []string \"\" block-referers []string \"\" proxy-ssl-location-only bool \"false\" default-type string \"text/html\" global-rate-limit-memcached-host string \"\" global-rate-limit-memcached-port int 11211 global-rate-limit-memcached-connect-timeout int 50 global-rate-limit-memcached-max-idle-timeout int 10000 global-rate-limit-memcached-pool-size int 50 global-rate-limit-status-code int 429 service-upstream bool \"false\" ssl-reject-handshake bool \"false\" debug-connections []string \"127.0.0.1,1.1.1.1/24\" add-headers \u00b6 Sets custom headers from named configmap before sending traffic to the client. See proxy-set-headers . example allow-backend-server-header \u00b6 Enables the return of the header Server from the backend instead of the generic nginx string. default: is disabled allow-snippet-annotations \u00b6 Enables Ingress to parse and add -snippet annotations/directives created by the user. _**default:* _ true Warning: We recommend enabling this option only if you TRUST users with permission to create Ingress objects, as this may allow a user to add restricted configurations to the final nginx.conf file annotation-value-word-blocklist \u00b6 Contains a comma-separated value of chars/words that are well known of being used to abuse Ingress configuration and must be blocked. Related to CVE-2021-25742 When an annotation is detected with a value that matches one of the blocked bad words, the whole Ingress won't be configured. default: \"\" When doing this, the default blocklist is override, which means that the Ingress admin should add all the words that should be blocked, here is a suggested block list. suggested: \"load_module,lua_package,_by_lua,location,root,proxy_pass,serviceaccount,{,},',\\\"\" hide-headers \u00b6 Sets additional header that will not be passed from the upstream server to the client response. default: empty References: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header access-log-params \u00b6 Additional params for access_log. For example, buffer=16k, gzip, flush=1m References: https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log access-log-path \u00b6 Access log path for both http and stream context. Goes to /var/log/nginx/access.log by default. Note: the file /var/log/nginx/access.log is a symlink to /dev/stdout http-access-log-path \u00b6 Access log path for http context globally. default: \"\" Note: If not specified, the access-log-path will be used. stream-access-log-path \u00b6 Access log path for stream context globally. default: \"\" Note: If not specified, the access-log-path will be used. enable-access-log-for-default-backend \u00b6 Enables logging access to default backend. default: is disabled. error-log-path \u00b6 Error log path. Goes to /var/log/nginx/error.log by default. Note: the file /var/log/nginx/error.log is a symlink to /dev/stderr References: https://nginx.org/en/docs/ngx_core_module.html#error_log enable-modsecurity \u00b6 Enables the modsecurity module for NGINX. default: is disabled enable-owasp-modsecurity-crs \u00b6 Enables the OWASP ModSecurity Core Rule Set (CRS). default: is disabled modsecurity-snippet \u00b6 Adds custom rules to modsecurity section of nginx configuration client-header-buffer-size \u00b6 Allows to configure a custom buffer size for reading client request header. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_buffer_size client-header-timeout \u00b6 Defines a timeout for reading client request header, in seconds. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_timeout client-body-buffer-size \u00b6 Sets buffer size for reading client request body. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size client-body-timeout \u00b6 Defines a timeout for reading client request body, in seconds. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_timeout disable-access-log \u00b6 Disables the Access Log from the entire Ingress Controller. default: false References: https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log disable-ipv6 \u00b6 Disable listening on IPV6. default: false ; IPv6 listening is enabled disable-ipv6-dns \u00b6 Disable IPV6 for nginx DNS resolver. default: false ; IPv6 resolving enabled. enable-underscores-in-headers \u00b6 Enables underscores in header names. default: is disabled enable-ocsp \u00b6 Enables Online Certificate Status Protocol stapling (OCSP) support. default: is disabled ignore-invalid-headers \u00b6 Set if header fields with invalid names should be ignored. default: is enabled retry-non-idempotent \u00b6 Since 1.9.13 NGINX will not retry non-idempotent requests (POST, LOCK, PATCH) in case of an error in the upstream server. The previous behavior can be restored using the value \"true\". error-log-level \u00b6 Configures the logging level of errors. Log levels above are listed in the order of increasing severity. References: https://nginx.org/en/docs/ngx_core_module.html#error_log http2-max-field-size \u00b6 Warning This feature was deprecated in 1.1.3 and will be removed in 1.3.0. Use large-client-header-buffers instead. Limits the maximum size of an HPACK-compressed request header field. References: https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_field_size http2-max-header-size \u00b6 Warning This feature was deprecated in 1.1.3 and will be removed in 1.3.0. Use large-client-header-buffers instead. Limits the maximum size of the entire request header list after HPACK decompression. References: https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_header_size http2-max-requests \u00b6 Warning This feature was deprecated in 1.1.3 and will be removed in 1.3.0. Use upstream-keepalive-requests instead. Sets the maximum number of requests (including push requests) that can be served through one HTTP/2 connection, after which the next client request will lead to connection closing and the need of establishing a new connection. References: https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_requests http2-max-concurrent-streams \u00b6 Sets the maximum number of concurrent HTTP/2 streams in a connection. References: https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_concurrent_streams hsts \u00b6 Enables or disables the header HSTS in servers running SSL. HTTP Strict Transport Security (often abbreviated as HSTS) is a security feature (HTTP header) that tell browsers that it should only be communicated with using HTTPS, instead of using HTTP. It provides protection against protocol downgrade attacks and cookie theft. References: https://developer.mozilla.org/en-US/docs/Web/Security/HTTP_strict_transport_security https://blog.qualys.com/securitylabs/2016/03/28/the-importance-of-a-proper-http-strict-transport-security-implementation-on-your-web-server hsts-include-subdomains \u00b6 Enables or disables the use of HSTS in all the subdomains of the server-name. hsts-max-age \u00b6 Sets the time, in seconds, that the browser should remember that this site is only to be accessed using HTTPS. hsts-preload \u00b6 Enables or disables the preload attribute in the HSTS feature (when it is enabled). keep-alive \u00b6 Sets the time, in seconds, during which a keep-alive client connection will stay open on the server side. The zero value disables keep-alive client connections. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout Important Setting keep-alive: '0' will most likely break concurrent http/2 requests due to changes introduced with nginx 1.19.7 Changes with nginx 1.19.7 16 Feb 2021 *) Change: connections handling in HTTP/2 has been changed to better match HTTP/1.x; the \"http2_recv_timeout\", \"http2_idle_timeout\", and \"http2_max_requests\" directives have been removed, the \"keepalive_timeout\" and \"keepalive_requests\" directives should be used instead. References: nginx change log nginx issue tracker nginx mailing list keep-alive-requests \u00b6 Sets the maximum number of requests that can be served through one keep-alive connection. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests large-client-header-buffers \u00b6 Sets the maximum number and size of buffers used for reading large client request header. default: 4 8k References: https://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers log-format-escape-none \u00b6 Sets if the escape parameter is disabled entirely for character escaping in variables (\"true\") or controlled by log-format-escape-json (\"false\") Sets the nginx log format . log-format-escape-json \u00b6 Sets if the escape parameter allows JSON (\"true\") or default characters escaping in variables (\"false\") Sets the nginx log format . log-format-upstream \u00b6 Sets the nginx log format . Example for json output: log - f orma t - ups trea m : ' { \"time\" : \"$time_iso8601\" , \"remote_addr\" : \"$proxy_protocol_addr\" , \"x_forwarded_for\" : \"$proxy_add_x_forwarded_for\" , \"request_id\" : \"$req_id\" , \"remote_user\" : \"$remote_user\" , \"bytes_sent\" : $by tes _se nt , \"request_time\" : $reques t _ t ime , \"status\" : $s tatus , \"vhost\" : \"$host\" , \"request_proto\" : \"$server_protocol\" , \"path\" : \"$uri\" , \"request_query\" : \"$args\" , \"request_length\" : $reques t _le n g t h , \"duration\" : $reques t _ t ime , \"method\" : \"$request_method\" , \"http_referrer\" : \"$http_referer\" , \"http_user_agent\" : \"$http_user_agent\" } ' Please check the log-format for definition of each field. log-format-stream \u00b6 Sets the nginx stream format . enable-multi-accept \u00b6 If disabled, a worker process will accept one new connection at a time. Otherwise, a worker process will accept all new connections at a time. default: true References: https://nginx.org/en/docs/ngx_core_module.html#multi_accept max-worker-connections \u00b6 Sets the maximum number of simultaneous connections that can be opened by each worker process. 0 will use the value of max-worker-open-files . default: 16384 Tip Using 0 in scenarios of high load improves performance at the cost of increasing RAM utilization (even on idle). max-worker-open-files \u00b6 Sets the maximum number of files that can be opened by each worker process. The default of 0 means \"max open files (system's limit) - 1024\". default: 0 map-hash-bucket-size \u00b6 Sets the bucket size for the map variables hash tables . The details of setting up hash tables are provided in a separate document . proxy-real-ip-cidr \u00b6 If use-forwarded-headers or use-proxy-protocol is enabled, proxy-real-ip-cidr defines the default IP/network address of your external load balancer. Can be a comma-separated list of CIDR blocks. default: \"0.0.0.0/0\" proxy-set-headers \u00b6 Sets custom headers from named configmap before sending traffic to backends. The value format is namespace/name. See example server-name-hash-max-size \u00b6 Sets the maximum size of the server names hash tables used in server names,map directive\u2019s values, MIME types, names of request header strings, etc. References: https://nginx.org/en/docs/hash.html server-name-hash-bucket-size \u00b6 Sets the size of the bucket for the server names hash tables. References: https://nginx.org/en/docs/hash.html https://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_bucket_size proxy-headers-hash-max-size \u00b6 Sets the maximum size of the proxy headers hash tables. References: https://nginx.org/en/docs/hash.html https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_headers_hash_max_size reuse-port \u00b6 Instructs NGINX to create an individual listening socket for each worker process (using the SO_REUSEPORT socket option), allowing a kernel to distribute incoming connections between worker processes default: true proxy-headers-hash-bucket-size \u00b6 Sets the size of the bucket for the proxy headers hash tables. References: https://nginx.org/en/docs/hash.html https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_headers_hash_bucket_size plugins \u00b6 Activates plugins installed in /etc/nginx/lua/plugins . Refer to ingress-nginx plugins README for more information on how to write and install a plugin. server-tokens \u00b6 Send NGINX Server header in responses and display NGINX version in error pages. default: is disabled ssl-ciphers \u00b6 Sets the ciphers list to enable. The ciphers are specified in the format understood by the OpenSSL library. The default cipher list is: ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 . The ordering of a ciphersuite is very important because it decides which algorithms are going to be selected in priority. The recommendation above prioritizes algorithms that provide perfect forward secrecy . DHE-based cyphers will not be available until DH parameter is configured Custom DH parameters for perfect forward secrecy Please check the Mozilla SSL Configuration Generator . Note: ssl_prefer_server_ciphers directive will be enabled by default for http context. ssl-ecdh-curve \u00b6 Specifies a curve for ECDHE ciphers. References: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ecdh_curve ssl-dh-param \u00b6 Sets the name of the secret that contains Diffie-Hellman key to help with \"Perfect Forward Secrecy\". References: https://wiki.openssl.org/index.php/Diffie-Hellman_parameters https://wiki.mozilla.org/Security/Server_Side_TLS#DHE_handshake_and_dhparam https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_dhparam ssl-protocols \u00b6 Sets the SSL protocols to use. The default is: TLSv1.2 TLSv1.3 . Please check the result of the configuration using https://ssllabs.com/ssltest/analyze.html or https://testssl.sh . ssl-early-data \u00b6 Enables or disables TLS 1.3 early data , also known as Zero Round Trip Time Resumption (0-RTT). This requires ssl-protocols to have TLSv1.3 enabled. Enable this with caution, because requests sent within early data are subject to replay attacks . ssl_early_data . The default is: false . ssl-session-cache \u00b6 Enables or disables the use of shared SSL cache among worker processes. ssl-session-cache-size \u00b6 Sets the size of the SSL shared session cache between all worker processes. ssl-session-tickets \u00b6 Enables or disables session resumption through TLS session tickets . ssl-session-ticket-key \u00b6 Sets the secret key used to encrypt and decrypt TLS session tickets. The value must be a valid base64 string. To create a ticket: openssl rand 80 | openssl enc -A -base64 TLS session ticket-key , by default, a randomly generated key is used. ssl-session-timeout \u00b6 Sets the time during which a client may reuse the session parameters stored in a cache. ssl-buffer-size \u00b6 Sets the size of the SSL buffer used for sending data. The default of 4k helps NGINX to improve TLS Time To First Byte (TTTFB). References: https://www.igvita.com/2013/12/16/optimizing-nginx-tls-time-to-first-byte/ use-proxy-protocol \u00b6 Enables or disables the PROXY protocol to receive client connection (real IP address) information passed through proxy servers and load balancers such as HAProxy and Amazon Elastic Load Balancer (ELB). proxy-protocol-header-timeout \u00b6 Sets the timeout value for receiving the proxy-protocol headers. The default of 5 seconds prevents the TLS passthrough handler from waiting indefinitely on a dropped connection. default: 5s use-gzip \u00b6 Enables or disables compression of HTTP responses using the \"gzip\" module . MIME types to compress are controlled by gzip-types . default: false use-geoip \u00b6 Enables or disables \"geoip\" module that creates variables with values depending on the client IP address, using the precompiled MaxMind databases. default: true Note: MaxMind legacy databases are discontinued and will not receive updates after 2019-01-02, cf. discontinuation notice . Consider use-geoip2 below. use-geoip2 \u00b6 Enables the geoip2 module for NGINX. Since 0.27.0 and due to a change in the MaxMind databases a license is required to have access to the databases. For this reason, it is required to define a new flag --maxmind-license-key in the ingress controller deployment to download the databases needed during the initialization of the ingress controller. Alternatively, it is possible to use a volume to mount the files /etc/nginx/geoip/GeoLite2-City.mmdb and /etc/nginx/geoip/GeoLite2-ASN.mmdb , avoiding the overhead of the download. Important If the feature is enabled but the files are missing, GeoIP2 will not be enabled. default: false enable-brotli \u00b6 Enables or disables compression of HTTP responses using the \"brotli\" module . The default mime type list to compress is: application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component . default: is disabled Note: Brotli does not works in Safari < 11. For more information see https://caniuse.com/#feat=brotli brotli-level \u00b6 Sets the Brotli Compression Level that will be used. default: 4 brotli-min-length \u00b6 Minimum length of responses, in bytes, that will be eligible for brotli compression. default: 20 brotli-types \u00b6 Sets the MIME Types that will be compressed on-the-fly by brotli. default: application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component use-http2 \u00b6 Enables or disables HTTP/2 support in secure connections. gzip-level \u00b6 Sets the gzip Compression Level that will be used. default: 1 gzip-min-length \u00b6 Minimum length of responses to be returned to the client before it is eligible for gzip compression, in bytes. default: 256 gzip-types \u00b6 Sets the MIME types in addition to \"text/html\" to compress. The special value \"*\" matches any MIME type. Responses with the \"text/html\" type are always compressed if use-gzip is enabled. default: application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component . worker-processes \u00b6 Sets the number of worker processes . The default of \"auto\" means number of available CPU cores. worker-cpu-affinity \u00b6 Binds worker processes to the sets of CPUs. worker_cpu_affinity . By default worker processes are not bound to any specific CPUs. The value can be: \"\": empty string indicate no affinity is applied. cpumask: e.g. 0001 0010 0100 1000 to bind processes to specific cpus. auto: binding worker processes automatically to available CPUs. worker-shutdown-timeout \u00b6 Sets a timeout for Nginx to wait for worker to gracefully shutdown . default: \"240s\" load-balance \u00b6 Sets the algorithm to use for load balancing. The value can either be: round_robin: to use the default round robin loadbalancer ewma: to use the Peak EWMA method for routing ( implementation ) The default is round_robin . To load balance using consistent hashing of IP or other variables, consider the nginx.ingress.kubernetes.io/upstream-hash-by annotation. To load balance using session cookies, consider the nginx.ingress.kubernetes.io/affinity annotation. References: https://nginx.org/en/docs/http/load_balancing.html variables-hash-bucket-size \u00b6 Sets the bucket size for the variables hash table. References: https://nginx.org/en/docs/http/ngx_http_map_module.html#variables_hash_bucket_size variables-hash-max-size \u00b6 Sets the maximum size of the variables hash table. References: https://nginx.org/en/docs/http/ngx_http_map_module.html#variables_hash_max_size upstream-keepalive-connections \u00b6 Activates the cache for connections to upstream servers. The connections parameter sets the maximum number of idle keepalive connections to upstream servers that are preserved in the cache of each worker process. When this number is exceeded, the least recently used connections are closed. default: 320 References: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive upstream-keepalive-time \u00b6 Sets the maximum time during which requests can be processed through one keepalive connection. default: \"1h\" References: http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_time upstream-keepalive-timeout \u00b6 Sets a timeout during which an idle keepalive connection to an upstream server will stay open. default: 60 References: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_timeout upstream-keepalive-requests \u00b6 Sets the maximum number of requests that can be served through one keepalive connection. After the maximum number of requests is made, the connection is closed. default: 10000 References: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_requests limit-conn-zone-variable \u00b6 Sets parameters for a shared memory zone that will keep states for various keys of limit_conn_zone . The default of \"$binary_remote_addr\" variable\u2019s size is always 4 bytes for IPv4 addresses or 16 bytes for IPv6 addresses. proxy-stream-timeout \u00b6 Sets the timeout between two successive read or write operations on client or proxied server connections. If no data is transmitted within this time, the connection is closed. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout proxy-stream-next-upstream \u00b6 When a connection to the proxied server cannot be established, determines whether a client connection will be passed to the next server. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream proxy-stream-next-upstream-timeout \u00b6 Limits the time allowed to pass a connection to the next server. The 0 value turns off this limitation. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_timeout proxy-stream-next-upstream-tries \u00b6 Limits the number of possible tries a request should be passed to the next server. The 0 value turns off this limitation. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_tries proxy-stream-responses \u00b6 Sets the number of datagrams expected from the proxied server in response to the client request if the UDP protocol is used. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_responses bind-address \u00b6 Sets the addresses on which the server will accept requests instead of *. It should be noted that these addresses must exist in the runtime environment or the controller will crash loop. use-forwarded-headers \u00b6 If true, NGINX passes the incoming X-Forwarded-* headers to upstreams. Use this option when NGINX is behind another L7 proxy / load balancer that is setting these headers. If false, NGINX ignores incoming X-Forwarded-* headers, filling them with the request information it sees. Use this option if NGINX is exposed directly to the internet, or it's behind a L3/packet-based load balancer that doesn't alter the source IP in the packets. enable-real-ip \u00b6 enable-real-ip enables the configuration of https://nginx.org/en/docs/http/ngx_http_realip_module.html . Specific attributes of the module can be configured further by using forwarded-for-header and proxy-real-ip-cidr settings. forwarded-for-header \u00b6 Sets the header field for identifying the originating IP address of a client. default: X-Forwarded-For compute-full-forwarded-for \u00b6 Append the remote address to the X-Forwarded-For header instead of replacing it. When this option is enabled, the upstream application is responsible for extracting the client IP based on its own list of trusted proxies. proxy-add-original-uri-header \u00b6 Adds an X-Original-Uri header with the original request URI to the backend request generate-request-id \u00b6 Ensures that X-Request-ID is defaulted to a random value, if no X-Request-ID is present in the request enable-opentracing \u00b6 Enables the nginx Opentracing extension. default: is disabled References: https://github.com/opentracing-contrib/nginx-opentracing opentracing-operation-name \u00b6 Specifies a custom name for the server span. default: is empty For example, set to \"HTTP $request_method $uri\". opentracing-location-operation-name \u00b6 Specifies a custom name for the location span. default: is empty For example, set to \"HTTP $request_method $uri\". zipkin-collector-host \u00b6 Specifies the host to use when uploading traces. It must be a valid URL. zipkin-collector-port \u00b6 Specifies the port to use when uploading traces. default: 9411 zipkin-service-name \u00b6 Specifies the service name to use for any traces created. default: nginx zipkin-sample-rate \u00b6 Specifies sample rate for any traces created. default: 1.0 jaeger-collector-host \u00b6 Specifies the host to use when uploading traces. It must be a valid URL. jaeger-collector-port \u00b6 Specifies the port to use when uploading traces. default: 6831 jaeger-endpoint \u00b6 Specifies the endpoint to use when uploading traces to a collector. This takes priority over jaeger-collector-host if both are specified. jaeger-service-name \u00b6 Specifies the service name to use for any traces created. default: nginx jaeger-propagation-format \u00b6 Specifies the traceparent/tracestate propagation format. default: jaeger jaeger-sampler-type \u00b6 Specifies the sampler to be used when sampling traces. The available samplers are: const, probabilistic, ratelimiting, remote. default: const jaeger-sampler-param \u00b6 Specifies the argument to be passed to the sampler constructor. Must be a number. For const this should be 0 to never sample and 1 to always sample. default: 1 jaeger-sampler-host \u00b6 Specifies the custom remote sampler host to be passed to the sampler constructor. Must be a valid URL. Leave blank to use default value (localhost). default: http://127.0.0.1 jaeger-sampler-port \u00b6 Specifies the custom remote sampler port to be passed to the sampler constructor. Must be a number. default: 5778 jaeger-trace-context-header-name \u00b6 Specifies the header name used for passing trace context. default: uber-trace-id jaeger-debug-header \u00b6 Specifies the header name used for force sampling. default: jaeger-debug-id jaeger-baggage-header \u00b6 Specifies the header name used to submit baggage if there is no root span. default: jaeger-baggage jaeger-tracer-baggage-header-prefix \u00b6 Specifies the header prefix used to propagate baggage. default: uberctx- datadog-collector-host \u00b6 Specifies the datadog agent host to use when uploading traces. It must be a valid URL. datadog-collector-port \u00b6 Specifies the port to use when uploading traces. default: 8126 datadog-service-name \u00b6 Specifies the service name to use for any traces created. default: nginx datadog-environment \u00b6 Specifies the environment this trace belongs to. default: prod datadog-operation-name-override \u00b6 Overrides the operation name to use for any traces crated. default: nginx.handle datadog-priority-sampling \u00b6 Specifies to use client-side sampling. If true disables client-side sampling (thus ignoring sample_rate ) and enables distributed priority sampling, where traces are sampled based on a combination of user-assigned priorities and configuration from the agent. default: true datadog-sample-rate \u00b6 Specifies sample rate for any traces created. This is effective only when datadog-priority-sampling is false default: 1.0 main-snippet \u00b6 Adds custom configuration to the main section of the nginx configuration. http-snippet \u00b6 Adds custom configuration to the http section of the nginx configuration. server-snippet \u00b6 Adds custom configuration to all the servers in the nginx configuration. stream-snippet \u00b6 Adds custom configuration to the stream section of the nginx configuration. location-snippet \u00b6 Adds custom configuration to all the locations in the nginx configuration. You can not use this to add new locations that proxy to the Kubernetes pods, as the snippet does not have access to the Go template functions. If you want to add custom locations you will have to provide your own nginx.tmpl . custom-http-errors \u00b6 Enables which HTTP codes should be passed for processing with the error_page directive Setting at least one code also enables proxy_intercept_errors which are required to process error_page. Example usage: custom-http-errors: 404,415 proxy-body-size \u00b6 Sets the maximum allowed size of the client request body. See NGINX client_max_body_size . proxy-connect-timeout \u00b6 Sets the timeout for establishing a connection with a proxied server . It should be noted that this timeout cannot usually exceed 75 seconds. proxy-read-timeout \u00b6 Sets the timeout in seconds for reading a response from the proxied server . The timeout is set only between two successive read operations, not for the transmission of the whole response. proxy-send-timeout \u00b6 Sets the timeout in seconds for transmitting a request to the proxied server . The timeout is set only between two successive write operations, not for the transmission of the whole request. proxy-buffers-number \u00b6 Sets the number of the buffer used for reading the first part of the response received from the proxied server. This part usually contains a small response header. proxy-buffer-size \u00b6 Sets the size of the buffer used for reading the first part of the response received from the proxied server. This part usually contains a small response header. proxy-cookie-path \u00b6 Sets a text that should be changed in the path attribute of the \u201cSet-Cookie\u201d header fields of a proxied server response. proxy-cookie-domain \u00b6 Sets a text that should be changed in the domain attribute of the \u201cSet-Cookie\u201d header fields of a proxied server response. proxy-next-upstream \u00b6 Specifies in which cases a request should be passed to the next server. proxy-next-upstream-timeout \u00b6 Limits the time in seconds during which a request can be passed to the next server. proxy-next-upstream-tries \u00b6 Limit the number of possible tries a request should be passed to the next server. proxy-redirect-from \u00b6 Sets the original text that should be changed in the \"Location\" and \"Refresh\" header fields of a proxied server response. default: off References: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_redirect proxy-request-buffering \u00b6 Enables or disables buffering of a client request body . ssl-redirect \u00b6 Sets the global value of redirects (301) to HTTPS if the server has a TLS certificate (defined in an Ingress rule). default: \"true\" force-ssl-redirect \u00b6 Sets the global value of redirects (308) to HTTPS if the server has a default TLS certificate (defined in extra-args). default: \"false\" whitelist-source-range \u00b6 Sets the default whitelisted IPs for each server block. This can be overwritten by an annotation on an Ingress rule. See ngx_http_access_module . skip-access-log-urls \u00b6 Sets a list of URLs that should not appear in the NGINX access log. This is useful with urls like /health or health-check that make \"complex\" reading the logs. default: is empty limit-rate \u00b6 Limits the rate of response transmission to a client. The rate is specified in bytes per second. The zero value disables rate limiting. The limit is set per a request, and so if a client simultaneously opens two connections, the overall rate will be twice as much as the specified limit. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate limit-rate-after \u00b6 Sets the initial amount after which the further transmission of a response to a client will be rate limited. lua-shared-dicts \u00b6 Customize default Lua shared dictionaries or define more. You can use the following syntax to do so: lua-shared-dicts: \"<my dict name>: <my dict size>, [<my dict name>: <my dict size>], ...\" For example following will set default certificate_data dictionary to 100M and will introduce a new dictionary called my_custom_plugin : lua-shared-dicts: \"certificate_data: 100, my_custom_plugin: 5\" You can optionally set a size unit to allow for kilobyte-granularity. Allowed units are 'm' or 'k' (case-insensitive), and it defaults to MB if no unit is provided. Here is a similar example, but the my_custom_plugin dict is only 512KB. lua-shared-dicts: \"certificate_data: 100, my_custom_plugin: 512k\" References: https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate_after http-redirect-code \u00b6 Sets the HTTP status code to be used in redirects. Supported codes are 301 , 302 , 307 and 308 default: 308 Why the default code is 308? RFC 7238 was created to define the 308 (Permanent Redirect) status code that is similar to 301 (Moved Permanently) but it keeps the payload in the redirect. This is important if we send a redirect in methods like POST. proxy-buffering \u00b6 Enables or disables buffering of responses from the proxied server . limit-req-status-code \u00b6 Sets the status code to return in response to rejected requests . default: 503 limit-conn-status-code \u00b6 Sets the status code to return in response to rejected connections . default: 503 enable-syslog \u00b6 Enable syslog feature for access log and error log. default: false syslog-host \u00b6 Sets the address of syslog server. The address can be specified as a domain name or IP address. syslog-port \u00b6 Sets the port of syslog server. default: 514 no-tls-redirect-locations \u00b6 A comma-separated list of locations on which http requests will never get redirected to their https counterpart. default: \"/.well-known/acme-challenge\" global-auth-url \u00b6 A url to an existing service that provides authentication for all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-url . Locations that should not get authenticated can be listed using no-auth-locations See no-auth-locations . In addition, each service can be excluded from authentication via annotation enable-global-auth set to \"false\". default: \"\" References: https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/annotations.md#external-authentication global-auth-method \u00b6 A HTTP method to use for an existing service that provides authentication for all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-method . default: \"\" global-auth-signin \u00b6 Sets the location of the error page for an existing service that provides authentication for all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-signin . default: \"\" global-auth-signin-redirect-param \u00b6 Sets the query parameter in the error page signin URL which contains the original URL of the request that failed authentication. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-signin-redirect-param . default: \"rd\" global-auth-response-headers \u00b6 Sets the headers to pass to backend once authentication request completes. Applied to all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-response-headers . default: \"\" global-auth-request-redirect \u00b6 Sets the X-Auth-Request-Redirect header value. Applied to all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-request-redirect . default: \"\" global-auth-snippet \u00b6 Sets a custom snippet to use with external authentication. Applied to all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-request-redirect . default: \"\" global-auth-cache-key \u00b6 Enables caching for global auth requests. Specify a lookup key for auth responses, e.g. $remote_user$http_authorization . global-auth-cache-duration \u00b6 Set a caching time for auth responses based on their response codes, e.g. 200 202 30m . See proxy_cache_valid for details. You may specify multiple, comma-separated values: 200 202 10m, 401 5m . defaults to 200 202 401 5m . global-auth-always-set-cookie \u00b6 Always set a cookie returned by auth request. By default, the cookie will be set only if an upstream reports with the code 200, 201, 204, 206, 301, 302, 303, 304, 307, or 308. default: false no-auth-locations \u00b6 A comma-separated list of locations that should not get authenticated. default: \"/.well-known/acme-challenge\" block-cidrs \u00b6 A comma-separated list of IP addresses (or subnets), request from which have to be blocked globally. References: https://nginx.org/en/docs/http/ngx_http_access_module.html#deny block-user-agents \u00b6 A comma-separated list of User-Agent, request from which have to be blocked globally. It's possible to use here full strings and regular expressions. More details about valid patterns can be found at map Nginx directive documentation. References: https://nginx.org/en/docs/http/ngx_http_map_module.html#map block-referers \u00b6 A comma-separated list of Referers, request from which have to be blocked globally. It's possible to use here full strings and regular expressions. More details about valid patterns can be found at map Nginx directive documentation. References: https://nginx.org/en/docs/http/ngx_http_map_module.html#map proxy-ssl-location-only \u00b6 Set if proxy-ssl parameters should be applied only on locations and not on servers. default: is disabled default-type \u00b6 Sets the default MIME type of a response. default: text/html References: https://nginx.org/en/docs/http/ngx_http_core_module.html#default_type global-rate-limit \u00b6 global-rate-limit-status-code : configure HTTP status code to return when rejecting requests. Defaults to 429. Configure memcached client for Global Rate Limiting . global-rate-limit-memcached-host : IP/FQDN of memcached server to use. Required to enable Global Rate Limiting. global-rate-limit-memcached-port : port of memcached server to use. Defaults default memcached port of 11211 . global-rate-limit-memcached-connect-timeout : configure timeout for connect, send and receive operations. Unit is millisecond. Defaults to 50ms. global-rate-limit-memcached-max-idle-timeout : configure timeout for cleaning idle connections. Unit is millisecond. Defaults to 50ms. global-rate-limit-memcached-pool-size : configure number of max connections to keep alive. Make sure your memcached server can handle global-rate-limit-memcached-pool-size * worker-processes * <number of ingress-nginx replicas> simultaneous connections. These settings get used by lua-resty-global-throttle that ingress-nginx includes. Refer to the link to learn more about lua-resty-global-throttle . service-upstream \u00b6 Set if the service's Cluster IP and port should be used instead of a list of all endpoints. This can be overwritten by an annotation on an Ingress rule. default: \"false\" ssl-reject-handshake \u00b6 Set to reject SSL handshake to an unknown virtualhost. This parameter helps to mitigate the fingerprinting using default certificate of ingress. default: \"false\" References: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_reject_handshake debug-connections \u00b6 Enables debugging log for selected client connections. default: \"\" References: http://nginx.org/en/docs/ngx_core_module.html#debug_connection","title":"ConfigMap"},{"location":"user-guide/nginx-configuration/configmap/#configmaps","text":"ConfigMaps allow you to decouple configuration artifacts from image content to keep containerized applications portable. The ConfigMap API resource stores configuration data as key-value pairs. The data provides the configurations for system components for the nginx-controller. In order to overwrite nginx-controller configuration values as seen in config.go , you can add key-value pairs to the data section of the config-map. For Example: data : map-hash-bucket-size : \"128\" ssl-protocols : SSLv2 Important The key and values in a ConfigMap can only be strings. This means that we want a value with boolean values we need to quote the values, like \"true\" or \"false\". Same for numbers, like \"100\". \"Slice\" types (defined below as []string or []int ) can be provided as a comma-delimited string.","title":"ConfigMaps"},{"location":"user-guide/nginx-configuration/configmap/#configuration-options","text":"The following table shows a configuration option's name, type, and the default value: name type default add-headers string \"\" allow-backend-server-header bool \"false\" allow-snippet-annotations bool true annotation-value-word-blocklist string array \"\" hide-headers string array empty access-log-params string \"\" access-log-path string \"/var/log/nginx/access.log\" http-access-log-path string \"\" stream-access-log-path string \"\" enable-access-log-for-default-backend bool \"false\" error-log-path string \"/var/log/nginx/error.log\" enable-modsecurity bool \"false\" modsecurity-snippet string \"\" enable-owasp-modsecurity-crs bool \"false\" client-header-buffer-size string \"1k\" client-header-timeout int 60 client-body-buffer-size string \"8k\" client-body-timeout int 60 disable-access-log bool false disable-ipv6 bool false disable-ipv6-dns bool false enable-underscores-in-headers bool false enable-ocsp bool false ignore-invalid-headers bool true retry-non-idempotent bool \"false\" error-log-level string \"notice\" http2-max-field-size string \"4k\" http2-max-header-size string \"16k\" http2-max-requests int 1000 http2-max-concurrent-streams int 128 hsts bool \"true\" hsts-include-subdomains bool \"true\" hsts-max-age string \"15724800\" hsts-preload bool \"false\" keep-alive int 75 keep-alive-requests int 100 large-client-header-buffers string \"4 8k\" log-format-escape-none bool \"false\" log-format-escape-json bool \"false\" log-format-upstream string $remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id log-format-stream string [$remote_addr] [$time_local] $protocol $status $bytes_sent $bytes_received $session_time enable-multi-accept bool \"true\" max-worker-connections int 16384 max-worker-open-files int 0 map-hash-bucket-size int 64 nginx-status-ipv4-whitelist []string \"127.0.0.1\" nginx-status-ipv6-whitelist []string \"::1\" proxy-real-ip-cidr []string \"0.0.0.0/0\" proxy-set-headers string \"\" server-name-hash-max-size int 1024 server-name-hash-bucket-size int <size of the processor\u2019s cache line> proxy-headers-hash-max-size int 512 proxy-headers-hash-bucket-size int 64 plugins []string reuse-port bool \"true\" server-tokens bool \"false\" ssl-ciphers string \"ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384\" ssl-ecdh-curve string \"auto\" ssl-dh-param string \"\" ssl-protocols string \"TLSv1.2 TLSv1.3\" ssl-session-cache bool \"true\" ssl-session-cache-size string \"10m\" ssl-session-tickets bool \"false\" ssl-session-ticket-key string <Randomly Generated> ssl-session-timeout string \"10m\" ssl-buffer-size string \"4k\" use-proxy-protocol bool \"false\" proxy-protocol-header-timeout string \"5s\" use-gzip bool \"false\" use-geoip bool \"true\" use-geoip2 bool \"false\" enable-brotli bool \"false\" brotli-level int 4 brotli-min-length int 20 brotli-types string \"application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/javascript text/plain text/x-component\" use-http2 bool \"true\" gzip-level int 1 gzip-types string \"application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/javascript text/plain text/x-component\" worker-processes string <Number of CPUs> worker-cpu-affinity string \"\" worker-shutdown-timeout string \"240s\" load-balance string \"round_robin\" variables-hash-bucket-size int 128 variables-hash-max-size int 2048 upstream-keepalive-connections int 320 upstream-keepalive-time string \"1h\" upstream-keepalive-timeout int 60 upstream-keepalive-requests int 10000 limit-conn-zone-variable string \"$binary_remote_addr\" proxy-stream-timeout string \"600s\" proxy-stream-next-upstream bool \"true\" proxy-stream-next-upstream-timeout string \"600s\" proxy-stream-next-upstream-tries int 3 proxy-stream-responses int 1 bind-address []string \"\" use-forwarded-headers bool \"false\" enable-real-ip bool \"false\" forwarded-for-header string \"X-Forwarded-For\" compute-full-forwarded-for bool \"false\" proxy-add-original-uri-header bool \"false\" generate-request-id bool \"true\" enable-opentracing bool \"false\" opentracing-operation-name string \"\" opentracing-location-operation-name string \"\" zipkin-collector-host string \"\" zipkin-collector-port int 9411 zipkin-service-name string \"nginx\" zipkin-sample-rate float 1.0 jaeger-collector-host string \"\" jaeger-collector-port int 6831 jaeger-endpoint string \"\" jaeger-service-name string \"nginx\" jaeger-propagation-format string \"jaeger\" jaeger-sampler-type string \"const\" jaeger-sampler-param string \"1\" jaeger-sampler-host string \"http://127.0.0.1\" jaeger-sampler-port int 5778 jaeger-trace-context-header-name string uber-trace-id jaeger-debug-header string uber-debug-id jaeger-baggage-header string jaeger-baggage jaeger-trace-baggage-header-prefix string uberctx- datadog-collector-host string \"\" datadog-collector-port int 8126 datadog-service-name string \"nginx\" datadog-environment string \"prod\" datadog-operation-name-override string \"nginx.handle\" datadog-priority-sampling bool \"true\" datadog-sample-rate float 1.0 main-snippet string \"\" http-snippet string \"\" server-snippet string \"\" stream-snippet string \"\" location-snippet string \"\" custom-http-errors []int []int{} proxy-body-size string \"1m\" proxy-connect-timeout int 5 proxy-read-timeout int 60 proxy-send-timeout int 60 proxy-buffers-number int 4 proxy-buffer-size string \"4k\" proxy-cookie-path string \"off\" proxy-cookie-domain string \"off\" proxy-next-upstream string \"error timeout\" proxy-next-upstream-timeout int 0 proxy-next-upstream-tries int 3 proxy-redirect-from string \"off\" proxy-request-buffering string \"on\" ssl-redirect bool \"true\" force-ssl-redirect bool \"false\" whitelist-source-range []string []string{} skip-access-log-urls []string []string{} limit-rate int 0 limit-rate-after int 0 lua-shared-dicts string \"\" http-redirect-code int 308 proxy-buffering string \"off\" limit-req-status-code int 503 limit-conn-status-code int 503 enable-syslog bool false syslog-host string \"\" syslog-port int 514 no-tls-redirect-locations string \"/.well-known/acme-challenge\" global-auth-url string \"\" global-auth-method string \"\" global-auth-signin string \"\" global-auth-signin-redirect-param string \"rd\" global-auth-response-headers string \"\" global-auth-request-redirect string \"\" global-auth-snippet string \"\" global-auth-cache-key string \"\" global-auth-cache-duration string \"200 202 401 5m\" no-auth-locations string \"/.well-known/acme-challenge\" block-cidrs []string \"\" block-user-agents []string \"\" block-referers []string \"\" proxy-ssl-location-only bool \"false\" default-type string \"text/html\" global-rate-limit-memcached-host string \"\" global-rate-limit-memcached-port int 11211 global-rate-limit-memcached-connect-timeout int 50 global-rate-limit-memcached-max-idle-timeout int 10000 global-rate-limit-memcached-pool-size int 50 global-rate-limit-status-code int 429 service-upstream bool \"false\" ssl-reject-handshake bool \"false\" debug-connections []string \"127.0.0.1,1.1.1.1/24\"","title":"Configuration options"},{"location":"user-guide/nginx-configuration/configmap/#add-headers","text":"Sets custom headers from named configmap before sending traffic to the client. See proxy-set-headers . example","title":"add-headers"},{"location":"user-guide/nginx-configuration/configmap/#allow-backend-server-header","text":"Enables the return of the header Server from the backend instead of the generic nginx string. default: is disabled","title":"allow-backend-server-header"},{"location":"user-guide/nginx-configuration/configmap/#allow-snippet-annotations","text":"Enables Ingress to parse and add -snippet annotations/directives created by the user. _**default:* _ true Warning: We recommend enabling this option only if you TRUST users with permission to create Ingress objects, as this may allow a user to add restricted configurations to the final nginx.conf file","title":"allow-snippet-annotations"},{"location":"user-guide/nginx-configuration/configmap/#annotation-value-word-blocklist","text":"Contains a comma-separated value of chars/words that are well known of being used to abuse Ingress configuration and must be blocked. Related to CVE-2021-25742 When an annotation is detected with a value that matches one of the blocked bad words, the whole Ingress won't be configured. default: \"\" When doing this, the default blocklist is override, which means that the Ingress admin should add all the words that should be blocked, here is a suggested block list. suggested: \"load_module,lua_package,_by_lua,location,root,proxy_pass,serviceaccount,{,},',\\\"\"","title":"annotation-value-word-blocklist"},{"location":"user-guide/nginx-configuration/configmap/#hide-headers","text":"Sets additional header that will not be passed from the upstream server to the client response. default: empty References: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header","title":"hide-headers"},{"location":"user-guide/nginx-configuration/configmap/#access-log-params","text":"Additional params for access_log. For example, buffer=16k, gzip, flush=1m References: https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log","title":"access-log-params"},{"location":"user-guide/nginx-configuration/configmap/#access-log-path","text":"Access log path for both http and stream context. Goes to /var/log/nginx/access.log by default. Note: the file /var/log/nginx/access.log is a symlink to /dev/stdout","title":"access-log-path"},{"location":"user-guide/nginx-configuration/configmap/#http-access-log-path","text":"Access log path for http context globally. default: \"\" Note: If not specified, the access-log-path will be used.","title":"http-access-log-path"},{"location":"user-guide/nginx-configuration/configmap/#stream-access-log-path","text":"Access log path for stream context globally. default: \"\" Note: If not specified, the access-log-path will be used.","title":"stream-access-log-path"},{"location":"user-guide/nginx-configuration/configmap/#enable-access-log-for-default-backend","text":"Enables logging access to default backend. default: is disabled.","title":"enable-access-log-for-default-backend"},{"location":"user-guide/nginx-configuration/configmap/#error-log-path","text":"Error log path. Goes to /var/log/nginx/error.log by default. Note: the file /var/log/nginx/error.log is a symlink to /dev/stderr References: https://nginx.org/en/docs/ngx_core_module.html#error_log","title":"error-log-path"},{"location":"user-guide/nginx-configuration/configmap/#enable-modsecurity","text":"Enables the modsecurity module for NGINX. default: is disabled","title":"enable-modsecurity"},{"location":"user-guide/nginx-configuration/configmap/#enable-owasp-modsecurity-crs","text":"Enables the OWASP ModSecurity Core Rule Set (CRS). default: is disabled","title":"enable-owasp-modsecurity-crs"},{"location":"user-guide/nginx-configuration/configmap/#modsecurity-snippet","text":"Adds custom rules to modsecurity section of nginx configuration","title":"modsecurity-snippet"},{"location":"user-guide/nginx-configuration/configmap/#client-header-buffer-size","text":"Allows to configure a custom buffer size for reading client request header. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_buffer_size","title":"client-header-buffer-size"},{"location":"user-guide/nginx-configuration/configmap/#client-header-timeout","text":"Defines a timeout for reading client request header, in seconds. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_timeout","title":"client-header-timeout"},{"location":"user-guide/nginx-configuration/configmap/#client-body-buffer-size","text":"Sets buffer size for reading client request body. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size","title":"client-body-buffer-size"},{"location":"user-guide/nginx-configuration/configmap/#client-body-timeout","text":"Defines a timeout for reading client request body, in seconds. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_timeout","title":"client-body-timeout"},{"location":"user-guide/nginx-configuration/configmap/#disable-access-log","text":"Disables the Access Log from the entire Ingress Controller. default: false References: https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log","title":"disable-access-log"},{"location":"user-guide/nginx-configuration/configmap/#disable-ipv6","text":"Disable listening on IPV6. default: false ; IPv6 listening is enabled","title":"disable-ipv6"},{"location":"user-guide/nginx-configuration/configmap/#disable-ipv6-dns","text":"Disable IPV6 for nginx DNS resolver. default: false ; IPv6 resolving enabled.","title":"disable-ipv6-dns"},{"location":"user-guide/nginx-configuration/configmap/#enable-underscores-in-headers","text":"Enables underscores in header names. default: is disabled","title":"enable-underscores-in-headers"},{"location":"user-guide/nginx-configuration/configmap/#enable-ocsp","text":"Enables Online Certificate Status Protocol stapling (OCSP) support. default: is disabled","title":"enable-ocsp"},{"location":"user-guide/nginx-configuration/configmap/#ignore-invalid-headers","text":"Set if header fields with invalid names should be ignored. default: is enabled","title":"ignore-invalid-headers"},{"location":"user-guide/nginx-configuration/configmap/#retry-non-idempotent","text":"Since 1.9.13 NGINX will not retry non-idempotent requests (POST, LOCK, PATCH) in case of an error in the upstream server. The previous behavior can be restored using the value \"true\".","title":"retry-non-idempotent"},{"location":"user-guide/nginx-configuration/configmap/#error-log-level","text":"Configures the logging level of errors. Log levels above are listed in the order of increasing severity. References: https://nginx.org/en/docs/ngx_core_module.html#error_log","title":"error-log-level"},{"location":"user-guide/nginx-configuration/configmap/#http2-max-field-size","text":"Warning This feature was deprecated in 1.1.3 and will be removed in 1.3.0. Use large-client-header-buffers instead. Limits the maximum size of an HPACK-compressed request header field. References: https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_field_size","title":"http2-max-field-size"},{"location":"user-guide/nginx-configuration/configmap/#http2-max-header-size","text":"Warning This feature was deprecated in 1.1.3 and will be removed in 1.3.0. Use large-client-header-buffers instead. Limits the maximum size of the entire request header list after HPACK decompression. References: https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_header_size","title":"http2-max-header-size"},{"location":"user-guide/nginx-configuration/configmap/#http2-max-requests","text":"Warning This feature was deprecated in 1.1.3 and will be removed in 1.3.0. Use upstream-keepalive-requests instead. Sets the maximum number of requests (including push requests) that can be served through one HTTP/2 connection, after which the next client request will lead to connection closing and the need of establishing a new connection. References: https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_requests","title":"http2-max-requests"},{"location":"user-guide/nginx-configuration/configmap/#http2-max-concurrent-streams","text":"Sets the maximum number of concurrent HTTP/2 streams in a connection. References: https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_concurrent_streams","title":"http2-max-concurrent-streams"},{"location":"user-guide/nginx-configuration/configmap/#hsts","text":"Enables or disables the header HSTS in servers running SSL. HTTP Strict Transport Security (often abbreviated as HSTS) is a security feature (HTTP header) that tell browsers that it should only be communicated with using HTTPS, instead of using HTTP. It provides protection against protocol downgrade attacks and cookie theft. References: https://developer.mozilla.org/en-US/docs/Web/Security/HTTP_strict_transport_security https://blog.qualys.com/securitylabs/2016/03/28/the-importance-of-a-proper-http-strict-transport-security-implementation-on-your-web-server","title":"hsts"},{"location":"user-guide/nginx-configuration/configmap/#hsts-include-subdomains","text":"Enables or disables the use of HSTS in all the subdomains of the server-name.","title":"hsts-include-subdomains"},{"location":"user-guide/nginx-configuration/configmap/#hsts-max-age","text":"Sets the time, in seconds, that the browser should remember that this site is only to be accessed using HTTPS.","title":"hsts-max-age"},{"location":"user-guide/nginx-configuration/configmap/#hsts-preload","text":"Enables or disables the preload attribute in the HSTS feature (when it is enabled).","title":"hsts-preload"},{"location":"user-guide/nginx-configuration/configmap/#keep-alive","text":"Sets the time, in seconds, during which a keep-alive client connection will stay open on the server side. The zero value disables keep-alive client connections. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout Important Setting keep-alive: '0' will most likely break concurrent http/2 requests due to changes introduced with nginx 1.19.7 Changes with nginx 1.19.7 16 Feb 2021 *) Change: connections handling in HTTP/2 has been changed to better match HTTP/1.x; the \"http2_recv_timeout\", \"http2_idle_timeout\", and \"http2_max_requests\" directives have been removed, the \"keepalive_timeout\" and \"keepalive_requests\" directives should be used instead. References: nginx change log nginx issue tracker nginx mailing list","title":"keep-alive"},{"location":"user-guide/nginx-configuration/configmap/#keep-alive-requests","text":"Sets the maximum number of requests that can be served through one keep-alive connection. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests","title":"keep-alive-requests"},{"location":"user-guide/nginx-configuration/configmap/#large-client-header-buffers","text":"Sets the maximum number and size of buffers used for reading large client request header. default: 4 8k References: https://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers","title":"large-client-header-buffers"},{"location":"user-guide/nginx-configuration/configmap/#log-format-escape-none","text":"Sets if the escape parameter is disabled entirely for character escaping in variables (\"true\") or controlled by log-format-escape-json (\"false\") Sets the nginx log format .","title":"log-format-escape-none"},{"location":"user-guide/nginx-configuration/configmap/#log-format-escape-json","text":"Sets if the escape parameter allows JSON (\"true\") or default characters escaping in variables (\"false\") Sets the nginx log format .","title":"log-format-escape-json"},{"location":"user-guide/nginx-configuration/configmap/#log-format-upstream","text":"Sets the nginx log format . Example for json output: log - f orma t - ups trea m : ' { \"time\" : \"$time_iso8601\" , \"remote_addr\" : \"$proxy_protocol_addr\" , \"x_forwarded_for\" : \"$proxy_add_x_forwarded_for\" , \"request_id\" : \"$req_id\" , \"remote_user\" : \"$remote_user\" , \"bytes_sent\" : $by tes _se nt , \"request_time\" : $reques t _ t ime , \"status\" : $s tatus , \"vhost\" : \"$host\" , \"request_proto\" : \"$server_protocol\" , \"path\" : \"$uri\" , \"request_query\" : \"$args\" , \"request_length\" : $reques t _le n g t h , \"duration\" : $reques t _ t ime , \"method\" : \"$request_method\" , \"http_referrer\" : \"$http_referer\" , \"http_user_agent\" : \"$http_user_agent\" } ' Please check the log-format for definition of each field.","title":"log-format-upstream"},{"location":"user-guide/nginx-configuration/configmap/#log-format-stream","text":"Sets the nginx stream format .","title":"log-format-stream"},{"location":"user-guide/nginx-configuration/configmap/#enable-multi-accept","text":"If disabled, a worker process will accept one new connection at a time. Otherwise, a worker process will accept all new connections at a time. default: true References: https://nginx.org/en/docs/ngx_core_module.html#multi_accept","title":"enable-multi-accept"},{"location":"user-guide/nginx-configuration/configmap/#max-worker-connections","text":"Sets the maximum number of simultaneous connections that can be opened by each worker process. 0 will use the value of max-worker-open-files . default: 16384 Tip Using 0 in scenarios of high load improves performance at the cost of increasing RAM utilization (even on idle).","title":"max-worker-connections"},{"location":"user-guide/nginx-configuration/configmap/#max-worker-open-files","text":"Sets the maximum number of files that can be opened by each worker process. The default of 0 means \"max open files (system's limit) - 1024\". default: 0","title":"max-worker-open-files"},{"location":"user-guide/nginx-configuration/configmap/#map-hash-bucket-size","text":"Sets the bucket size for the map variables hash tables . The details of setting up hash tables are provided in a separate document .","title":"map-hash-bucket-size"},{"location":"user-guide/nginx-configuration/configmap/#proxy-real-ip-cidr","text":"If use-forwarded-headers or use-proxy-protocol is enabled, proxy-real-ip-cidr defines the default IP/network address of your external load balancer. Can be a comma-separated list of CIDR blocks. default: \"0.0.0.0/0\"","title":"proxy-real-ip-cidr"},{"location":"user-guide/nginx-configuration/configmap/#proxy-set-headers","text":"Sets custom headers from named configmap before sending traffic to backends. The value format is namespace/name. See example","title":"proxy-set-headers"},{"location":"user-guide/nginx-configuration/configmap/#server-name-hash-max-size","text":"Sets the maximum size of the server names hash tables used in server names,map directive\u2019s values, MIME types, names of request header strings, etc. References: https://nginx.org/en/docs/hash.html","title":"server-name-hash-max-size"},{"location":"user-guide/nginx-configuration/configmap/#server-name-hash-bucket-size","text":"Sets the size of the bucket for the server names hash tables. References: https://nginx.org/en/docs/hash.html https://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_bucket_size","title":"server-name-hash-bucket-size"},{"location":"user-guide/nginx-configuration/configmap/#proxy-headers-hash-max-size","text":"Sets the maximum size of the proxy headers hash tables. References: https://nginx.org/en/docs/hash.html https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_headers_hash_max_size","title":"proxy-headers-hash-max-size"},{"location":"user-guide/nginx-configuration/configmap/#reuse-port","text":"Instructs NGINX to create an individual listening socket for each worker process (using the SO_REUSEPORT socket option), allowing a kernel to distribute incoming connections between worker processes default: true","title":"reuse-port"},{"location":"user-guide/nginx-configuration/configmap/#proxy-headers-hash-bucket-size","text":"Sets the size of the bucket for the proxy headers hash tables. References: https://nginx.org/en/docs/hash.html https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_headers_hash_bucket_size","title":"proxy-headers-hash-bucket-size"},{"location":"user-guide/nginx-configuration/configmap/#plugins","text":"Activates plugins installed in /etc/nginx/lua/plugins . Refer to ingress-nginx plugins README for more information on how to write and install a plugin.","title":"plugins"},{"location":"user-guide/nginx-configuration/configmap/#server-tokens","text":"Send NGINX Server header in responses and display NGINX version in error pages. default: is disabled","title":"server-tokens"},{"location":"user-guide/nginx-configuration/configmap/#ssl-ciphers","text":"Sets the ciphers list to enable. The ciphers are specified in the format understood by the OpenSSL library. The default cipher list is: ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 . The ordering of a ciphersuite is very important because it decides which algorithms are going to be selected in priority. The recommendation above prioritizes algorithms that provide perfect forward secrecy . DHE-based cyphers will not be available until DH parameter is configured Custom DH parameters for perfect forward secrecy Please check the Mozilla SSL Configuration Generator . Note: ssl_prefer_server_ciphers directive will be enabled by default for http context.","title":"ssl-ciphers"},{"location":"user-guide/nginx-configuration/configmap/#ssl-ecdh-curve","text":"Specifies a curve for ECDHE ciphers. References: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ecdh_curve","title":"ssl-ecdh-curve"},{"location":"user-guide/nginx-configuration/configmap/#ssl-dh-param","text":"Sets the name of the secret that contains Diffie-Hellman key to help with \"Perfect Forward Secrecy\". References: https://wiki.openssl.org/index.php/Diffie-Hellman_parameters https://wiki.mozilla.org/Security/Server_Side_TLS#DHE_handshake_and_dhparam https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_dhparam","title":"ssl-dh-param"},{"location":"user-guide/nginx-configuration/configmap/#ssl-protocols","text":"Sets the SSL protocols to use. The default is: TLSv1.2 TLSv1.3 . Please check the result of the configuration using https://ssllabs.com/ssltest/analyze.html or https://testssl.sh .","title":"ssl-protocols"},{"location":"user-guide/nginx-configuration/configmap/#ssl-early-data","text":"Enables or disables TLS 1.3 early data , also known as Zero Round Trip Time Resumption (0-RTT). This requires ssl-protocols to have TLSv1.3 enabled. Enable this with caution, because requests sent within early data are subject to replay attacks . ssl_early_data . The default is: false .","title":"ssl-early-data"},{"location":"user-guide/nginx-configuration/configmap/#ssl-session-cache","text":"Enables or disables the use of shared SSL cache among worker processes.","title":"ssl-session-cache"},{"location":"user-guide/nginx-configuration/configmap/#ssl-session-cache-size","text":"Sets the size of the SSL shared session cache between all worker processes.","title":"ssl-session-cache-size"},{"location":"user-guide/nginx-configuration/configmap/#ssl-session-tickets","text":"Enables or disables session resumption through TLS session tickets .","title":"ssl-session-tickets"},{"location":"user-guide/nginx-configuration/configmap/#ssl-session-ticket-key","text":"Sets the secret key used to encrypt and decrypt TLS session tickets. The value must be a valid base64 string. To create a ticket: openssl rand 80 | openssl enc -A -base64 TLS session ticket-key , by default, a randomly generated key is used.","title":"ssl-session-ticket-key"},{"location":"user-guide/nginx-configuration/configmap/#ssl-session-timeout","text":"Sets the time during which a client may reuse the session parameters stored in a cache.","title":"ssl-session-timeout"},{"location":"user-guide/nginx-configuration/configmap/#ssl-buffer-size","text":"Sets the size of the SSL buffer used for sending data. The default of 4k helps NGINX to improve TLS Time To First Byte (TTTFB). References: https://www.igvita.com/2013/12/16/optimizing-nginx-tls-time-to-first-byte/","title":"ssl-buffer-size"},{"location":"user-guide/nginx-configuration/configmap/#use-proxy-protocol","text":"Enables or disables the PROXY protocol to receive client connection (real IP address) information passed through proxy servers and load balancers such as HAProxy and Amazon Elastic Load Balancer (ELB).","title":"use-proxy-protocol"},{"location":"user-guide/nginx-configuration/configmap/#proxy-protocol-header-timeout","text":"Sets the timeout value for receiving the proxy-protocol headers. The default of 5 seconds prevents the TLS passthrough handler from waiting indefinitely on a dropped connection. default: 5s","title":"proxy-protocol-header-timeout"},{"location":"user-guide/nginx-configuration/configmap/#use-gzip","text":"Enables or disables compression of HTTP responses using the \"gzip\" module . MIME types to compress are controlled by gzip-types . default: false","title":"use-gzip"},{"location":"user-guide/nginx-configuration/configmap/#use-geoip","text":"Enables or disables \"geoip\" module that creates variables with values depending on the client IP address, using the precompiled MaxMind databases. default: true Note: MaxMind legacy databases are discontinued and will not receive updates after 2019-01-02, cf. discontinuation notice . Consider use-geoip2 below.","title":"use-geoip"},{"location":"user-guide/nginx-configuration/configmap/#use-geoip2","text":"Enables the geoip2 module for NGINX. Since 0.27.0 and due to a change in the MaxMind databases a license is required to have access to the databases. For this reason, it is required to define a new flag --maxmind-license-key in the ingress controller deployment to download the databases needed during the initialization of the ingress controller. Alternatively, it is possible to use a volume to mount the files /etc/nginx/geoip/GeoLite2-City.mmdb and /etc/nginx/geoip/GeoLite2-ASN.mmdb , avoiding the overhead of the download. Important If the feature is enabled but the files are missing, GeoIP2 will not be enabled. default: false","title":"use-geoip2"},{"location":"user-guide/nginx-configuration/configmap/#enable-brotli","text":"Enables or disables compression of HTTP responses using the \"brotli\" module . The default mime type list to compress is: application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component . default: is disabled Note: Brotli does not works in Safari < 11. For more information see https://caniuse.com/#feat=brotli","title":"enable-brotli"},{"location":"user-guide/nginx-configuration/configmap/#brotli-level","text":"Sets the Brotli Compression Level that will be used. default: 4","title":"brotli-level"},{"location":"user-guide/nginx-configuration/configmap/#brotli-min-length","text":"Minimum length of responses, in bytes, that will be eligible for brotli compression. default: 20","title":"brotli-min-length"},{"location":"user-guide/nginx-configuration/configmap/#brotli-types","text":"Sets the MIME Types that will be compressed on-the-fly by brotli. default: application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component","title":"brotli-types"},{"location":"user-guide/nginx-configuration/configmap/#use-http2","text":"Enables or disables HTTP/2 support in secure connections.","title":"use-http2"},{"location":"user-guide/nginx-configuration/configmap/#gzip-level","text":"Sets the gzip Compression Level that will be used. default: 1","title":"gzip-level"},{"location":"user-guide/nginx-configuration/configmap/#gzip-min-length","text":"Minimum length of responses to be returned to the client before it is eligible for gzip compression, in bytes. default: 256","title":"gzip-min-length"},{"location":"user-guide/nginx-configuration/configmap/#gzip-types","text":"Sets the MIME types in addition to \"text/html\" to compress. The special value \"*\" matches any MIME type. Responses with the \"text/html\" type are always compressed if use-gzip is enabled. default: application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component .","title":"gzip-types"},{"location":"user-guide/nginx-configuration/configmap/#worker-processes","text":"Sets the number of worker processes . The default of \"auto\" means number of available CPU cores.","title":"worker-processes"},{"location":"user-guide/nginx-configuration/configmap/#worker-cpu-affinity","text":"Binds worker processes to the sets of CPUs. worker_cpu_affinity . By default worker processes are not bound to any specific CPUs. The value can be: \"\": empty string indicate no affinity is applied. cpumask: e.g. 0001 0010 0100 1000 to bind processes to specific cpus. auto: binding worker processes automatically to available CPUs.","title":"worker-cpu-affinity"},{"location":"user-guide/nginx-configuration/configmap/#worker-shutdown-timeout","text":"Sets a timeout for Nginx to wait for worker to gracefully shutdown . default: \"240s\"","title":"worker-shutdown-timeout"},{"location":"user-guide/nginx-configuration/configmap/#load-balance","text":"Sets the algorithm to use for load balancing. The value can either be: round_robin: to use the default round robin loadbalancer ewma: to use the Peak EWMA method for routing ( implementation ) The default is round_robin . To load balance using consistent hashing of IP or other variables, consider the nginx.ingress.kubernetes.io/upstream-hash-by annotation. To load balance using session cookies, consider the nginx.ingress.kubernetes.io/affinity annotation. References: https://nginx.org/en/docs/http/load_balancing.html","title":"load-balance"},{"location":"user-guide/nginx-configuration/configmap/#variables-hash-bucket-size","text":"Sets the bucket size for the variables hash table. References: https://nginx.org/en/docs/http/ngx_http_map_module.html#variables_hash_bucket_size","title":"variables-hash-bucket-size"},{"location":"user-guide/nginx-configuration/configmap/#variables-hash-max-size","text":"Sets the maximum size of the variables hash table. References: https://nginx.org/en/docs/http/ngx_http_map_module.html#variables_hash_max_size","title":"variables-hash-max-size"},{"location":"user-guide/nginx-configuration/configmap/#upstream-keepalive-connections","text":"Activates the cache for connections to upstream servers. The connections parameter sets the maximum number of idle keepalive connections to upstream servers that are preserved in the cache of each worker process. When this number is exceeded, the least recently used connections are closed. default: 320 References: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive","title":"upstream-keepalive-connections"},{"location":"user-guide/nginx-configuration/configmap/#upstream-keepalive-time","text":"Sets the maximum time during which requests can be processed through one keepalive connection. default: \"1h\" References: http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_time","title":"upstream-keepalive-time"},{"location":"user-guide/nginx-configuration/configmap/#upstream-keepalive-timeout","text":"Sets a timeout during which an idle keepalive connection to an upstream server will stay open. default: 60 References: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_timeout","title":"upstream-keepalive-timeout"},{"location":"user-guide/nginx-configuration/configmap/#upstream-keepalive-requests","text":"Sets the maximum number of requests that can be served through one keepalive connection. After the maximum number of requests is made, the connection is closed. default: 10000 References: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_requests","title":"upstream-keepalive-requests"},{"location":"user-guide/nginx-configuration/configmap/#limit-conn-zone-variable","text":"Sets parameters for a shared memory zone that will keep states for various keys of limit_conn_zone . The default of \"$binary_remote_addr\" variable\u2019s size is always 4 bytes for IPv4 addresses or 16 bytes for IPv6 addresses.","title":"limit-conn-zone-variable"},{"location":"user-guide/nginx-configuration/configmap/#proxy-stream-timeout","text":"Sets the timeout between two successive read or write operations on client or proxied server connections. If no data is transmitted within this time, the connection is closed. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout","title":"proxy-stream-timeout"},{"location":"user-guide/nginx-configuration/configmap/#proxy-stream-next-upstream","text":"When a connection to the proxied server cannot be established, determines whether a client connection will be passed to the next server. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream","title":"proxy-stream-next-upstream"},{"location":"user-guide/nginx-configuration/configmap/#proxy-stream-next-upstream-timeout","text":"Limits the time allowed to pass a connection to the next server. The 0 value turns off this limitation. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_timeout","title":"proxy-stream-next-upstream-timeout"},{"location":"user-guide/nginx-configuration/configmap/#proxy-stream-next-upstream-tries","text":"Limits the number of possible tries a request should be passed to the next server. The 0 value turns off this limitation. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_tries","title":"proxy-stream-next-upstream-tries"},{"location":"user-guide/nginx-configuration/configmap/#proxy-stream-responses","text":"Sets the number of datagrams expected from the proxied server in response to the client request if the UDP protocol is used. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_responses","title":"proxy-stream-responses"},{"location":"user-guide/nginx-configuration/configmap/#bind-address","text":"Sets the addresses on which the server will accept requests instead of *. It should be noted that these addresses must exist in the runtime environment or the controller will crash loop.","title":"bind-address"},{"location":"user-guide/nginx-configuration/configmap/#use-forwarded-headers","text":"If true, NGINX passes the incoming X-Forwarded-* headers to upstreams. Use this option when NGINX is behind another L7 proxy / load balancer that is setting these headers. If false, NGINX ignores incoming X-Forwarded-* headers, filling them with the request information it sees. Use this option if NGINX is exposed directly to the internet, or it's behind a L3/packet-based load balancer that doesn't alter the source IP in the packets.","title":"use-forwarded-headers"},{"location":"user-guide/nginx-configuration/configmap/#enable-real-ip","text":"enable-real-ip enables the configuration of https://nginx.org/en/docs/http/ngx_http_realip_module.html . Specific attributes of the module can be configured further by using forwarded-for-header and proxy-real-ip-cidr settings.","title":"enable-real-ip"},{"location":"user-guide/nginx-configuration/configmap/#forwarded-for-header","text":"Sets the header field for identifying the originating IP address of a client. default: X-Forwarded-For","title":"forwarded-for-header"},{"location":"user-guide/nginx-configuration/configmap/#compute-full-forwarded-for","text":"Append the remote address to the X-Forwarded-For header instead of replacing it. When this option is enabled, the upstream application is responsible for extracting the client IP based on its own list of trusted proxies.","title":"compute-full-forwarded-for"},{"location":"user-guide/nginx-configuration/configmap/#proxy-add-original-uri-header","text":"Adds an X-Original-Uri header with the original request URI to the backend request","title":"proxy-add-original-uri-header"},{"location":"user-guide/nginx-configuration/configmap/#generate-request-id","text":"Ensures that X-Request-ID is defaulted to a random value, if no X-Request-ID is present in the request","title":"generate-request-id"},{"location":"user-guide/nginx-configuration/configmap/#enable-opentracing","text":"Enables the nginx Opentracing extension. default: is disabled References: https://github.com/opentracing-contrib/nginx-opentracing","title":"enable-opentracing"},{"location":"user-guide/nginx-configuration/configmap/#opentracing-operation-name","text":"Specifies a custom name for the server span. default: is empty For example, set to \"HTTP $request_method $uri\".","title":"opentracing-operation-name"},{"location":"user-guide/nginx-configuration/configmap/#opentracing-location-operation-name","text":"Specifies a custom name for the location span. default: is empty For example, set to \"HTTP $request_method $uri\".","title":"opentracing-location-operation-name"},{"location":"user-guide/nginx-configuration/configmap/#zipkin-collector-host","text":"Specifies the host to use when uploading traces. It must be a valid URL.","title":"zipkin-collector-host"},{"location":"user-guide/nginx-configuration/configmap/#zipkin-collector-port","text":"Specifies the port to use when uploading traces. default: 9411","title":"zipkin-collector-port"},{"location":"user-guide/nginx-configuration/configmap/#zipkin-service-name","text":"Specifies the service name to use for any traces created. default: nginx","title":"zipkin-service-name"},{"location":"user-guide/nginx-configuration/configmap/#zipkin-sample-rate","text":"Specifies sample rate for any traces created. default: 1.0","title":"zipkin-sample-rate"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-collector-host","text":"Specifies the host to use when uploading traces. It must be a valid URL.","title":"jaeger-collector-host"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-collector-port","text":"Specifies the port to use when uploading traces. default: 6831","title":"jaeger-collector-port"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-endpoint","text":"Specifies the endpoint to use when uploading traces to a collector. This takes priority over jaeger-collector-host if both are specified.","title":"jaeger-endpoint"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-service-name","text":"Specifies the service name to use for any traces created. default: nginx","title":"jaeger-service-name"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-propagation-format","text":"Specifies the traceparent/tracestate propagation format. default: jaeger","title":"jaeger-propagation-format"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-sampler-type","text":"Specifies the sampler to be used when sampling traces. The available samplers are: const, probabilistic, ratelimiting, remote. default: const","title":"jaeger-sampler-type"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-sampler-param","text":"Specifies the argument to be passed to the sampler constructor. Must be a number. For const this should be 0 to never sample and 1 to always sample. default: 1","title":"jaeger-sampler-param"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-sampler-host","text":"Specifies the custom remote sampler host to be passed to the sampler constructor. Must be a valid URL. Leave blank to use default value (localhost). default: http://127.0.0.1","title":"jaeger-sampler-host"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-sampler-port","text":"Specifies the custom remote sampler port to be passed to the sampler constructor. Must be a number. default: 5778","title":"jaeger-sampler-port"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-trace-context-header-name","text":"Specifies the header name used for passing trace context. default: uber-trace-id","title":"jaeger-trace-context-header-name"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-debug-header","text":"Specifies the header name used for force sampling. default: jaeger-debug-id","title":"jaeger-debug-header"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-baggage-header","text":"Specifies the header name used to submit baggage if there is no root span. default: jaeger-baggage","title":"jaeger-baggage-header"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-tracer-baggage-header-prefix","text":"Specifies the header prefix used to propagate baggage. default: uberctx-","title":"jaeger-tracer-baggage-header-prefix"},{"location":"user-guide/nginx-configuration/configmap/#datadog-collector-host","text":"Specifies the datadog agent host to use when uploading traces. It must be a valid URL.","title":"datadog-collector-host"},{"location":"user-guide/nginx-configuration/configmap/#datadog-collector-port","text":"Specifies the port to use when uploading traces. default: 8126","title":"datadog-collector-port"},{"location":"user-guide/nginx-configuration/configmap/#datadog-service-name","text":"Specifies the service name to use for any traces created. default: nginx","title":"datadog-service-name"},{"location":"user-guide/nginx-configuration/configmap/#datadog-environment","text":"Specifies the environment this trace belongs to. default: prod","title":"datadog-environment"},{"location":"user-guide/nginx-configuration/configmap/#datadog-operation-name-override","text":"Overrides the operation name to use for any traces crated. default: nginx.handle","title":"datadog-operation-name-override"},{"location":"user-guide/nginx-configuration/configmap/#datadog-priority-sampling","text":"Specifies to use client-side sampling. If true disables client-side sampling (thus ignoring sample_rate ) and enables distributed priority sampling, where traces are sampled based on a combination of user-assigned priorities and configuration from the agent. default: true","title":"datadog-priority-sampling"},{"location":"user-guide/nginx-configuration/configmap/#datadog-sample-rate","text":"Specifies sample rate for any traces created. This is effective only when datadog-priority-sampling is false default: 1.0","title":"datadog-sample-rate"},{"location":"user-guide/nginx-configuration/configmap/#main-snippet","text":"Adds custom configuration to the main section of the nginx configuration.","title":"main-snippet"},{"location":"user-guide/nginx-configuration/configmap/#http-snippet","text":"Adds custom configuration to the http section of the nginx configuration.","title":"http-snippet"},{"location":"user-guide/nginx-configuration/configmap/#server-snippet","text":"Adds custom configuration to all the servers in the nginx configuration.","title":"server-snippet"},{"location":"user-guide/nginx-configuration/configmap/#stream-snippet","text":"Adds custom configuration to the stream section of the nginx configuration.","title":"stream-snippet"},{"location":"user-guide/nginx-configuration/configmap/#location-snippet","text":"Adds custom configuration to all the locations in the nginx configuration. You can not use this to add new locations that proxy to the Kubernetes pods, as the snippet does not have access to the Go template functions. If you want to add custom locations you will have to provide your own nginx.tmpl .","title":"location-snippet"},{"location":"user-guide/nginx-configuration/configmap/#custom-http-errors","text":"Enables which HTTP codes should be passed for processing with the error_page directive Setting at least one code also enables proxy_intercept_errors which are required to process error_page. Example usage: custom-http-errors: 404,415","title":"custom-http-errors"},{"location":"user-guide/nginx-configuration/configmap/#proxy-body-size","text":"Sets the maximum allowed size of the client request body. See NGINX client_max_body_size .","title":"proxy-body-size"},{"location":"user-guide/nginx-configuration/configmap/#proxy-connect-timeout","text":"Sets the timeout for establishing a connection with a proxied server . It should be noted that this timeout cannot usually exceed 75 seconds.","title":"proxy-connect-timeout"},{"location":"user-guide/nginx-configuration/configmap/#proxy-read-timeout","text":"Sets the timeout in seconds for reading a response from the proxied server . The timeout is set only between two successive read operations, not for the transmission of the whole response.","title":"proxy-read-timeout"},{"location":"user-guide/nginx-configuration/configmap/#proxy-send-timeout","text":"Sets the timeout in seconds for transmitting a request to the proxied server . The timeout is set only between two successive write operations, not for the transmission of the whole request.","title":"proxy-send-timeout"},{"location":"user-guide/nginx-configuration/configmap/#proxy-buffers-number","text":"Sets the number of the buffer used for reading the first part of the response received from the proxied server. This part usually contains a small response header.","title":"proxy-buffers-number"},{"location":"user-guide/nginx-configuration/configmap/#proxy-buffer-size","text":"Sets the size of the buffer used for reading the first part of the response received from the proxied server. This part usually contains a small response header.","title":"proxy-buffer-size"},{"location":"user-guide/nginx-configuration/configmap/#proxy-cookie-path","text":"Sets a text that should be changed in the path attribute of the \u201cSet-Cookie\u201d header fields of a proxied server response.","title":"proxy-cookie-path"},{"location":"user-guide/nginx-configuration/configmap/#proxy-cookie-domain","text":"Sets a text that should be changed in the domain attribute of the \u201cSet-Cookie\u201d header fields of a proxied server response.","title":"proxy-cookie-domain"},{"location":"user-guide/nginx-configuration/configmap/#proxy-next-upstream","text":"Specifies in which cases a request should be passed to the next server.","title":"proxy-next-upstream"},{"location":"user-guide/nginx-configuration/configmap/#proxy-next-upstream-timeout","text":"Limits the time in seconds during which a request can be passed to the next server.","title":"proxy-next-upstream-timeout"},{"location":"user-guide/nginx-configuration/configmap/#proxy-next-upstream-tries","text":"Limit the number of possible tries a request should be passed to the next server.","title":"proxy-next-upstream-tries"},{"location":"user-guide/nginx-configuration/configmap/#proxy-redirect-from","text":"Sets the original text that should be changed in the \"Location\" and \"Refresh\" header fields of a proxied server response. default: off References: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_redirect","title":"proxy-redirect-from"},{"location":"user-guide/nginx-configuration/configmap/#proxy-request-buffering","text":"Enables or disables buffering of a client request body .","title":"proxy-request-buffering"},{"location":"user-guide/nginx-configuration/configmap/#ssl-redirect","text":"Sets the global value of redirects (301) to HTTPS if the server has a TLS certificate (defined in an Ingress rule). default: \"true\"","title":"ssl-redirect"},{"location":"user-guide/nginx-configuration/configmap/#force-ssl-redirect","text":"Sets the global value of redirects (308) to HTTPS if the server has a default TLS certificate (defined in extra-args). default: \"false\"","title":"force-ssl-redirect"},{"location":"user-guide/nginx-configuration/configmap/#whitelist-source-range","text":"Sets the default whitelisted IPs for each server block. This can be overwritten by an annotation on an Ingress rule. See ngx_http_access_module .","title":"whitelist-source-range"},{"location":"user-guide/nginx-configuration/configmap/#skip-access-log-urls","text":"Sets a list of URLs that should not appear in the NGINX access log. This is useful with urls like /health or health-check that make \"complex\" reading the logs. default: is empty","title":"skip-access-log-urls"},{"location":"user-guide/nginx-configuration/configmap/#limit-rate","text":"Limits the rate of response transmission to a client. The rate is specified in bytes per second. The zero value disables rate limiting. The limit is set per a request, and so if a client simultaneously opens two connections, the overall rate will be twice as much as the specified limit. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate","title":"limit-rate"},{"location":"user-guide/nginx-configuration/configmap/#limit-rate-after","text":"Sets the initial amount after which the further transmission of a response to a client will be rate limited.","title":"limit-rate-after"},{"location":"user-guide/nginx-configuration/configmap/#lua-shared-dicts","text":"Customize default Lua shared dictionaries or define more. You can use the following syntax to do so: lua-shared-dicts: \"<my dict name>: <my dict size>, [<my dict name>: <my dict size>], ...\" For example following will set default certificate_data dictionary to 100M and will introduce a new dictionary called my_custom_plugin : lua-shared-dicts: \"certificate_data: 100, my_custom_plugin: 5\" You can optionally set a size unit to allow for kilobyte-granularity. Allowed units are 'm' or 'k' (case-insensitive), and it defaults to MB if no unit is provided. Here is a similar example, but the my_custom_plugin dict is only 512KB. lua-shared-dicts: \"certificate_data: 100, my_custom_plugin: 512k\" References: https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate_after","title":"lua-shared-dicts"},{"location":"user-guide/nginx-configuration/configmap/#http-redirect-code","text":"Sets the HTTP status code to be used in redirects. Supported codes are 301 , 302 , 307 and 308 default: 308 Why the default code is 308? RFC 7238 was created to define the 308 (Permanent Redirect) status code that is similar to 301 (Moved Permanently) but it keeps the payload in the redirect. This is important if we send a redirect in methods like POST.","title":"http-redirect-code"},{"location":"user-guide/nginx-configuration/configmap/#proxy-buffering","text":"Enables or disables buffering of responses from the proxied server .","title":"proxy-buffering"},{"location":"user-guide/nginx-configuration/configmap/#limit-req-status-code","text":"Sets the status code to return in response to rejected requests . default: 503","title":"limit-req-status-code"},{"location":"user-guide/nginx-configuration/configmap/#limit-conn-status-code","text":"Sets the status code to return in response to rejected connections . default: 503","title":"limit-conn-status-code"},{"location":"user-guide/nginx-configuration/configmap/#enable-syslog","text":"Enable syslog feature for access log and error log. default: false","title":"enable-syslog"},{"location":"user-guide/nginx-configuration/configmap/#syslog-host","text":"Sets the address of syslog server. The address can be specified as a domain name or IP address.","title":"syslog-host"},{"location":"user-guide/nginx-configuration/configmap/#syslog-port","text":"Sets the port of syslog server. default: 514","title":"syslog-port"},{"location":"user-guide/nginx-configuration/configmap/#no-tls-redirect-locations","text":"A comma-separated list of locations on which http requests will never get redirected to their https counterpart. default: \"/.well-known/acme-challenge\"","title":"no-tls-redirect-locations"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-url","text":"A url to an existing service that provides authentication for all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-url . Locations that should not get authenticated can be listed using no-auth-locations See no-auth-locations . In addition, each service can be excluded from authentication via annotation enable-global-auth set to \"false\". default: \"\" References: https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/annotations.md#external-authentication","title":"global-auth-url"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-method","text":"A HTTP method to use for an existing service that provides authentication for all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-method . default: \"\"","title":"global-auth-method"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-signin","text":"Sets the location of the error page for an existing service that provides authentication for all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-signin . default: \"\"","title":"global-auth-signin"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-signin-redirect-param","text":"Sets the query parameter in the error page signin URL which contains the original URL of the request that failed authentication. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-signin-redirect-param . default: \"rd\"","title":"global-auth-signin-redirect-param"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-response-headers","text":"Sets the headers to pass to backend once authentication request completes. Applied to all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-response-headers . default: \"\"","title":"global-auth-response-headers"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-request-redirect","text":"Sets the X-Auth-Request-Redirect header value. Applied to all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-request-redirect . default: \"\"","title":"global-auth-request-redirect"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-snippet","text":"Sets a custom snippet to use with external authentication. Applied to all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-request-redirect . default: \"\"","title":"global-auth-snippet"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-cache-key","text":"Enables caching for global auth requests. Specify a lookup key for auth responses, e.g. $remote_user$http_authorization .","title":"global-auth-cache-key"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-cache-duration","text":"Set a caching time for auth responses based on their response codes, e.g. 200 202 30m . See proxy_cache_valid for details. You may specify multiple, comma-separated values: 200 202 10m, 401 5m . defaults to 200 202 401 5m .","title":"global-auth-cache-duration"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-always-set-cookie","text":"Always set a cookie returned by auth request. By default, the cookie will be set only if an upstream reports with the code 200, 201, 204, 206, 301, 302, 303, 304, 307, or 308. default: false","title":"global-auth-always-set-cookie"},{"location":"user-guide/nginx-configuration/configmap/#no-auth-locations","text":"A comma-separated list of locations that should not get authenticated. default: \"/.well-known/acme-challenge\"","title":"no-auth-locations"},{"location":"user-guide/nginx-configuration/configmap/#block-cidrs","text":"A comma-separated list of IP addresses (or subnets), request from which have to be blocked globally. References: https://nginx.org/en/docs/http/ngx_http_access_module.html#deny","title":"block-cidrs"},{"location":"user-guide/nginx-configuration/configmap/#block-user-agents","text":"A comma-separated list of User-Agent, request from which have to be blocked globally. It's possible to use here full strings and regular expressions. More details about valid patterns can be found at map Nginx directive documentation. References: https://nginx.org/en/docs/http/ngx_http_map_module.html#map","title":"block-user-agents"},{"location":"user-guide/nginx-configuration/configmap/#block-referers","text":"A comma-separated list of Referers, request from which have to be blocked globally. It's possible to use here full strings and regular expressions. More details about valid patterns can be found at map Nginx directive documentation. References: https://nginx.org/en/docs/http/ngx_http_map_module.html#map","title":"block-referers"},{"location":"user-guide/nginx-configuration/configmap/#proxy-ssl-location-only","text":"Set if proxy-ssl parameters should be applied only on locations and not on servers. default: is disabled","title":"proxy-ssl-location-only"},{"location":"user-guide/nginx-configuration/configmap/#default-type","text":"Sets the default MIME type of a response. default: text/html References: https://nginx.org/en/docs/http/ngx_http_core_module.html#default_type","title":"default-type"},{"location":"user-guide/nginx-configuration/configmap/#global-rate-limit","text":"global-rate-limit-status-code : configure HTTP status code to return when rejecting requests. Defaults to 429. Configure memcached client for Global Rate Limiting . global-rate-limit-memcached-host : IP/FQDN of memcached server to use. Required to enable Global Rate Limiting. global-rate-limit-memcached-port : port of memcached server to use. Defaults default memcached port of 11211 . global-rate-limit-memcached-connect-timeout : configure timeout for connect, send and receive operations. Unit is millisecond. Defaults to 50ms. global-rate-limit-memcached-max-idle-timeout : configure timeout for cleaning idle connections. Unit is millisecond. Defaults to 50ms. global-rate-limit-memcached-pool-size : configure number of max connections to keep alive. Make sure your memcached server can handle global-rate-limit-memcached-pool-size * worker-processes * <number of ingress-nginx replicas> simultaneous connections. These settings get used by lua-resty-global-throttle that ingress-nginx includes. Refer to the link to learn more about lua-resty-global-throttle .","title":"global-rate-limit"},{"location":"user-guide/nginx-configuration/configmap/#service-upstream","text":"Set if the service's Cluster IP and port should be used instead of a list of all endpoints. This can be overwritten by an annotation on an Ingress rule. default: \"false\"","title":"service-upstream"},{"location":"user-guide/nginx-configuration/configmap/#ssl-reject-handshake","text":"Set to reject SSL handshake to an unknown virtualhost. This parameter helps to mitigate the fingerprinting using default certificate of ingress. default: \"false\" References: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_reject_handshake","title":"ssl-reject-handshake"},{"location":"user-guide/nginx-configuration/configmap/#debug-connections","text":"Enables debugging log for selected client connections. default: \"\" References: http://nginx.org/en/docs/ngx_core_module.html#debug_connection","title":"debug-connections"},{"location":"user-guide/nginx-configuration/custom-template/","text":"Custom NGINX template \u00b6 The NGINX template is located in the file /etc/nginx/template/nginx.tmpl . Using a Volume it is possible to use a custom template. This includes using a Configmap as source of the template volumeMounts : - mountPath : /etc/nginx/template name : nginx-template-volume readOnly : true volumes : - name : nginx-template-volume configMap : name : nginx-template items : - key : nginx.tmpl path : nginx.tmpl Please note the template is tied to the Go code. Do not change names in the variable $cfg . For more information about the template syntax please check the Go template package . In addition to the built-in functions provided by the Go package the following functions are also available: empty: returns true if the specified parameter (string) is empty contains: strings.Contains hasPrefix: strings.HasPrefix hasSuffix: strings.HasSuffix toUpper: strings.ToUpper toLower: strings.ToLower split: strings.Split quote: wraps a string in double quotes buildLocation: helps to build the NGINX Location section in each server buildProxyPass: builds the reverse proxy configuration buildRateLimit: helps to build a limit zone inside a location if contains a rate limit annotation TODO: buildAuthLocation: buildAuthResponseHeaders: buildResolvers: buildDenyVariable: buildUpstreamName: buildForwardedFor: buildAuthSignURL: buildNextUpstream: filterRateLimits: formatIP: getenv: getIngressInformation: serverConfig: isLocationAllowed: isValidClientBodyBufferSize:","title":"Custom NGINX template"},{"location":"user-guide/nginx-configuration/custom-template/#custom-nginx-template","text":"The NGINX template is located in the file /etc/nginx/template/nginx.tmpl . Using a Volume it is possible to use a custom template. This includes using a Configmap as source of the template volumeMounts : - mountPath : /etc/nginx/template name : nginx-template-volume readOnly : true volumes : - name : nginx-template-volume configMap : name : nginx-template items : - key : nginx.tmpl path : nginx.tmpl Please note the template is tied to the Go code. Do not change names in the variable $cfg . For more information about the template syntax please check the Go template package . In addition to the built-in functions provided by the Go package the following functions are also available: empty: returns true if the specified parameter (string) is empty contains: strings.Contains hasPrefix: strings.HasPrefix hasSuffix: strings.HasSuffix toUpper: strings.ToUpper toLower: strings.ToLower split: strings.Split quote: wraps a string in double quotes buildLocation: helps to build the NGINX Location section in each server buildProxyPass: builds the reverse proxy configuration buildRateLimit: helps to build a limit zone inside a location if contains a rate limit annotation TODO: buildAuthLocation: buildAuthResponseHeaders: buildResolvers: buildDenyVariable: buildUpstreamName: buildForwardedFor: buildAuthSignURL: buildNextUpstream: filterRateLimits: formatIP: getenv: getIngressInformation: serverConfig: isLocationAllowed: isValidClientBodyBufferSize:","title":"Custom NGINX template"},{"location":"user-guide/nginx-configuration/log-format/","text":"Log format \u00b6 The default configuration uses a custom logging format to add additional information about upstreams, response time and status. log_format upstreaminfo '$remote_addr - $remote_user [$time_local] \"$request\" ' '$status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" ' '$request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr ' '$upstream_response_length $upstream_response_time $upstream_status $req_id'; Placeholder Description $proxy_protocol_addr remote address if proxy protocol is enabled $remote_addr the source IP address of the client $remote_user user name supplied with the Basic authentication $time_local local time in the Common Log Format $request full original request line $status response status $body_bytes_sent number of bytes sent to a client, not counting the response header $http_referer value of the Referer header $http_user_agent value of User-Agent header $request_length request length (including request line, header, and request body) $request_time time elapsed since the first bytes were read from the client $proxy_upstream_name name of the upstream. The format is upstream-<namespace>-<service name>-<service port> $proxy_alternative_upstream_name name of the alternative upstream. The format is upstream-<namespace>-<service name>-<service port> $upstream_addr the IP address and port (or the path to the domain socket) of the upstream server. If several servers were contacted during request processing, their addresses are separated by commas. $upstream_response_length the length of the response obtained from the upstream server $upstream_response_time time spent on receiving the response from the upstream server as seconds with millisecond resolution $upstream_status status code of the response obtained from the upstream server $req_id value of the X-Request-ID HTTP header. If the header is not set, a randomly generated ID. Additional available variables: Placeholder Description $namespace namespace of the ingress $ingress_name name of the ingress $service_name name of the service $service_port port of the service Sources: Upstream variables Embedded variables","title":"Log format"},{"location":"user-guide/nginx-configuration/log-format/#log-format","text":"The default configuration uses a custom logging format to add additional information about upstreams, response time and status. log_format upstreaminfo '$remote_addr - $remote_user [$time_local] \"$request\" ' '$status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" ' '$request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr ' '$upstream_response_length $upstream_response_time $upstream_status $req_id'; Placeholder Description $proxy_protocol_addr remote address if proxy protocol is enabled $remote_addr the source IP address of the client $remote_user user name supplied with the Basic authentication $time_local local time in the Common Log Format $request full original request line $status response status $body_bytes_sent number of bytes sent to a client, not counting the response header $http_referer value of the Referer header $http_user_agent value of User-Agent header $request_length request length (including request line, header, and request body) $request_time time elapsed since the first bytes were read from the client $proxy_upstream_name name of the upstream. The format is upstream-<namespace>-<service name>-<service port> $proxy_alternative_upstream_name name of the alternative upstream. The format is upstream-<namespace>-<service name>-<service port> $upstream_addr the IP address and port (or the path to the domain socket) of the upstream server. If several servers were contacted during request processing, their addresses are separated by commas. $upstream_response_length the length of the response obtained from the upstream server $upstream_response_time time spent on receiving the response from the upstream server as seconds with millisecond resolution $upstream_status status code of the response obtained from the upstream server $req_id value of the X-Request-ID HTTP header. If the header is not set, a randomly generated ID. Additional available variables: Placeholder Description $namespace namespace of the ingress $ingress_name name of the ingress $service_name name of the service $service_port port of the service Sources: Upstream variables Embedded variables","title":"Log format"},{"location":"user-guide/third-party-addons/modsecurity/","text":"ModSecurity Web Application Firewall \u00b6 ModSecurity is an open source, cross platform web application firewall (WAF) engine for Apache, IIS and Nginx that is developed by Trustwave's SpiderLabs. It has a robust event-based programming language which provides protection from a range of attacks against web applications and allows for HTTP traffic monitoring, logging and real-time analysis - https://www.modsecurity.org The ModSecurity-nginx connector is the connection point between NGINX and libmodsecurity (ModSecurity v3). The default ModSecurity configuration file is located in /etc/nginx/modsecurity/modsecurity.conf . This is the only file located in this directory and contains the default recommended configuration. Using a volume we can replace this file with the desired configuration. To enable the ModSecurity feature we need to specify enable-modsecurity: \"true\" in the configuration configmap. Note: the default configuration use detection only, because that minimizes the chances of post-installation disruption. Due to the value of the setting SecAuditLogType=Concurrent the ModSecurity log is stored in multiple files inside the directory /var/log/audit . The default Serial value in SecAuditLogType can impact performance. The OWASP ModSecurity Core Rule Set (CRS) is a set of generic attack detection rules for use with ModSecurity or compatible web application firewalls. The CRS aims to protect web applications from a wide range of attacks, including the OWASP Top Ten, with a minimum of false alerts. The directory /etc/nginx/owasp-modsecurity-crs contains the OWASP ModSecurity Core Rule Set repository . Using enable-owasp-modsecurity-crs: \"true\" we enable the use of the rules.","title":"ModSecurity Web Application Firewall"},{"location":"user-guide/third-party-addons/modsecurity/#modsecurity-web-application-firewall","text":"ModSecurity is an open source, cross platform web application firewall (WAF) engine for Apache, IIS and Nginx that is developed by Trustwave's SpiderLabs. It has a robust event-based programming language which provides protection from a range of attacks against web applications and allows for HTTP traffic monitoring, logging and real-time analysis - https://www.modsecurity.org The ModSecurity-nginx connector is the connection point between NGINX and libmodsecurity (ModSecurity v3). The default ModSecurity configuration file is located in /etc/nginx/modsecurity/modsecurity.conf . This is the only file located in this directory and contains the default recommended configuration. Using a volume we can replace this file with the desired configuration. To enable the ModSecurity feature we need to specify enable-modsecurity: \"true\" in the configuration configmap. Note: the default configuration use detection only, because that minimizes the chances of post-installation disruption. Due to the value of the setting SecAuditLogType=Concurrent the ModSecurity log is stored in multiple files inside the directory /var/log/audit . The default Serial value in SecAuditLogType can impact performance. The OWASP ModSecurity Core Rule Set (CRS) is a set of generic attack detection rules for use with ModSecurity or compatible web application firewalls. The CRS aims to protect web applications from a wide range of attacks, including the OWASP Top Ten, with a minimum of false alerts. The directory /etc/nginx/owasp-modsecurity-crs contains the OWASP ModSecurity Core Rule Set repository . Using enable-owasp-modsecurity-crs: \"true\" we enable the use of the rules.","title":"ModSecurity Web Application Firewall"},{"location":"user-guide/third-party-addons/opentracing/","text":"OpenTracing \u00b6 Enables requests served by NGINX for distributed tracing via The OpenTracing Project. Using the third party module opentracing-contrib/nginx-opentracing the NGINX ingress controller can configure NGINX to enable OpenTracing instrumentation. By default this feature is disabled. Usage \u00b6 To enable the instrumentation we must enable OpenTracing in the configuration ConfigMap: data: enable-opentracing: \"true\" To enable or disable instrumentation for a single Ingress, use the enable-opentracing annotation: kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/enable-opentracing: \"true\" We must also set the host to use when uploading traces: zipkin-collector-host: zipkin.default.svc.cluster.local jaeger-collector-host: jaeger-agent.default.svc.cluster.local datadog-collector-host: datadog-agent.default.svc.cluster.local NOTE: While the option is called jaeger-collector-host , you will need to point this to a jaeger-agent , and not the jaeger-collector component. Alternatively, you can set jaeger-endpoint and specify the full endpoint for uploading traces. This will use TCP and should be used for a collector rather than an agent. Next you will need to deploy a distributed tracing system which uses OpenTracing. Zipkin and Jaeger and Datadog have been tested. Other optional configuration options: # specifies the name to use for the server span opentracing-operation-name # specifies specifies the name to use for the location span opentracing-location-operation-name # sets whether or not to trust incoming tracing spans opentracing-trust-incoming-span # specifies the port to use when uploading traces, Default: 9411 zipkin-collector-port # specifies the service name to use for any traces created, Default: nginx zipkin-service-name # specifies sample rate for any traces created, Default: 1.0 zipkin-sample-rate # specifies the port to use when uploading traces, Default: 6831 jaeger-collector-port # specifies the endpoint to use when uploading traces to a collector instead of an agent jaeger-endpoint # specifies the service name to use for any traces created, Default: nginx jaeger-service-name # specifies the traceparent/tracestate propagation format jaeger-propagation-format # specifies the sampler to be used when sampling traces. # The available samplers are: const, probabilistic, ratelimiting, remote, Default: const jaeger-sampler-type # specifies the argument to be passed to the sampler constructor, Default: 1 jaeger-sampler-param # Specifies the custom remote sampler host to be passed to the sampler constructor. Must be a valid URL. # Default: http://127.0.0.1 jaeger-sampler-host # Specifies the custom remote sampler port to be passed to the sampler constructor. Must be a number. Default: 5778 jaeger-sampler-port # Specifies the header name used for passing trace context. Must be a string. Default: uber-trace-id jaeger-trace-context-header-name # Specifies the header name used for force sampling. Must be a string. Default: jaeger-debug-id jaeger-debug-header # Specifies the header name used to submit baggage if there is no root span. Must be a string. Default: jaeger-baggage jaeger-baggage-header # Specifies the header prefix used to propagate baggage. Must be a string. Default: uberctx- jaeger-tracer-baggage-header-prefix # specifies the port to use when uploading traces, Default 8126 datadog-collector-port # specifies the service name to use for any traces created, Default: nginx datadog-service-name # specifies the environment this trace belongs to, Default: prod datadog-environment # specifies the operation name to use for any traces collected, Default: nginx.handle datadog-operation-name-override # Specifies to use client-side sampling for distributed priority sampling and ignore sample rate, Default: true datadog-priority-sampling # specifies sample rate for any traces created, Default: 1.0 datadog-sample-rate All these options (including host) allow environment variables, such as $HOSTNAME or $HOST_IP . In the case of Jaeger, if you have a Jaeger agent running on each machine in your cluster, you can use something like $HOST_IP (which can be 'mounted' with the status.hostIP fieldpath, as described here ) to make sure traces will be sent to the local agent. Note that you can also set whether to trust incoming spans (global default is true) per-location using annotations like the following: kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/opentracing-trust-incoming-span: \"true\" Examples \u00b6 The following examples show how to deploy and test different distributed tracing systems. These example can be performed using Minikube. Zipkin \u00b6 In the rnburn/zipkin-date-server GitHub repository is an example of a dockerized date service. To install the example and Zipkin collector run: kubectl create -f https://raw.githubusercontent.com/rnburn/zipkin-date-server/master/kubernetes/zipkin.yaml kubectl create -f https://raw.githubusercontent.com/rnburn/zipkin-date-server/master/kubernetes/deployment.yaml Also we need to configure the Ingress-NGINX controller ConfigMap with the required values: $ echo ' apiVersion: v1 kind: ConfigMap data: enable-opentracing: \"true\" zipkin-collector-host: zipkin.default.svc.cluster.local metadata: name: ingress-nginx-controller namespace: kube-system ' | kubectl replace -f - In the Zipkin interface we can see the details: Jaeger \u00b6 Enable Ingress addon in Minikube: $ minikube addons enable ingress Add Minikube IP to /etc/hosts: $ echo \"$(minikube ip) example.com\" | sudo tee -a /etc/hosts Apply a basic Service and Ingress Resource: # Create Echoheaders Deployment $ kubectl run echoheaders --image=registry.k8s.io/echoserver:1.4 --replicas=1 --port=8080 # Expose as a Cluster-IP $ kubectl expose deployment echoheaders --port=80 --target-port=8080 --name=echoheaders-x # Apply the Ingress Resource $ echo ' apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: echo-ingress spec: ingressClassName: nginx rules: - host: example.com http: paths: - path: /echo pathType: Prefix backend: service: name: echoheaders-x port: number: 80 ' | kubectl apply -f - Enable OpenTracing and set the jaeger-collector-host: $ echo ' apiVersion: v1 kind: ConfigMap data: enable-opentracing: \"true\" jaeger-collector-host: jaeger-agent.default.svc.cluster.local metadata: name: ingress-nginx-controller namespace: kube-system ' | kubectl replace -f - Apply the Jaeger All-In-One Template: $ kubectl apply -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml Make a few requests to the Service: $ curl example.com/echo -d \"meow\" CLIENT VALUES: client_address=172.17.0.5 command=POST real path=/echo query=nil request_version=1.1 request_uri=http://example.com:8080/echo SERVER VALUES: server_version=nginx: 1.10.0 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=close content-length=4 content-type=application/x-www-form-urlencoded host=example.com user-agent=curl/7.54.0 x-forwarded-for=192.168.99.1 x-forwarded-host=example.com x-forwarded-port=80 x-forwarded-proto=http x-original-uri=/echo x-real-ip=192.168.99.1 x-scheme=http BODY: meow View the Jaeger UI: $ minikube service jaeger-query --url http://192.168.99.100:30183 In the Jaeger interface we can see the details:","title":"OpenTracing"},{"location":"user-guide/third-party-addons/opentracing/#opentracing","text":"Enables requests served by NGINX for distributed tracing via The OpenTracing Project. Using the third party module opentracing-contrib/nginx-opentracing the NGINX ingress controller can configure NGINX to enable OpenTracing instrumentation. By default this feature is disabled.","title":"OpenTracing"},{"location":"user-guide/third-party-addons/opentracing/#usage","text":"To enable the instrumentation we must enable OpenTracing in the configuration ConfigMap: data: enable-opentracing: \"true\" To enable or disable instrumentation for a single Ingress, use the enable-opentracing annotation: kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/enable-opentracing: \"true\" We must also set the host to use when uploading traces: zipkin-collector-host: zipkin.default.svc.cluster.local jaeger-collector-host: jaeger-agent.default.svc.cluster.local datadog-collector-host: datadog-agent.default.svc.cluster.local NOTE: While the option is called jaeger-collector-host , you will need to point this to a jaeger-agent , and not the jaeger-collector component. Alternatively, you can set jaeger-endpoint and specify the full endpoint for uploading traces. This will use TCP and should be used for a collector rather than an agent. Next you will need to deploy a distributed tracing system which uses OpenTracing. Zipkin and Jaeger and Datadog have been tested. Other optional configuration options: # specifies the name to use for the server span opentracing-operation-name # specifies specifies the name to use for the location span opentracing-location-operation-name # sets whether or not to trust incoming tracing spans opentracing-trust-incoming-span # specifies the port to use when uploading traces, Default: 9411 zipkin-collector-port # specifies the service name to use for any traces created, Default: nginx zipkin-service-name # specifies sample rate for any traces created, Default: 1.0 zipkin-sample-rate # specifies the port to use when uploading traces, Default: 6831 jaeger-collector-port # specifies the endpoint to use when uploading traces to a collector instead of an agent jaeger-endpoint # specifies the service name to use for any traces created, Default: nginx jaeger-service-name # specifies the traceparent/tracestate propagation format jaeger-propagation-format # specifies the sampler to be used when sampling traces. # The available samplers are: const, probabilistic, ratelimiting, remote, Default: const jaeger-sampler-type # specifies the argument to be passed to the sampler constructor, Default: 1 jaeger-sampler-param # Specifies the custom remote sampler host to be passed to the sampler constructor. Must be a valid URL. # Default: http://127.0.0.1 jaeger-sampler-host # Specifies the custom remote sampler port to be passed to the sampler constructor. Must be a number. Default: 5778 jaeger-sampler-port # Specifies the header name used for passing trace context. Must be a string. Default: uber-trace-id jaeger-trace-context-header-name # Specifies the header name used for force sampling. Must be a string. Default: jaeger-debug-id jaeger-debug-header # Specifies the header name used to submit baggage if there is no root span. Must be a string. Default: jaeger-baggage jaeger-baggage-header # Specifies the header prefix used to propagate baggage. Must be a string. Default: uberctx- jaeger-tracer-baggage-header-prefix # specifies the port to use when uploading traces, Default 8126 datadog-collector-port # specifies the service name to use for any traces created, Default: nginx datadog-service-name # specifies the environment this trace belongs to, Default: prod datadog-environment # specifies the operation name to use for any traces collected, Default: nginx.handle datadog-operation-name-override # Specifies to use client-side sampling for distributed priority sampling and ignore sample rate, Default: true datadog-priority-sampling # specifies sample rate for any traces created, Default: 1.0 datadog-sample-rate All these options (including host) allow environment variables, such as $HOSTNAME or $HOST_IP . In the case of Jaeger, if you have a Jaeger agent running on each machine in your cluster, you can use something like $HOST_IP (which can be 'mounted' with the status.hostIP fieldpath, as described here ) to make sure traces will be sent to the local agent. Note that you can also set whether to trust incoming spans (global default is true) per-location using annotations like the following: kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/opentracing-trust-incoming-span: \"true\"","title":"Usage"},{"location":"user-guide/third-party-addons/opentracing/#examples","text":"The following examples show how to deploy and test different distributed tracing systems. These example can be performed using Minikube.","title":"Examples"},{"location":"user-guide/third-party-addons/opentracing/#zipkin","text":"In the rnburn/zipkin-date-server GitHub repository is an example of a dockerized date service. To install the example and Zipkin collector run: kubectl create -f https://raw.githubusercontent.com/rnburn/zipkin-date-server/master/kubernetes/zipkin.yaml kubectl create -f https://raw.githubusercontent.com/rnburn/zipkin-date-server/master/kubernetes/deployment.yaml Also we need to configure the Ingress-NGINX controller ConfigMap with the required values: $ echo ' apiVersion: v1 kind: ConfigMap data: enable-opentracing: \"true\" zipkin-collector-host: zipkin.default.svc.cluster.local metadata: name: ingress-nginx-controller namespace: kube-system ' | kubectl replace -f - In the Zipkin interface we can see the details:","title":"Zipkin"},{"location":"user-guide/third-party-addons/opentracing/#jaeger","text":"Enable Ingress addon in Minikube: $ minikube addons enable ingress Add Minikube IP to /etc/hosts: $ echo \"$(minikube ip) example.com\" | sudo tee -a /etc/hosts Apply a basic Service and Ingress Resource: # Create Echoheaders Deployment $ kubectl run echoheaders --image=registry.k8s.io/echoserver:1.4 --replicas=1 --port=8080 # Expose as a Cluster-IP $ kubectl expose deployment echoheaders --port=80 --target-port=8080 --name=echoheaders-x # Apply the Ingress Resource $ echo ' apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: echo-ingress spec: ingressClassName: nginx rules: - host: example.com http: paths: - path: /echo pathType: Prefix backend: service: name: echoheaders-x port: number: 80 ' | kubectl apply -f - Enable OpenTracing and set the jaeger-collector-host: $ echo ' apiVersion: v1 kind: ConfigMap data: enable-opentracing: \"true\" jaeger-collector-host: jaeger-agent.default.svc.cluster.local metadata: name: ingress-nginx-controller namespace: kube-system ' | kubectl replace -f - Apply the Jaeger All-In-One Template: $ kubectl apply -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml Make a few requests to the Service: $ curl example.com/echo -d \"meow\" CLIENT VALUES: client_address=172.17.0.5 command=POST real path=/echo query=nil request_version=1.1 request_uri=http://example.com:8080/echo SERVER VALUES: server_version=nginx: 1.10.0 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=close content-length=4 content-type=application/x-www-form-urlencoded host=example.com user-agent=curl/7.54.0 x-forwarded-for=192.168.99.1 x-forwarded-host=example.com x-forwarded-port=80 x-forwarded-proto=http x-original-uri=/echo x-real-ip=192.168.99.1 x-scheme=http BODY: meow View the Jaeger UI: $ minikube service jaeger-query --url http://192.168.99.100:30183 In the Jaeger interface we can see the details:","title":"Jaeger"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Overview \u00b6 This is the documentation for the Ingress NGINX Controller. It is built around the Kubernetes Ingress resource , using a ConfigMap to store the controller configuration. You can learn more about using Ingress in the official Kubernetes documentation . Getting Started \u00b6 See Deployment for a whirlwind tour that will get you started. FAQ - Migration to apiVersion networking.k8s.io/v1 \u00b6 If you are using Ingress objects in your cluster (running Kubernetes older than v1.22), and you plan to upgrade to Kubernetes v1.22, this section is relevant to you. Please read this official blog on deprecated Ingress API versions Please read this official documentation on the IngressClass object What is an IngressClass and why is it important for users of Ingress-NGINX controller now ? \u00b6 IngressClass is a Kubernetes resource. See the description below. It's important because until now, a default install of the Ingress-NGINX controller did not require any IngressClass object. From version 1.0.0 of the Ingress-NGINX Controller, an IngressClass object is required. On clusters with more than one instance of the Ingress-NGINX controller, all instances of the controllers must be aware of which Ingress objects they serve. The ingressClassName field of an Ingress is the way to let the controller know about that. kubectl explain ingressclass KIND: IngressClass VERSION: networking.k8s.io/v1 DESCRIPTION: IngressClass represents the class of the Ingress, referenced by the Ingress Spec. The `ingressclass.kubernetes.io/is-default-class` annotation can be used to indicate that an IngressClass should be considered default. When a single IngressClass resource has this annotation set to true, new Ingress resources without a class specified will be assigned this default class. FIELDS: apiVersion <string> APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources kind <string> Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds metadata <Object> Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata spec <Object> Spec is the desired state of the IngressClass. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status` What has caused this change in behavior? \u00b6 There are 2 reasons primarily. Reason #1 \u00b6 Until K8s version 1.21, it was possible to create an Ingress resource using deprecated versions of the Ingress API, such as: extensions/v1beta1 networking.k8s.io/v1beta1 You would get a message about deprecation, but the Ingress resource would get created. From K8s version 1.22 onwards, you can only access the Ingress API via the stable, networking.k8s.io/v1 API. The reason is explained in the official blog on deprecated ingress API versions . Reason #2 \u00b6 If you are already using the Ingress-NGINX controller and then upgrade to K8s version v1.22 , there are several scenarios where your existing Ingress objects will not work how you expect. Read this FAQ to check which scenario matches your use case. What is ingressClassName field ? \u00b6 ingressClassName is a field in the specs of an Ingress object. kubectl explain ingress.spec.ingressClassName KIND: Ingress VERSION: networking.k8s.io/v1 FIELD: ingressClassName <string> DESCRIPTION: IngressClassName is the name of the IngressClass cluster resource. The associated IngressClass defines which controller will implement the resource. This replaces the deprecated `kubernetes.io/ingress.class` annotation. For backwards compatibility, when that annotation is set, it must be given precedence over this field. The controller may emit a warning if the field and annotation have different values. Implementations of this API should ignore Ingresses without a class specified. An IngressClass resource may be marked as default, which can be used to set a default value for this field. For more information, refer to the IngressClass documentation. The .spec.ingressClassName behavior has precedence over the deprecated kubernetes.io/ingress.class annotation. I have only one ingress controller in my cluster. What should I do? \u00b6 If a single instance of the Ingress-NGINX controller is the sole Ingress controller running in your cluster, you should add the annotation \"ingressclass.kubernetes.io/is-default-class\" in your IngressClass, so any new Ingress objects will have this one as default IngressClass. When using Helm, you can enable this annotation by setting .controller.ingressClassResource.default: true in your Helm chart installation's values file. If you have any old Ingress objects remaining without an IngressClass set, you can do one or more of the following to make the Ingress-NGINX controller aware of the old objects: You can manually set the .spec.ingressClassName field in the manifest of your own Ingress resources. You can re-create them after setting the ingressclass.kubernetes.io/is-default-class annotation to true on the IngressClass Alternatively you can make the Ingress-NGINX controller watch Ingress objects without the ingressClassName field set by starting your Ingress-NGINX with the flag --watch-ingress-without-class=true . When using Helm, you can configure your Helm chart installation's values file with .controller.watchIngressWithoutClass: true You can configure your Helm chart installation's values file with .controller.watchIngressWithoutClass: true . We recommend that you create the IngressClass as shown below: --- apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: labels: app.kubernetes.io/component: controller name: nginx annotations: ingressclass.kubernetes.io/is-default-class: \"true\" spec: controller: k8s.io/ingress-nginx And add the value spec.ingressClassName=nginx in your Ingress objects. I have multiple ingress objects in my cluster. What should I do ? \u00b6 If you have lot of ingress objects without ingressClass configuration, you can run the ingress-controller with the flag --watch-ingress-without-class=true . What is the flag '--watch-ingress-without-class' ? \u00b6 Its a flag that is passed,as an argument, to the nginx-ingress-controller executable. In the configuration, it looks like this: ... ... args: - /nginx-ingress-controller - --watch-ingress-without-class=true - --publish-service=$(POD_NAMESPACE)/ingress-nginx-dev-v1-test-controller - --election-id=ingress-controller-leader - --controller-class=k8s.io/ingress-nginx - --configmap=$(POD_NAMESPACE)/ingress-nginx-dev-v1-test-controller - --validating-webhook=:8443 - --validating-webhook-certificate=/usr/local/certificates/cert - --validating-webhook-key=/usr/local/certificates/key ... ... I have more than one controller in my cluster and already use the annotation ? \u00b6 No problem. This should still keep working, but we highly recommend you to test! Even though kubernetes.io/ingress.class is deprecated, the Ingress-NGINX controller still understands that annotation. If you want to follow good practice, you should consider migrating to use IngressClass and .spec.ingressClassName . I have more than one controller running in my cluster, and I want to use the new API ? \u00b6 In this scenario, you need to create multiple IngressClasses (see example one). But be aware that IngressClass works in a very specific way: you will need to change the .spec.controller value in your IngressClass and configure the controller to expect the exact same value. Let's see some example, supposing that you have three IngressClasses: IngressClass ingress-nginx-one , with .spec.controller equal to example.com/ingress-nginx1 IngressClass ingress-nginx-two , with .spec.controller equal to example.com/ingress-nginx2 IngressClass ingress-nginx-three , with .spec.controller equal to example.com/ingress-nginx1 (for private use, you can also use a controller name that doesn't contain a / ; for example: ingress-nginx1 ) When deploying your ingress controllers, you will have to change the --controller-class field as follows: Ingress-Nginx A, configured to use controller class name example.com/ingress-nginx1 Ingress-Nginx B, configured to use controller class name example.com/ingress-nginx2 Then, when you create an Ingress object with its ingressClassName set to ingress-nginx-two , only controllers looking for the example.com/ingress-nginx2 controller class pay attention to the new object. Given that Ingress-Nginx B is set up that way, it will serve that object, whereas Ingress-Nginx A ignores the new Ingress. Bear in mind that, if you start Ingress-Nginx B with the command line argument --watch-ingress-without-class=true , then it will serve: Ingresses without any ingressClassName set Ingresses where the deprecated annotation ( kubernetes.io/ingress.class ) matches the value set in the command line argument --ingress-class Ingresses that refer to any IngressClass that has the same spec.controller as configured in --controller-class If you start Ingress-Nginx B with the command line argument --watch-ingress-without-class=true and you run Ingress-Nginx A with the command line argument --watch-ingress-without-class=false then this is a supported configuration. If you have two Ingress-NGINX controllers for the same cluster, both running with --watch-ingress-without-class=true then there is likely to be a conflict. I am seeing this error message in the logs of the Ingress-NGINX controller: \"ingress class annotation is not equal to the expected by Ingress Controller\". Why ? \u00b6 It is highly likely that you will also see the name of the ingress resource in the same error message. This error message has been observed on use the deprecated annotation ( kubernetes.io/ingress.class ) in a Ingress resource manifest. It is recommended to use the .spec.ingressClassName field of the Ingress resource, to specify the name of the IngressClass of the Ingress you are defining. How to easily install multiple instances of the ingress-NGINX controller in the same cluster ? \u00b6 Create a new namespace kubectl create namespace ingress-nginx-2 Use Helm to install the additional instance of the ingress controller Ensure you have Helm working (refer to the Helm documentation ) We have to assume that you have the helm repo for the ingress-NGINX controller already added to your Helm config. But, if you have not added the helm repo then you can do this to add the repo to your helm config; helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx Make sure you have updated the helm repo data; helm repo update Now, install an additional instance of the ingress-NGINX controller like this: helm install ingress-nginx-2 ingress-nginx/ingress-nginx \\ --namespace ingress-nginx-2 \\ --set controller.ingressClassResource.name=nginx-two \\ --set controller.ingressClass=nginx-two \\ --set controller.ingressClassResource.controllerValue=\"example.com/ingress-nginx-2\" \\ --set controller.ingressClassResource.enabled=true \\ --set controller.ingressClassByName=true If you need to install yet another instance, then repeat the procedure to create a new namespace, change the values such as names & namespaces (for example from \"-2\" to \"-3\"), or anything else that meets your needs. If you need to install all instances in the same namespace, then you need to specify a different election id , like this: helm install ingress-nginx-2 ingress-nginx/ingress-nginx \\ --namespace kube-system \\ --set controller.electionID=nginx-two-leader \\ --set controller.ingressClassResource.name=nginx-two \\ --set controller.ingressClass=nginx-two \\ --set controller.ingressClassResource.controllerValue=\"example.com/ingress-nginx-2\" \\ --set controller.ingressClassResource.enabled=true \\ --set controller.ingressClassByName=true Note, controller.ingressClassResource.name and controller.ingressClass have to be set with the value of the new class as the first is to create the IngressClass object and the other is to modify the deployment of the actuall ingress controller pod.","title":"Welcome"},{"location":"#overview","text":"This is the documentation for the Ingress NGINX Controller. It is built around the Kubernetes Ingress resource , using a ConfigMap to store the controller configuration. You can learn more about using Ingress in the official Kubernetes documentation .","title":"Overview"},{"location":"#getting-started","text":"See Deployment for a whirlwind tour that will get you started.","title":"Getting Started"},{"location":"#faq-migration-to-apiversion-networkingk8siov1","text":"If you are using Ingress objects in your cluster (running Kubernetes older than v1.22), and you plan to upgrade to Kubernetes v1.22, this section is relevant to you. Please read this official blog on deprecated Ingress API versions Please read this official documentation on the IngressClass object","title":"FAQ - Migration to apiVersion networking.k8s.io/v1"},{"location":"#what-is-an-ingressclass-and-why-is-it-important-for-users-of-ingress-nginx-controller-now","text":"IngressClass is a Kubernetes resource. See the description below. It's important because until now, a default install of the Ingress-NGINX controller did not require any IngressClass object. From version 1.0.0 of the Ingress-NGINX Controller, an IngressClass object is required. On clusters with more than one instance of the Ingress-NGINX controller, all instances of the controllers must be aware of which Ingress objects they serve. The ingressClassName field of an Ingress is the way to let the controller know about that. kubectl explain ingressclass KIND: IngressClass VERSION: networking.k8s.io/v1 DESCRIPTION: IngressClass represents the class of the Ingress, referenced by the Ingress Spec. The `ingressclass.kubernetes.io/is-default-class` annotation can be used to indicate that an IngressClass should be considered default. When a single IngressClass resource has this annotation set to true, new Ingress resources without a class specified will be assigned this default class. FIELDS: apiVersion <string> APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources kind <string> Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds metadata <Object> Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata spec <Object> Spec is the desired state of the IngressClass. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status`","title":"What is an IngressClass and why is it important for users of Ingress-NGINX controller now ?"},{"location":"#what-has-caused-this-change-in-behavior","text":"There are 2 reasons primarily.","title":"What has caused this change in behavior?"},{"location":"#reason-1","text":"Until K8s version 1.21, it was possible to create an Ingress resource using deprecated versions of the Ingress API, such as: extensions/v1beta1 networking.k8s.io/v1beta1 You would get a message about deprecation, but the Ingress resource would get created. From K8s version 1.22 onwards, you can only access the Ingress API via the stable, networking.k8s.io/v1 API. The reason is explained in the official blog on deprecated ingress API versions .","title":"Reason #1"},{"location":"#reason-2","text":"If you are already using the Ingress-NGINX controller and then upgrade to K8s version v1.22 , there are several scenarios where your existing Ingress objects will not work how you expect. Read this FAQ to check which scenario matches your use case.","title":"Reason #2"},{"location":"#what-is-ingressclassname-field","text":"ingressClassName is a field in the specs of an Ingress object. kubectl explain ingress.spec.ingressClassName KIND: Ingress VERSION: networking.k8s.io/v1 FIELD: ingressClassName <string> DESCRIPTION: IngressClassName is the name of the IngressClass cluster resource. The associated IngressClass defines which controller will implement the resource. This replaces the deprecated `kubernetes.io/ingress.class` annotation. For backwards compatibility, when that annotation is set, it must be given precedence over this field. The controller may emit a warning if the field and annotation have different values. Implementations of this API should ignore Ingresses without a class specified. An IngressClass resource may be marked as default, which can be used to set a default value for this field. For more information, refer to the IngressClass documentation. The .spec.ingressClassName behavior has precedence over the deprecated kubernetes.io/ingress.class annotation.","title":"What is ingressClassName field ?"},{"location":"#i-have-only-one-ingress-controller-in-my-cluster-what-should-i-do","text":"If a single instance of the Ingress-NGINX controller is the sole Ingress controller running in your cluster, you should add the annotation \"ingressclass.kubernetes.io/is-default-class\" in your IngressClass, so any new Ingress objects will have this one as default IngressClass. When using Helm, you can enable this annotation by setting .controller.ingressClassResource.default: true in your Helm chart installation's values file. If you have any old Ingress objects remaining without an IngressClass set, you can do one or more of the following to make the Ingress-NGINX controller aware of the old objects: You can manually set the .spec.ingressClassName field in the manifest of your own Ingress resources. You can re-create them after setting the ingressclass.kubernetes.io/is-default-class annotation to true on the IngressClass Alternatively you can make the Ingress-NGINX controller watch Ingress objects without the ingressClassName field set by starting your Ingress-NGINX with the flag --watch-ingress-without-class=true . When using Helm, you can configure your Helm chart installation's values file with .controller.watchIngressWithoutClass: true You can configure your Helm chart installation's values file with .controller.watchIngressWithoutClass: true . We recommend that you create the IngressClass as shown below: --- apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: labels: app.kubernetes.io/component: controller name: nginx annotations: ingressclass.kubernetes.io/is-default-class: \"true\" spec: controller: k8s.io/ingress-nginx And add the value spec.ingressClassName=nginx in your Ingress objects.","title":"I have only one ingress controller in my cluster. What should I do?"},{"location":"#i-have-multiple-ingress-objects-in-my-cluster-what-should-i-do","text":"If you have lot of ingress objects without ingressClass configuration, you can run the ingress-controller with the flag --watch-ingress-without-class=true .","title":"I have multiple ingress objects in my cluster. What should I do ?"},{"location":"#what-is-the-flag-watch-ingress-without-class","text":"Its a flag that is passed,as an argument, to the nginx-ingress-controller executable. In the configuration, it looks like this: ... ... args: - /nginx-ingress-controller - --watch-ingress-without-class=true - --publish-service=$(POD_NAMESPACE)/ingress-nginx-dev-v1-test-controller - --election-id=ingress-controller-leader - --controller-class=k8s.io/ingress-nginx - --configmap=$(POD_NAMESPACE)/ingress-nginx-dev-v1-test-controller - --validating-webhook=:8443 - --validating-webhook-certificate=/usr/local/certificates/cert - --validating-webhook-key=/usr/local/certificates/key ... ...","title":"What is the flag '--watch-ingress-without-class' ?"},{"location":"#i-have-more-than-one-controller-in-my-cluster-and-already-use-the-annotation","text":"No problem. This should still keep working, but we highly recommend you to test! Even though kubernetes.io/ingress.class is deprecated, the Ingress-NGINX controller still understands that annotation. If you want to follow good practice, you should consider migrating to use IngressClass and .spec.ingressClassName .","title":"I have more than one controller in my cluster and already use the annotation ?"},{"location":"#i-have-more-than-one-controller-running-in-my-cluster-and-i-want-to-use-the-new-api","text":"In this scenario, you need to create multiple IngressClasses (see example one). But be aware that IngressClass works in a very specific way: you will need to change the .spec.controller value in your IngressClass and configure the controller to expect the exact same value. Let's see some example, supposing that you have three IngressClasses: IngressClass ingress-nginx-one , with .spec.controller equal to example.com/ingress-nginx1 IngressClass ingress-nginx-two , with .spec.controller equal to example.com/ingress-nginx2 IngressClass ingress-nginx-three , with .spec.controller equal to example.com/ingress-nginx1 (for private use, you can also use a controller name that doesn't contain a / ; for example: ingress-nginx1 ) When deploying your ingress controllers, you will have to change the --controller-class field as follows: Ingress-Nginx A, configured to use controller class name example.com/ingress-nginx1 Ingress-Nginx B, configured to use controller class name example.com/ingress-nginx2 Then, when you create an Ingress object with its ingressClassName set to ingress-nginx-two , only controllers looking for the example.com/ingress-nginx2 controller class pay attention to the new object. Given that Ingress-Nginx B is set up that way, it will serve that object, whereas Ingress-Nginx A ignores the new Ingress. Bear in mind that, if you start Ingress-Nginx B with the command line argument --watch-ingress-without-class=true , then it will serve: Ingresses without any ingressClassName set Ingresses where the deprecated annotation ( kubernetes.io/ingress.class ) matches the value set in the command line argument --ingress-class Ingresses that refer to any IngressClass that has the same spec.controller as configured in --controller-class If you start Ingress-Nginx B with the command line argument --watch-ingress-without-class=true and you run Ingress-Nginx A with the command line argument --watch-ingress-without-class=false then this is a supported configuration. If you have two Ingress-NGINX controllers for the same cluster, both running with --watch-ingress-without-class=true then there is likely to be a conflict.","title":"I have more than one controller running in my cluster, and I want to use the new API ?"},{"location":"#i-am-seeing-this-error-message-in-the-logs-of-the-ingress-nginx-controller-ingress-class-annotation-is-not-equal-to-the-expected-by-ingress-controller-why","text":"It is highly likely that you will also see the name of the ingress resource in the same error message. This error message has been observed on use the deprecated annotation ( kubernetes.io/ingress.class ) in a Ingress resource manifest. It is recommended to use the .spec.ingressClassName field of the Ingress resource, to specify the name of the IngressClass of the Ingress you are defining.","title":"I am seeing this error message in the logs of the Ingress-NGINX controller: \"ingress class annotation is not equal to the expected by Ingress Controller\". Why ?"},{"location":"#how-to-easily-install-multiple-instances-of-the-ingress-nginx-controller-in-the-same-cluster","text":"Create a new namespace kubectl create namespace ingress-nginx-2 Use Helm to install the additional instance of the ingress controller Ensure you have Helm working (refer to the Helm documentation ) We have to assume that you have the helm repo for the ingress-NGINX controller already added to your Helm config. But, if you have not added the helm repo then you can do this to add the repo to your helm config; helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx Make sure you have updated the helm repo data; helm repo update Now, install an additional instance of the ingress-NGINX controller like this: helm install ingress-nginx-2 ingress-nginx/ingress-nginx \\ --namespace ingress-nginx-2 \\ --set controller.ingressClassResource.name=nginx-two \\ --set controller.ingressClass=nginx-two \\ --set controller.ingressClassResource.controllerValue=\"example.com/ingress-nginx-2\" \\ --set controller.ingressClassResource.enabled=true \\ --set controller.ingressClassByName=true If you need to install yet another instance, then repeat the procedure to create a new namespace, change the values such as names & namespaces (for example from \"-2\" to \"-3\"), or anything else that meets your needs. If you need to install all instances in the same namespace, then you need to specify a different election id , like this: helm install ingress-nginx-2 ingress-nginx/ingress-nginx \\ --namespace kube-system \\ --set controller.electionID=nginx-two-leader \\ --set controller.ingressClassResource.name=nginx-two \\ --set controller.ingressClass=nginx-two \\ --set controller.ingressClassResource.controllerValue=\"example.com/ingress-nginx-2\" \\ --set controller.ingressClassResource.enabled=true \\ --set controller.ingressClassByName=true Note, controller.ingressClassResource.name and controller.ingressClass have to be set with the value of the new class as the first is to create the IngressClass object and the other is to modify the deployment of the actuall ingress controller pod.","title":"How to easily install multiple instances of the ingress-NGINX controller in the same cluster ?"},{"location":"e2e-tests/","text":"e2e test suite for Ingress NGINX Controller \u00b6 Geoip2 \u00b6 should include geoip2 line in config when enabled and db file exists should only allow requests from specific countries [Security] global-auth-url \u00b6 should return status code 401 when request any protected service should return status code 200 when request whitelisted (via no-auth-locations) service and 401 when request protected service should return status code 200 when request whitelisted (via ingress annotation) service and 401 when request protected service should still return status code 200 after auth backend is deleted using cache should proxy_method method when global-auth-method is configured should add custom error page when global-auth-signin url is configured should add auth headers when global-auth-response-headers is configured should set request-redirect when global-auth-request-redirect is configured should set snippet when global external auth is configured user retains cookie by default user does not retain cookie if upstream returns error status code user with global-auth-always-set-cookie key in configmap retains cookie if upstream returns error status code [Security] Pod Security Policies \u00b6 should be running with a Pod Security Policy log-format-* \u00b6 should disable the log-format-escape-json by default should enable the log-format-escape-json should disable the log-format-escape-json log-format-escape-json enabled log-format-escape-json disabled server-tokens \u00b6 should not exists Server header in the response should exists Server header in the response when is enabled proxy-connect-timeout \u00b6 should set valid proxy timeouts using configmap values should not set invalid proxy timeouts using configmap values ssl-ciphers \u00b6 Add ssl ciphers use-proxy-protocol \u00b6 should respect port passed by the PROXY Protocol should respect proto passed by the PROXY Protocol server port should enable PROXY Protocol for HTTPS should enable PROXY Protocol for TCP plugins \u00b6 should exist a x-hello-world header configmap server-snippet \u00b6 should add value of server-snippet setting to all ingress config should add global server-snippet and drop annotations per admin config [Flag] disable-catch-all \u00b6 should ignore catch all Ingress with backend should ignore catch all Ingress with backend and rules should delete Ingress updated to catch-all should allow Ingress with rules enable-real-ip \u00b6 trusts X-Forwarded-For header only when setting is true should not trust X-Forwarded-For header when setting is false keep-alive keep-alive-requests \u00b6 should set keepalive_timeout should set keepalive_requests should set keepalive connection to upstream server should set keep alive connection timeout to upstream server should set the request count to upstream server through one keep alive connection enable-multi-accept \u00b6 should be enabled by default should be enabled when set to true should be disabled when set to false [Flag] watch namespace selector \u00b6 should ingore Ingress of namespace without label foo=bar and accept those of namespace with label foo=bar [Flag] disable-service-external-name \u00b6 should ignore services of external-name type Configure OpenTracing \u00b6 should not exists opentracing directive should exists opentracing directive when is enabled should include opentracing_trust_incoming_span off directive when disabled should not exists opentracing_operation_name directive when is empty should exists opentracing_operation_name directive when is configured should not exists opentracing_location_operation_name directive when is empty should exists opentracing_location_operation_name directive when is configured should enable opentracing using zipkin should enable opentracing using jaeger should enable opentracing using jaeger with sampler host should propagate the w3c header when configured with jaeger should enable opentracing using jaeger with an HTTP endpoint should enable opentracing using datadog Configmap change \u00b6 should reload after an update in the configuration Configmap - limit-rate \u00b6 Check limit-rate config access-log \u00b6 use the default configuration use the specified configuration use the specified configuration use the specified configuration use the specified configuration global-options \u00b6 should have worker_rlimit_nofile option should have worker_rlimit_nofile option and be independent on amount of worker processes [Flag] ingress-class \u00b6 should ignore Ingress with a different class annotation should ignore Ingress with different controller class should accept both Ingresses with default IngressClassName and IngressClass annotation should ignore Ingress without IngressClass configuration should delete Ingress when class is removed should serve Ingress when class is added should serve Ingress when class is updated between annotation and ingressClassName should ignore Ingress with no class and accept the correctly configured Ingresses should watch Ingress with no class and ignore ingress with a different class should watch Ingress that uses the class name even if spec is different should watch Ingress with correct annotation should ignore Ingress with only IngressClassName reuse-port \u00b6 reuse port should be enabled by default reuse port should be disabled reuse port should be enabled proxy-send-timeout \u00b6 should set valid proxy send timeouts using configmap values should not set invalid proxy send timeouts using configmap values [SSL] [Flag] default-ssl-certificate \u00b6 uses default ssl certificate for catch-all ingress uses default ssl certificate for host based ingress when configured certificate does not match host brotli \u00b6 condition [Security] Pod Security Policies with volumes \u00b6 should be running with a Pod Security Policy Dynamic $proxy_host \u00b6 should exist a proxy_host should exist a proxy_host using the upstream-vhost annotation value [Security] block-* \u00b6 should block CIDRs defined in the ConfigMap should block User-Agents defined in the ConfigMap should block Referers defined in the ConfigMap settings-global-rate-limit \u00b6 generates correct NGINX configuration Add no tls redirect locations \u00b6 Check no tls redirect locations config main-snippet \u00b6 should add value of main-snippet setting to nginx config [Lua] lua-shared-dicts \u00b6 configures lua shared dicts Bad annotation values \u00b6 [BAD_ANNOTATIONS] should drop an ingress if there is an invalid character in some annotation [BAD_ANNOTATIONS] should drop an ingress if there is a forbidden word in some annotation [BAD_ANNOTATIONS] should allow an ingress if there is a default blocklist config in place [BAD_ANNOTATIONS] should drop an ingress if there is a custom blocklist config in place and allow others to pass [SSL] TLS protocols, ciphers and headers) \u00b6 setting cipher suite enforcing TLS v1.0 setting max-age parameter setting includeSubDomains parameter setting preload parameter overriding what's set from the upstream should not use ports during the HTTP to HTTPS redirection should not use ports or X-Forwarded-Host during the HTTP to HTTPS redirection [Security] modsecurity-snippet \u00b6 should add value of modsecurity-snippet setting to nginx config OCSP \u00b6 should enable OCSP and contain stapling information in the connection configmap stream-snippet \u00b6 should add value of stream-snippet via config map to nginx config proxy-read-timeout \u00b6 should set valid proxy read timeouts using configmap values should not set invalid proxy read timeouts using configmap values proxy-next-upstream \u00b6 should build proxy next upstream using configmap values hash size \u00b6 should set server_names_hash_bucket_size should set server_names_hash_max_size should set proxy-headers-hash-bucket-size should set proxy-headers-hash-max-size should set variables-hash-bucket-size should set variables-hash-max-size should set vmap-hash-bucket-size [Security] no-auth-locations \u00b6 should return status code 401 when accessing '/' unauthentication should return status code 200 when accessing '/' authentication should return status code 200 when accessing '/noauth' unauthenticated [Flag] custom HTTP and HTTPS ports \u00b6 should set X-Forwarded-Port headers accordingly when listening on a non-default HTTP port should set X-Forwarded-Port header to 443 should set the X-Forwarded-Port header to 443 use-forwarded-headers \u00b6 should trust X-Forwarded headers when setting is true should not trust X-Forwarded headers when setting is false add-headers \u00b6 Add a custom header Add multiple custom headers [Load Balancer] round-robin \u00b6 should evenly distribute requests with round-robin (default algorithm) [Load Balancer] EWMA \u00b6 does not fail requests [Load Balancer] load-balance \u00b6 should apply the configmap load-balance setting [SSL] redirect to HTTPS \u00b6 should redirect from HTTP to HTTPS when secret is missing [SSL] secret update \u00b6 should not appear references to secret updates not used in ingress rules should return the fake SSL certificate if the secret is invalid [Service] Type ExternalName \u00b6 works with external name set to incomplete fqdn should return 200 for service type=ExternalName without a port defined should return 200 for service type=ExternalName with a port defined should return status 502 for service type=ExternalName with an invalid host should return 200 for service type=ExternalName using a port name should return 200 for service type=ExternalName using FQDN with trailing dot should update the external name after a service update should sync ingress on external name service addition/deletion [Service] Nil Service Backend \u00b6 should return 404 when backend service is nil [Service] backend status code 503 \u00b6 should return 503 when backend service does not exist should return 503 when all backend service endpoints are unavailable","title":"E2e tests"},{"location":"e2e-tests/#e2e-test-suite-for-ingress-nginx-controller","text":"","title":"e2e test suite for Ingress NGINX Controller"},{"location":"e2e-tests/#geoip2","text":"should include geoip2 line in config when enabled and db file exists should only allow requests from specific countries","title":"Geoip2"},{"location":"e2e-tests/#security-global-auth-url","text":"should return status code 401 when request any protected service should return status code 200 when request whitelisted (via no-auth-locations) service and 401 when request protected service should return status code 200 when request whitelisted (via ingress annotation) service and 401 when request protected service should still return status code 200 after auth backend is deleted using cache should proxy_method method when global-auth-method is configured should add custom error page when global-auth-signin url is configured should add auth headers when global-auth-response-headers is configured should set request-redirect when global-auth-request-redirect is configured should set snippet when global external auth is configured user retains cookie by default user does not retain cookie if upstream returns error status code user with global-auth-always-set-cookie key in configmap retains cookie if upstream returns error status code","title":"[Security] global-auth-url"},{"location":"e2e-tests/#security-pod-security-policies","text":"should be running with a Pod Security Policy","title":"[Security] Pod Security Policies"},{"location":"e2e-tests/#log-format-","text":"should disable the log-format-escape-json by default should enable the log-format-escape-json should disable the log-format-escape-json log-format-escape-json enabled log-format-escape-json disabled","title":"log-format-*"},{"location":"e2e-tests/#server-tokens","text":"should not exists Server header in the response should exists Server header in the response when is enabled","title":"server-tokens"},{"location":"e2e-tests/#proxy-connect-timeout","text":"should set valid proxy timeouts using configmap values should not set invalid proxy timeouts using configmap values","title":"proxy-connect-timeout"},{"location":"e2e-tests/#ssl-ciphers","text":"Add ssl ciphers","title":"ssl-ciphers"},{"location":"e2e-tests/#use-proxy-protocol","text":"should respect port passed by the PROXY Protocol should respect proto passed by the PROXY Protocol server port should enable PROXY Protocol for HTTPS should enable PROXY Protocol for TCP","title":"use-proxy-protocol"},{"location":"e2e-tests/#plugins","text":"should exist a x-hello-world header","title":"plugins"},{"location":"e2e-tests/#configmap-server-snippet","text":"should add value of server-snippet setting to all ingress config should add global server-snippet and drop annotations per admin config","title":"configmap server-snippet"},{"location":"e2e-tests/#flag-disable-catch-all","text":"should ignore catch all Ingress with backend should ignore catch all Ingress with backend and rules should delete Ingress updated to catch-all should allow Ingress with rules","title":"[Flag] disable-catch-all"},{"location":"e2e-tests/#enable-real-ip","text":"trusts X-Forwarded-For header only when setting is true should not trust X-Forwarded-For header when setting is false","title":"enable-real-ip"},{"location":"e2e-tests/#keep-alive-keep-alive-requests","text":"should set keepalive_timeout should set keepalive_requests should set keepalive connection to upstream server should set keep alive connection timeout to upstream server should set the request count to upstream server through one keep alive connection","title":"keep-alive keep-alive-requests"},{"location":"e2e-tests/#enable-multi-accept","text":"should be enabled by default should be enabled when set to true should be disabled when set to false","title":"enable-multi-accept"},{"location":"e2e-tests/#flag-watch-namespace-selector","text":"should ingore Ingress of namespace without label foo=bar and accept those of namespace with label foo=bar","title":"[Flag] watch namespace selector"},{"location":"e2e-tests/#flag-disable-service-external-name","text":"should ignore services of external-name type","title":"[Flag] disable-service-external-name"},{"location":"e2e-tests/#configure-opentracing","text":"should not exists opentracing directive should exists opentracing directive when is enabled should include opentracing_trust_incoming_span off directive when disabled should not exists opentracing_operation_name directive when is empty should exists opentracing_operation_name directive when is configured should not exists opentracing_location_operation_name directive when is empty should exists opentracing_location_operation_name directive when is configured should enable opentracing using zipkin should enable opentracing using jaeger should enable opentracing using jaeger with sampler host should propagate the w3c header when configured with jaeger should enable opentracing using jaeger with an HTTP endpoint should enable opentracing using datadog","title":"Configure OpenTracing"},{"location":"e2e-tests/#configmap-change","text":"should reload after an update in the configuration","title":"Configmap change"},{"location":"e2e-tests/#configmap-limit-rate","text":"Check limit-rate config","title":"Configmap - limit-rate"},{"location":"e2e-tests/#access-log","text":"use the default configuration use the specified configuration use the specified configuration use the specified configuration use the specified configuration","title":"access-log"},{"location":"e2e-tests/#global-options","text":"should have worker_rlimit_nofile option should have worker_rlimit_nofile option and be independent on amount of worker processes","title":"global-options"},{"location":"e2e-tests/#flag-ingress-class","text":"should ignore Ingress with a different class annotation should ignore Ingress with different controller class should accept both Ingresses with default IngressClassName and IngressClass annotation should ignore Ingress without IngressClass configuration should delete Ingress when class is removed should serve Ingress when class is added should serve Ingress when class is updated between annotation and ingressClassName should ignore Ingress with no class and accept the correctly configured Ingresses should watch Ingress with no class and ignore ingress with a different class should watch Ingress that uses the class name even if spec is different should watch Ingress with correct annotation should ignore Ingress with only IngressClassName","title":"[Flag] ingress-class"},{"location":"e2e-tests/#reuse-port","text":"reuse port should be enabled by default reuse port should be disabled reuse port should be enabled","title":"reuse-port"},{"location":"e2e-tests/#proxy-send-timeout","text":"should set valid proxy send timeouts using configmap values should not set invalid proxy send timeouts using configmap values","title":"proxy-send-timeout"},{"location":"e2e-tests/#ssl-flag-default-ssl-certificate","text":"uses default ssl certificate for catch-all ingress uses default ssl certificate for host based ingress when configured certificate does not match host","title":"[SSL] [Flag] default-ssl-certificate"},{"location":"e2e-tests/#brotli","text":"condition","title":"brotli"},{"location":"e2e-tests/#security-pod-security-policies-with-volumes","text":"should be running with a Pod Security Policy","title":"[Security] Pod Security Policies with volumes"},{"location":"e2e-tests/#dynamic-proxy_host","text":"should exist a proxy_host should exist a proxy_host using the upstream-vhost annotation value","title":"Dynamic $proxy_host"},{"location":"e2e-tests/#security-block-","text":"should block CIDRs defined in the ConfigMap should block User-Agents defined in the ConfigMap should block Referers defined in the ConfigMap","title":"[Security] block-*"},{"location":"e2e-tests/#settings-global-rate-limit","text":"generates correct NGINX configuration","title":"settings-global-rate-limit"},{"location":"e2e-tests/#add-no-tls-redirect-locations","text":"Check no tls redirect locations config","title":"Add no tls redirect locations"},{"location":"e2e-tests/#main-snippet","text":"should add value of main-snippet setting to nginx config","title":"main-snippet"},{"location":"e2e-tests/#lua-lua-shared-dicts","text":"configures lua shared dicts","title":"[Lua] lua-shared-dicts"},{"location":"e2e-tests/#bad-annotation-values","text":"[BAD_ANNOTATIONS] should drop an ingress if there is an invalid character in some annotation [BAD_ANNOTATIONS] should drop an ingress if there is a forbidden word in some annotation [BAD_ANNOTATIONS] should allow an ingress if there is a default blocklist config in place [BAD_ANNOTATIONS] should drop an ingress if there is a custom blocklist config in place and allow others to pass","title":"Bad annotation values"},{"location":"e2e-tests/#ssl-tls-protocols-ciphers-and-headers","text":"setting cipher suite enforcing TLS v1.0 setting max-age parameter setting includeSubDomains parameter setting preload parameter overriding what's set from the upstream should not use ports during the HTTP to HTTPS redirection should not use ports or X-Forwarded-Host during the HTTP to HTTPS redirection","title":"[SSL] TLS protocols, ciphers and headers)"},{"location":"e2e-tests/#security-modsecurity-snippet","text":"should add value of modsecurity-snippet setting to nginx config","title":"[Security] modsecurity-snippet"},{"location":"e2e-tests/#ocsp","text":"should enable OCSP and contain stapling information in the connection","title":"OCSP"},{"location":"e2e-tests/#configmap-stream-snippet","text":"should add value of stream-snippet via config map to nginx config","title":"configmap stream-snippet"},{"location":"e2e-tests/#proxy-read-timeout","text":"should set valid proxy read timeouts using configmap values should not set invalid proxy read timeouts using configmap values","title":"proxy-read-timeout"},{"location":"e2e-tests/#proxy-next-upstream","text":"should build proxy next upstream using configmap values","title":"proxy-next-upstream"},{"location":"e2e-tests/#hash-size","text":"should set server_names_hash_bucket_size should set server_names_hash_max_size should set proxy-headers-hash-bucket-size should set proxy-headers-hash-max-size should set variables-hash-bucket-size should set variables-hash-max-size should set vmap-hash-bucket-size","title":"hash size"},{"location":"e2e-tests/#security-no-auth-locations","text":"should return status code 401 when accessing '/' unauthentication should return status code 200 when accessing '/' authentication should return status code 200 when accessing '/noauth' unauthenticated","title":"[Security] no-auth-locations"},{"location":"e2e-tests/#flag-custom-http-and-https-ports","text":"should set X-Forwarded-Port headers accordingly when listening on a non-default HTTP port should set X-Forwarded-Port header to 443 should set the X-Forwarded-Port header to 443","title":"[Flag] custom HTTP and HTTPS ports"},{"location":"e2e-tests/#use-forwarded-headers","text":"should trust X-Forwarded headers when setting is true should not trust X-Forwarded headers when setting is false","title":"use-forwarded-headers"},{"location":"e2e-tests/#add-headers","text":"Add a custom header Add multiple custom headers","title":"add-headers"},{"location":"e2e-tests/#load-balancer-round-robin","text":"should evenly distribute requests with round-robin (default algorithm)","title":"[Load Balancer] round-robin"},{"location":"e2e-tests/#load-balancer-ewma","text":"does not fail requests","title":"[Load Balancer] EWMA"},{"location":"e2e-tests/#load-balancer-load-balance","text":"should apply the configmap load-balance setting","title":"[Load Balancer] load-balance"},{"location":"e2e-tests/#ssl-redirect-to-https","text":"should redirect from HTTP to HTTPS when secret is missing","title":"[SSL] redirect to HTTPS"},{"location":"e2e-tests/#ssl-secret-update","text":"should not appear references to secret updates not used in ingress rules should return the fake SSL certificate if the secret is invalid","title":"[SSL] secret update"},{"location":"e2e-tests/#service-type-externalname","text":"works with external name set to incomplete fqdn should return 200 for service type=ExternalName without a port defined should return 200 for service type=ExternalName with a port defined should return status 502 for service type=ExternalName with an invalid host should return 200 for service type=ExternalName using a port name should return 200 for service type=ExternalName using FQDN with trailing dot should update the external name after a service update should sync ingress on external name service addition/deletion","title":"[Service] Type ExternalName"},{"location":"e2e-tests/#service-nil-service-backend","text":"should return 404 when backend service is nil","title":"[Service] Nil Service Backend"},{"location":"e2e-tests/#service-backend-status-code-503","text":"should return 503 when backend service does not exist should return 503 when all backend service endpoints are unavailable","title":"[Service] backend status code 503"},{"location":"how-it-works/","text":"How it works \u00b6 The objective of this document is to explain how the Ingress-NGINX controller works, in particular how the NGINX model is built and why we need one. NGINX configuration \u00b6 The goal of this Ingress controller is the assembly of a configuration file (nginx.conf). The main implication of this requirement is the need to reload NGINX after any change in the configuration file. Though it is important to note that we don't reload Nginx on changes that impact only an upstream configuration (i.e Endpoints change when you deploy your app) . We use lua-nginx-module to achieve this. Check below to learn more about how it's done. NGINX model \u00b6 Usually, a Kubernetes Controller utilizes the synchronization loop pattern to check if the desired state in the controller is updated or a change is required. To this purpose, we need to build a model using different objects from the cluster, in particular (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster. To get this object from the cluster, we use Kubernetes Informers , in particular, FilteredSharedInformer . This informers allows reacting to changes in using callbacks to individual changes when a new object is added, modified or removed. Unfortunately, there is no way to know if a particular change is going to affect the final configuration file. Therefore on every change, we have to rebuild a new model from scratch based on the state of cluster and compare it to the current model. If the new model equals to the current one, then we avoid generating a new NGINX configuration and triggering a reload. Otherwise, we check if the difference is only about Endpoints. If so we then send the new list of Endpoints to a Lua handler running inside Nginx using HTTP POST request and again avoid generating a new NGINX configuration and triggering a reload. If the difference between running and new model is about more than just Endpoints we create a new NGINX configuration based on the new model, replace the current model and trigger a reload. One of the uses of the model is to avoid unnecessary reloads when there's no change in the state and to detect conflicts in definitions. The final representation of the NGINX configuration is generated from a Go template using the new model as input for the variables required by the template. Building the NGINX model \u00b6 Building a model is an expensive operation, for this reason, the use of the synchronization loop is a must. By using a work queue it is possible to not lose changes and remove the use of sync.Mutex to force a single execution of the sync loop and additionally it is possible to create a time window between the start and end of the sync loop that allows us to discard unnecessary updates. It is important to understand that any change in the cluster could generate events that the informer will send to the controller and one of the reasons for the work queue . Operations to build the model: Order Ingress rules by CreationTimestamp field, i.e., old rules first. If the same path for the same host is defined in more than one Ingress, the oldest rule wins. If more than one Ingress contains a TLS section for the same host, the oldest rule wins. If multiple Ingresses define an annotation that affects the configuration of the Server block, the oldest rule wins. Create a list of NGINX Servers (per hostname) Create a list of NGINX Upstreams If multiple Ingresses define different paths for the same host, the ingress controller will merge the definitions. Annotations are applied to all the paths in the Ingress. Multiple Ingresses can define different annotations. These definitions are not shared between Ingresses. When a reload is required \u00b6 The next list describes the scenarios when a reload is required: New Ingress Resource Created. TLS section is added to existing Ingress. Change in Ingress annotations that impacts more than just upstream configuration. For instance load-balance annotation does not require a reload. A path is added/removed from an Ingress. An Ingress, Service, Secret is removed. Some missing referenced object from the Ingress is available, like a Service or Secret. A Secret is updated. Avoiding reloads \u00b6 In some cases, it is possible to avoid reloads, in particular when there is a change in the endpoints, i.e., a pod is started or replaced. It is out of the scope of this Ingress controller to remove reloads completely. This would require an incredible amount of work and at some point makes no sense. This can change only if NGINX changes the way new configurations are read, basically, new changes do not replace worker processes. Avoiding reloads on Endpoints changes \u00b6 On every endpoint change the controller fetches endpoints from all the services it sees and generates corresponding Backend objects. It then sends these objects to a Lua handler running inside Nginx. The Lua code in turn stores those backends in a shared memory zone. Then for every request Lua code running in balancer_by_lua context detects what endpoints it should choose upstream peer from and applies the configured load balancing algorithm to choose the peer. Then Nginx takes care of the rest. This way we avoid reloading Nginx on endpoint changes. Note that this includes annotation changes that affects only upstream configuration in Nginx as well. In a relatively big cluster with frequently deploying apps this feature saves significant number of Nginx reloads which can otherwise affect response latency, load balancing quality (after every reload Nginx resets the state of load balancing) and so on. Avoiding outage from wrong configuration \u00b6 Because the ingress controller works using the synchronization loop pattern , it is applying the configuration for all matching objects. In case some Ingress objects have a broken configuration, for example a syntax error in the nginx.ingress.kubernetes.io/configuration-snippet annotation, the generated configuration becomes invalid, does not reload and hence no more ingresses will be taken into account. To prevent this situation to happen, the nginx ingress controller optionally exposes a validating admission webhook server to ensure the validity of incoming ingress objects. This webhook appends the incoming ingress objects to the list of ingresses, generates the configuration and calls nginx to ensure the configuration has no syntax errors.","title":"How it works"},{"location":"how-it-works/#how-it-works","text":"The objective of this document is to explain how the Ingress-NGINX controller works, in particular how the NGINX model is built and why we need one.","title":"How it works"},{"location":"how-it-works/#nginx-configuration","text":"The goal of this Ingress controller is the assembly of a configuration file (nginx.conf). The main implication of this requirement is the need to reload NGINX after any change in the configuration file. Though it is important to note that we don't reload Nginx on changes that impact only an upstream configuration (i.e Endpoints change when you deploy your app) . We use lua-nginx-module to achieve this. Check below to learn more about how it's done.","title":"NGINX configuration"},{"location":"how-it-works/#nginx-model","text":"Usually, a Kubernetes Controller utilizes the synchronization loop pattern to check if the desired state in the controller is updated or a change is required. To this purpose, we need to build a model using different objects from the cluster, in particular (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster. To get this object from the cluster, we use Kubernetes Informers , in particular, FilteredSharedInformer . This informers allows reacting to changes in using callbacks to individual changes when a new object is added, modified or removed. Unfortunately, there is no way to know if a particular change is going to affect the final configuration file. Therefore on every change, we have to rebuild a new model from scratch based on the state of cluster and compare it to the current model. If the new model equals to the current one, then we avoid generating a new NGINX configuration and triggering a reload. Otherwise, we check if the difference is only about Endpoints. If so we then send the new list of Endpoints to a Lua handler running inside Nginx using HTTP POST request and again avoid generating a new NGINX configuration and triggering a reload. If the difference between running and new model is about more than just Endpoints we create a new NGINX configuration based on the new model, replace the current model and trigger a reload. One of the uses of the model is to avoid unnecessary reloads when there's no change in the state and to detect conflicts in definitions. The final representation of the NGINX configuration is generated from a Go template using the new model as input for the variables required by the template.","title":"NGINX model"},{"location":"how-it-works/#building-the-nginx-model","text":"Building a model is an expensive operation, for this reason, the use of the synchronization loop is a must. By using a work queue it is possible to not lose changes and remove the use of sync.Mutex to force a single execution of the sync loop and additionally it is possible to create a time window between the start and end of the sync loop that allows us to discard unnecessary updates. It is important to understand that any change in the cluster could generate events that the informer will send to the controller and one of the reasons for the work queue . Operations to build the model: Order Ingress rules by CreationTimestamp field, i.e., old rules first. If the same path for the same host is defined in more than one Ingress, the oldest rule wins. If more than one Ingress contains a TLS section for the same host, the oldest rule wins. If multiple Ingresses define an annotation that affects the configuration of the Server block, the oldest rule wins. Create a list of NGINX Servers (per hostname) Create a list of NGINX Upstreams If multiple Ingresses define different paths for the same host, the ingress controller will merge the definitions. Annotations are applied to all the paths in the Ingress. Multiple Ingresses can define different annotations. These definitions are not shared between Ingresses.","title":"Building the NGINX model"},{"location":"how-it-works/#when-a-reload-is-required","text":"The next list describes the scenarios when a reload is required: New Ingress Resource Created. TLS section is added to existing Ingress. Change in Ingress annotations that impacts more than just upstream configuration. For instance load-balance annotation does not require a reload. A path is added/removed from an Ingress. An Ingress, Service, Secret is removed. Some missing referenced object from the Ingress is available, like a Service or Secret. A Secret is updated.","title":"When a reload is required"},{"location":"how-it-works/#avoiding-reloads","text":"In some cases, it is possible to avoid reloads, in particular when there is a change in the endpoints, i.e., a pod is started or replaced. It is out of the scope of this Ingress controller to remove reloads completely. This would require an incredible amount of work and at some point makes no sense. This can change only if NGINX changes the way new configurations are read, basically, new changes do not replace worker processes.","title":"Avoiding reloads"},{"location":"how-it-works/#avoiding-reloads-on-endpoints-changes","text":"On every endpoint change the controller fetches endpoints from all the services it sees and generates corresponding Backend objects. It then sends these objects to a Lua handler running inside Nginx. The Lua code in turn stores those backends in a shared memory zone. Then for every request Lua code running in balancer_by_lua context detects what endpoints it should choose upstream peer from and applies the configured load balancing algorithm to choose the peer. Then Nginx takes care of the rest. This way we avoid reloading Nginx on endpoint changes. Note that this includes annotation changes that affects only upstream configuration in Nginx as well. In a relatively big cluster with frequently deploying apps this feature saves significant number of Nginx reloads which can otherwise affect response latency, load balancing quality (after every reload Nginx resets the state of load balancing) and so on.","title":"Avoiding reloads on Endpoints changes"},{"location":"how-it-works/#avoiding-outage-from-wrong-configuration","text":"Because the ingress controller works using the synchronization loop pattern , it is applying the configuration for all matching objects. In case some Ingress objects have a broken configuration, for example a syntax error in the nginx.ingress.kubernetes.io/configuration-snippet annotation, the generated configuration becomes invalid, does not reload and hence no more ingresses will be taken into account. To prevent this situation to happen, the nginx ingress controller optionally exposes a validating admission webhook server to ensure the validity of incoming ingress objects. This webhook appends the incoming ingress objects to the list of ingresses, generates the configuration and calls nginx to ensure the configuration has no syntax errors.","title":"Avoiding outage from wrong configuration"},{"location":"kubectl-plugin/","text":"The ingress-nginx kubectl plugin \u00b6 Installation \u00b6 Install krew , then run kubectl krew install ingress-nginx to install the plugin. Then run kubectl ingress-nginx --help to make sure the plugin is properly installed and to get a list of commands: kubectl ingress-nginx --help A kubectl plugin for inspecting your ingress-nginx deployments Usage: ingress-nginx [command] Available Commands: backends Inspect the dynamic backend information of an ingress-nginx instance certs Output the certificate data stored in an ingress-nginx pod conf Inspect the generated nginx.conf exec Execute a command inside an ingress-nginx pod general Inspect the other dynamic ingress-nginx information help Help about any command info Show information about the ingress-nginx service ingresses Provide a short summary of all of the ingress definitions lint Inspect kubernetes resources for possible issues logs Get the kubernetes logs for an ingress-nginx pod ssh ssh into a running ingress-nginx pod Flags: --as string Username to impersonate for the operation --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. --cache-dir string Default HTTP cache directory (default \"/Users/alexkursell/.kube/http-cache\") --certificate-authority string Path to a cert file for the certificate authority --client-certificate string Path to a client certificate file for TLS --client-key string Path to a client key file for TLS --cluster string The name of the kubeconfig cluster to use --context string The name of the kubeconfig context to use -h, --help help for ingress-nginx --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure --kubeconfig string Path to the kubeconfig file to use for CLI requests. -n, --namespace string If present, the namespace scope for this CLI request --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\") -s, --server string The address and port of the Kubernetes API server --token string Bearer token for authentication to the API server --user string The name of the kubeconfig user to use Use \"ingress-nginx [command] --help\" for more information about a command. Common Flags \u00b6 Every subcommand supports the basic kubectl configuration flags like --namespace , --context , --client-key and so on. Subcommands that act on a particular ingress-nginx pod ( backends , certs , conf , exec , general , logs , ssh ), support the --deployment <deployment> and --pod <pod> flags to select either a pod from a deployment with the given name, or a pod with the given name. The --deployment flag defaults to ingress-nginx-controller . Subcommands that inspect resources ( ingresses , lint ) support the --all-namespaces flag, which causes them to inspect resources in every namespace. Subcommands \u00b6 Note that backends , general , certs , and conf require ingress-nginx version 0.23.0 or higher. backends \u00b6 Run kubectl ingress-nginx backends to get a JSON array of the backends that an ingress-nginx controller currently knows about: $ kubectl ingress-nginx backends -n ingress-nginx [ { \"name\": \"default-apple-service-5678\", \"service\": { \"metadata\": { \"creationTimestamp\": null }, \"spec\": { \"ports\": [ { \"protocol\": \"TCP\", \"port\": 5678, \"targetPort\": 5678 } ], \"selector\": { \"app\": \"apple\" }, \"clusterIP\": \"10.97.230.121\", \"type\": \"ClusterIP\", \"sessionAffinity\": \"None\" }, \"status\": { \"loadBalancer\": {} } }, \"port\": 0, \"sslPassthrough\": false, \"endpoints\": [ { \"address\": \"10.1.3.86\", \"port\": \"5678\" } ], \"sessionAffinityConfig\": { \"name\": \"\", \"cookieSessionAffinity\": { \"name\": \"\" } }, \"upstreamHashByConfig\": { \"upstream-hash-by-subset-size\": 3 }, \"noServer\": false, \"trafficShapingPolicy\": { \"weight\": 0, \"header\": \"\", \"headerValue\": \"\", \"cookie\": \"\" } }, { \"name\": \"default-echo-service-8080\", ... }, { \"name\": \"upstream-default-backend\", ... } ] Add the --list option to show only the backend names. Add the --backend <backend> option to show only the backend with the given name. certs \u00b6 Use kubectl ingress-nginx certs --host <hostname> to dump the SSL cert/key information for a given host. WARNING: This command will dump sensitive private key information. Don't blindly share the output, and certainly don't log it anywhere. $ kubectl ingress-nginx certs -n ingress-nginx --host testaddr.local -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN RSA PRIVATE KEY----- <REDACTED! DO NOT SHARE THIS!> -----END RSA PRIVATE KEY----- conf \u00b6 Use kubectl ingress-nginx conf to dump the generated nginx.conf file. Add the --host <hostname> option to view only the server block for that host: kubectl ingress-nginx conf -n ingress-nginx --host testaddr.local server { server_name testaddr.local ; listen 80; set $proxy_upstream_name \"-\"; set $pass_access_scheme $scheme; set $pass_server_port $server_port; set $best_http_host $http_host; set $pass_port $pass_server_port; location / { set $namespace \"\"; set $ingress_name \"\"; set $service_name \"\"; set $service_port \"0\"; set $location_path \"/\"; ... exec \u00b6 kubectl ingress-nginx exec is exactly the same as kubectl exec , with the same command flags. It will automatically choose an ingress-nginx pod to run the command in. $ kubectl ingress-nginx exec -i -n ingress-nginx -- ls /etc/nginx fastcgi_params geoip lua mime.types modsecurity modules nginx.conf opentracing.json owasp-modsecurity-crs template info \u00b6 Shows the internal and external IP/CNAMES for an ingress-nginx service. $ kubectl ingress-nginx info -n ingress-nginx Service cluster IP address: 10.187.253.31 LoadBalancer IP|CNAME: 35.123.123.123 Use the --service <service> flag if your ingress-nginx LoadBalancer service is not named ingress-nginx . ingresses \u00b6 kubectl ingress-nginx ingresses , alternately kubectl ingress-nginx ing , shows a more detailed view of the ingress definitions in a namespace. Compare: $ kubectl get ingresses --all-namespaces NAMESPACE NAME HOSTS ADDRESS PORTS AGE default example-ingress1 testaddr.local,testaddr2.local localhost 80 5d default test-ingress-2 * localhost 80 5d vs. $ kubectl ingress-nginx ingresses --all-namespaces NAMESPACE INGRESS NAME HOST+PATH ADDRESSES TLS SERVICE SERVICE PORT ENDPOINTS default example-ingress1 testaddr.local/etameta localhost NO pear-service 5678 5 default example-ingress1 testaddr2.local/otherpath localhost NO apple-service 5678 1 default example-ingress1 testaddr2.local/otherotherpath localhost NO pear-service 5678 5 default test-ingress-2 * localhost NO echo-service 8080 2 lint \u00b6 kubectl ingress-nginx lint can check a namespace or entire cluster for potential configuration issues. This command is especially useful when upgrading between ingress-nginx versions. $ kubectl ingress-nginx lint --all-namespaces --verbose Checking ingresses... \u2717 anamespace/this-nginx - Contains the removed session-cookie-hash annotation. Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3743 \u2717 othernamespace/ingress-definition-blah - The rewrite-target annotation value does not reference a capture group Lint added for version 0.22.0 https://github.com/kubernetes/ingress-nginx/issues/3174 Checking deployments... \u2717 namespace2/ingress-nginx-controller - Uses removed config flag --sort-backends Lint added for version 0.22.0 https://github.com/kubernetes/ingress-nginx/issues/3655 - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808 To show the lints added only for a particular ingress-nginx release, use the --from-version and --to-version flags: $ kubectl ingress-nginx lint --all-namespaces --verbose --from-version 0 .24.0 --to-version 0 .24.0 Checking ingresses... \u2717 anamespace/this-nginx - Contains the removed session-cookie-hash annotation. Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3743 Checking deployments... \u2717 namespace2/ingress-nginx-controller - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808 logs \u00b6 kubectl ingress-nginx logs is almost the same as kubectl logs , with fewer flags. It will automatically choose an ingress-nginx pod to read logs from. $ kubectl ingress-nginx logs -n ingress-nginx ------------------------------------------------------------------------------- NGINX Ingress controller Release: dev Build: git-48dc3a867 Repository: git@github.com:kubernetes/ingress-nginx.git ------------------------------------------------------------------------------- W0405 16:53:46.061589 7 flags.go:214] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false) nginx version: nginx/1.15.9 W0405 16:53:46.070093 7 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0405 16:53:46.070499 7 main.go:205] Creating API client for https://10.96.0.1:443 I0405 16:53:46.077784 7 main.go:249] Running in Kubernetes cluster version v1.10 (v1.10.11) - git (clean) commit 637c7e288581ee40ab4ca210618a89a555b6e7e9 - platform linux/amd64 I0405 16:53:46.183359 7 nginx.go:265] Starting NGINX Ingress controller I0405 16:53:46.193913 7 event.go:209] Event(v1.ObjectReference{Kind:\"ConfigMap\", Namespace:\"ingress-nginx\", Name:\"udp-services\", UID:\"82258915-563e-11e9-9c52-025000000001\", APIVersion:\"v1\", ResourceVersion:\"494\", FieldPath:\"\"}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services ... ssh \u00b6 kubectl ingress-nginx ssh is exactly the same as kubectl ingress-nginx exec -it -- /bin/bash . Use it when you want to quickly be dropped into a shell inside a running ingress-nginx container. $ kubectl ingress-nginx ssh -n ingress-nginx www-data@ingress-nginx-controller-7cbf77c976-wx5pn:/etc/nginx$","title":"kubectl plugin"},{"location":"kubectl-plugin/#the-ingress-nginx-kubectl-plugin","text":"","title":"The ingress-nginx kubectl plugin"},{"location":"kubectl-plugin/#installation","text":"Install krew , then run kubectl krew install ingress-nginx to install the plugin. Then run kubectl ingress-nginx --help to make sure the plugin is properly installed and to get a list of commands: kubectl ingress-nginx --help A kubectl plugin for inspecting your ingress-nginx deployments Usage: ingress-nginx [command] Available Commands: backends Inspect the dynamic backend information of an ingress-nginx instance certs Output the certificate data stored in an ingress-nginx pod conf Inspect the generated nginx.conf exec Execute a command inside an ingress-nginx pod general Inspect the other dynamic ingress-nginx information help Help about any command info Show information about the ingress-nginx service ingresses Provide a short summary of all of the ingress definitions lint Inspect kubernetes resources for possible issues logs Get the kubernetes logs for an ingress-nginx pod ssh ssh into a running ingress-nginx pod Flags: --as string Username to impersonate for the operation --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. --cache-dir string Default HTTP cache directory (default \"/Users/alexkursell/.kube/http-cache\") --certificate-authority string Path to a cert file for the certificate authority --client-certificate string Path to a client certificate file for TLS --client-key string Path to a client key file for TLS --cluster string The name of the kubeconfig cluster to use --context string The name of the kubeconfig context to use -h, --help help for ingress-nginx --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure --kubeconfig string Path to the kubeconfig file to use for CLI requests. -n, --namespace string If present, the namespace scope for this CLI request --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\") -s, --server string The address and port of the Kubernetes API server --token string Bearer token for authentication to the API server --user string The name of the kubeconfig user to use Use \"ingress-nginx [command] --help\" for more information about a command.","title":"Installation"},{"location":"kubectl-plugin/#common-flags","text":"Every subcommand supports the basic kubectl configuration flags like --namespace , --context , --client-key and so on. Subcommands that act on a particular ingress-nginx pod ( backends , certs , conf , exec , general , logs , ssh ), support the --deployment <deployment> and --pod <pod> flags to select either a pod from a deployment with the given name, or a pod with the given name. The --deployment flag defaults to ingress-nginx-controller . Subcommands that inspect resources ( ingresses , lint ) support the --all-namespaces flag, which causes them to inspect resources in every namespace.","title":"Common Flags"},{"location":"kubectl-plugin/#subcommands","text":"Note that backends , general , certs , and conf require ingress-nginx version 0.23.0 or higher.","title":"Subcommands"},{"location":"kubectl-plugin/#backends","text":"Run kubectl ingress-nginx backends to get a JSON array of the backends that an ingress-nginx controller currently knows about: $ kubectl ingress-nginx backends -n ingress-nginx [ { \"name\": \"default-apple-service-5678\", \"service\": { \"metadata\": { \"creationTimestamp\": null }, \"spec\": { \"ports\": [ { \"protocol\": \"TCP\", \"port\": 5678, \"targetPort\": 5678 } ], \"selector\": { \"app\": \"apple\" }, \"clusterIP\": \"10.97.230.121\", \"type\": \"ClusterIP\", \"sessionAffinity\": \"None\" }, \"status\": { \"loadBalancer\": {} } }, \"port\": 0, \"sslPassthrough\": false, \"endpoints\": [ { \"address\": \"10.1.3.86\", \"port\": \"5678\" } ], \"sessionAffinityConfig\": { \"name\": \"\", \"cookieSessionAffinity\": { \"name\": \"\" } }, \"upstreamHashByConfig\": { \"upstream-hash-by-subset-size\": 3 }, \"noServer\": false, \"trafficShapingPolicy\": { \"weight\": 0, \"header\": \"\", \"headerValue\": \"\", \"cookie\": \"\" } }, { \"name\": \"default-echo-service-8080\", ... }, { \"name\": \"upstream-default-backend\", ... } ] Add the --list option to show only the backend names. Add the --backend <backend> option to show only the backend with the given name.","title":"backends"},{"location":"kubectl-plugin/#certs","text":"Use kubectl ingress-nginx certs --host <hostname> to dump the SSL cert/key information for a given host. WARNING: This command will dump sensitive private key information. Don't blindly share the output, and certainly don't log it anywhere. $ kubectl ingress-nginx certs -n ingress-nginx --host testaddr.local -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN RSA PRIVATE KEY----- <REDACTED! DO NOT SHARE THIS!> -----END RSA PRIVATE KEY-----","title":"certs"},{"location":"kubectl-plugin/#conf","text":"Use kubectl ingress-nginx conf to dump the generated nginx.conf file. Add the --host <hostname> option to view only the server block for that host: kubectl ingress-nginx conf -n ingress-nginx --host testaddr.local server { server_name testaddr.local ; listen 80; set $proxy_upstream_name \"-\"; set $pass_access_scheme $scheme; set $pass_server_port $server_port; set $best_http_host $http_host; set $pass_port $pass_server_port; location / { set $namespace \"\"; set $ingress_name \"\"; set $service_name \"\"; set $service_port \"0\"; set $location_path \"/\"; ...","title":"conf"},{"location":"kubectl-plugin/#exec","text":"kubectl ingress-nginx exec is exactly the same as kubectl exec , with the same command flags. It will automatically choose an ingress-nginx pod to run the command in. $ kubectl ingress-nginx exec -i -n ingress-nginx -- ls /etc/nginx fastcgi_params geoip lua mime.types modsecurity modules nginx.conf opentracing.json owasp-modsecurity-crs template","title":"exec"},{"location":"kubectl-plugin/#info","text":"Shows the internal and external IP/CNAMES for an ingress-nginx service. $ kubectl ingress-nginx info -n ingress-nginx Service cluster IP address: 10.187.253.31 LoadBalancer IP|CNAME: 35.123.123.123 Use the --service <service> flag if your ingress-nginx LoadBalancer service is not named ingress-nginx .","title":"info"},{"location":"kubectl-plugin/#ingresses","text":"kubectl ingress-nginx ingresses , alternately kubectl ingress-nginx ing , shows a more detailed view of the ingress definitions in a namespace. Compare: $ kubectl get ingresses --all-namespaces NAMESPACE NAME HOSTS ADDRESS PORTS AGE default example-ingress1 testaddr.local,testaddr2.local localhost 80 5d default test-ingress-2 * localhost 80 5d vs. $ kubectl ingress-nginx ingresses --all-namespaces NAMESPACE INGRESS NAME HOST+PATH ADDRESSES TLS SERVICE SERVICE PORT ENDPOINTS default example-ingress1 testaddr.local/etameta localhost NO pear-service 5678 5 default example-ingress1 testaddr2.local/otherpath localhost NO apple-service 5678 1 default example-ingress1 testaddr2.local/otherotherpath localhost NO pear-service 5678 5 default test-ingress-2 * localhost NO echo-service 8080 2","title":"ingresses"},{"location":"kubectl-plugin/#lint","text":"kubectl ingress-nginx lint can check a namespace or entire cluster for potential configuration issues. This command is especially useful when upgrading between ingress-nginx versions. $ kubectl ingress-nginx lint --all-namespaces --verbose Checking ingresses... \u2717 anamespace/this-nginx - Contains the removed session-cookie-hash annotation. Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3743 \u2717 othernamespace/ingress-definition-blah - The rewrite-target annotation value does not reference a capture group Lint added for version 0.22.0 https://github.com/kubernetes/ingress-nginx/issues/3174 Checking deployments... \u2717 namespace2/ingress-nginx-controller - Uses removed config flag --sort-backends Lint added for version 0.22.0 https://github.com/kubernetes/ingress-nginx/issues/3655 - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808 To show the lints added only for a particular ingress-nginx release, use the --from-version and --to-version flags: $ kubectl ingress-nginx lint --all-namespaces --verbose --from-version 0 .24.0 --to-version 0 .24.0 Checking ingresses... \u2717 anamespace/this-nginx - Contains the removed session-cookie-hash annotation. Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3743 Checking deployments... \u2717 namespace2/ingress-nginx-controller - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808","title":"lint"},{"location":"kubectl-plugin/#logs","text":"kubectl ingress-nginx logs is almost the same as kubectl logs , with fewer flags. It will automatically choose an ingress-nginx pod to read logs from. $ kubectl ingress-nginx logs -n ingress-nginx ------------------------------------------------------------------------------- NGINX Ingress controller Release: dev Build: git-48dc3a867 Repository: git@github.com:kubernetes/ingress-nginx.git ------------------------------------------------------------------------------- W0405 16:53:46.061589 7 flags.go:214] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false) nginx version: nginx/1.15.9 W0405 16:53:46.070093 7 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0405 16:53:46.070499 7 main.go:205] Creating API client for https://10.96.0.1:443 I0405 16:53:46.077784 7 main.go:249] Running in Kubernetes cluster version v1.10 (v1.10.11) - git (clean) commit 637c7e288581ee40ab4ca210618a89a555b6e7e9 - platform linux/amd64 I0405 16:53:46.183359 7 nginx.go:265] Starting NGINX Ingress controller I0405 16:53:46.193913 7 event.go:209] Event(v1.ObjectReference{Kind:\"ConfigMap\", Namespace:\"ingress-nginx\", Name:\"udp-services\", UID:\"82258915-563e-11e9-9c52-025000000001\", APIVersion:\"v1\", ResourceVersion:\"494\", FieldPath:\"\"}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services ...","title":"logs"},{"location":"kubectl-plugin/#ssh","text":"kubectl ingress-nginx ssh is exactly the same as kubectl ingress-nginx exec -it -- /bin/bash . Use it when you want to quickly be dropped into a shell inside a running ingress-nginx container. $ kubectl ingress-nginx ssh -n ingress-nginx www-data@ingress-nginx-controller-7cbf77c976-wx5pn:/etc/nginx$","title":"ssh"},{"location":"troubleshooting/","text":"Troubleshooting \u00b6 Ingress-Controller Logs and Events \u00b6 There are many ways to troubleshoot the ingress-controller. The following are basic troubleshooting methods to obtain more information. Check the Ingress Resource Events \u00b6 $ kubectl get ing -n <namespace-of-ingress-resource> NAME HOSTS ADDRESS PORTS AGE cafe-ingress cafe.com 10.0.2.15 80 25s $ kubectl describe ing <ingress-resource-name> -n <namespace-of-ingress-resource> Name: cafe-ingress Namespace: default Address: 10.0.2.15 Default backend: default-http-backend:80 (172.17.0.5:8080) Rules: Host Path Backends ---- ---- -------- cafe.com /tea tea-svc:80 (<none>) /coffee coffee-svc:80 (<none>) Annotations: kubectl.kubernetes.io/last-applied-configuration: {\"apiVersion\":\"networking.k8s.io/v1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{},\"name\":\"cafe-ingress\",\"namespace\":\"default\",\"selfLink\":\"/apis/networking/v1/namespaces/default/ingresses/cafe-ingress\"},\"spec\":{\"rules\":[{\"host\":\"cafe.com\",\"http\":{\"paths\":[{\"backend\":{\"serviceName\":\"tea-svc\",\"servicePort\":80},\"path\":\"/tea\"},{\"backend\":{\"serviceName\":\"coffee-svc\",\"servicePort\":80},\"path\":\"/coffee\"}]}}]},\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"169.48.142.110\"}]}}} Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal CREATE 1m ingress-nginx-controller Ingress default/cafe-ingress Normal UPDATE 58s ingress-nginx-controller Ingress default/cafe-ingress Check the Ingress Controller Logs \u00b6 $ kubectl get pods -n <namespace-of-ingress-controller> NAME READY STATUS RESTARTS AGE ingress-nginx-controller-67956bf89d-fv58j 1/1 Running 0 1m $ kubectl logs -n <namespace> ingress-nginx-controller-67956bf89d-fv58j ------------------------------------------------------------------------------- NGINX Ingress controller Release: 0.14.0 Build: git-734361d Repository: https://github.com/kubernetes/ingress-nginx ------------------------------------------------------------------------------- .... Check the Nginx Configuration \u00b6 $ kubectl get pods -n <namespace-of-ingress-controller> NAME READY STATUS RESTARTS AGE ingress-nginx-controller-67956bf89d-fv58j 1/1 Running 0 1m $ kubectl exec -it -n <namespace-of-ingress-controller> ingress-nginx-controller-67956bf89d-fv58j -- cat /etc/nginx/nginx.conf daemon off; worker_processes 2; pid /run/nginx.pid; worker_rlimit_nofile 523264; worker_shutdown_timeout 240s; events { multi_accept on; worker_connections 16384; use epoll; } http { .... Check if used Services Exist \u00b6 $ kubectl get svc --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default coffee-svc ClusterIP 10.106.154.35 <none> 80/TCP 18m default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 30m default tea-svc ClusterIP 10.104.172.12 <none> 80/TCP 18m kube-system default-http-backend NodePort 10.108.189.236 <none> 80:30001/TCP 30m kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 30m kube-system kubernetes-dashboard NodePort 10.103.128.17 <none> 80:30000/TCP 30m Debug Logging \u00b6 Using the flag --v=XX it is possible to increase the level of logging. This is performed by editing the deployment. $ kubectl get deploy -n <namespace-of-ingress-controller> NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE default-http-backend 1 1 1 1 35m ingress-nginx-controller 1 1 1 1 35m $ kubectl edit deploy -n <namespace-of-ingress-controller> ingress-nginx-controller # Add --v = X to \"- args\" , where X is an integer --v=2 shows details using diff about the changes in the configuration in nginx --v=3 shows details about the service, Ingress rule, endpoint changes and it dumps the nginx configuration in JSON format --v=5 configures NGINX in debug mode Authentication to the Kubernetes API Server \u00b6 A number of components are involved in the authentication process and the first step is to narrow down the source of the problem, namely whether it is a problem with service authentication or with the kubeconfig file. Both authentications must work: +-------------+ service +------------+ | | authentication | | + apiserver +<-------------------+ ingress | | | | controller | +-------------+ +------------+ Service authentication The Ingress controller needs information from apiserver. Therefore, authentication is required, which can be achieved in a couple of ways: Service Account: This is recommended, because nothing has to be configured. The Ingress controller will use information provided by the system to communicate with the API server. See 'Service Account' section for details. Kubeconfig file: In some Kubernetes environments service accounts are not available. In this case a manual configuration is required. The Ingress controller binary can be started with the --kubeconfig flag. The value of the flag is a path to a file specifying how to connect to the API server. Using the --kubeconfig does not requires the flag --apiserver-host . The format of the file is identical to ~/.kube/config which is used by kubectl to connect to the API server. See 'kubeconfig' section for details. Using the flag --apiserver-host : Using this flag --apiserver-host=http://localhost:8080 it is possible to specify an unsecured API server or reach a remote kubernetes cluster using kubectl proxy . Please do not use this approach in production. In the diagram below you can see the full authentication flow with all options, starting with the browser on the lower left hand side. Kubernetes Workstation +---------------------------------------------------+ +------------------+ | | | | | +-----------+ apiserver +------------+ | | +------------+ | | | | proxy | | | | | | | | | apiserver | | ingress | | | | ingress | | | | | | controller | | | | controller | | | | | | | | | | | | | | | | | | | | | | | | | service account/ | | | | | | | | | | kubeconfig | | | | | | | | | +<-------------------+ | | | | | | | | | | | | | | | | | +------+----+ kubeconfig +------+-----+ | | +------+-----+ | | |<--------------------------------------------------------| | | | | | +---------------------------------------------------+ +------------------+ Service Account \u00b6 If using a service account to connect to the API server, the ingress-controller expects the file /var/run/secrets/kubernetes.io/serviceaccount/token to be present. It provides a secret token that is required to authenticate with the API server. Verify with the following commands: # start a container that contains curl $ kubectl run -it --rm test --image = curlimages/curl --restart = Never -- /bin/sh # check if secret exists / $ ls /var/run/secrets/kubernetes.io/serviceaccount/ ca.crt namespace token / $ # check base connectivity from cluster inside / $ curl -k https://kubernetes.default.svc.cluster.local { \"kind\": \"Status\", \"apiVersion\": \"v1\", \"metadata\": { }, \"status\": \"Failure\", \"message\": \"forbidden: User \\\"system:anonymous\\\" cannot get path \\\"/\\\"\", \"reason\": \"Forbidden\", \"details\": { }, \"code\": 403 }/ $ # connect using tokens }/ $ curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H \"Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://kubernetes.default.svc.cluster.local && echo { \"paths\": [ \"/api\", \"/api/v1\", \"/apis\", \"/apis/\", ... TRUNCATED \"/readyz/shutdown\", \"/version\" ] } / $ # when you type ` exit ` or ` ^D ` the test pod will be deleted. If it is not working, there are two possible reasons: The contents of the tokens are invalid. Find the secret name with kubectl get secrets | grep service-account and delete it with kubectl delete secret <name> . It will automatically be recreated. You have a non-standard Kubernetes installation and the file containing the token may not be present. The API server will mount a volume containing this file, but only if the API server is configured to use the ServiceAccount admission controller. If you experience this error, verify that your API server is using the ServiceAccount admission controller. If you are configuring the API server by hand, you can set this with the --admission-control parameter. Note that you should use other admission controllers as well. Before configuring this option, you should read about admission controllers. More information: User Guide: Service Accounts Cluster Administrator Guide: Managing Service Accounts Kube-Config \u00b6 If you want to use a kubeconfig file for authentication, follow the deploy procedure and add the flag --kubeconfig=/etc/kubernetes/kubeconfig.yaml to the args section of the deployment. Using GDB with Nginx \u00b6 Gdb can be used to with nginx to perform a configuration dump. This allows us to see which configuration is being used, as well as older configurations. Note: The below is based on the nginx documentation . SSH into the worker $ ssh user@workerIP Obtain the Docker Container Running nginx $ docker ps | grep ingress-nginx-controller CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d9e1d243156a registry.k8s.io/ingress-nginx/controller \"/usr/bin/dumb-init \u2026\" 19 minutes ago Up 19 minutes k8s_ingress-nginx-controller_ingress-nginx-controller-67956bf89d-mqxzt_kube-system_079f31ec-aa37-11e8-ad39-080027a227db_0 Exec into the container $ docker exec -it --user = 0 --privileged d9e1d243156a bash Make sure nginx is running in --with-debug $ nginx -V 2 > & 1 | grep -- '--with-debug' Get list of processes running on container $ ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 20:23 ? 00:00:00 /usr/bin/dumb-init /nginx-ingres root 5 1 0 20:23 ? 00:00:05 /ingress-nginx-controller --defa root 21 5 0 20:23 ? 00:00:00 nginx: master process /usr/sbin/ nobody 106 21 0 20:23 ? 00:00:00 nginx: worker process nobody 107 21 0 20:23 ? 00:00:00 nginx: worker process root 172 0 0 20:43 pts/0 00:00:00 bash Attach gdb to the nginx master process $ gdb -p 21 .... Attaching to process 21 Reading symbols from /usr/sbin/nginx...done. .... (gdb) Copy and paste the following: set $cd = ngx_cycle->config_dump set $nelts = $cd.nelts set $elts = (ngx_conf_dump_t*)($cd.elts) while ($nelts-- > 0) set $name = $elts[$nelts]->name.data printf \"Dumping %s to nginx_conf.txt\\n\", $name append memory nginx_conf.txt \\ $ elts [ $nelts ] ->buffer.start $elts [ $nelts ] ->buffer.end end Quit GDB by pressing CTRL+D Open nginx_conf.txt cat nginx_conf.txt Image related issues faced on Nginx 4.2.5 or other versions (Helm chart versions) \u00b6 Incase you face below error while installing Nginx using helm chart (either by helm commands or helm_release terraform provider ) Warning Failed 5m5s (x4 over 6m34s) kubelet Failed to pull image \"registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47\": rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47\": failed to resolve reference \"registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47\": failed to do request: Head \"https://eu.gcr.io/v2/k8s-artifacts-prod/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47\": EOF Then please follow the below steps. During troubleshooting you can also execute the below commands to test the connectivities from you local machines and repositories details a. curl registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 > /dev/null (\u2388 |myprompt)\u279c ~ curl registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 > /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 (\u2388 |myprompt)\u279c ~ b. curl -I https://eu.gcr.io/v2/k8s-artifacts-prod/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 (\u2388 |myprompt)\u279c ~ curl -I https://eu.gcr.io/v2/k8s-artifacts-prod/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 HTTP/2 200 docker-distribution-api-version: registry/2.0 content-type: application/vnd.docker.distribution.manifest.list.v2+json docker-content-digest: sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 content-length: 1384 date: Wed, 28 Sep 2022 16:46:28 GMT server: Docker Registry x-xss-protection: 0 x-frame-options: SAMEORIGIN alt-svc: h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\" (\u2388 |myprompt)\u279c ~ Redirection in the proxy is implemented to ensure the pulling of the images. This is the solution recommended to whitelist the below image repositories : *.appspot.com *.k8s.io *.pkg.dev *.gcr.io More details about the above repos : a. *.k8s.io -> To ensure you can pull any images from registry.k8s.io b. *.gcr.io -> GCP services are used for image hosting. This is part of the domains suggested by GCP to allow and ensure users can pull images from their container registry services. c. *.appspot.com -> This a Google domain. part of the domain used for GCR. Unable to listen on port (80/443) \u00b6 One possible reason for this error is lack of permission to bind to the port. Ports 80, 443, and any other port < 1024 are Linux privileged ports which historically could only be bound by root. The ingress-nginx-controller uses the CAP_NET_BIND_SERVICE linux capability to allow binding these ports as a normal user (www-data / 101). This involves two components: 1. In the image, the /nginx-ingress-controller file has the cap_net_bind_service capability added (e.g. via setcap ) 2. The NET_BIND_SERVICE capability is added to the container in the containerSecurityContext of the deployment. If encountering this on one/some node(s) and not on others, try to purge and pull a fresh copy of the image to the affected node(s), in case there has been corruption of the underlying layers to lose the capability on the executable. Create a test pod \u00b6 The /nginx-ingress-controller process exits/crashes when encountering this error, making it difficult to troubleshoot what is happening inside the container. To get around this, start an equivalent container running \"sleep 3600\", and exec into it for further troubleshooting. For example: apiVersion : v1 kind : Pod metadata : name : ingress-nginx-sleep namespace : default labels : app : nginx spec : containers : - name : nginx image : ##_CONTROLLER_IMAGE_## resources : requests : memory : \"512Mi\" cpu : \"500m\" limits : memory : \"1Gi\" cpu : \"1\" command : [ \"sleep\" ] args : [ \"3600\" ] ports : - containerPort : 80 name : http protocol : TCP - containerPort : 443 name : https protocol : TCP securityContext : allowPrivilegeEscalation : true capabilities : add : - NET_BIND_SERVICE drop : - ALL runAsUser : 101 restartPolicy : Never nodeSelector : kubernetes.io/hostname : ##_NODE_NAME_## tolerations : - key : \"node.kubernetes.io/unschedulable\" operator : \"Exists\" effect : NoSchedule * update the namespace if applicable/desired * replace ##_NODE_NAME_## with the problematic node (or remove nodeSelector section if problem is not confined to one node) * replace ##_CONTROLLER_IMAGE_## with the same image as in use by your ingress-nginx deployment * confirm the securityContext section matches what is in place for ingress-nginx-controller pods in your cluster Apply the YAML and open a shell into the pod. Try to manually run the controller process: $ /nginx-ingress-controller You should get the same error as from the ingress controller pod logs. Confirm the capabilities are properly surfacing into the pod: $ grep CapBnd /proc/1/status CapBnd: 0000000000000400 The above value has only net_bind_service enabled (per security context in YAML which adds that and drops all). If you get a different value, then you can decode it on another linux box (capsh not available in this container) like below, and then figure out why specified capabilities are not propagating into the pod/container. $ capsh --decode = 0000000000000400 0x0000000000000400=cap_net_bind_service Create a test pod as root \u00b6 (Note, this may be restricted by PodSecurityPolicy, PodSecurityAdmission/Standards, OPA Gatekeeper, etc. in which case you will need to do the appropriate workaround for testing, e.g. deploy in a new namespace without the restrictions.) To test further you may want to install additional utilities, etc. Modify the pod yaml by: * changing runAsUser from 101 to 0 * removing the \"drop..ALL\" section from the capabilities. Some things to try after shelling into this container: Try running the controller as the www-data (101) user: $ chmod 4755 /nginx-ingress-controller $ /nginx-ingress-controller Examine the errors to see if there is still an issue listening on the port or if it passed that and moved on to other expected errors due to running out of context. Install the libcap package and check capabilities on the file: $ apk add libcap (1/1) Installing libcap (2.50-r0) Executing busybox-1.33.1-r7.trigger OK: 26 MiB in 41 packages $ getcap /nginx-ingress-controller /nginx-ingress-controller cap_net_bind_service=ep (if missing, see above about purging image on the server and re-pulling) Strace the executable to see what system calls are being executed when it fails: $ apk add strace (1/1) Installing strace (5.12-r0) Executing busybox-1.33.1-r7.trigger OK: 28 MiB in 42 packages $ strace /nginx-ingress-controller execve(\"/nginx-ingress-controller\", [\"/nginx-ingress-controller\"], 0x7ffeb9eb3240 /* 131 vars */) = 0 arch_prctl(ARCH_SET_FS, 0x29ea690) = 0 ...","title":"Troubleshooting"},{"location":"troubleshooting/#troubleshooting","text":"","title":"Troubleshooting"},{"location":"troubleshooting/#ingress-controller-logs-and-events","text":"There are many ways to troubleshoot the ingress-controller. The following are basic troubleshooting methods to obtain more information.","title":"Ingress-Controller Logs and Events"},{"location":"troubleshooting/#check-the-ingress-resource-events","text":"$ kubectl get ing -n <namespace-of-ingress-resource> NAME HOSTS ADDRESS PORTS AGE cafe-ingress cafe.com 10.0.2.15 80 25s $ kubectl describe ing <ingress-resource-name> -n <namespace-of-ingress-resource> Name: cafe-ingress Namespace: default Address: 10.0.2.15 Default backend: default-http-backend:80 (172.17.0.5:8080) Rules: Host Path Backends ---- ---- -------- cafe.com /tea tea-svc:80 (<none>) /coffee coffee-svc:80 (<none>) Annotations: kubectl.kubernetes.io/last-applied-configuration: {\"apiVersion\":\"networking.k8s.io/v1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{},\"name\":\"cafe-ingress\",\"namespace\":\"default\",\"selfLink\":\"/apis/networking/v1/namespaces/default/ingresses/cafe-ingress\"},\"spec\":{\"rules\":[{\"host\":\"cafe.com\",\"http\":{\"paths\":[{\"backend\":{\"serviceName\":\"tea-svc\",\"servicePort\":80},\"path\":\"/tea\"},{\"backend\":{\"serviceName\":\"coffee-svc\",\"servicePort\":80},\"path\":\"/coffee\"}]}}]},\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"169.48.142.110\"}]}}} Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal CREATE 1m ingress-nginx-controller Ingress default/cafe-ingress Normal UPDATE 58s ingress-nginx-controller Ingress default/cafe-ingress","title":"Check the Ingress Resource Events"},{"location":"troubleshooting/#check-the-ingress-controller-logs","text":"$ kubectl get pods -n <namespace-of-ingress-controller> NAME READY STATUS RESTARTS AGE ingress-nginx-controller-67956bf89d-fv58j 1/1 Running 0 1m $ kubectl logs -n <namespace> ingress-nginx-controller-67956bf89d-fv58j ------------------------------------------------------------------------------- NGINX Ingress controller Release: 0.14.0 Build: git-734361d Repository: https://github.com/kubernetes/ingress-nginx ------------------------------------------------------------------------------- ....","title":"Check the Ingress Controller Logs"},{"location":"troubleshooting/#check-the-nginx-configuration","text":"$ kubectl get pods -n <namespace-of-ingress-controller> NAME READY STATUS RESTARTS AGE ingress-nginx-controller-67956bf89d-fv58j 1/1 Running 0 1m $ kubectl exec -it -n <namespace-of-ingress-controller> ingress-nginx-controller-67956bf89d-fv58j -- cat /etc/nginx/nginx.conf daemon off; worker_processes 2; pid /run/nginx.pid; worker_rlimit_nofile 523264; worker_shutdown_timeout 240s; events { multi_accept on; worker_connections 16384; use epoll; } http { ....","title":"Check the Nginx Configuration"},{"location":"troubleshooting/#check-if-used-services-exist","text":"$ kubectl get svc --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default coffee-svc ClusterIP 10.106.154.35 <none> 80/TCP 18m default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 30m default tea-svc ClusterIP 10.104.172.12 <none> 80/TCP 18m kube-system default-http-backend NodePort 10.108.189.236 <none> 80:30001/TCP 30m kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 30m kube-system kubernetes-dashboard NodePort 10.103.128.17 <none> 80:30000/TCP 30m","title":"Check if used Services Exist"},{"location":"troubleshooting/#debug-logging","text":"Using the flag --v=XX it is possible to increase the level of logging. This is performed by editing the deployment. $ kubectl get deploy -n <namespace-of-ingress-controller> NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE default-http-backend 1 1 1 1 35m ingress-nginx-controller 1 1 1 1 35m $ kubectl edit deploy -n <namespace-of-ingress-controller> ingress-nginx-controller # Add --v = X to \"- args\" , where X is an integer --v=2 shows details using diff about the changes in the configuration in nginx --v=3 shows details about the service, Ingress rule, endpoint changes and it dumps the nginx configuration in JSON format --v=5 configures NGINX in debug mode","title":"Debug Logging"},{"location":"troubleshooting/#authentication-to-the-kubernetes-api-server","text":"A number of components are involved in the authentication process and the first step is to narrow down the source of the problem, namely whether it is a problem with service authentication or with the kubeconfig file. Both authentications must work: +-------------+ service +------------+ | | authentication | | + apiserver +<-------------------+ ingress | | | | controller | +-------------+ +------------+ Service authentication The Ingress controller needs information from apiserver. Therefore, authentication is required, which can be achieved in a couple of ways: Service Account: This is recommended, because nothing has to be configured. The Ingress controller will use information provided by the system to communicate with the API server. See 'Service Account' section for details. Kubeconfig file: In some Kubernetes environments service accounts are not available. In this case a manual configuration is required. The Ingress controller binary can be started with the --kubeconfig flag. The value of the flag is a path to a file specifying how to connect to the API server. Using the --kubeconfig does not requires the flag --apiserver-host . The format of the file is identical to ~/.kube/config which is used by kubectl to connect to the API server. See 'kubeconfig' section for details. Using the flag --apiserver-host : Using this flag --apiserver-host=http://localhost:8080 it is possible to specify an unsecured API server or reach a remote kubernetes cluster using kubectl proxy . Please do not use this approach in production. In the diagram below you can see the full authentication flow with all options, starting with the browser on the lower left hand side. Kubernetes Workstation +---------------------------------------------------+ +------------------+ | | | | | +-----------+ apiserver +------------+ | | +------------+ | | | | proxy | | | | | | | | | apiserver | | ingress | | | | ingress | | | | | | controller | | | | controller | | | | | | | | | | | | | | | | | | | | | | | | | service account/ | | | | | | | | | | kubeconfig | | | | | | | | | +<-------------------+ | | | | | | | | | | | | | | | | | +------+----+ kubeconfig +------+-----+ | | +------+-----+ | | |<--------------------------------------------------------| | | | | | +---------------------------------------------------+ +------------------+","title":"Authentication to the Kubernetes API Server"},{"location":"troubleshooting/#service-account","text":"If using a service account to connect to the API server, the ingress-controller expects the file /var/run/secrets/kubernetes.io/serviceaccount/token to be present. It provides a secret token that is required to authenticate with the API server. Verify with the following commands: # start a container that contains curl $ kubectl run -it --rm test --image = curlimages/curl --restart = Never -- /bin/sh # check if secret exists / $ ls /var/run/secrets/kubernetes.io/serviceaccount/ ca.crt namespace token / $ # check base connectivity from cluster inside / $ curl -k https://kubernetes.default.svc.cluster.local { \"kind\": \"Status\", \"apiVersion\": \"v1\", \"metadata\": { }, \"status\": \"Failure\", \"message\": \"forbidden: User \\\"system:anonymous\\\" cannot get path \\\"/\\\"\", \"reason\": \"Forbidden\", \"details\": { }, \"code\": 403 }/ $ # connect using tokens }/ $ curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H \"Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://kubernetes.default.svc.cluster.local && echo { \"paths\": [ \"/api\", \"/api/v1\", \"/apis\", \"/apis/\", ... TRUNCATED \"/readyz/shutdown\", \"/version\" ] } / $ # when you type ` exit ` or ` ^D ` the test pod will be deleted. If it is not working, there are two possible reasons: The contents of the tokens are invalid. Find the secret name with kubectl get secrets | grep service-account and delete it with kubectl delete secret <name> . It will automatically be recreated. You have a non-standard Kubernetes installation and the file containing the token may not be present. The API server will mount a volume containing this file, but only if the API server is configured to use the ServiceAccount admission controller. If you experience this error, verify that your API server is using the ServiceAccount admission controller. If you are configuring the API server by hand, you can set this with the --admission-control parameter. Note that you should use other admission controllers as well. Before configuring this option, you should read about admission controllers. More information: User Guide: Service Accounts Cluster Administrator Guide: Managing Service Accounts","title":"Service Account"},{"location":"troubleshooting/#kube-config","text":"If you want to use a kubeconfig file for authentication, follow the deploy procedure and add the flag --kubeconfig=/etc/kubernetes/kubeconfig.yaml to the args section of the deployment.","title":"Kube-Config"},{"location":"troubleshooting/#using-gdb-with-nginx","text":"Gdb can be used to with nginx to perform a configuration dump. This allows us to see which configuration is being used, as well as older configurations. Note: The below is based on the nginx documentation . SSH into the worker $ ssh user@workerIP Obtain the Docker Container Running nginx $ docker ps | grep ingress-nginx-controller CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d9e1d243156a registry.k8s.io/ingress-nginx/controller \"/usr/bin/dumb-init \u2026\" 19 minutes ago Up 19 minutes k8s_ingress-nginx-controller_ingress-nginx-controller-67956bf89d-mqxzt_kube-system_079f31ec-aa37-11e8-ad39-080027a227db_0 Exec into the container $ docker exec -it --user = 0 --privileged d9e1d243156a bash Make sure nginx is running in --with-debug $ nginx -V 2 > & 1 | grep -- '--with-debug' Get list of processes running on container $ ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 20:23 ? 00:00:00 /usr/bin/dumb-init /nginx-ingres root 5 1 0 20:23 ? 00:00:05 /ingress-nginx-controller --defa root 21 5 0 20:23 ? 00:00:00 nginx: master process /usr/sbin/ nobody 106 21 0 20:23 ? 00:00:00 nginx: worker process nobody 107 21 0 20:23 ? 00:00:00 nginx: worker process root 172 0 0 20:43 pts/0 00:00:00 bash Attach gdb to the nginx master process $ gdb -p 21 .... Attaching to process 21 Reading symbols from /usr/sbin/nginx...done. .... (gdb) Copy and paste the following: set $cd = ngx_cycle->config_dump set $nelts = $cd.nelts set $elts = (ngx_conf_dump_t*)($cd.elts) while ($nelts-- > 0) set $name = $elts[$nelts]->name.data printf \"Dumping %s to nginx_conf.txt\\n\", $name append memory nginx_conf.txt \\ $ elts [ $nelts ] ->buffer.start $elts [ $nelts ] ->buffer.end end Quit GDB by pressing CTRL+D Open nginx_conf.txt cat nginx_conf.txt","title":"Using GDB with Nginx"},{"location":"troubleshooting/#image-related-issues-faced-on-nginx-425-or-other-versions-helm-chart-versions","text":"Incase you face below error while installing Nginx using helm chart (either by helm commands or helm_release terraform provider ) Warning Failed 5m5s (x4 over 6m34s) kubelet Failed to pull image \"registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47\": rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47\": failed to resolve reference \"registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47\": failed to do request: Head \"https://eu.gcr.io/v2/k8s-artifacts-prod/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47\": EOF Then please follow the below steps. During troubleshooting you can also execute the below commands to test the connectivities from you local machines and repositories details a. curl registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 > /dev/null (\u2388 |myprompt)\u279c ~ curl registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 > /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 (\u2388 |myprompt)\u279c ~ b. curl -I https://eu.gcr.io/v2/k8s-artifacts-prod/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 (\u2388 |myprompt)\u279c ~ curl -I https://eu.gcr.io/v2/k8s-artifacts-prod/ingress-nginx/kube-webhook-certgen/manifests/sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 HTTP/2 200 docker-distribution-api-version: registry/2.0 content-type: application/vnd.docker.distribution.manifest.list.v2+json docker-content-digest: sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47 content-length: 1384 date: Wed, 28 Sep 2022 16:46:28 GMT server: Docker Registry x-xss-protection: 0 x-frame-options: SAMEORIGIN alt-svc: h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\" (\u2388 |myprompt)\u279c ~ Redirection in the proxy is implemented to ensure the pulling of the images. This is the solution recommended to whitelist the below image repositories : *.appspot.com *.k8s.io *.pkg.dev *.gcr.io More details about the above repos : a. *.k8s.io -> To ensure you can pull any images from registry.k8s.io b. *.gcr.io -> GCP services are used for image hosting. This is part of the domains suggested by GCP to allow and ensure users can pull images from their container registry services. c. *.appspot.com -> This a Google domain. part of the domain used for GCR.","title":"Image related issues faced on Nginx 4.2.5 or other versions (Helm chart versions)"},{"location":"troubleshooting/#unable-to-listen-on-port-80443","text":"One possible reason for this error is lack of permission to bind to the port. Ports 80, 443, and any other port < 1024 are Linux privileged ports which historically could only be bound by root. The ingress-nginx-controller uses the CAP_NET_BIND_SERVICE linux capability to allow binding these ports as a normal user (www-data / 101). This involves two components: 1. In the image, the /nginx-ingress-controller file has the cap_net_bind_service capability added (e.g. via setcap ) 2. The NET_BIND_SERVICE capability is added to the container in the containerSecurityContext of the deployment. If encountering this on one/some node(s) and not on others, try to purge and pull a fresh copy of the image to the affected node(s), in case there has been corruption of the underlying layers to lose the capability on the executable.","title":"Unable to listen on port (80/443)"},{"location":"troubleshooting/#create-a-test-pod","text":"The /nginx-ingress-controller process exits/crashes when encountering this error, making it difficult to troubleshoot what is happening inside the container. To get around this, start an equivalent container running \"sleep 3600\", and exec into it for further troubleshooting. For example: apiVersion : v1 kind : Pod metadata : name : ingress-nginx-sleep namespace : default labels : app : nginx spec : containers : - name : nginx image : ##_CONTROLLER_IMAGE_## resources : requests : memory : \"512Mi\" cpu : \"500m\" limits : memory : \"1Gi\" cpu : \"1\" command : [ \"sleep\" ] args : [ \"3600\" ] ports : - containerPort : 80 name : http protocol : TCP - containerPort : 443 name : https protocol : TCP securityContext : allowPrivilegeEscalation : true capabilities : add : - NET_BIND_SERVICE drop : - ALL runAsUser : 101 restartPolicy : Never nodeSelector : kubernetes.io/hostname : ##_NODE_NAME_## tolerations : - key : \"node.kubernetes.io/unschedulable\" operator : \"Exists\" effect : NoSchedule * update the namespace if applicable/desired * replace ##_NODE_NAME_## with the problematic node (or remove nodeSelector section if problem is not confined to one node) * replace ##_CONTROLLER_IMAGE_## with the same image as in use by your ingress-nginx deployment * confirm the securityContext section matches what is in place for ingress-nginx-controller pods in your cluster Apply the YAML and open a shell into the pod. Try to manually run the controller process: $ /nginx-ingress-controller You should get the same error as from the ingress controller pod logs. Confirm the capabilities are properly surfacing into the pod: $ grep CapBnd /proc/1/status CapBnd: 0000000000000400 The above value has only net_bind_service enabled (per security context in YAML which adds that and drops all). If you get a different value, then you can decode it on another linux box (capsh not available in this container) like below, and then figure out why specified capabilities are not propagating into the pod/container. $ capsh --decode = 0000000000000400 0x0000000000000400=cap_net_bind_service","title":"Create a test pod"},{"location":"troubleshooting/#create-a-test-pod-as-root","text":"(Note, this may be restricted by PodSecurityPolicy, PodSecurityAdmission/Standards, OPA Gatekeeper, etc. in which case you will need to do the appropriate workaround for testing, e.g. deploy in a new namespace without the restrictions.) To test further you may want to install additional utilities, etc. Modify the pod yaml by: * changing runAsUser from 101 to 0 * removing the \"drop..ALL\" section from the capabilities. Some things to try after shelling into this container: Try running the controller as the www-data (101) user: $ chmod 4755 /nginx-ingress-controller $ /nginx-ingress-controller Examine the errors to see if there is still an issue listening on the port or if it passed that and moved on to other expected errors due to running out of context. Install the libcap package and check capabilities on the file: $ apk add libcap (1/1) Installing libcap (2.50-r0) Executing busybox-1.33.1-r7.trigger OK: 26 MiB in 41 packages $ getcap /nginx-ingress-controller /nginx-ingress-controller cap_net_bind_service=ep (if missing, see above about purging image on the server and re-pulling) Strace the executable to see what system calls are being executed when it fails: $ apk add strace (1/1) Installing strace (5.12-r0) Executing busybox-1.33.1-r7.trigger OK: 28 MiB in 42 packages $ strace /nginx-ingress-controller execve(\"/nginx-ingress-controller\", [\"/nginx-ingress-controller\"], 0x7ffeb9eb3240 /* 131 vars */) = 0 arch_prctl(ARCH_SET_FS, 0x29ea690) = 0 ...","title":"Create a test pod as root"},{"location":"deploy/","text":"Installation Guide \u00b6 There are multiple ways to install the NGINX ingress controller: with Helm , using the project repository chart; with kubectl apply , using YAML manifests; with specific addons (e.g. for minikube or MicroK8s ). On most Kubernetes clusters, the ingress controller will work without requiring any extra configuration. If you want to get started as fast as possible, you can check the quick start instructions. However, in many environments, you can improve the performance or get better logs by enabling extra features. We recommend that you check the environment-specific instructions for details about optimizing the ingress controller for your particular environment or cloud provider. Contents \u00b6 Quick start Environment-specific instructions ... Docker Desktop ... Rancher Desktop ... minikube ... MicroK8s ... AWS ... GCE - GKE ... Azure ... Digital Ocean ... Scaleway ... Exoscale ... Oracle Cloud Infrastructure ... OVHcloud ... Bare-metal Miscellaneous Quick start \u00b6 If you have Helm, you can deploy the ingress controller with the following command: helm upgrade --install ingress-nginx ingress-nginx \\ --repo https://kubernetes.github.io/ingress-nginx \\ --namespace ingress-nginx --create-namespace It will install the controller in the ingress-nginx namespace, creating that namespace if it doesn't already exist. Info This command is idempotent : if the ingress controller is not installed, it will install it, if the ingress controller is already installed, it will upgrade it. If you don't have Helm or if you prefer to use a YAML manifest, you can run the following command instead: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml Info The YAML manifest in the command above was generated with helm template , so you will end up with almost the same resources as if you had used Helm to install the controller. Attention If you are running an old version of Kubernetes (1.18 or earlier), please read this paragraph for specific instructions. Because of api deprecations, the default manifest may not work on your cluster. Specific manifests for supported Kubernetes versions are available within a sub-folder of each provider. Pre-flight check \u00b6 A few pods should start in the ingress-nginx namespace: kubectl get pods --namespace=ingress-nginx After a while, they should all be running. The following command will wait for the ingress controller pod to be up, running, and ready: kubectl wait --namespace ingress-nginx \\ --for=condition=ready pod \\ --selector=app.kubernetes.io/component=controller \\ --timeout=120s Local testing \u00b6 Let's create a simple web server and the associated service: kubectl create deployment demo --image=httpd --port=80 kubectl expose deployment demo Then create an ingress resource. The following example uses a host that maps to localhost : kubectl create ingress demo-localhost --class=nginx \\ --rule=\"demo.localdev.me/*=demo:80\" Now, forward a local port to the ingress controller: kubectl port-forward --namespace=ingress-nginx service/ingress-nginx-controller 8080:80 At this point, if you access http://demo.localdev.me:8080/, you should see an HTML page telling you \"It works!\". Online testing \u00b6 If your Kubernetes cluster is a \"real\" cluster that supports services of type LoadBalancer , it will have allocated an external IP address or FQDN to the ingress controller. You can see that IP address or FQDN with the following command: kubectl get service ingress-nginx-controller --namespace=ingress-nginx It will be the EXTERNAL-IP field. If that field shows <pending> , this means that your Kubernetes cluster wasn't able to provision the load balancer (generally, this is because it doesn't support services of type LoadBalancer ). Once you have the external IP address (or FQDN), set up a DNS record pointing to it. Then you can create an ingress resource. The following example assumes that you have set up a DNS record for www.demo.io : kubectl create ingress demo --class=nginx \\ --rule=\"www.demo.io/*=demo:80\" Alternatively, the above command can be rewritten as follows for the --rule command and below. kubectl create ingress demo --class=nginx \\ --rule www.demo.io/=demo:80 You should then be able to see the \"It works!\" page when you connect to http://www.demo.io/. Congratulations, you are serving a public website hosted on a Kubernetes cluster! \ud83c\udf89 Environment-specific instructions \u00b6 Local development clusters \u00b6 minikube \u00b6 The ingress controller can be installed through minikube's addons system: minikube addons enable ingress MicroK8s \u00b6 The ingress controller can be installed through MicroK8s's addons system: microk8s enable ingress Please check the MicroK8s documentation page for details. Docker Desktop \u00b6 Kubernetes is available in Docker Desktop: Mac, from version 18.06.0-ce Windows, from version 18.06.0-ce First, make sure that Kubernetes is enabled in the Docker settings. The command kubectl get nodes should show a single node called docker-desktop . The ingress controller can be installed on Docker Desktop using the default quick start instructions. On most systems, if you don't have any other service of type LoadBalancer bound to port 80, the ingress controller will be assigned the EXTERNAL-IP of localhost , which means that it will be reachable on localhost:80. If that doesn't work, you might have to fall back to the kubectl port-forward method described in the local testing section . Rancher Desktop \u00b6 Rancher Desktop provides Kubernetes and Container Management on the desktop. Kubernetes is enabled by default in Rancher Desktop. Rancher Desktop uses K3s under the hood, which in turn uses Traefik as the default ingress controller for the Kubernetes cluster. To use NGINX ingress controller in place of the default Traefik, disable Traefik from Preference > Kubernetes menu. Once traefik is disabled, the NGINX ingress controller can be installed on Rancher Desktop using the default quick start instructions. Follow the instructions described in the local testing section to try a sample. Cloud deployments \u00b6 If the load balancers of your cloud provider do active healthchecks on their backends (most do), you can change the externalTrafficPolicy of the ingress controller Service to Local (instead of the default Cluster ) to save an extra hop in some cases. If you're installing with Helm, this can be done by adding --set controller.service.externalTrafficPolicy=Local to the helm install or helm upgrade command. Furthermore, if the load balancers of your cloud provider support the PROXY protocol, you can enable it, and it will let the ingress controller see the real IP address of the clients. Otherwise, it will generally see the IP address of the upstream load balancer. This must be done both in the ingress controller (with e.g. --set controller.config.use-proxy-protocol=true ) and in the cloud provider's load balancer configuration to function correctly. In the following sections, we provide YAML manifests that enable these options when possible, using the specific options of various cloud providers. AWS \u00b6 In AWS, we use a Network load balancer (NLB) to expose the NGINX Ingress controller behind a Service of Type=LoadBalancer . Info The provided templates illustrate the setup for legacy in-tree service load balancer for AWS NLB. AWS provides the documentation on how to use Network load balancing on Amazon EKS with AWS Load Balancer Controller . Network Load Balancer (NLB) \u00b6 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/aws/deploy.yaml TLS termination in AWS Load Balancer (NLB) \u00b6 By default, TLS is terminated in the ingress controller. But it is also possible to terminate TLS in the Load Balancer. This section explains how to do that on AWS using an NLB. Download the deploy.yaml template wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/aws/nlb-with-tls-termination/deploy.yaml Edit the file and change the VPC CIDR in use for the Kubernetes cluster: proxy-real-ip-cidr: XXX.XXX.XXX/XX Change the AWS Certificate Manager (ACM) ID as well: arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX Deploy the manifest: kubectl apply -f deploy.yaml NLB Idle Timeouts \u00b6 Idle timeout value for TCP flows is 350 seconds and cannot be modified . For this reason, you need to ensure the keepalive_timeout value is configured less than 350 seconds to work as expected. By default, NGINX keepalive_timeout is set to 75s . More information with regard to timeouts can be found in the official AWS documentation GCE-GKE \u00b6 First, your user needs to have cluster-admin permissions on the cluster. This can be done with the following command: kubectl create clusterrolebinding cluster-admin-binding \\ --clusterrole cluster-admin \\ --user $(gcloud config get-value account) Then, the ingress controller can be installed like this: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml Warning For private clusters, you will need to either add a firewall rule that allows master nodes access to port 8443/tcp on worker nodes, or change the existing rule that allows access to port 80/tcp , 443/tcp and 10254/tcp to also allow access to port 8443/tcp . More information can be found in the Official GCP Documentation . See the GKE documentation on adding rules and the Kubernetes issue for more detail. Proxy-protocol is supported in GCE check the Official Documentations on how to enable. Azure \u00b6 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml More information with regard to Azure annotations for ingress controller can be found in the official AKS documentation . Digital Ocean \u00b6 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/do/deploy.yaml - By default the service object of the ingress-nginx-controller for Digital-Ocean, only configures one annotation. Its this one service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: \"true\" . While this makes the service functional, it was reported that the Digital-Ocean LoadBalancer graphs shows no data , unless a few other annotations are also configured. Some of these other annotations require values that can not be generic and hence not forced in a out-of-the-box installation. These annotations and a discussion on them is well documented in this issue . Please refer to the issue to add annotations, with values specific to user, to get graphs of the DO-LB populated with data. Scaleway \u00b6 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/scw/deploy.yaml Exoscale \u00b6 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/exoscale/deploy.yaml The full list of annotations supported by Exoscale is available in the Exoscale Cloud Controller Manager documentation . Oracle Cloud Infrastructure \u00b6 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml A complete list of available annotations for Oracle Cloud Infrastructure can be found in the OCI Cloud Controller Manager documentation. OVHcloud \u00b6 helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update helm -n ingress-nginx install ingress-nginx ingress-nginx/ingress-nginx --create-namespace You can find the complete tutorial . Bare metal clusters \u00b6 This section is applicable to Kubernetes clusters deployed on bare metal servers, as well as \"raw\" VMs where Kubernetes was installed manually, using generic Linux distros (like CentOS, Ubuntu...) For quick testing, you can use a NodePort . This should work on almost every cluster, but it will typically use a port in the range 30000-32767. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/baremetal/deploy.yaml For more information about bare metal deployments (and how to use port 80 instead of a random port in the 30000-32767 range), see bare-metal considerations . Miscellaneous \u00b6 Checking ingress controller version \u00b6 Run /nginx-ingress-controller --version within the pod, for instance with kubectl exec : POD_NAMESPACE=ingress-nginx POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx --field-selector=status.phase=Running -o name) kubectl exec $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version Scope \u00b6 By default, the controller watches Ingress objects from all namespaces. If you want to change this behavior, use the flag --watch-namespace or check the Helm chart value controller.scope to limit the controller to a single namespace. See also \u201cHow to easily install multiple instances of the Ingress NGINX controller in the same cluster\u201d for more details. Webhook network access \u00b6 Warning The controller uses an admission webhook to validate Ingress definitions. Make sure that you don't have Network policies or additional firewalls preventing connections from the API server to the ingress-nginx-controller-admission service. Certificate generation \u00b6 Attention The first time the ingress controller starts, two Jobs create the SSL Certificate used by the admission webhook. This can cause an initial delay of up to two minutes until it is possible to create and validate Ingress definitions. You can wait until it is ready to run the next command: kubectl wait --namespace ingress-nginx \\ --for=condition=ready pod \\ --selector=app.kubernetes.io/component=controller \\ --timeout=120s Running on Kubernetes versions older than 1.19 \u00b6 Ingress resources evolved over time. They started with apiVersion: extensions/v1beta1 , then moved to apiVersion: networking.k8s.io/v1beta1 and more recently to apiVersion: networking.k8s.io/v1 . Here is how these Ingress versions are supported in Kubernetes: - before Kubernetes 1.19, only v1beta1 Ingress resources are supported - from Kubernetes 1.19 to 1.21, both v1beta1 and v1 Ingress resources are supported - in Kubernetes 1.22 and above, only v1 Ingress resources are supported And here is how these Ingress versions are supported in NGINX Ingress Controller: - before version 1.0, only v1beta1 Ingress resources are supported - in version 1.0 and above, only v1 Ingress resources are As a result, if you're running Kubernetes 1.19 or later, you should be able to use the latest version of the NGINX Ingress Controller; but if you're using an old version of Kubernetes (1.18 or earlier) you will have to use version 0.X of the NGINX Ingress Controller (e.g. version 0.49). The Helm chart of the NGINX Ingress Controller switched to version 1 in version 4 of the chart. In other words, if you're running Kubernetes 1.19 or earlier, you should use version 3.X of the chart (this can be done by adding --version='<4' to the helm install command).","title":"Installation Guide"},{"location":"deploy/#installation-guide","text":"There are multiple ways to install the NGINX ingress controller: with Helm , using the project repository chart; with kubectl apply , using YAML manifests; with specific addons (e.g. for minikube or MicroK8s ). On most Kubernetes clusters, the ingress controller will work without requiring any extra configuration. If you want to get started as fast as possible, you can check the quick start instructions. However, in many environments, you can improve the performance or get better logs by enabling extra features. We recommend that you check the environment-specific instructions for details about optimizing the ingress controller for your particular environment or cloud provider.","title":"Installation Guide"},{"location":"deploy/#contents","text":"Quick start Environment-specific instructions ... Docker Desktop ... Rancher Desktop ... minikube ... MicroK8s ... AWS ... GCE - GKE ... Azure ... Digital Ocean ... Scaleway ... Exoscale ... Oracle Cloud Infrastructure ... OVHcloud ... Bare-metal Miscellaneous","title":"Contents"},{"location":"deploy/#quick-start","text":"If you have Helm, you can deploy the ingress controller with the following command: helm upgrade --install ingress-nginx ingress-nginx \\ --repo https://kubernetes.github.io/ingress-nginx \\ --namespace ingress-nginx --create-namespace It will install the controller in the ingress-nginx namespace, creating that namespace if it doesn't already exist. Info This command is idempotent : if the ingress controller is not installed, it will install it, if the ingress controller is already installed, it will upgrade it. If you don't have Helm or if you prefer to use a YAML manifest, you can run the following command instead: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml Info The YAML manifest in the command above was generated with helm template , so you will end up with almost the same resources as if you had used Helm to install the controller. Attention If you are running an old version of Kubernetes (1.18 or earlier), please read this paragraph for specific instructions. Because of api deprecations, the default manifest may not work on your cluster. Specific manifests for supported Kubernetes versions are available within a sub-folder of each provider.","title":"Quick start"},{"location":"deploy/#pre-flight-check","text":"A few pods should start in the ingress-nginx namespace: kubectl get pods --namespace=ingress-nginx After a while, they should all be running. The following command will wait for the ingress controller pod to be up, running, and ready: kubectl wait --namespace ingress-nginx \\ --for=condition=ready pod \\ --selector=app.kubernetes.io/component=controller \\ --timeout=120s","title":"Pre-flight check"},{"location":"deploy/#local-testing","text":"Let's create a simple web server and the associated service: kubectl create deployment demo --image=httpd --port=80 kubectl expose deployment demo Then create an ingress resource. The following example uses a host that maps to localhost : kubectl create ingress demo-localhost --class=nginx \\ --rule=\"demo.localdev.me/*=demo:80\" Now, forward a local port to the ingress controller: kubectl port-forward --namespace=ingress-nginx service/ingress-nginx-controller 8080:80 At this point, if you access http://demo.localdev.me:8080/, you should see an HTML page telling you \"It works!\".","title":"Local testing"},{"location":"deploy/#online-testing","text":"If your Kubernetes cluster is a \"real\" cluster that supports services of type LoadBalancer , it will have allocated an external IP address or FQDN to the ingress controller. You can see that IP address or FQDN with the following command: kubectl get service ingress-nginx-controller --namespace=ingress-nginx It will be the EXTERNAL-IP field. If that field shows <pending> , this means that your Kubernetes cluster wasn't able to provision the load balancer (generally, this is because it doesn't support services of type LoadBalancer ). Once you have the external IP address (or FQDN), set up a DNS record pointing to it. Then you can create an ingress resource. The following example assumes that you have set up a DNS record for www.demo.io : kubectl create ingress demo --class=nginx \\ --rule=\"www.demo.io/*=demo:80\" Alternatively, the above command can be rewritten as follows for the --rule command and below. kubectl create ingress demo --class=nginx \\ --rule www.demo.io/=demo:80 You should then be able to see the \"It works!\" page when you connect to http://www.demo.io/. Congratulations, you are serving a public website hosted on a Kubernetes cluster! \ud83c\udf89","title":"Online testing"},{"location":"deploy/#environment-specific-instructions","text":"","title":"Environment-specific instructions"},{"location":"deploy/#local-development-clusters","text":"","title":"Local development clusters"},{"location":"deploy/#minikube","text":"The ingress controller can be installed through minikube's addons system: minikube addons enable ingress","title":"minikube"},{"location":"deploy/#microk8s","text":"The ingress controller can be installed through MicroK8s's addons system: microk8s enable ingress Please check the MicroK8s documentation page for details.","title":"MicroK8s"},{"location":"deploy/#docker-desktop","text":"Kubernetes is available in Docker Desktop: Mac, from version 18.06.0-ce Windows, from version 18.06.0-ce First, make sure that Kubernetes is enabled in the Docker settings. The command kubectl get nodes should show a single node called docker-desktop . The ingress controller can be installed on Docker Desktop using the default quick start instructions. On most systems, if you don't have any other service of type LoadBalancer bound to port 80, the ingress controller will be assigned the EXTERNAL-IP of localhost , which means that it will be reachable on localhost:80. If that doesn't work, you might have to fall back to the kubectl port-forward method described in the local testing section .","title":"Docker Desktop"},{"location":"deploy/#rancher-desktop","text":"Rancher Desktop provides Kubernetes and Container Management on the desktop. Kubernetes is enabled by default in Rancher Desktop. Rancher Desktop uses K3s under the hood, which in turn uses Traefik as the default ingress controller for the Kubernetes cluster. To use NGINX ingress controller in place of the default Traefik, disable Traefik from Preference > Kubernetes menu. Once traefik is disabled, the NGINX ingress controller can be installed on Rancher Desktop using the default quick start instructions. Follow the instructions described in the local testing section to try a sample.","title":"Rancher Desktop"},{"location":"deploy/#cloud-deployments","text":"If the load balancers of your cloud provider do active healthchecks on their backends (most do), you can change the externalTrafficPolicy of the ingress controller Service to Local (instead of the default Cluster ) to save an extra hop in some cases. If you're installing with Helm, this can be done by adding --set controller.service.externalTrafficPolicy=Local to the helm install or helm upgrade command. Furthermore, if the load balancers of your cloud provider support the PROXY protocol, you can enable it, and it will let the ingress controller see the real IP address of the clients. Otherwise, it will generally see the IP address of the upstream load balancer. This must be done both in the ingress controller (with e.g. --set controller.config.use-proxy-protocol=true ) and in the cloud provider's load balancer configuration to function correctly. In the following sections, we provide YAML manifests that enable these options when possible, using the specific options of various cloud providers.","title":"Cloud deployments"},{"location":"deploy/#aws","text":"In AWS, we use a Network load balancer (NLB) to expose the NGINX Ingress controller behind a Service of Type=LoadBalancer . Info The provided templates illustrate the setup for legacy in-tree service load balancer for AWS NLB. AWS provides the documentation on how to use Network load balancing on Amazon EKS with AWS Load Balancer Controller .","title":"AWS"},{"location":"deploy/#network-load-balancer-nlb","text":"kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/aws/deploy.yaml","title":"Network Load Balancer (NLB)"},{"location":"deploy/#tls-termination-in-aws-load-balancer-nlb","text":"By default, TLS is terminated in the ingress controller. But it is also possible to terminate TLS in the Load Balancer. This section explains how to do that on AWS using an NLB. Download the deploy.yaml template wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/aws/nlb-with-tls-termination/deploy.yaml Edit the file and change the VPC CIDR in use for the Kubernetes cluster: proxy-real-ip-cidr: XXX.XXX.XXX/XX Change the AWS Certificate Manager (ACM) ID as well: arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX Deploy the manifest: kubectl apply -f deploy.yaml","title":"TLS termination in AWS Load Balancer (NLB)"},{"location":"deploy/#nlb-idle-timeouts","text":"Idle timeout value for TCP flows is 350 seconds and cannot be modified . For this reason, you need to ensure the keepalive_timeout value is configured less than 350 seconds to work as expected. By default, NGINX keepalive_timeout is set to 75s . More information with regard to timeouts can be found in the official AWS documentation","title":"NLB Idle Timeouts"},{"location":"deploy/#gce-gke","text":"First, your user needs to have cluster-admin permissions on the cluster. This can be done with the following command: kubectl create clusterrolebinding cluster-admin-binding \\ --clusterrole cluster-admin \\ --user $(gcloud config get-value account) Then, the ingress controller can be installed like this: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml Warning For private clusters, you will need to either add a firewall rule that allows master nodes access to port 8443/tcp on worker nodes, or change the existing rule that allows access to port 80/tcp , 443/tcp and 10254/tcp to also allow access to port 8443/tcp . More information can be found in the Official GCP Documentation . See the GKE documentation on adding rules and the Kubernetes issue for more detail. Proxy-protocol is supported in GCE check the Official Documentations on how to enable.","title":"GCE-GKE"},{"location":"deploy/#azure","text":"kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml More information with regard to Azure annotations for ingress controller can be found in the official AKS documentation .","title":"Azure"},{"location":"deploy/#digital-ocean","text":"kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/do/deploy.yaml - By default the service object of the ingress-nginx-controller for Digital-Ocean, only configures one annotation. Its this one service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: \"true\" . While this makes the service functional, it was reported that the Digital-Ocean LoadBalancer graphs shows no data , unless a few other annotations are also configured. Some of these other annotations require values that can not be generic and hence not forced in a out-of-the-box installation. These annotations and a discussion on them is well documented in this issue . Please refer to the issue to add annotations, with values specific to user, to get graphs of the DO-LB populated with data.","title":"Digital Ocean"},{"location":"deploy/#scaleway","text":"kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/scw/deploy.yaml","title":"Scaleway"},{"location":"deploy/#exoscale","text":"kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/exoscale/deploy.yaml The full list of annotations supported by Exoscale is available in the Exoscale Cloud Controller Manager documentation .","title":"Exoscale"},{"location":"deploy/#oracle-cloud-infrastructure","text":"kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml A complete list of available annotations for Oracle Cloud Infrastructure can be found in the OCI Cloud Controller Manager documentation.","title":"Oracle Cloud Infrastructure"},{"location":"deploy/#ovhcloud","text":"helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update helm -n ingress-nginx install ingress-nginx ingress-nginx/ingress-nginx --create-namespace You can find the complete tutorial .","title":"OVHcloud"},{"location":"deploy/#bare-metal-clusters","text":"This section is applicable to Kubernetes clusters deployed on bare metal servers, as well as \"raw\" VMs where Kubernetes was installed manually, using generic Linux distros (like CentOS, Ubuntu...) For quick testing, you can use a NodePort . This should work on almost every cluster, but it will typically use a port in the range 30000-32767. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/baremetal/deploy.yaml For more information about bare metal deployments (and how to use port 80 instead of a random port in the 30000-32767 range), see bare-metal considerations .","title":"Bare metal clusters"},{"location":"deploy/#miscellaneous","text":"","title":"Miscellaneous"},{"location":"deploy/#checking-ingress-controller-version","text":"Run /nginx-ingress-controller --version within the pod, for instance with kubectl exec : POD_NAMESPACE=ingress-nginx POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx --field-selector=status.phase=Running -o name) kubectl exec $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version","title":"Checking ingress controller version"},{"location":"deploy/#scope","text":"By default, the controller watches Ingress objects from all namespaces. If you want to change this behavior, use the flag --watch-namespace or check the Helm chart value controller.scope to limit the controller to a single namespace. See also \u201cHow to easily install multiple instances of the Ingress NGINX controller in the same cluster\u201d for more details.","title":"Scope"},{"location":"deploy/#webhook-network-access","text":"Warning The controller uses an admission webhook to validate Ingress definitions. Make sure that you don't have Network policies or additional firewalls preventing connections from the API server to the ingress-nginx-controller-admission service.","title":"Webhook network access"},{"location":"deploy/#certificate-generation","text":"Attention The first time the ingress controller starts, two Jobs create the SSL Certificate used by the admission webhook. This can cause an initial delay of up to two minutes until it is possible to create and validate Ingress definitions. You can wait until it is ready to run the next command: kubectl wait --namespace ingress-nginx \\ --for=condition=ready pod \\ --selector=app.kubernetes.io/component=controller \\ --timeout=120s","title":"Certificate generation"},{"location":"deploy/#running-on-kubernetes-versions-older-than-119","text":"Ingress resources evolved over time. They started with apiVersion: extensions/v1beta1 , then moved to apiVersion: networking.k8s.io/v1beta1 and more recently to apiVersion: networking.k8s.io/v1 . Here is how these Ingress versions are supported in Kubernetes: - before Kubernetes 1.19, only v1beta1 Ingress resources are supported - from Kubernetes 1.19 to 1.21, both v1beta1 and v1 Ingress resources are supported - in Kubernetes 1.22 and above, only v1 Ingress resources are supported And here is how these Ingress versions are supported in NGINX Ingress Controller: - before version 1.0, only v1beta1 Ingress resources are supported - in version 1.0 and above, only v1 Ingress resources are As a result, if you're running Kubernetes 1.19 or later, you should be able to use the latest version of the NGINX Ingress Controller; but if you're using an old version of Kubernetes (1.18 or earlier) you will have to use version 0.X of the NGINX Ingress Controller (e.g. version 0.49). The Helm chart of the NGINX Ingress Controller switched to version 1 in version 4 of the chart. In other words, if you're running Kubernetes 1.19 or earlier, you should use version 3.X of the chart (this can be done by adding --version='<4' to the helm install command).","title":"Running on Kubernetes versions older than 1.19"},{"location":"deploy/baremetal/","text":"Bare-metal considerations \u00b6 In traditional cloud environments, where network load balancers are available on-demand, a single Kubernetes manifest suffices to provide a single point of contact to the NGINX Ingress controller to external clients and, indirectly, to any application running inside the cluster. Bare-metal environments lack this commodity, requiring a slightly different setup to offer the same kind of access to external consumers. The rest of this document describes a few recommended approaches to deploying the NGINX Ingress controller inside a Kubernetes cluster running on bare-metal. A pure software solution: MetalLB \u00b6 MetalLB provides a network load-balancer implementation for Kubernetes clusters that do not run on a supported cloud provider, effectively allowing the usage of LoadBalancer Services within any cluster. This section demonstrates how to use the Layer 2 configuration mode of MetalLB together with the NGINX Ingress controller in a Kubernetes cluster that has publicly accessible nodes . In this mode, one node attracts all the traffic for the ingress-nginx Service IP. See Traffic policies for more details. Note The description of other supported configuration modes is off-scope for this document. Warning MetalLB is currently in beta . Read about the Project maturity and make sure you inform yourself by reading the official documentation thoroughly. MetalLB can be deployed either with a simple Kubernetes manifest or with Helm. The rest of this example assumes MetalLB was deployed following the Installation instructions, and that the NGINX Ingress controller was installed using the steps described in the quickstart section of the installation guide . MetalLB requires a pool of IP addresses in order to be able to take ownership of the ingress-nginx Service. This pool can be defined through IPAddressPool objects in the same namespace as the MetalLB controller. This pool of IPs must be dedicated to MetalLB's use, you can't reuse the Kubernetes node IPs or IPs handed out by a DHCP server. Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 After creating the following objects, MetalLB takes ownership of one of the IP addresses in the pool and updates the loadBalancer IP field of the ingress-nginx Service accordingly. --- apiVersion : metallb.io/v1beta1 kind : IPAddressPool metadata : name : default namespace : metallb-system spec : addresses : - 203.0.113.10-203.0.113.15 autoAssign : true --- apiVersion : metallb.io/v1beta1 kind : L2Advertisement metadata : name : default namespace : metallb-system spec : ipAddressPools : - default $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 <none> 80/TCP ingress-nginx LoadBalancer 10.0.220.217 203.0.113.10 80:30100/TCP,443:30101/TCP As soon as MetalLB sets the external IP address of the ingress-nginx LoadBalancer Service, the corresponding entries are created in the iptables NAT table and the node with the selected IP address starts responding to HTTP requests on the ports configured in the LoadBalancer Service: $ curl -D- http://203.0.113.10 -H 'Host: myapp.example.com' HTTP/1.1 200 OK Server: nginx/1.15.2 Tip In order to preserve the source IP address in HTTP requests sent to NGINX, it is necessary to use the Local traffic policy. Traffic policies are described in more details in Traffic policies as well as in the next section. Over a NodePort Service \u00b6 Due to its simplicity, this is the setup a user will deploy by default when following the steps described in the installation guide . Info A Service of type NodePort exposes, via the kube-proxy component, the same unprivileged port (default: 30000-32767) on every Kubernetes node, masters included. For more information, see Services . In this configuration, the NGINX container remains isolated from the host network. As a result, it can safely bind to any port, including the standard HTTP ports 80 and 443. However, due to the container namespace isolation, a client located outside the cluster network (e.g. on the public internet) is not able to access Ingress hosts directly on ports 80 and 443. Instead, the external client must append the NodePort allocated to the ingress-nginx Service to HTTP requests. Example Given the NodePort 30100 allocated to the ingress-nginx Service $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 80/TCP ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP and a Kubernetes node with the public IP address 203.0.113.2 (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 a client would reach an Ingress with host: myapp.example.com at http://myapp.example.com:30100 , where the myapp.example.com subdomain resolves to the 203.0.113.2 IP address. Impact on the host system While it may sound tempting to reconfigure the NodePort range using the --service-node-port-range API server flag to include unprivileged ports and be able to expose ports 80 and 443, doing so may result in unexpected issues including (but not limited to) the use of ports otherwise reserved to system daemons and the necessity to grant kube-proxy privileges it may otherwise not require. This practice is therefore discouraged . See the other approaches proposed in this page for alternatives. This approach has a few other limitations one ought to be aware of: Source IP address Services of type NodePort perform source address translation by default. This means the source IP of a HTTP request is always the IP address of the Kubernetes node that received the request from the perspective of NGINX. The recommended way to preserve the source IP in a NodePort setup is to set the value of the externalTrafficPolicy field of the ingress-nginx Service spec to Local ( example ). Warning This setting effectively drops packets sent to Kubernetes nodes which are not running any instance of the NGINX Ingress controller. Consider assigning NGINX Pods to specific nodes in order to control on what nodes the NGINX Ingress controller should be scheduled or not scheduled. Example In a Kubernetes cluster composed of 3 nodes (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 with a ingress-nginx-controller Deployment composed of 2 replicas $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 ingress-nginx-controller-cf9ff8c96-8vvf8 1/1 Running 172.17.0.3 host-3 ingress-nginx-controller-cf9ff8c96-pxsds 1/1 Running 172.17.1.4 host-2 Requests sent to host-2 and host-3 would be forwarded to NGINX and original client's IP would be preserved, while requests to host-1 would get dropped because there is no NGINX replica running on that node. Ingress status Because NodePort Services do not get a LoadBalancerIP assigned by definition, the NGINX Ingress controller does not update the status of Ingress objects it manages . $ kubectl get ingress NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 80 Despite the fact there is no load balancer providing a public IP address to the NGINX Ingress controller, it is possible to force the status update of all managed Ingress objects by setting the externalIPs field of the ingress-nginx Service. Warning There is more to setting externalIPs than just enabling the NGINX Ingress controller to update the status of Ingress objects. Please read about this option in the Services page of official Kubernetes documentation as well as the section about External IPs in this document for more information. Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 one could edit the ingress-nginx Service and add the following field to the object spec spec : externalIPs : - 203.0.113.1 - 203.0.113.2 - 203.0.113.3 which would in turn be reflected on Ingress objects as follows: $ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.1,203.0.113.2,203.0.113.3 80 Redirects As NGINX is not aware of the port translation operated by the NodePort Service , backend applications are responsible for generating redirect URLs that take into account the URL used by external clients, including the NodePort. Example Redirects generated by NGINX, for instance HTTP to HTTPS or domain to www.domain , are generated without NodePort: $ curl -D- http://myapp.example.com:30100 ` HTTP/1.1 308 Permanent Redirect Server: nginx/1.15.2 Location: https://myapp.example.com/ #-> missing NodePort in HTTPS redirect Via the host network \u00b6 In a setup where there is no external load balancer available but using NodePorts is not an option, one can configure ingress-nginx Pods to use the network of the host they run on instead of a dedicated network namespace. The benefit of this approach is that the NGINX Ingress controller can bind ports 80 and 443 directly to Kubernetes nodes' network interfaces, without the extra network translation imposed by NodePort Services. Note This approach does not leverage any Service object to expose the NGINX Ingress controller. If the ingress-nginx Service exists in the target cluster, it is recommended to delete it . This can be achieved by enabling the hostNetwork option in the Pods' spec. template : spec : hostNetwork : true Security considerations Enabling this option exposes every system daemon to the NGINX Ingress controller on any network interface, including the host's loopback. Please evaluate the impact this may have on the security of your system carefully. Example Consider this ingress-nginx-controller Deployment composed of 2 replicas, NGINX Pods inherit from the IP address of their host instead of an internal Pod IP. $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 ingress-nginx-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3 ingress-nginx-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2 One major limitation of this deployment approach is that only a single NGINX Ingress controller Pod may be scheduled on each cluster node, because binding the same port multiple times on the same network interface is technically impossible. Pods that are unschedulable due to such situation fail with the following event: $ kubectl -n ingress-nginx describe pod <unschedulable-ingress-nginx-controller-pod> ... Events: Type Reason From Message ---- ------ ---- ------- Warning FailedScheduling default-scheduler 0/3 nodes are available: 3 node(s) didn't have free ports for the requested pod ports. One way to ensure only schedulable Pods are created is to deploy the NGINX Ingress controller as a DaemonSet instead of a traditional Deployment. Info A DaemonSet schedules exactly one type of Pod per cluster node, masters included, unless a node is configured to repel those Pods . For more information, see DaemonSet . Because most properties of DaemonSet objects are identical to Deployment objects, this documentation page leaves the configuration of the corresponding manifest at the user's discretion. Like with NodePorts, this approach has a few quirks it is important to be aware of. DNS resolution Pods configured with hostNetwork: true do not use the internal DNS resolver (i.e. kube-dns or CoreDNS ), unless their dnsPolicy spec field is set to ClusterFirstWithHostNet . Consider using this setting if NGINX is expected to resolve internal names for any reason. Ingress status Because there is no Service exposing the NGINX Ingress controller in a configuration using the host network, the default --publish-service flag used in standard cloud setups does not apply and the status of all Ingress objects remains blank. $ kubectl get ingress NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 80 Instead, and because bare-metal nodes usually don't have an ExternalIP, one has to enable the --report-node-internal-ip-address flag, which sets the status of all Ingress objects to the internal IP address of all nodes running the NGINX Ingress controller. Example Given a ingress-nginx-controller DaemonSet composed of 2 replicas $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 ingress-nginx-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3 ingress-nginx-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2 the controller sets the status of all Ingress objects it manages to the following value: $ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.2,203.0.113.3 80 Note Alternatively, it is possible to override the address written to Ingress objects using the --publish-status-address flag. See Command line arguments . Using a self-provisioned edge \u00b6 Similarly to cloud environments, this deployment approach requires an edge network component providing a public entrypoint to the Kubernetes cluster. This edge component can be either hardware (e.g. vendor appliance) or software (e.g. HAproxy ) and is usually managed outside of the Kubernetes landscape by operations teams. Such deployment builds upon the NodePort Service described above in Over a NodePort Service , with one significant difference: external clients do not access cluster nodes directly, only the edge component does. This is particularly suitable for private Kubernetes clusters where none of the nodes has a public IP address. On the edge side, the only prerequisite is to dedicate a public IP address that forwards all HTTP traffic to Kubernetes nodes and/or masters. Incoming traffic on TCP ports 80 and 443 is forwarded to the corresponding HTTP and HTTPS NodePort on the target nodes as shown in the diagram below: External IPs \u00b6 Source IP address This method does not allow preserving the source IP of HTTP requests in any manner, it is therefore not recommended to use it despite its apparent simplicity. The externalIPs Service option was previously mentioned in the NodePort section. As per the Services page of the official Kubernetes documentation, the externalIPs option causes kube-proxy to route traffic sent to arbitrary IP addresses and on the Service ports to the endpoints of that Service. These IP addresses must belong to the target node . Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 and the following ingress-nginx NodePort Service $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP One could set the following external IPs in the Service spec, and NGINX would become available on both the NodePort and the Service port: spec : externalIPs : - 203.0.113.2 - 203.0.113.3 $ curl -D- http://myapp.example.com:30100 HTTP/1.1 200 OK Server: nginx/1.15.2 $ curl -D- http://myapp.example.com HTTP/1.1 200 OK Server: nginx/1.15.2 We assume the myapp.example.com subdomain above resolves to both 203.0.113.2 and 203.0.113.3 IP addresses.","title":"Bare-metal considerations"},{"location":"deploy/baremetal/#bare-metal-considerations","text":"In traditional cloud environments, where network load balancers are available on-demand, a single Kubernetes manifest suffices to provide a single point of contact to the NGINX Ingress controller to external clients and, indirectly, to any application running inside the cluster. Bare-metal environments lack this commodity, requiring a slightly different setup to offer the same kind of access to external consumers. The rest of this document describes a few recommended approaches to deploying the NGINX Ingress controller inside a Kubernetes cluster running on bare-metal.","title":"Bare-metal considerations"},{"location":"deploy/baremetal/#a-pure-software-solution-metallb","text":"MetalLB provides a network load-balancer implementation for Kubernetes clusters that do not run on a supported cloud provider, effectively allowing the usage of LoadBalancer Services within any cluster. This section demonstrates how to use the Layer 2 configuration mode of MetalLB together with the NGINX Ingress controller in a Kubernetes cluster that has publicly accessible nodes . In this mode, one node attracts all the traffic for the ingress-nginx Service IP. See Traffic policies for more details. Note The description of other supported configuration modes is off-scope for this document. Warning MetalLB is currently in beta . Read about the Project maturity and make sure you inform yourself by reading the official documentation thoroughly. MetalLB can be deployed either with a simple Kubernetes manifest or with Helm. The rest of this example assumes MetalLB was deployed following the Installation instructions, and that the NGINX Ingress controller was installed using the steps described in the quickstart section of the installation guide . MetalLB requires a pool of IP addresses in order to be able to take ownership of the ingress-nginx Service. This pool can be defined through IPAddressPool objects in the same namespace as the MetalLB controller. This pool of IPs must be dedicated to MetalLB's use, you can't reuse the Kubernetes node IPs or IPs handed out by a DHCP server. Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 After creating the following objects, MetalLB takes ownership of one of the IP addresses in the pool and updates the loadBalancer IP field of the ingress-nginx Service accordingly. --- apiVersion : metallb.io/v1beta1 kind : IPAddressPool metadata : name : default namespace : metallb-system spec : addresses : - 203.0.113.10-203.0.113.15 autoAssign : true --- apiVersion : metallb.io/v1beta1 kind : L2Advertisement metadata : name : default namespace : metallb-system spec : ipAddressPools : - default $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 <none> 80/TCP ingress-nginx LoadBalancer 10.0.220.217 203.0.113.10 80:30100/TCP,443:30101/TCP As soon as MetalLB sets the external IP address of the ingress-nginx LoadBalancer Service, the corresponding entries are created in the iptables NAT table and the node with the selected IP address starts responding to HTTP requests on the ports configured in the LoadBalancer Service: $ curl -D- http://203.0.113.10 -H 'Host: myapp.example.com' HTTP/1.1 200 OK Server: nginx/1.15.2 Tip In order to preserve the source IP address in HTTP requests sent to NGINX, it is necessary to use the Local traffic policy. Traffic policies are described in more details in Traffic policies as well as in the next section.","title":"A pure software solution: MetalLB"},{"location":"deploy/baremetal/#over-a-nodeport-service","text":"Due to its simplicity, this is the setup a user will deploy by default when following the steps described in the installation guide . Info A Service of type NodePort exposes, via the kube-proxy component, the same unprivileged port (default: 30000-32767) on every Kubernetes node, masters included. For more information, see Services . In this configuration, the NGINX container remains isolated from the host network. As a result, it can safely bind to any port, including the standard HTTP ports 80 and 443. However, due to the container namespace isolation, a client located outside the cluster network (e.g. on the public internet) is not able to access Ingress hosts directly on ports 80 and 443. Instead, the external client must append the NodePort allocated to the ingress-nginx Service to HTTP requests. Example Given the NodePort 30100 allocated to the ingress-nginx Service $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 80/TCP ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP and a Kubernetes node with the public IP address 203.0.113.2 (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 a client would reach an Ingress with host: myapp.example.com at http://myapp.example.com:30100 , where the myapp.example.com subdomain resolves to the 203.0.113.2 IP address. Impact on the host system While it may sound tempting to reconfigure the NodePort range using the --service-node-port-range API server flag to include unprivileged ports and be able to expose ports 80 and 443, doing so may result in unexpected issues including (but not limited to) the use of ports otherwise reserved to system daemons and the necessity to grant kube-proxy privileges it may otherwise not require. This practice is therefore discouraged . See the other approaches proposed in this page for alternatives. This approach has a few other limitations one ought to be aware of: Source IP address Services of type NodePort perform source address translation by default. This means the source IP of a HTTP request is always the IP address of the Kubernetes node that received the request from the perspective of NGINX. The recommended way to preserve the source IP in a NodePort setup is to set the value of the externalTrafficPolicy field of the ingress-nginx Service spec to Local ( example ). Warning This setting effectively drops packets sent to Kubernetes nodes which are not running any instance of the NGINX Ingress controller. Consider assigning NGINX Pods to specific nodes in order to control on what nodes the NGINX Ingress controller should be scheduled or not scheduled. Example In a Kubernetes cluster composed of 3 nodes (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 with a ingress-nginx-controller Deployment composed of 2 replicas $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 ingress-nginx-controller-cf9ff8c96-8vvf8 1/1 Running 172.17.0.3 host-3 ingress-nginx-controller-cf9ff8c96-pxsds 1/1 Running 172.17.1.4 host-2 Requests sent to host-2 and host-3 would be forwarded to NGINX and original client's IP would be preserved, while requests to host-1 would get dropped because there is no NGINX replica running on that node. Ingress status Because NodePort Services do not get a LoadBalancerIP assigned by definition, the NGINX Ingress controller does not update the status of Ingress objects it manages . $ kubectl get ingress NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 80 Despite the fact there is no load balancer providing a public IP address to the NGINX Ingress controller, it is possible to force the status update of all managed Ingress objects by setting the externalIPs field of the ingress-nginx Service. Warning There is more to setting externalIPs than just enabling the NGINX Ingress controller to update the status of Ingress objects. Please read about this option in the Services page of official Kubernetes documentation as well as the section about External IPs in this document for more information. Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 one could edit the ingress-nginx Service and add the following field to the object spec spec : externalIPs : - 203.0.113.1 - 203.0.113.2 - 203.0.113.3 which would in turn be reflected on Ingress objects as follows: $ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.1,203.0.113.2,203.0.113.3 80 Redirects As NGINX is not aware of the port translation operated by the NodePort Service , backend applications are responsible for generating redirect URLs that take into account the URL used by external clients, including the NodePort. Example Redirects generated by NGINX, for instance HTTP to HTTPS or domain to www.domain , are generated without NodePort: $ curl -D- http://myapp.example.com:30100 ` HTTP/1.1 308 Permanent Redirect Server: nginx/1.15.2 Location: https://myapp.example.com/ #-> missing NodePort in HTTPS redirect","title":"Over a NodePort Service"},{"location":"deploy/baremetal/#via-the-host-network","text":"In a setup where there is no external load balancer available but using NodePorts is not an option, one can configure ingress-nginx Pods to use the network of the host they run on instead of a dedicated network namespace. The benefit of this approach is that the NGINX Ingress controller can bind ports 80 and 443 directly to Kubernetes nodes' network interfaces, without the extra network translation imposed by NodePort Services. Note This approach does not leverage any Service object to expose the NGINX Ingress controller. If the ingress-nginx Service exists in the target cluster, it is recommended to delete it . This can be achieved by enabling the hostNetwork option in the Pods' spec. template : spec : hostNetwork : true Security considerations Enabling this option exposes every system daemon to the NGINX Ingress controller on any network interface, including the host's loopback. Please evaluate the impact this may have on the security of your system carefully. Example Consider this ingress-nginx-controller Deployment composed of 2 replicas, NGINX Pods inherit from the IP address of their host instead of an internal Pod IP. $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 ingress-nginx-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3 ingress-nginx-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2 One major limitation of this deployment approach is that only a single NGINX Ingress controller Pod may be scheduled on each cluster node, because binding the same port multiple times on the same network interface is technically impossible. Pods that are unschedulable due to such situation fail with the following event: $ kubectl -n ingress-nginx describe pod <unschedulable-ingress-nginx-controller-pod> ... Events: Type Reason From Message ---- ------ ---- ------- Warning FailedScheduling default-scheduler 0/3 nodes are available: 3 node(s) didn't have free ports for the requested pod ports. One way to ensure only schedulable Pods are created is to deploy the NGINX Ingress controller as a DaemonSet instead of a traditional Deployment. Info A DaemonSet schedules exactly one type of Pod per cluster node, masters included, unless a node is configured to repel those Pods . For more information, see DaemonSet . Because most properties of DaemonSet objects are identical to Deployment objects, this documentation page leaves the configuration of the corresponding manifest at the user's discretion. Like with NodePorts, this approach has a few quirks it is important to be aware of. DNS resolution Pods configured with hostNetwork: true do not use the internal DNS resolver (i.e. kube-dns or CoreDNS ), unless their dnsPolicy spec field is set to ClusterFirstWithHostNet . Consider using this setting if NGINX is expected to resolve internal names for any reason. Ingress status Because there is no Service exposing the NGINX Ingress controller in a configuration using the host network, the default --publish-service flag used in standard cloud setups does not apply and the status of all Ingress objects remains blank. $ kubectl get ingress NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 80 Instead, and because bare-metal nodes usually don't have an ExternalIP, one has to enable the --report-node-internal-ip-address flag, which sets the status of all Ingress objects to the internal IP address of all nodes running the NGINX Ingress controller. Example Given a ingress-nginx-controller DaemonSet composed of 2 replicas $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 ingress-nginx-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3 ingress-nginx-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2 the controller sets the status of all Ingress objects it manages to the following value: $ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.2,203.0.113.3 80 Note Alternatively, it is possible to override the address written to Ingress objects using the --publish-status-address flag. See Command line arguments .","title":"Via the host network"},{"location":"deploy/baremetal/#using-a-self-provisioned-edge","text":"Similarly to cloud environments, this deployment approach requires an edge network component providing a public entrypoint to the Kubernetes cluster. This edge component can be either hardware (e.g. vendor appliance) or software (e.g. HAproxy ) and is usually managed outside of the Kubernetes landscape by operations teams. Such deployment builds upon the NodePort Service described above in Over a NodePort Service , with one significant difference: external clients do not access cluster nodes directly, only the edge component does. This is particularly suitable for private Kubernetes clusters where none of the nodes has a public IP address. On the edge side, the only prerequisite is to dedicate a public IP address that forwards all HTTP traffic to Kubernetes nodes and/or masters. Incoming traffic on TCP ports 80 and 443 is forwarded to the corresponding HTTP and HTTPS NodePort on the target nodes as shown in the diagram below:","title":"Using a self-provisioned edge"},{"location":"deploy/baremetal/#external-ips","text":"Source IP address This method does not allow preserving the source IP of HTTP requests in any manner, it is therefore not recommended to use it despite its apparent simplicity. The externalIPs Service option was previously mentioned in the NodePort section. As per the Services page of the official Kubernetes documentation, the externalIPs option causes kube-proxy to route traffic sent to arbitrary IP addresses and on the Service ports to the endpoints of that Service. These IP addresses must belong to the target node . Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is <None>) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 and the following ingress-nginx NodePort Service $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP One could set the following external IPs in the Service spec, and NGINX would become available on both the NodePort and the Service port: spec : externalIPs : - 203.0.113.2 - 203.0.113.3 $ curl -D- http://myapp.example.com:30100 HTTP/1.1 200 OK Server: nginx/1.15.2 $ curl -D- http://myapp.example.com HTTP/1.1 200 OK Server: nginx/1.15.2 We assume the myapp.example.com subdomain above resolves to both 203.0.113.2 and 203.0.113.3 IP addresses.","title":"External IPs"},{"location":"deploy/hardening-guide/","text":"Hardening Guide \u00b6 Overview \u00b6 There are several ways to do hardening and securing of nginx. In this documentation two guides are used, the guides are overlapping in some points: nginx CIS Benchmark cipherlist.eu (one of many forks of the now dead project cipherli.st) This guide describes, what of the different configurations described in those guides is already implemented as default in the nginx implementation of kubernetes ingress, what needs to be configured, what is obsolete due to the fact that the nginx is running as container (the CIS benchmark relates to a non-containerized installation) and what is difficult or not possible. Be aware that this is only a guide and you are responsible for your own implementation. Some of the configurations may lead to have specific clients unable to reach your site or similar consequences. This guide refers to chapters in the CIS Benchmark. For full explanation you should refer to the benchmark document itself Configuration Guide \u00b6 Chapter in CIS benchmark Status Default Action to do if not default 1 Initial Setup 1.1 Installation 1.1.1 Ensure NGINX is installed (Scored) OK done through helm charts / following documentation to deploy nginx ingress 1.1.2 Ensure NGINX is installed from source (Not Scored) OK done through helm charts / following documentation to deploy nginx ingress 1.2 Configure Software Updates 1.2.1 Ensure package manager repositories are properly configured (Not Scored) OK done via helm, nginx version could be overwritten, however compatibility is not ensured then 1.2.2 Ensure the latest software package is installed (Not Scored) ACTION NEEDED done via helm, nginx version could be overwritten, however compatibility is not ensured then Plan for periodic updates 2 Basic Configuration 2.1 Minimize NGINX Modules 2.1.1 Ensure only required modules are installed (Not Scored) OK Already only needed modules are installed, however proposals for further reduction are welcome 2.1.2 Ensure HTTP WebDAV module is not installed (Scored) OK 2.1.3 Ensure modules with gzip functionality are disabled (Scored) OK 2.1.4 Ensure the autoindex module is disabled (Scored) OK No autoindex configs so far in ingress defaults 2.2 Account Security 2.2.1 Ensure that NGINX is run using a non-privileged, dedicated service account (Not Scored) OK Pod configured as user www-data: See this line in helm chart values . Compiled with user www-data: See this line in build script 2.2.2 Ensure the NGINX service account is locked (Scored) OK Docker design ensures this 2.2.3 Ensure the NGINX service account has an invalid shell (Scored) OK Shell is nologin: see this line in build script 2.3 Permissions and Ownership 2.3.1 Ensure NGINX directories and files are owned by root (Scored) OK Obsolete through docker-design and ingress controller needs to update the configs dynamically 2.3.2 Ensure access to NGINX directories and files is restricted (Scored) OK See previous answer 2.3.3 Ensure the NGINX process ID (PID) file is secured (Scored) OK No PID-File due to docker design 2.3.4 Ensure the core dump directory is secured (Not Scored) OK No working_directory configured by default 2.4 Network Configuration 2.4.1 Ensure NGINX only listens for network connections on authorized ports (Not Scored) OK Ensured by automatic nginx.conf configuration 2.4.2 Ensure requests for unknown host names are rejected (Not Scored) OK They are not rejected but send to the \"default backend\" delivering appropriate errors (mostly 404) 2.4.3 Ensure keepalive_timeout is 10 seconds or less, but not 0 (Scored) ACTION NEEDED Default is 75s configure keep-alive to 10 seconds according to this documentation 2.4.4 Ensure send_timeout is set to 10 seconds or less, but not 0 (Scored) RISK TO BE ACCEPTED Not configured, however the nginx default is 60s Not configurable 2.5 Information Disclosure 2.5.1 Ensure server_tokens directive is set to off (Scored) OK server_tokens is configured to off by default 2.5.2 Ensure default error and index.html pages do not reference NGINX (Scored) ACTION NEEDED 404 shows no version at all, 503 and 403 show \"nginx\", which is hardcoded see this line in nginx source code configure custom error pages at least for 403, 404 and 503 and 500 2.5.3 Ensure hidden file serving is disabled (Not Scored) ACTION NEEDED config not set configure a config.server-snippet Snippet, but beware of .well-known challenges or similar. Refer to the benchmark here please 2.5.4 Ensure the NGINX reverse proxy does not enable information disclosure (Scored) ACTION NEEDED hide not configured configure hide-headers with array of \"X-Powered-By\" and \"Server\": according to this documentation 3 Logging 3.1 Ensure detailed logging is enabled (Not Scored) OK nginx ingress has a very detailed log format by default 3.2 Ensure access logging is enabled (Scored) OK Access log is enabled by default 3.3 Ensure error logging is enabled and set to the info logging level (Scored) OK Error log is configured by default. The log level does not matter, because it is all sent to STDOUT anyway 3.4 Ensure log files are rotated (Scored) OBSOLETE Log file handling is not part of the nginx ingress and should be handled separately 3.5 Ensure error logs are sent to a remote syslog server (Not Scored) OBSOLETE See previous answer 3.6 Ensure access logs are sent to a remote syslog server (Not Scored) OBSOLETE See previous answer 3.7 Ensure proxies pass source IP information (Scored) OK Headers are set by default 4 Encryption 4.1 TLS / SSL Configuration 4.1.1 Ensure HTTP is redirected to HTTPS (Scored) OK Redirect to TLS is default 4.1.2 Ensure a trusted certificate and trust chain is installed (Not Scored) ACTION NEEDED For installing certs there are enough manuals in the web. A good way is to use lets encrypt through cert-manager Install proper certificates or use lets encrypt with cert-manager 4.1.3 Ensure private key permissions are restricted (Scored) ACTION NEEDED See previous answer 4.1.4 Ensure only modern TLS protocols are used (Scored) OK/ACTION NEEDED Default is TLS 1.2 + 1.3, while this is okay for CIS Benchmark, cipherlist.eu only recommends 1.3. This may cut off old OS's Set controller.config.ssl-protocols to \"TLSv1.3\" 4.1.5 Disable weak ciphers (Scored) ACTION NEEDED Default ciphers are already good, but cipherlist.eu recommends even stronger ciphers Set controller.config.ssl-ciphers to \"EECDH+AESGCM:EDH+AESGCM\" 4.1.6 Ensure custom Diffie-Hellman parameters are used (Scored) ACTION NEEDED No custom DH parameters are generated Generate dh parameters for each ingress deployment you use - see here for a how to 4.1.7 Ensure Online Certificate Status Protocol (OCSP) stapling is enabled (Scored) ACTION NEEDED Not enabled set via this configuration parameter 4.1.8 Ensure HTTP Strict Transport Security (HSTS) is enabled (Scored) OK HSTS is enabled by default 4.1.9 Ensure HTTP Public Key Pinning is enabled (Not Scored) ACTION NEEDED / RISK TO BE ACCEPTED HKPK not enabled by default If lets encrypt is not used, set correct HPKP header. There are several ways to implement this - with the helm charts it works via controller.add-headers. If lets encrypt is used, this is complicated, a solution here is yet unknown 4.1.10 Ensure upstream server traffic is authenticated with a client certificate (Scored) DEPENDS ON BACKEND Highly dependent on backends, not every backend allows configuring this, can also be mitigated via a service mesh If backend allows it, manual is here 4.1.11 Ensure the upstream traffic server certificate is trusted (Not Scored) DEPENDS ON BACKEND Highly dependent on backends, not every backend allows configuring this, can also be mitigated via a service mesh If backend allows it, see configuration here 4.1.12 Ensure your domain is preloaded (Not Scored) ACTION NEEDED Preload is not active by default Set controller.config.hsts-preload to true 4.1.13 Ensure session resumption is disabled to enable perfect forward security (Scored) OK Session tickets are disabled by default 4.1.14 Ensure HTTP/2.0 is used (Not Scored) OK http2 is set by default 5 Request Filtering and Restrictions 5.1 Access Control 5.1.1 Ensure allow and deny filters limit access to specific IP addresses (Not Scored) OK/ACTION NEEDED Depends on use case, geo ip module is compiled into nginx ingress controller, there are several ways to use it If needed set IP restrictions via annotations or work with config snippets (be careful with lets-encrypt-http-challenge!) 5.1.2 Ensure only whitelisted HTTP methods are allowed (Not Scored) OK/ACTION NEEDED Depends on use case If required it can be set via config snippet 5.2 Request Limits 5.2.1 Ensure timeout values for reading the client header and body are set correctly (Scored) ACTION NEEDED Default timeout is 60s Set via this configuration parameter and respective body equivalent 5.2.2 Ensure the maximum request body size is set correctly (Scored) ACTION NEEDED Default is 1m set via this configuration parameter 5.2.3 Ensure the maximum buffer size for URIs is defined (Scored) ACTION NEEDED Default is 4 8k Set via this configuration parameter 5.2.4 Ensure the number of connections per IP address is limited (Not Scored) OK/ACTION NEEDED No limit set Depends on use case, limit can be set via these annotations 5.2.5 Ensure rate limits by IP address are set (Not Scored) OK/ACTION NEEDED No limit set Depends on use case, limit can be set via these annotations 5.3 Browser Security 5.3.1 Ensure X-Frame-Options header is configured and enabled (Scored) ACTION NEEDED Header not set by default Several ways to implement this - with the helm charts it works via controller.add-headers 5.3.2 Ensure X-Content-Type-Options header is configured and enabled (Scored) ACTION NEEDED See previous answer See previous answer 5.3.3 Ensure the X-XSS-Protection Header is enabled and configured properly (Scored) ACTION NEEDED See previous answer See previous answer 5.3.4 Ensure that Content Security Policy (CSP) is enabled and configured properly (Not Scored) ACTION NEEDED See previous answer See previous answer 5.3.5 Ensure the Referrer Policy is enabled and configured properly (Not Scored) ACTION NEEDED Depends on application. It should be handled in the applications webserver itself, not in the load balancing ingress check backend webserver 6 Mandatory Access Control n/a too high level, depends on backends @media only screen and (min-width: 768px) { td:nth-child(1){ white-space:normal !important; } .md-typeset table:not([class]) td { padding: .2rem .3rem; } }","title":"Hardening guide"},{"location":"deploy/hardening-guide/#hardening-guide","text":"","title":"Hardening Guide"},{"location":"deploy/hardening-guide/#overview","text":"There are several ways to do hardening and securing of nginx. In this documentation two guides are used, the guides are overlapping in some points: nginx CIS Benchmark cipherlist.eu (one of many forks of the now dead project cipherli.st) This guide describes, what of the different configurations described in those guides is already implemented as default in the nginx implementation of kubernetes ingress, what needs to be configured, what is obsolete due to the fact that the nginx is running as container (the CIS benchmark relates to a non-containerized installation) and what is difficult or not possible. Be aware that this is only a guide and you are responsible for your own implementation. Some of the configurations may lead to have specific clients unable to reach your site or similar consequences. This guide refers to chapters in the CIS Benchmark. For full explanation you should refer to the benchmark document itself","title":"Overview"},{"location":"deploy/hardening-guide/#configuration-guide","text":"Chapter in CIS benchmark Status Default Action to do if not default 1 Initial Setup 1.1 Installation 1.1.1 Ensure NGINX is installed (Scored) OK done through helm charts / following documentation to deploy nginx ingress 1.1.2 Ensure NGINX is installed from source (Not Scored) OK done through helm charts / following documentation to deploy nginx ingress 1.2 Configure Software Updates 1.2.1 Ensure package manager repositories are properly configured (Not Scored) OK done via helm, nginx version could be overwritten, however compatibility is not ensured then 1.2.2 Ensure the latest software package is installed (Not Scored) ACTION NEEDED done via helm, nginx version could be overwritten, however compatibility is not ensured then Plan for periodic updates 2 Basic Configuration 2.1 Minimize NGINX Modules 2.1.1 Ensure only required modules are installed (Not Scored) OK Already only needed modules are installed, however proposals for further reduction are welcome 2.1.2 Ensure HTTP WebDAV module is not installed (Scored) OK 2.1.3 Ensure modules with gzip functionality are disabled (Scored) OK 2.1.4 Ensure the autoindex module is disabled (Scored) OK No autoindex configs so far in ingress defaults 2.2 Account Security 2.2.1 Ensure that NGINX is run using a non-privileged, dedicated service account (Not Scored) OK Pod configured as user www-data: See this line in helm chart values . Compiled with user www-data: See this line in build script 2.2.2 Ensure the NGINX service account is locked (Scored) OK Docker design ensures this 2.2.3 Ensure the NGINX service account has an invalid shell (Scored) OK Shell is nologin: see this line in build script 2.3 Permissions and Ownership 2.3.1 Ensure NGINX directories and files are owned by root (Scored) OK Obsolete through docker-design and ingress controller needs to update the configs dynamically 2.3.2 Ensure access to NGINX directories and files is restricted (Scored) OK See previous answer 2.3.3 Ensure the NGINX process ID (PID) file is secured (Scored) OK No PID-File due to docker design 2.3.4 Ensure the core dump directory is secured (Not Scored) OK No working_directory configured by default 2.4 Network Configuration 2.4.1 Ensure NGINX only listens for network connections on authorized ports (Not Scored) OK Ensured by automatic nginx.conf configuration 2.4.2 Ensure requests for unknown host names are rejected (Not Scored) OK They are not rejected but send to the \"default backend\" delivering appropriate errors (mostly 404) 2.4.3 Ensure keepalive_timeout is 10 seconds or less, but not 0 (Scored) ACTION NEEDED Default is 75s configure keep-alive to 10 seconds according to this documentation 2.4.4 Ensure send_timeout is set to 10 seconds or less, but not 0 (Scored) RISK TO BE ACCEPTED Not configured, however the nginx default is 60s Not configurable 2.5 Information Disclosure 2.5.1 Ensure server_tokens directive is set to off (Scored) OK server_tokens is configured to off by default 2.5.2 Ensure default error and index.html pages do not reference NGINX (Scored) ACTION NEEDED 404 shows no version at all, 503 and 403 show \"nginx\", which is hardcoded see this line in nginx source code configure custom error pages at least for 403, 404 and 503 and 500 2.5.3 Ensure hidden file serving is disabled (Not Scored) ACTION NEEDED config not set configure a config.server-snippet Snippet, but beware of .well-known challenges or similar. Refer to the benchmark here please 2.5.4 Ensure the NGINX reverse proxy does not enable information disclosure (Scored) ACTION NEEDED hide not configured configure hide-headers with array of \"X-Powered-By\" and \"Server\": according to this documentation 3 Logging 3.1 Ensure detailed logging is enabled (Not Scored) OK nginx ingress has a very detailed log format by default 3.2 Ensure access logging is enabled (Scored) OK Access log is enabled by default 3.3 Ensure error logging is enabled and set to the info logging level (Scored) OK Error log is configured by default. The log level does not matter, because it is all sent to STDOUT anyway 3.4 Ensure log files are rotated (Scored) OBSOLETE Log file handling is not part of the nginx ingress and should be handled separately 3.5 Ensure error logs are sent to a remote syslog server (Not Scored) OBSOLETE See previous answer 3.6 Ensure access logs are sent to a remote syslog server (Not Scored) OBSOLETE See previous answer 3.7 Ensure proxies pass source IP information (Scored) OK Headers are set by default 4 Encryption 4.1 TLS / SSL Configuration 4.1.1 Ensure HTTP is redirected to HTTPS (Scored) OK Redirect to TLS is default 4.1.2 Ensure a trusted certificate and trust chain is installed (Not Scored) ACTION NEEDED For installing certs there are enough manuals in the web. A good way is to use lets encrypt through cert-manager Install proper certificates or use lets encrypt with cert-manager 4.1.3 Ensure private key permissions are restricted (Scored) ACTION NEEDED See previous answer 4.1.4 Ensure only modern TLS protocols are used (Scored) OK/ACTION NEEDED Default is TLS 1.2 + 1.3, while this is okay for CIS Benchmark, cipherlist.eu only recommends 1.3. This may cut off old OS's Set controller.config.ssl-protocols to \"TLSv1.3\" 4.1.5 Disable weak ciphers (Scored) ACTION NEEDED Default ciphers are already good, but cipherlist.eu recommends even stronger ciphers Set controller.config.ssl-ciphers to \"EECDH+AESGCM:EDH+AESGCM\" 4.1.6 Ensure custom Diffie-Hellman parameters are used (Scored) ACTION NEEDED No custom DH parameters are generated Generate dh parameters for each ingress deployment you use - see here for a how to 4.1.7 Ensure Online Certificate Status Protocol (OCSP) stapling is enabled (Scored) ACTION NEEDED Not enabled set via this configuration parameter 4.1.8 Ensure HTTP Strict Transport Security (HSTS) is enabled (Scored) OK HSTS is enabled by default 4.1.9 Ensure HTTP Public Key Pinning is enabled (Not Scored) ACTION NEEDED / RISK TO BE ACCEPTED HKPK not enabled by default If lets encrypt is not used, set correct HPKP header. There are several ways to implement this - with the helm charts it works via controller.add-headers. If lets encrypt is used, this is complicated, a solution here is yet unknown 4.1.10 Ensure upstream server traffic is authenticated with a client certificate (Scored) DEPENDS ON BACKEND Highly dependent on backends, not every backend allows configuring this, can also be mitigated via a service mesh If backend allows it, manual is here 4.1.11 Ensure the upstream traffic server certificate is trusted (Not Scored) DEPENDS ON BACKEND Highly dependent on backends, not every backend allows configuring this, can also be mitigated via a service mesh If backend allows it, see configuration here 4.1.12 Ensure your domain is preloaded (Not Scored) ACTION NEEDED Preload is not active by default Set controller.config.hsts-preload to true 4.1.13 Ensure session resumption is disabled to enable perfect forward security (Scored) OK Session tickets are disabled by default 4.1.14 Ensure HTTP/2.0 is used (Not Scored) OK http2 is set by default 5 Request Filtering and Restrictions 5.1 Access Control 5.1.1 Ensure allow and deny filters limit access to specific IP addresses (Not Scored) OK/ACTION NEEDED Depends on use case, geo ip module is compiled into nginx ingress controller, there are several ways to use it If needed set IP restrictions via annotations or work with config snippets (be careful with lets-encrypt-http-challenge!) 5.1.2 Ensure only whitelisted HTTP methods are allowed (Not Scored) OK/ACTION NEEDED Depends on use case If required it can be set via config snippet 5.2 Request Limits 5.2.1 Ensure timeout values for reading the client header and body are set correctly (Scored) ACTION NEEDED Default timeout is 60s Set via this configuration parameter and respective body equivalent 5.2.2 Ensure the maximum request body size is set correctly (Scored) ACTION NEEDED Default is 1m set via this configuration parameter 5.2.3 Ensure the maximum buffer size for URIs is defined (Scored) ACTION NEEDED Default is 4 8k Set via this configuration parameter 5.2.4 Ensure the number of connections per IP address is limited (Not Scored) OK/ACTION NEEDED No limit set Depends on use case, limit can be set via these annotations 5.2.5 Ensure rate limits by IP address are set (Not Scored) OK/ACTION NEEDED No limit set Depends on use case, limit can be set via these annotations 5.3 Browser Security 5.3.1 Ensure X-Frame-Options header is configured and enabled (Scored) ACTION NEEDED Header not set by default Several ways to implement this - with the helm charts it works via controller.add-headers 5.3.2 Ensure X-Content-Type-Options header is configured and enabled (Scored) ACTION NEEDED See previous answer See previous answer 5.3.3 Ensure the X-XSS-Protection Header is enabled and configured properly (Scored) ACTION NEEDED See previous answer See previous answer 5.3.4 Ensure that Content Security Policy (CSP) is enabled and configured properly (Not Scored) ACTION NEEDED See previous answer See previous answer 5.3.5 Ensure the Referrer Policy is enabled and configured properly (Not Scored) ACTION NEEDED Depends on application. It should be handled in the applications webserver itself, not in the load balancing ingress check backend webserver 6 Mandatory Access Control n/a too high level, depends on backends @media only screen and (min-width: 768px) { td:nth-child(1){ white-space:normal !important; } .md-typeset table:not([class]) td { padding: .2rem .3rem; } }","title":"Configuration Guide"},{"location":"deploy/rbac/","text":"Role Based Access Control (RBAC) \u00b6 Overview \u00b6 This example applies to ingress-nginx-controllers being deployed in an environment with RBAC enabled. Role Based Access Control is comprised of four layers: ClusterRole - permissions assigned to a role that apply to an entire cluster ClusterRoleBinding - binding a ClusterRole to a specific account Role - permissions assigned to a role that apply to a specific namespace RoleBinding - binding a Role to a specific account In order for RBAC to be applied to an ingress-nginx-controller, that controller should be assigned to a ServiceAccount . That ServiceAccount should be bound to the Role s and ClusterRole s defined for the ingress-nginx-controller. Service Accounts created in this example \u00b6 One ServiceAccount is created in this example, ingress-nginx . Permissions Granted in this example \u00b6 There are two sets of permissions defined in this example. Cluster-wide permissions defined by the ClusterRole named ingress-nginx , and namespace specific permissions defined by the Role named ingress-nginx . Cluster Permissions \u00b6 These permissions are granted in order for the ingress-nginx-controller to be able to function as an ingress across the cluster. These permissions are granted to the ClusterRole named ingress-nginx configmaps , endpoints , nodes , pods , secrets : list, watch nodes : get services , ingresses : get, list, watch events : create, patch ingresses/status : update Namespace Permissions \u00b6 These permissions are granted specific to the ingress-nginx namespace. These permissions are granted to the Role named ingress-nginx configmaps , pods , secrets : get endpoints : get Furthermore to support leader-election, the ingress-nginx-controller needs to have access to a configmap using the resourceName ingress-controller-leader-nginx Note that resourceNames can NOT be used to limit requests using the \u201ccreate\u201d verb because authorizers only have access to information that can be obtained from the request URL, method, and headers (resource names in a \u201ccreate\u201d request are part of the request body). configmaps : get, update (for resourceName ingress-controller-leader-nginx ) configmaps : create This resourceName is the concatenation of the election-id and the ingress-class as defined by the ingress-controller, which defaults to: election-id : ingress-controller-leader ingress-class : nginx resourceName : <election-id>-<ingress-class> Please adapt accordingly if you overwrite either parameter when launching the ingress-nginx-controller. Bindings \u00b6 The ServiceAccount ingress-nginx is bound to the Role ingress-nginx and the ClusterRole ingress-nginx . The serviceAccountName associated with the containers in the deployment must match the serviceAccount. The namespace references in the Deployment metadata, container arguments, and POD_NAMESPACE should be in the ingress-nginx namespace.","title":"Role Based Access Control (RBAC)"},{"location":"deploy/rbac/#role-based-access-control-rbac","text":"","title":"Role Based Access Control (RBAC)"},{"location":"deploy/rbac/#overview","text":"This example applies to ingress-nginx-controllers being deployed in an environment with RBAC enabled. Role Based Access Control is comprised of four layers: ClusterRole - permissions assigned to a role that apply to an entire cluster ClusterRoleBinding - binding a ClusterRole to a specific account Role - permissions assigned to a role that apply to a specific namespace RoleBinding - binding a Role to a specific account In order for RBAC to be applied to an ingress-nginx-controller, that controller should be assigned to a ServiceAccount . That ServiceAccount should be bound to the Role s and ClusterRole s defined for the ingress-nginx-controller.","title":"Overview"},{"location":"deploy/rbac/#service-accounts-created-in-this-example","text":"One ServiceAccount is created in this example, ingress-nginx .","title":"Service Accounts created in this example"},{"location":"deploy/rbac/#permissions-granted-in-this-example","text":"There are two sets of permissions defined in this example. Cluster-wide permissions defined by the ClusterRole named ingress-nginx , and namespace specific permissions defined by the Role named ingress-nginx .","title":"Permissions Granted in this example"},{"location":"deploy/rbac/#cluster-permissions","text":"These permissions are granted in order for the ingress-nginx-controller to be able to function as an ingress across the cluster. These permissions are granted to the ClusterRole named ingress-nginx configmaps , endpoints , nodes , pods , secrets : list, watch nodes : get services , ingresses : get, list, watch events : create, patch ingresses/status : update","title":"Cluster Permissions"},{"location":"deploy/rbac/#namespace-permissions","text":"These permissions are granted specific to the ingress-nginx namespace. These permissions are granted to the Role named ingress-nginx configmaps , pods , secrets : get endpoints : get Furthermore to support leader-election, the ingress-nginx-controller needs to have access to a configmap using the resourceName ingress-controller-leader-nginx Note that resourceNames can NOT be used to limit requests using the \u201ccreate\u201d verb because authorizers only have access to information that can be obtained from the request URL, method, and headers (resource names in a \u201ccreate\u201d request are part of the request body). configmaps : get, update (for resourceName ingress-controller-leader-nginx ) configmaps : create This resourceName is the concatenation of the election-id and the ingress-class as defined by the ingress-controller, which defaults to: election-id : ingress-controller-leader ingress-class : nginx resourceName : <election-id>-<ingress-class> Please adapt accordingly if you overwrite either parameter when launching the ingress-nginx-controller.","title":"Namespace Permissions"},{"location":"deploy/rbac/#bindings","text":"The ServiceAccount ingress-nginx is bound to the Role ingress-nginx and the ClusterRole ingress-nginx . The serviceAccountName associated with the containers in the deployment must match the serviceAccount. The namespace references in the Deployment metadata, container arguments, and POD_NAMESPACE should be in the ingress-nginx namespace.","title":"Bindings"},{"location":"deploy/upgrade/","text":"Upgrading \u00b6 Important No matter the method you use for upgrading, if you use template overrides, make sure your templates are compatible with the new version of ingress-nginx . Without Helm \u00b6 To upgrade your ingress-nginx installation, it should be enough to change the version of the image in the controller Deployment. I.e. if your deployment resource looks like (partial example): kind : Deployment metadata : name : ingress-nginx-controller namespace : ingress-nginx spec : replicas : 1 selector : ... template : metadata : ... spec : containers : - name : ingress-nginx-controller image : registry.k8s.io/ingress-nginx/controller:v1.0.4@sha256:545cff00370f28363dad31e3b59a94ba377854d3a11f18988f5f9e56841ef9ef args : ... simply change the v1.0.4 tag to the version you wish to upgrade to. The easiest way to do this is e.g. (do note you may need to change the name parameter according to your installation): kubectl set image deployment/ingress-nginx-controller \\ controller=registry.k8s.io/ingress-nginx/controller:v1.0.5@sha256:55a1fcda5b7657c372515fe402c3e39ad93aa59f6e4378e82acd99912fe6028d \\ -n ingress-nginx For interactive editing, use kubectl edit deployment ingress-nginx-controller -n ingress-nginx . With Helm \u00b6 If you installed ingress-nginx using the Helm command in the deployment docs so its name is ingress-nginx , you should be able to upgrade using helm upgrade --reuse-values ingress-nginx ingress-nginx/ingress-nginx Migrating from stable/nginx-ingress \u00b6 See detailed steps in the upgrading section of the ingress-nginx chart README .","title":"Upgrade"},{"location":"deploy/upgrade/#upgrading","text":"Important No matter the method you use for upgrading, if you use template overrides, make sure your templates are compatible with the new version of ingress-nginx .","title":"Upgrading"},{"location":"deploy/upgrade/#without-helm","text":"To upgrade your ingress-nginx installation, it should be enough to change the version of the image in the controller Deployment. I.e. if your deployment resource looks like (partial example): kind : Deployment metadata : name : ingress-nginx-controller namespace : ingress-nginx spec : replicas : 1 selector : ... template : metadata : ... spec : containers : - name : ingress-nginx-controller image : registry.k8s.io/ingress-nginx/controller:v1.0.4@sha256:545cff00370f28363dad31e3b59a94ba377854d3a11f18988f5f9e56841ef9ef args : ... simply change the v1.0.4 tag to the version you wish to upgrade to. The easiest way to do this is e.g. (do note you may need to change the name parameter according to your installation): kubectl set image deployment/ingress-nginx-controller \\ controller=registry.k8s.io/ingress-nginx/controller:v1.0.5@sha256:55a1fcda5b7657c372515fe402c3e39ad93aa59f6e4378e82acd99912fe6028d \\ -n ingress-nginx For interactive editing, use kubectl edit deployment ingress-nginx-controller -n ingress-nginx .","title":"Without Helm"},{"location":"deploy/upgrade/#with-helm","text":"If you installed ingress-nginx using the Helm command in the deployment docs so its name is ingress-nginx , you should be able to upgrade using helm upgrade --reuse-values ingress-nginx ingress-nginx/ingress-nginx","title":"With Helm"},{"location":"deploy/upgrade/#migrating-from-stablenginx-ingress","text":"See detailed steps in the upgrading section of the ingress-nginx chart README .","title":"Migrating from stable/nginx-ingress"},{"location":"developer-guide/code-overview/","text":"Ingress NGINX - Code Overview \u00b6 This document provides an overview of Ingress NGINX code. Core Golang code \u00b6 This part of the code is responsible for the main logic of Ingress NGINX. It contains all the logics that parses Ingress Objects , annotations , watches Endpoints and turn them into usable nginx.conf configuration. Core Sync Logics: \u00b6 Ingress-nginx has an internal model of the ingresses, secrets and endpoints in a given cluster. It maintains two copies of that: One copy is the currently running configuration model Second copy is the one generated in response to some changes in the cluster The sync logic diffs the two models and if there's a change it tries to converge the running configuration to the new one. There are static and dynamic configuration changes. All endpoints and certificate changes are handled dynamically by posting the payload to an internal NGINX endpoint that is handled by Lua. The following parts of the code can be found: Entrypoint \u00b6 The main package is responsible for starting ingress-nginx program, which can be found in cmd/nginx directory. Version \u00b6 Is the package of the code responsible for adding version subcommand, and can be found in version directory. Internal code \u00b6 This part of the code contains the internal logics that compose Ingress NGINX Controller, and it's split into: Admission Controller \u00b6 Contains the code of Kubernetes Admission Controller which validates the syntax of ingress objects before accepting it. This code can be found in internal/admission/controller directory. File functions \u00b6 Contains auxiliary codes that deal with files, such as generating the SHA1 checksum of a file, or creating required directories. This code can be found in internal/file directory. Ingress functions \u00b6 Contains all the logics from NGINX Ingress Controller, with some examples being: Expected Golang structures that will be used in templates and other parts of the code - internal/ingress/types.go . supported annotations and its parsing logics - internal/ingress/annotations . reconciliation loops and logics - internal/ingress/controller defaults - define the default struct - internal/ingress/defaults . Error interface and types implementation - internal/ingress/errors Metrics collectors for Prometheus exporting - internal/ingress/metric . Resolver - Extracts information from a controller - internal/ingress/resolver . Ingress Object status publisher - internal/ingress/status . And other parts of the code that will be written in this document in a future. K8s functions \u00b6 Contains helper functions for parsing Kubernetes objects. This part of the code can be found in internal/k8s directory. Networking functions \u00b6 Contains helper functions for networking, such as IPv4 and IPv6 parsing, SSL certificate parsing, etc. This part of the code can be found in internal/net directory. NGINX functions \u00b6 Contains helper function to deal with NGINX, such as verify if it's running and reading it's configuration file parts. This part of the code can be found in internal/nginx directory. Tasks / Queue \u00b6 Contains the functions responsible for the sync queue part of the controller. This part of the code can be found in internal/task directory. Other parts of internal \u00b6 Other parts of internal code might not be covered here, like runtime and watch but they can be added in a future. E2E Test \u00b6 The e2e tests code is in test directory. Other programs \u00b6 Describe here kubectl plugin , dbg , waitshutdown and cover the hack scripts. kubectl plugin \u00b6 It containes kubectl plugin for inspecting your ingress-nginx deployments. This part of code can be found in cmd/plugin directory Detail functions flow and available flow can be found in kubectl-plugin Deploy files \u00b6 This directory contains the yaml deploy files used as examples or references in the docs to deploy Ingress NGINX and other components. Those files are in deploy directory. Helm Chart \u00b6 Used to generate the Helm chart published. Code is in charts/ingress-nginx . Documentation/Website \u00b6 The documentation used to generate the website https://kubernetes.github.io/ingress-nginx/ This code is available in docs and it's main \"language\" is Markdown , used by mkdocs file to generate static pages. Container Images \u00b6 Container images used to run ingress-nginx, or to build the final image. Base Images \u00b6 Contains the Dockerfiles and scripts used to build base images that are used in other parts of the repo. They are present in images repo. Some examples: * nginx - The base NGINX image ingress-nginx uses is not a vanilla NGINX. It bundles many libraries together and it is a job in itself to maintain that and keep things up-to-date. * custom-error-pages - Used on the custom error page examples. There are other images inside this directory. Ingress Controller Image \u00b6 The image used to build the final ingress controller, used in deploy scripts and Helm charts. This is NGINX with some Lua enhancement. We do dynamic certificate, endpoints handling, canary traffic split, custom load balancing etc at this component. One can also add new functionalities using Lua plugin system. The files are in rootfs directory and contains: The Dockerfile nginx config Ingress NGINX Lua Scripts \u00b6 Ingress NGINX uses Lua Scripts to enable features like hot reloading, rate limiting and monitoring. Some are written using the OpenResty helper. The directory containing Lua scripts is rootfs/etc/nginx/lua . Nginx Go template file \u00b6 One of the functions of Ingress NGINX is to turn Ingress objects into nginx.conf file. To do so, the final step is to apply those configurations in nginx.tmpl turning it into a final nginx.conf file.","title":"Code Overview"},{"location":"developer-guide/code-overview/#ingress-nginx-code-overview","text":"This document provides an overview of Ingress NGINX code.","title":"Ingress NGINX - Code Overview"},{"location":"developer-guide/code-overview/#core-golang-code","text":"This part of the code is responsible for the main logic of Ingress NGINX. It contains all the logics that parses Ingress Objects , annotations , watches Endpoints and turn them into usable nginx.conf configuration.","title":"Core Golang code"},{"location":"developer-guide/code-overview/#core-sync-logics","text":"Ingress-nginx has an internal model of the ingresses, secrets and endpoints in a given cluster. It maintains two copies of that: One copy is the currently running configuration model Second copy is the one generated in response to some changes in the cluster The sync logic diffs the two models and if there's a change it tries to converge the running configuration to the new one. There are static and dynamic configuration changes. All endpoints and certificate changes are handled dynamically by posting the payload to an internal NGINX endpoint that is handled by Lua. The following parts of the code can be found:","title":"Core Sync Logics:"},{"location":"developer-guide/code-overview/#entrypoint","text":"The main package is responsible for starting ingress-nginx program, which can be found in cmd/nginx directory.","title":"Entrypoint"},{"location":"developer-guide/code-overview/#version","text":"Is the package of the code responsible for adding version subcommand, and can be found in version directory.","title":"Version"},{"location":"developer-guide/code-overview/#internal-code","text":"This part of the code contains the internal logics that compose Ingress NGINX Controller, and it's split into:","title":"Internal code"},{"location":"developer-guide/code-overview/#admission-controller","text":"Contains the code of Kubernetes Admission Controller which validates the syntax of ingress objects before accepting it. This code can be found in internal/admission/controller directory.","title":"Admission Controller"},{"location":"developer-guide/code-overview/#file-functions","text":"Contains auxiliary codes that deal with files, such as generating the SHA1 checksum of a file, or creating required directories. This code can be found in internal/file directory.","title":"File functions"},{"location":"developer-guide/code-overview/#ingress-functions","text":"Contains all the logics from NGINX Ingress Controller, with some examples being: Expected Golang structures that will be used in templates and other parts of the code - internal/ingress/types.go . supported annotations and its parsing logics - internal/ingress/annotations . reconciliation loops and logics - internal/ingress/controller defaults - define the default struct - internal/ingress/defaults . Error interface and types implementation - internal/ingress/errors Metrics collectors for Prometheus exporting - internal/ingress/metric . Resolver - Extracts information from a controller - internal/ingress/resolver . Ingress Object status publisher - internal/ingress/status . And other parts of the code that will be written in this document in a future.","title":"Ingress functions"},{"location":"developer-guide/code-overview/#k8s-functions","text":"Contains helper functions for parsing Kubernetes objects. This part of the code can be found in internal/k8s directory.","title":"K8s functions"},{"location":"developer-guide/code-overview/#networking-functions","text":"Contains helper functions for networking, such as IPv4 and IPv6 parsing, SSL certificate parsing, etc. This part of the code can be found in internal/net directory.","title":"Networking functions"},{"location":"developer-guide/code-overview/#nginx-functions","text":"Contains helper function to deal with NGINX, such as verify if it's running and reading it's configuration file parts. This part of the code can be found in internal/nginx directory.","title":"NGINX functions"},{"location":"developer-guide/code-overview/#tasks-queue","text":"Contains the functions responsible for the sync queue part of the controller. This part of the code can be found in internal/task directory.","title":"Tasks / Queue"},{"location":"developer-guide/code-overview/#other-parts-of-internal","text":"Other parts of internal code might not be covered here, like runtime and watch but they can be added in a future.","title":"Other parts of internal"},{"location":"developer-guide/code-overview/#e2e-test","text":"The e2e tests code is in test directory.","title":"E2E Test"},{"location":"developer-guide/code-overview/#other-programs","text":"Describe here kubectl plugin , dbg , waitshutdown and cover the hack scripts.","title":"Other programs"},{"location":"developer-guide/code-overview/#kubectl-plugin","text":"It containes kubectl plugin for inspecting your ingress-nginx deployments. This part of code can be found in cmd/plugin directory Detail functions flow and available flow can be found in kubectl-plugin","title":"kubectl plugin"},{"location":"developer-guide/code-overview/#deploy-files","text":"This directory contains the yaml deploy files used as examples or references in the docs to deploy Ingress NGINX and other components. Those files are in deploy directory.","title":"Deploy files"},{"location":"developer-guide/code-overview/#helm-chart","text":"Used to generate the Helm chart published. Code is in charts/ingress-nginx .","title":"Helm Chart"},{"location":"developer-guide/code-overview/#documentationwebsite","text":"The documentation used to generate the website https://kubernetes.github.io/ingress-nginx/ This code is available in docs and it's main \"language\" is Markdown , used by mkdocs file to generate static pages.","title":"Documentation/Website"},{"location":"developer-guide/code-overview/#container-images","text":"Container images used to run ingress-nginx, or to build the final image.","title":"Container Images"},{"location":"developer-guide/code-overview/#base-images","text":"Contains the Dockerfiles and scripts used to build base images that are used in other parts of the repo. They are present in images repo. Some examples: * nginx - The base NGINX image ingress-nginx uses is not a vanilla NGINX. It bundles many libraries together and it is a job in itself to maintain that and keep things up-to-date. * custom-error-pages - Used on the custom error page examples. There are other images inside this directory.","title":"Base Images"},{"location":"developer-guide/code-overview/#ingress-controller-image","text":"The image used to build the final ingress controller, used in deploy scripts and Helm charts. This is NGINX with some Lua enhancement. We do dynamic certificate, endpoints handling, canary traffic split, custom load balancing etc at this component. One can also add new functionalities using Lua plugin system. The files are in rootfs directory and contains: The Dockerfile nginx config","title":"Ingress Controller Image"},{"location":"developer-guide/code-overview/#ingress-nginx-lua-scripts","text":"Ingress NGINX uses Lua Scripts to enable features like hot reloading, rate limiting and monitoring. Some are written using the OpenResty helper. The directory containing Lua scripts is rootfs/etc/nginx/lua .","title":"Ingress NGINX Lua Scripts"},{"location":"developer-guide/code-overview/#nginx-go-template-file","text":"One of the functions of Ingress NGINX is to turn Ingress objects into nginx.conf file. To do so, the final step is to apply those configurations in nginx.tmpl turning it into a final nginx.conf file.","title":"Nginx Go template file"},{"location":"developer-guide/getting-started/","text":"Developing for NGINX Ingress Controller This document explains how to get started with developing for NGINX Ingress controller. For the really new contributors, who want to contribute to the INGRESS-NGINX project, but need help with understanding some basic concepts, that are needed to work with the Kubernetes ingress resource, here is a link to the New Contributors Guide . This guide contains tips on how a http/https request travels, from a browser or a curl command, to the webserver process running inside a container, in a pod, in a Kubernetes cluster, but enters the cluster via a ingress resource. For those who are familiar with those basic networking concepts like routing of a packet with regards to a http request, termination of connection, reverseproxy etc. etc., you can skip this and move on to the sections below. (or read it anyways just for context and also provide feedbacks if any) Prerequisites \u00b6 Install Go 1.14 or later. Note The project uses Go Modules Install Docker (v19.03.0 or later with experimental feature on) Important The majority of make tasks run as docker containers Quick Start \u00b6 Fork the repository Clone the repository to any location in your work station Add a GO111MODULE environment variable with export GO111MODULE=on Run go mod download to install dependencies Local build \u00b6 Start a local Kubernetes cluster using kind , build and deploy the ingress controller make dev-env - If you are working on the v1.x.x version of this controller, and you want to create a cluster with kubernetes version 1.22, then please visit the documentation for kind , and look for how to set a custom image for the kind node (image: kindest/node...), in the kind config file. Testing \u00b6 Run go unit tests make test Run unit-tests for lua code make lua-test Lua tests are located in the directory rootfs/etc/nginx/lua/test Important Test files must follow the naming convention <mytest>_test.lua or it will be ignored Run e2e test suite make kind-e2e-test To limit the scope of the tests to execute, we can use the environment variable FOCUS FOCUS=\"no-auth-locations\" make kind-e2e-test Note The variable FOCUS defines Ginkgo Focused Specs Valid values are defined in the describe definition of the e2e tests like Default Backend The complete list of tests can be found here Custom docker image \u00b6 In some cases, it can be useful to build a docker image and publish such an image to a private or custom registry location. This can be done setting two environment variables, REGISTRY and TAG export TAG=\"dev\" export REGISTRY=\"$USER\" make build image and then publish such version with docker push $REGISTRY/controller:$TAG","title":"Getting Started"},{"location":"developer-guide/getting-started/#prerequisites","text":"Install Go 1.14 or later. Note The project uses Go Modules Install Docker (v19.03.0 or later with experimental feature on) Important The majority of make tasks run as docker containers","title":"Prerequisites"},{"location":"developer-guide/getting-started/#quick-start","text":"Fork the repository Clone the repository to any location in your work station Add a GO111MODULE environment variable with export GO111MODULE=on Run go mod download to install dependencies","title":"Quick Start"},{"location":"developer-guide/getting-started/#local-build","text":"Start a local Kubernetes cluster using kind , build and deploy the ingress controller make dev-env - If you are working on the v1.x.x version of this controller, and you want to create a cluster with kubernetes version 1.22, then please visit the documentation for kind , and look for how to set a custom image for the kind node (image: kindest/node...), in the kind config file.","title":"Local build"},{"location":"developer-guide/getting-started/#testing","text":"Run go unit tests make test Run unit-tests for lua code make lua-test Lua tests are located in the directory rootfs/etc/nginx/lua/test Important Test files must follow the naming convention <mytest>_test.lua or it will be ignored Run e2e test suite make kind-e2e-test To limit the scope of the tests to execute, we can use the environment variable FOCUS FOCUS=\"no-auth-locations\" make kind-e2e-test Note The variable FOCUS defines Ginkgo Focused Specs Valid values are defined in the describe definition of the e2e tests like Default Backend The complete list of tests can be found here","title":"Testing"},{"location":"developer-guide/getting-started/#custom-docker-image","text":"In some cases, it can be useful to build a docker image and publish such an image to a private or custom registry location. This can be done setting two environment variables, REGISTRY and TAG export TAG=\"dev\" export REGISTRY=\"$USER\" make build image and then publish such version with docker push $REGISTRY/controller:$TAG","title":"Custom docker image"},{"location":"enhancements/","text":"Kubernetes Enhancement Proposals (KEPs) \u00b6 A Kubernetes Enhancement Proposal (KEP) is a way to propose, communicate and coordinate on new efforts for the Kubernetes project. For this reason, the ingress-nginx project is adopting it. Quick start for the KEP process \u00b6 Follow the process outlined in the KEP template Do I have to use the KEP process? \u00b6 No... but we hope that you will. Over time having a rich set of KEPs in one place will make it easier for people to track what is going on in the community and find a structured historic record. KEPs are only required when the changes are wide ranging and impact most of the project. Why would I want to use the KEP process? \u00b6 Our aim with KEPs is to clearly communicate new efforts to the Kubernetes contributor community. As such, we want to build a well curated set of clear proposals in a common format with useful metadata. Benefits to KEP users (in the limit): Exposure on a kubernetes blessed web site that is findable via web search engines. Cross indexing of KEPs so that users can find connections and the current status of any KEP. A clear process with approvers and reviewers for making decisions. This will lead to more structured decisions that stick as there is a discoverable record around the decisions. We are inspired by IETF RFCs, Python PEPs, and Rust RFCs.","title":"Kubernetes Enhancement Proposals (KEPs)"},{"location":"enhancements/#kubernetes-enhancement-proposals-keps","text":"A Kubernetes Enhancement Proposal (KEP) is a way to propose, communicate and coordinate on new efforts for the Kubernetes project. For this reason, the ingress-nginx project is adopting it.","title":"Kubernetes Enhancement Proposals (KEPs)"},{"location":"enhancements/#quick-start-for-the-kep-process","text":"Follow the process outlined in the KEP template","title":"Quick start for the KEP process"},{"location":"enhancements/#do-i-have-to-use-the-kep-process","text":"No... but we hope that you will. Over time having a rich set of KEPs in one place will make it easier for people to track what is going on in the community and find a structured historic record. KEPs are only required when the changes are wide ranging and impact most of the project.","title":"Do I have to use the KEP process?"},{"location":"enhancements/#why-would-i-want-to-use-the-kep-process","text":"Our aim with KEPs is to clearly communicate new efforts to the Kubernetes contributor community. As such, we want to build a well curated set of clear proposals in a common format with useful metadata. Benefits to KEP users (in the limit): Exposure on a kubernetes blessed web site that is findable via web search engines. Cross indexing of KEPs so that users can find connections and the current status of any KEP. A clear process with approvers and reviewers for making decisions. This will lead to more structured decisions that stick as there is a discoverable record around the decisions. We are inspired by IETF RFCs, Python PEPs, and Rust RFCs.","title":"Why would I want to use the KEP process?"},{"location":"enhancements/20190724-only-dynamic-ssl/","text":"Remove static SSL configuration mode \u00b6 Table of Contents \u00b6 Summary Motivation Goals Non-Goals Proposal Implementation Details/Notes/Constraints Drawbacks Alternatives Summary \u00b6 Since release 0.19.0 is possible to configure SSL certificates without the need of NGINX reloads (thanks to lua) and after release 0.24.0 the default enabled mode is dynamic. Motivation \u00b6 The static configuration implies reloads, something that affects the majority of the users. Goals \u00b6 Deprecation of the flag --enable-dynamic-certificates . Cleanup of the codebase. Non-Goals \u00b6 Features related to certificate authentication are not changed in any way. Proposal \u00b6 Remove static SSL configuration Implementation Details/Notes/Constraints \u00b6 Deprecate the flag Move the directives ssl_certificate and ssl_certificate_key from each server block to the http section. These settings are required to avoid NGINX errors in the logs. Remove any action of the flag --enable-dynamic-certificates Drawbacks \u00b6 Alternatives \u00b6 Keep both implementations","title":"Remove static SSL configuration mode"},{"location":"enhancements/20190724-only-dynamic-ssl/#remove-static-ssl-configuration-mode","text":"","title":"Remove static SSL configuration mode"},{"location":"enhancements/20190724-only-dynamic-ssl/#table-of-contents","text":"Summary Motivation Goals Non-Goals Proposal Implementation Details/Notes/Constraints Drawbacks Alternatives","title":"Table of Contents"},{"location":"enhancements/20190724-only-dynamic-ssl/#summary","text":"Since release 0.19.0 is possible to configure SSL certificates without the need of NGINX reloads (thanks to lua) and after release 0.24.0 the default enabled mode is dynamic.","title":"Summary"},{"location":"enhancements/20190724-only-dynamic-ssl/#motivation","text":"The static configuration implies reloads, something that affects the majority of the users.","title":"Motivation"},{"location":"enhancements/20190724-only-dynamic-ssl/#goals","text":"Deprecation of the flag --enable-dynamic-certificates . Cleanup of the codebase.","title":"Goals"},{"location":"enhancements/20190724-only-dynamic-ssl/#non-goals","text":"Features related to certificate authentication are not changed in any way.","title":"Non-Goals"},{"location":"enhancements/20190724-only-dynamic-ssl/#proposal","text":"Remove static SSL configuration","title":"Proposal"},{"location":"enhancements/20190724-only-dynamic-ssl/#implementation-detailsnotesconstraints","text":"Deprecate the flag Move the directives ssl_certificate and ssl_certificate_key from each server block to the http section. These settings are required to avoid NGINX errors in the logs. Remove any action of the flag --enable-dynamic-certificates","title":"Implementation Details/Notes/Constraints"},{"location":"enhancements/20190724-only-dynamic-ssl/#drawbacks","text":"","title":"Drawbacks"},{"location":"enhancements/20190724-only-dynamic-ssl/#alternatives","text":"Keep both implementations","title":"Alternatives"},{"location":"enhancements/20190815-zone-aware-routing/","text":"Availability zone aware routing \u00b6 Table of Contents \u00b6 Availability zone aware routing Table of Contents Summary Motivation Goals Non-Goals Proposal Implementation History Drawbacks [optional] Summary \u00b6 Teach ingress-nginx about availability zones where endpoints are running in. This way ingress-nginx pod will do its best to proxy to zone-local endpoint. Motivation \u00b6 When users run their services across multiple availability zones they usually pay for egress traffic between zones. Providers such as GCP, and Amazon EC2 usually charge extra for this feature. ingress-nginx when picking an endpoint to route request to does not consider whether the endpoint is in a different zone or the same one. That means it's at least equally likely that it will pick an endpoint from another zone and proxy the request to it. In this situation response from the endpoint to the ingress-nginx pod is considered inter-zone traffic and usually costs extra money. At the time of this writing, GCP charges $0.01 per GB of inter-zone egress traffic according to https://cloud.google.com/compute/network-pricing. According to https://datapath.io/resources/blog/what-are-aws-data-transfer-costs-and-how-to-minimize-them/ Amazon also charges the same amount of money as GCP for cross-zone, egress traffic. This can be a lot of money depending on once's traffic. By teaching ingress-nginx about zones we can eliminate or at least decrease this cost. Arguably inter-zone network latency should also be better than cross-zone. Goals \u00b6 Given a regional cluster running ingress-nginx, ingress-nginx should do best-effort to pick a zone-local endpoint when proxying This should not impact canary feature ingress-nginx should be able to operate successfully if there are no zonal endpoints Non-Goals \u00b6 This feature inherently assumes that endpoints are distributed across zones in a way that they can handle all the traffic from ingress-nginx pod(s) in that zone This feature will be relying on https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#failure-domainbetakubernetesiozone, it is not this KEP's goal to support other cases Proposal \u00b6 The idea here is to have the controller part of ingress-nginx (1) detect what zone its current pod is running in and (2) detect the zone for every endpoint it knows about. After that, it will post that data as part of endpoints to Lua land. When picking an endpoint, the Lua balancer will try to pick zone-local endpoint first and if there is no zone-local endpoint then it will fall back to current behavior. Initially, this feature should be optional since it is going to make it harder to reason about the load balancing and not everyone might want that. How does controller know what zone it runs in? We can have the pod spec pass the node name using downward API as an environment variable. Upon startup, the controller can get node details from the API based on the node name. Once the node details are obtained we can extract the zone from the failure-domain.beta.kubernetes.io/zone annotation. Then we can pass that value to Lua land through Nginx configuration when loading lua_ingress.lua module in init_by_lua phase. How do we extract zones for endpoints? We can have the controller watch create and update events on nodes in the entire cluster and based on that keep the map of nodes to zones in the memory. And when we generate endpoints list, we can access node name using .subsets.addresses[i].nodeName and based on that fetch zone from the map in memory and store it as a field on the endpoint. This solution assumes failure-domain.beta.kubernetes.io/zone annotation does not change until the end of the node's life. Otherwise, we have to watch update events as well on the nodes and that'll add even more overhead. Alternatively, we can get the list of nodes only when there's no node in the memory for the given node name. This is probably a better solution because then we would avoid watching for API changes on node resources. We can eagerly fetch all the nodes and build node name to zone mapping on start. From there on, it will sync during endpoint building in the main event loop if there's no existing entry for the node of an endpoint. This means an extra API call in case cluster has expanded. How do we make sure we do our best to choose zone-local endpoint? This will be done on the Lua side. For every backend, we will initialize two balancer instances: (1) with all endpoints (2) with all endpoints corresponding to the current zone for the backend. Then given the request once we choose what backend needs to serve the request, we will first try to use a zonal balancer for that backend. If a zonal balancer does not exist (i.e. there's no zonal endpoint) then we will use a general balancer. In case of zonal outages, we assume that the readiness probe will fail and the controller will see no endpoints for the backend and therefore we will use a general balancer. We can enable the feature using a configmap setting. Doing it this way makes it easier to rollback in case of a problem. Implementation History \u00b6 initial version of KEP is shipped proposal and implementation details are done Drawbacks [optional] \u00b6 More load on the Kubernetes API server.","title":"Availability zone aware routing"},{"location":"enhancements/20190815-zone-aware-routing/#availability-zone-aware-routing","text":"","title":"Availability zone aware routing"},{"location":"enhancements/20190815-zone-aware-routing/#table-of-contents","text":"Availability zone aware routing Table of Contents Summary Motivation Goals Non-Goals Proposal Implementation History Drawbacks [optional]","title":"Table of Contents"},{"location":"enhancements/20190815-zone-aware-routing/#summary","text":"Teach ingress-nginx about availability zones where endpoints are running in. This way ingress-nginx pod will do its best to proxy to zone-local endpoint.","title":"Summary"},{"location":"enhancements/20190815-zone-aware-routing/#motivation","text":"When users run their services across multiple availability zones they usually pay for egress traffic between zones. Providers such as GCP, and Amazon EC2 usually charge extra for this feature. ingress-nginx when picking an endpoint to route request to does not consider whether the endpoint is in a different zone or the same one. That means it's at least equally likely that it will pick an endpoint from another zone and proxy the request to it. In this situation response from the endpoint to the ingress-nginx pod is considered inter-zone traffic and usually costs extra money. At the time of this writing, GCP charges $0.01 per GB of inter-zone egress traffic according to https://cloud.google.com/compute/network-pricing. According to https://datapath.io/resources/blog/what-are-aws-data-transfer-costs-and-how-to-minimize-them/ Amazon also charges the same amount of money as GCP for cross-zone, egress traffic. This can be a lot of money depending on once's traffic. By teaching ingress-nginx about zones we can eliminate or at least decrease this cost. Arguably inter-zone network latency should also be better than cross-zone.","title":"Motivation"},{"location":"enhancements/20190815-zone-aware-routing/#goals","text":"Given a regional cluster running ingress-nginx, ingress-nginx should do best-effort to pick a zone-local endpoint when proxying This should not impact canary feature ingress-nginx should be able to operate successfully if there are no zonal endpoints","title":"Goals"},{"location":"enhancements/20190815-zone-aware-routing/#non-goals","text":"This feature inherently assumes that endpoints are distributed across zones in a way that they can handle all the traffic from ingress-nginx pod(s) in that zone This feature will be relying on https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#failure-domainbetakubernetesiozone, it is not this KEP's goal to support other cases","title":"Non-Goals"},{"location":"enhancements/20190815-zone-aware-routing/#proposal","text":"The idea here is to have the controller part of ingress-nginx (1) detect what zone its current pod is running in and (2) detect the zone for every endpoint it knows about. After that, it will post that data as part of endpoints to Lua land. When picking an endpoint, the Lua balancer will try to pick zone-local endpoint first and if there is no zone-local endpoint then it will fall back to current behavior. Initially, this feature should be optional since it is going to make it harder to reason about the load balancing and not everyone might want that. How does controller know what zone it runs in? We can have the pod spec pass the node name using downward API as an environment variable. Upon startup, the controller can get node details from the API based on the node name. Once the node details are obtained we can extract the zone from the failure-domain.beta.kubernetes.io/zone annotation. Then we can pass that value to Lua land through Nginx configuration when loading lua_ingress.lua module in init_by_lua phase. How do we extract zones for endpoints? We can have the controller watch create and update events on nodes in the entire cluster and based on that keep the map of nodes to zones in the memory. And when we generate endpoints list, we can access node name using .subsets.addresses[i].nodeName and based on that fetch zone from the map in memory and store it as a field on the endpoint. This solution assumes failure-domain.beta.kubernetes.io/zone annotation does not change until the end of the node's life. Otherwise, we have to watch update events as well on the nodes and that'll add even more overhead. Alternatively, we can get the list of nodes only when there's no node in the memory for the given node name. This is probably a better solution because then we would avoid watching for API changes on node resources. We can eagerly fetch all the nodes and build node name to zone mapping on start. From there on, it will sync during endpoint building in the main event loop if there's no existing entry for the node of an endpoint. This means an extra API call in case cluster has expanded. How do we make sure we do our best to choose zone-local endpoint? This will be done on the Lua side. For every backend, we will initialize two balancer instances: (1) with all endpoints (2) with all endpoints corresponding to the current zone for the backend. Then given the request once we choose what backend needs to serve the request, we will first try to use a zonal balancer for that backend. If a zonal balancer does not exist (i.e. there's no zonal endpoint) then we will use a general balancer. In case of zonal outages, we assume that the readiness probe will fail and the controller will see no endpoints for the backend and therefore we will use a general balancer. We can enable the feature using a configmap setting. Doing it this way makes it easier to rollback in case of a problem.","title":"Proposal"},{"location":"enhancements/20190815-zone-aware-routing/#implementation-history","text":"initial version of KEP is shipped proposal and implementation details are done","title":"Implementation History"},{"location":"enhancements/20190815-zone-aware-routing/#drawbacks-optional","text":"More load on the Kubernetes API server.","title":"Drawbacks [optional]"},{"location":"enhancements/YYYYMMDD-kep-template/","text":"Title \u00b6 This is the title of the KEP. Keep it simple and descriptive. A good title can help communicate what the KEP is and should be considered as part of any review. The title should be lowercased and spaces/punctuation should be replaced with - . To get started with this template: Make a copy of this template. Create a copy of this template and name it YYYYMMDD-my-title.md , where YYYYMMDD is the date the KEP was first drafted. Fill out the \"overview\" sections. This includes the Summary and Motivation sections. These should be easy if you've preflighted the idea of the KEP in an issue. Create a PR. Assign it to folks that are sponsoring this process. Create an issue When filing an enhancement tracking issue, please ensure to complete all fields in the template. Merge early. Avoid getting hung up on specific details and instead aim to get the goal of the KEP merged quickly. The best way to do this is to just start with the \"Overview\" sections and fill out details incrementally in follow on PRs. View anything marked as a provisional as a working document and subject to change. Aim for single topic PRs to keep discussions focused. If you disagree with what is already in a document, open a new PR with suggested changes. The canonical place for the latest set of instructions (and the likely source of this file) is here . The Metadata section above is intended to support the creation of tooling around the KEP process. This will be a YAML section that is fenced as a code block. See the KEP process for details on each of these items. Table of Contents \u00b6 A table of contents is helpful for quickly jumping to sections of a KEP and for highlighting any additional information provided beyond the standard KEP template. Ensure the TOC is wrapped with <!-- toc --&rt;<!-- /toc --&rt; tags, and then generate with hack/update-toc.sh . Summary Motivation Goals Non-Goals Proposal User Stories [optional] Story 1 Story 2 Implementation Details/Notes/Constraints [optional] Risks and Mitigations Design Details Test Plan Removing a deprecated flag Implementation History Drawbacks [optional] Alternatives [optional] Summary \u00b6 The Summary section is incredibly important for producing high quality user-focused documentation such as release notes or a development roadmap. It should be possible to collect this information before implementation begins in order to avoid requiring implementers to split their attention between writing release notes and implementing the feature itself. A good summary is probably at least a paragraph in length. Motivation \u00b6 This section is for explicitly listing the motivation, goals and non-goals of this KEP. Describe why the change is important and the benefits to users. The motivation section can optionally provide links to experience reports to demonstrate the interest in a KEP within the wider Kubernetes community. Goals \u00b6 List the specific goals of the KEP. How will we know that this has succeeded? Non-Goals \u00b6 What is out of scope for this KEP? Listing non-goals helps to focus discussion and make progress. Proposal \u00b6 This is where we get down to the nitty gritty of what the proposal actually is. User Stories [optional] \u00b6 Detail the things that people will be able to do if this KEP is implemented. Include as much detail as possible so that people can understand the \"how\" of the system. The goal here is to make this feel real for users without getting bogged down. Story 1 \u00b6 Story 2 \u00b6 Implementation Details/Notes/Constraints [optional] \u00b6 What are the caveats to the implementation? What are some important details that didn't come across above. Go in to as much detail as necessary here. This might be a good place to talk about core concepts and how they relate. Risks and Mitigations \u00b6 What are the risks of this proposal and how do we mitigate. Think broadly. For example, consider both security and how this will impact the larger kubernetes ecosystem. How will security be reviewed and by whom? How will UX be reviewed and by whom? Consider including folks that also work outside project. Design Details \u00b6 Test Plan \u00b6 Note: Section not required until targeted at a release. Consider the following in developing a test plan for this enhancement: Will there be e2e and integration tests, in addition to unit tests? How will it be tested in isolation vs with other components? No need to outline all of the test cases, just the general strategy. Anything that would count as tricky in the implementation and anything particularly challenging to test should be called out. All code is expected to have adequate tests (eventually with coverage expectations). Please adhere to the Kubernetes testing guidelines when drafting this test plan. Removing a deprecated flag \u00b6 Announce deprecation and support policy of the existing flag Two versions passed since introducing the functionality which deprecates the flag (to address version skew) Address feedback on usage/changed behavior, provided on GitHub issues Deprecate the flag Implementation History \u00b6 Major milestones in the life cycle of a KEP should be tracked in Implementation History . Major milestones might include the Summary and Motivation sections being merged signaling acceptance the Proposal section being merged signaling agreement on a proposed design the date implementation started the first Kubernetes release where an initial version of the KEP was available the version of Kubernetes where the KEP graduated to general availability when the KEP was retired or superseded Drawbacks [optional] \u00b6 Why should this KEP not be implemented. Alternatives [optional] \u00b6 Similar to the Drawbacks section the Alternatives section is used to highlight and record other possible approaches to delivering the value proposed by a KEP.","title":"KEP Template"},{"location":"enhancements/YYYYMMDD-kep-template/#title","text":"This is the title of the KEP. Keep it simple and descriptive. A good title can help communicate what the KEP is and should be considered as part of any review. The title should be lowercased and spaces/punctuation should be replaced with - . To get started with this template: Make a copy of this template. Create a copy of this template and name it YYYYMMDD-my-title.md , where YYYYMMDD is the date the KEP was first drafted. Fill out the \"overview\" sections. This includes the Summary and Motivation sections. These should be easy if you've preflighted the idea of the KEP in an issue. Create a PR. Assign it to folks that are sponsoring this process. Create an issue When filing an enhancement tracking issue, please ensure to complete all fields in the template. Merge early. Avoid getting hung up on specific details and instead aim to get the goal of the KEP merged quickly. The best way to do this is to just start with the \"Overview\" sections and fill out details incrementally in follow on PRs. View anything marked as a provisional as a working document and subject to change. Aim for single topic PRs to keep discussions focused. If you disagree with what is already in a document, open a new PR with suggested changes. The canonical place for the latest set of instructions (and the likely source of this file) is here . The Metadata section above is intended to support the creation of tooling around the KEP process. This will be a YAML section that is fenced as a code block. See the KEP process for details on each of these items.","title":"Title"},{"location":"enhancements/YYYYMMDD-kep-template/#table-of-contents","text":"A table of contents is helpful for quickly jumping to sections of a KEP and for highlighting any additional information provided beyond the standard KEP template. Ensure the TOC is wrapped with <!-- toc --&rt;<!-- /toc --&rt; tags, and then generate with hack/update-toc.sh . Summary Motivation Goals Non-Goals Proposal User Stories [optional] Story 1 Story 2 Implementation Details/Notes/Constraints [optional] Risks and Mitigations Design Details Test Plan Removing a deprecated flag Implementation History Drawbacks [optional] Alternatives [optional]","title":"Table of Contents"},{"location":"enhancements/YYYYMMDD-kep-template/#summary","text":"The Summary section is incredibly important for producing high quality user-focused documentation such as release notes or a development roadmap. It should be possible to collect this information before implementation begins in order to avoid requiring implementers to split their attention between writing release notes and implementing the feature itself. A good summary is probably at least a paragraph in length.","title":"Summary"},{"location":"enhancements/YYYYMMDD-kep-template/#motivation","text":"This section is for explicitly listing the motivation, goals and non-goals of this KEP. Describe why the change is important and the benefits to users. The motivation section can optionally provide links to experience reports to demonstrate the interest in a KEP within the wider Kubernetes community.","title":"Motivation"},{"location":"enhancements/YYYYMMDD-kep-template/#goals","text":"List the specific goals of the KEP. How will we know that this has succeeded?","title":"Goals"},{"location":"enhancements/YYYYMMDD-kep-template/#non-goals","text":"What is out of scope for this KEP? Listing non-goals helps to focus discussion and make progress.","title":"Non-Goals"},{"location":"enhancements/YYYYMMDD-kep-template/#proposal","text":"This is where we get down to the nitty gritty of what the proposal actually is.","title":"Proposal"},{"location":"enhancements/YYYYMMDD-kep-template/#user-stories-optional","text":"Detail the things that people will be able to do if this KEP is implemented. Include as much detail as possible so that people can understand the \"how\" of the system. The goal here is to make this feel real for users without getting bogged down.","title":"User Stories [optional]"},{"location":"enhancements/YYYYMMDD-kep-template/#story-1","text":"","title":"Story 1"},{"location":"enhancements/YYYYMMDD-kep-template/#story-2","text":"","title":"Story 2"},{"location":"enhancements/YYYYMMDD-kep-template/#implementation-detailsnotesconstraints-optional","text":"What are the caveats to the implementation? What are some important details that didn't come across above. Go in to as much detail as necessary here. This might be a good place to talk about core concepts and how they relate.","title":"Implementation Details/Notes/Constraints [optional]"},{"location":"enhancements/YYYYMMDD-kep-template/#risks-and-mitigations","text":"What are the risks of this proposal and how do we mitigate. Think broadly. For example, consider both security and how this will impact the larger kubernetes ecosystem. How will security be reviewed and by whom? How will UX be reviewed and by whom? Consider including folks that also work outside project.","title":"Risks and Mitigations"},{"location":"enhancements/YYYYMMDD-kep-template/#design-details","text":"","title":"Design Details"},{"location":"enhancements/YYYYMMDD-kep-template/#test-plan","text":"Note: Section not required until targeted at a release. Consider the following in developing a test plan for this enhancement: Will there be e2e and integration tests, in addition to unit tests? How will it be tested in isolation vs with other components? No need to outline all of the test cases, just the general strategy. Anything that would count as tricky in the implementation and anything particularly challenging to test should be called out. All code is expected to have adequate tests (eventually with coverage expectations). Please adhere to the Kubernetes testing guidelines when drafting this test plan.","title":"Test Plan"},{"location":"enhancements/YYYYMMDD-kep-template/#removing-a-deprecated-flag","text":"Announce deprecation and support policy of the existing flag Two versions passed since introducing the functionality which deprecates the flag (to address version skew) Address feedback on usage/changed behavior, provided on GitHub issues Deprecate the flag","title":"Removing a deprecated flag"},{"location":"enhancements/YYYYMMDD-kep-template/#implementation-history","text":"Major milestones in the life cycle of a KEP should be tracked in Implementation History . Major milestones might include the Summary and Motivation sections being merged signaling acceptance the Proposal section being merged signaling agreement on a proposed design the date implementation started the first Kubernetes release where an initial version of the KEP was available the version of Kubernetes where the KEP graduated to general availability when the KEP was retired or superseded","title":"Implementation History"},{"location":"enhancements/YYYYMMDD-kep-template/#drawbacks-optional","text":"Why should this KEP not be implemented.","title":"Drawbacks [optional]"},{"location":"enhancements/YYYYMMDD-kep-template/#alternatives-optional","text":"Similar to the Drawbacks section the Alternatives section is used to highlight and record other possible approaches to delivering the value proposed by a KEP.","title":"Alternatives [optional]"},{"location":"examples/","text":"Ingress examples \u00b6 This directory contains a catalog of examples on how to run, configure and scale Ingress. Please review the prerequisites before trying them. The examples on these pages include the spec.ingressClassName field which replaces the deprecated kubernetes.io/ingress.class: nginx annotation. Users of ingress-nginx < 1.0.0 (Helm chart < 4.0.0) should use the legacy documentation . For more information, check out the Migration to apiVersion networking.k8s.io/v1 guide. Category Name Description Complexity Level Apps Docker Registry TODO TODO Auth Basic authentication password protect your website Intermediate Auth Client certificate authentication secure your website with client certificate authentication Intermediate Auth External authentication plugin defer to an external authentication service Intermediate Auth OAuth external auth TODO TODO Customization Configuration snippets customize nginx location configuration using annotations Advanced Customization Custom configuration TODO TODO Customization Custom DH parameters for perfect forward secrecy TODO TODO Customization Custom errors serve custom error pages from the default backend Intermediate Customization Custom headers set custom headers before sending traffic to backends Advanced Customization External authentication with response header propagation TODO TODO Customization Sysctl tuning TODO TODO Features Rewrite TODO TODO Features Session stickiness route requests consistently to the same endpoint Advanced Scaling Static IP a single ingress gets a single static IP Intermediate TLS Multi TLS certificate termination TODO TODO TLS TLS termination TODO TODO","title":"Introduction"},{"location":"examples/#ingress-examples","text":"This directory contains a catalog of examples on how to run, configure and scale Ingress. Please review the prerequisites before trying them. The examples on these pages include the spec.ingressClassName field which replaces the deprecated kubernetes.io/ingress.class: nginx annotation. Users of ingress-nginx < 1.0.0 (Helm chart < 4.0.0) should use the legacy documentation . For more information, check out the Migration to apiVersion networking.k8s.io/v1 guide. Category Name Description Complexity Level Apps Docker Registry TODO TODO Auth Basic authentication password protect your website Intermediate Auth Client certificate authentication secure your website with client certificate authentication Intermediate Auth External authentication plugin defer to an external authentication service Intermediate Auth OAuth external auth TODO TODO Customization Configuration snippets customize nginx location configuration using annotations Advanced Customization Custom configuration TODO TODO Customization Custom DH parameters for perfect forward secrecy TODO TODO Customization Custom errors serve custom error pages from the default backend Intermediate Customization Custom headers set custom headers before sending traffic to backends Advanced Customization External authentication with response header propagation TODO TODO Customization Sysctl tuning TODO TODO Features Rewrite TODO TODO Features Session stickiness route requests consistently to the same endpoint Advanced Scaling Static IP a single ingress gets a single static IP Intermediate TLS Multi TLS certificate termination TODO TODO TLS TLS termination TODO TODO","title":"Ingress examples"},{"location":"examples/PREREQUISITES/","text":"Prerequisites \u00b6 Many of the examples in this directory have common prerequisites. TLS certificates \u00b6 Unless otherwise mentioned, the TLS secret used in examples is a 2048 bit RSA key/cert pair with an arbitrarily chosen hostname, created as follows $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj \"/CN=nginxsvc/O=nginxsvc\" Generating a 2048 bit RSA private key ................+++ ................+++ writing new private key to 'tls.key' ----- $ kubectl create secret tls tls-secret --key tls.key --cert tls.crt secret \"tls-secret\" created Note: If using CA Authentication, described below, you will need to sign the server certificate with the CA. Client Certificate Authentication \u00b6 CA Authentication also known as Mutual Authentication allows both the server and client to verify each others identity via a common CA. We have a CA Certificate which we usually obtain from a Certificate Authority and use that to sign both our server certificate and client certificate. Then every time we want to access our backend, we must pass the client certificate. These instructions are based on the following blog Generate the CA Key and Certificate: openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=My Cert Authority' Generate the Server Key, and Certificate and Sign with the CA Certificate: openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=mydomain.com' openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt Generate the Client Key, and Certificate and Sign with the CA Certificate: openssl req -new -newkey rsa:4096 -keyout client.key -out client.csr -nodes -subj '/CN=My Client' openssl x509 -req -sha256 -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 02 -out client.crt Once this is complete you can continue to follow the instructions here Test HTTP Service \u00b6 All examples that require a test HTTP Service use the standard http-svc pod, which you can deploy as follows $ kubectl create -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/http-svc.yaml service \"http-svc\" created replicationcontroller \"http-svc\" created $ kubectl get po NAME READY STATUS RESTARTS AGE http-svc-p1t3t 1/1 Running 0 1d $ kubectl get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE http-svc 10.0.122.116 <pending> 80:30301/TCP 1d You can test that the HTTP Service works by exposing it temporarily $ kubectl patch svc http-svc -p '{\"spec\":{\"type\": \"LoadBalancer\"}}' \"http-svc\" patched $ kubectl get svc http-svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE http-svc 10.0.122.116 <pending> 80:30301/TCP 1d $ kubectl describe svc http-svc Name: http-svc Namespace: default Labels: app=http-svc Selector: app=http-svc Type: LoadBalancer IP: 10.0.122.116 LoadBalancer Ingress: 108.59.87.136 Port: http 80/TCP NodePort: http 30301/TCP Endpoints: 10.180.1.6:8080 Session Affinity: None Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 {service-controller } Normal Type ClusterIP -> LoadBalancer 1m 1m 1 {service-controller } Normal CreatingLoadBalancer Creating load balancer 16s 16s 1 {service-controller } Normal CreatedLoadBalancer Created load balancer $ curl 108 .59.87.136 CLIENT VALUES: client_address=10.240.0.3 command=GET real path=/ query=nil request_version=1.1 request_uri=http://108.59.87.136:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* host=108.59.87.136 user-agent=curl/7.46.0 BODY: -no body in request- $ kubectl patch svc http-svc -p '{\"spec\":{\"type\": \"NodePort\"}}' \"http-svc\" patched","title":"Prerequisites"},{"location":"examples/PREREQUISITES/#prerequisites","text":"Many of the examples in this directory have common prerequisites.","title":"Prerequisites"},{"location":"examples/PREREQUISITES/#tls-certificates","text":"Unless otherwise mentioned, the TLS secret used in examples is a 2048 bit RSA key/cert pair with an arbitrarily chosen hostname, created as follows $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj \"/CN=nginxsvc/O=nginxsvc\" Generating a 2048 bit RSA private key ................+++ ................+++ writing new private key to 'tls.key' ----- $ kubectl create secret tls tls-secret --key tls.key --cert tls.crt secret \"tls-secret\" created Note: If using CA Authentication, described below, you will need to sign the server certificate with the CA.","title":"TLS certificates"},{"location":"examples/PREREQUISITES/#client-certificate-authentication","text":"CA Authentication also known as Mutual Authentication allows both the server and client to verify each others identity via a common CA. We have a CA Certificate which we usually obtain from a Certificate Authority and use that to sign both our server certificate and client certificate. Then every time we want to access our backend, we must pass the client certificate. These instructions are based on the following blog Generate the CA Key and Certificate: openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=My Cert Authority' Generate the Server Key, and Certificate and Sign with the CA Certificate: openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=mydomain.com' openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt Generate the Client Key, and Certificate and Sign with the CA Certificate: openssl req -new -newkey rsa:4096 -keyout client.key -out client.csr -nodes -subj '/CN=My Client' openssl x509 -req -sha256 -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 02 -out client.crt Once this is complete you can continue to follow the instructions here","title":"Client Certificate Authentication"},{"location":"examples/PREREQUISITES/#test-http-service","text":"All examples that require a test HTTP Service use the standard http-svc pod, which you can deploy as follows $ kubectl create -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/http-svc.yaml service \"http-svc\" created replicationcontroller \"http-svc\" created $ kubectl get po NAME READY STATUS RESTARTS AGE http-svc-p1t3t 1/1 Running 0 1d $ kubectl get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE http-svc 10.0.122.116 <pending> 80:30301/TCP 1d You can test that the HTTP Service works by exposing it temporarily $ kubectl patch svc http-svc -p '{\"spec\":{\"type\": \"LoadBalancer\"}}' \"http-svc\" patched $ kubectl get svc http-svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE http-svc 10.0.122.116 <pending> 80:30301/TCP 1d $ kubectl describe svc http-svc Name: http-svc Namespace: default Labels: app=http-svc Selector: app=http-svc Type: LoadBalancer IP: 10.0.122.116 LoadBalancer Ingress: 108.59.87.136 Port: http 80/TCP NodePort: http 30301/TCP Endpoints: 10.180.1.6:8080 Session Affinity: None Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 {service-controller } Normal Type ClusterIP -> LoadBalancer 1m 1m 1 {service-controller } Normal CreatingLoadBalancer Creating load balancer 16s 16s 1 {service-controller } Normal CreatedLoadBalancer Created load balancer $ curl 108 .59.87.136 CLIENT VALUES: client_address=10.240.0.3 command=GET real path=/ query=nil request_version=1.1 request_uri=http://108.59.87.136:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* host=108.59.87.136 user-agent=curl/7.46.0 BODY: -no body in request- $ kubectl patch svc http-svc -p '{\"spec\":{\"type\": \"NodePort\"}}' \"http-svc\" patched","title":"Test HTTP Service"},{"location":"examples/affinity/cookie/","text":"Sticky sessions \u00b6 This example demonstrates how to achieve session affinity using cookies. Deployment \u00b6 Session affinity can be configured using the following annotations: Name Description Value nginx.ingress.kubernetes.io/affinity Type of the affinity, set this to cookie to enable session affinity string (NGINX only supports cookie ) nginx.ingress.kubernetes.io/affinity-mode The affinity mode defines how sticky a session is. Use balanced to redistribute some sessions when scaling pods or persistent for maximum stickiness. balanced (default) or persistent nginx.ingress.kubernetes.io/affinity-canary-behavior Defines session affinity behavior of canaries. By default the behavior is sticky , and canaries respect session affinity configuration. Set this to legacy to restore original canary behavior, when session affinity parameters were not respected. sticky (default) or legacy nginx.ingress.kubernetes.io/session-cookie-name Name of the cookie that will be created string (defaults to INGRESSCOOKIE ) nginx.ingress.kubernetes.io/session-cookie-secure Set the cookie as secure regardless the protocol of the incoming request \"true\" or \"false\" nginx.ingress.kubernetes.io/session-cookie-path Path that will be set on the cookie (required if your Ingress paths use regular expressions) string (defaults to the currently matched path ) nginx.ingress.kubernetes.io/session-cookie-domain Domain that will be set on the cookie string nginx.ingress.kubernetes.io/session-cookie-samesite SameSite attribute to apply to the cookie Browser accepted values are None , Lax , and Strict nginx.ingress.kubernetes.io/session-cookie-conditional-samesite-none Will omit SameSite=None attribute for older browsers which reject the more-recently defined SameSite=None value \"true\" or \"false\" nginx.ingress.kubernetes.io/session-cookie-max-age Time until the cookie expires, corresponds to the Max-Age cookie directive number of seconds nginx.ingress.kubernetes.io/session-cookie-expires Legacy version of the previous annotation for compatibility with older browsers, generates an Expires cookie directive by adding the seconds to the current date number of seconds nginx.ingress.kubernetes.io/session-cookie-change-on-failure When set to false nginx ingress will send request to upstream pointed by sticky cookie even if previous attempt failed. When set to true and previous attempt failed, sticky cookie will be changed to point to another upstream. true or false (defaults to false ) You can create the session affinity example Ingress to test this: kubectl create -f ingress.yaml Validation \u00b6 You can confirm that the Ingress works: $ kubectl describe ing nginx-test Name: nginx-test Namespace: default Address: Default backend: default-http-backend:80 (10.180.0.4:8080,10.240.0.2:8080) Rules: Host Path Backends ---- ---- -------- stickyingress.example.com / nginx-service:80 (<none>) Annotations: affinity: cookie session-cookie-name: INGRESSCOOKIE session-cookie-expires: 172800 session-cookie-max-age: 172800 Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 7s 7s 1 {ingress-nginx-controller } Normal CREATE default/nginx-test $ curl -I http://stickyingress.example.com HTTP/1.1 200 OK Server: nginx/1.11.9 Date: Fri, 10 Feb 2017 14:11:12 GMT Content-Type: text/html Content-Length: 612 Connection: keep-alive Set-Cookie: INGRESSCOOKIE=a9907b79b248140b56bb13723f72b67697baac3d; Expires=Sun, 12-Feb-17 14:11:12 GMT; Max-Age=172800; Path=/; HttpOnly Last-Modified: Tue, 24 Jan 2017 14:02:19 GMT ETag: \"58875e6b-264\" Accept-Ranges: bytes In the example above, you can see that the response contains a Set-Cookie header with the settings we have defined. This cookie is created by the NGINX Ingress Controller, it contains a randomly generated key corresponding to the upstream used for that request (selected using consistent hashing ) and has an Expires directive. If a client sends a cookie that doesn't correspond to an upstream, NGINX selects an upstream and creates a corresponding cookie. If the backend pool grows NGINX will keep sending the requests through the same server of the first request, even if it's overloaded. When the backend server is removed, the requests are re-routed to another upstream server. This does not require the cookie to be updated because the key's consistent hash will change. Caveats \u00b6 When you have a Service pointing to more than one Ingress, with only one containing affinity configuration, the first created Ingress will be used. This means that you can face the situation that you've configured session affinity on one Ingress and it doesn't work because the Service is pointing to another Ingress that doesn't configure this.","title":"Sticky Sessions"},{"location":"examples/affinity/cookie/#sticky-sessions","text":"This example demonstrates how to achieve session affinity using cookies.","title":"Sticky sessions"},{"location":"examples/affinity/cookie/#deployment","text":"Session affinity can be configured using the following annotations: Name Description Value nginx.ingress.kubernetes.io/affinity Type of the affinity, set this to cookie to enable session affinity string (NGINX only supports cookie ) nginx.ingress.kubernetes.io/affinity-mode The affinity mode defines how sticky a session is. Use balanced to redistribute some sessions when scaling pods or persistent for maximum stickiness. balanced (default) or persistent nginx.ingress.kubernetes.io/affinity-canary-behavior Defines session affinity behavior of canaries. By default the behavior is sticky , and canaries respect session affinity configuration. Set this to legacy to restore original canary behavior, when session affinity parameters were not respected. sticky (default) or legacy nginx.ingress.kubernetes.io/session-cookie-name Name of the cookie that will be created string (defaults to INGRESSCOOKIE ) nginx.ingress.kubernetes.io/session-cookie-secure Set the cookie as secure regardless the protocol of the incoming request \"true\" or \"false\" nginx.ingress.kubernetes.io/session-cookie-path Path that will be set on the cookie (required if your Ingress paths use regular expressions) string (defaults to the currently matched path ) nginx.ingress.kubernetes.io/session-cookie-domain Domain that will be set on the cookie string nginx.ingress.kubernetes.io/session-cookie-samesite SameSite attribute to apply to the cookie Browser accepted values are None , Lax , and Strict nginx.ingress.kubernetes.io/session-cookie-conditional-samesite-none Will omit SameSite=None attribute for older browsers which reject the more-recently defined SameSite=None value \"true\" or \"false\" nginx.ingress.kubernetes.io/session-cookie-max-age Time until the cookie expires, corresponds to the Max-Age cookie directive number of seconds nginx.ingress.kubernetes.io/session-cookie-expires Legacy version of the previous annotation for compatibility with older browsers, generates an Expires cookie directive by adding the seconds to the current date number of seconds nginx.ingress.kubernetes.io/session-cookie-change-on-failure When set to false nginx ingress will send request to upstream pointed by sticky cookie even if previous attempt failed. When set to true and previous attempt failed, sticky cookie will be changed to point to another upstream. true or false (defaults to false ) You can create the session affinity example Ingress to test this: kubectl create -f ingress.yaml","title":"Deployment"},{"location":"examples/affinity/cookie/#validation","text":"You can confirm that the Ingress works: $ kubectl describe ing nginx-test Name: nginx-test Namespace: default Address: Default backend: default-http-backend:80 (10.180.0.4:8080,10.240.0.2:8080) Rules: Host Path Backends ---- ---- -------- stickyingress.example.com / nginx-service:80 (<none>) Annotations: affinity: cookie session-cookie-name: INGRESSCOOKIE session-cookie-expires: 172800 session-cookie-max-age: 172800 Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 7s 7s 1 {ingress-nginx-controller } Normal CREATE default/nginx-test $ curl -I http://stickyingress.example.com HTTP/1.1 200 OK Server: nginx/1.11.9 Date: Fri, 10 Feb 2017 14:11:12 GMT Content-Type: text/html Content-Length: 612 Connection: keep-alive Set-Cookie: INGRESSCOOKIE=a9907b79b248140b56bb13723f72b67697baac3d; Expires=Sun, 12-Feb-17 14:11:12 GMT; Max-Age=172800; Path=/; HttpOnly Last-Modified: Tue, 24 Jan 2017 14:02:19 GMT ETag: \"58875e6b-264\" Accept-Ranges: bytes In the example above, you can see that the response contains a Set-Cookie header with the settings we have defined. This cookie is created by the NGINX Ingress Controller, it contains a randomly generated key corresponding to the upstream used for that request (selected using consistent hashing ) and has an Expires directive. If a client sends a cookie that doesn't correspond to an upstream, NGINX selects an upstream and creates a corresponding cookie. If the backend pool grows NGINX will keep sending the requests through the same server of the first request, even if it's overloaded. When the backend server is removed, the requests are re-routed to another upstream server. This does not require the cookie to be updated because the key's consistent hash will change.","title":"Validation"},{"location":"examples/affinity/cookie/#caveats","text":"When you have a Service pointing to more than one Ingress, with only one containing affinity configuration, the first created Ingress will be used. This means that you can face the situation that you've configured session affinity on one Ingress and it doesn't work because the Service is pointing to another Ingress that doesn't configure this.","title":"Caveats"},{"location":"examples/auth/basic/","text":"Basic Authentication \u00b6 This example shows how to add authentication in a Ingress rule using a secret that contains a file generated with htpasswd . It's important the file generated is named auth (actually - that the secret has a key data.auth ), otherwise the ingress-controller returns a 503. Create htpasswd file \u00b6 $ htpasswd -c auth foo New password: <bar> New password: Re-type new password: Adding password for user foo Convert htpasswd into a secret \u00b6 $ kubectl create secret generic basic-auth --from-file = auth secret \"basic-auth\" created Examine secret \u00b6 $ kubectl get secret basic-auth -o yaml apiVersion: v1 data: auth: Zm9vOiRhcHIxJE9GRzNYeWJwJGNrTDBGSERBa29YWUlsSDkuY3lzVDAK kind: Secret metadata: name: basic-auth namespace: default type: Opaque Using kubectl, create an ingress tied to the basic-auth secret \u00b6 $ echo \" apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-with-auth annotations: # type of authentication nginx.ingress.kubernetes.io/auth-type: basic # name of the secret that contains the user/password definitions nginx.ingress.kubernetes.io/auth-secret: basic-auth # message to display with an appropriate context why the authentication is required nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - foo' spec: ingressClassName: nginx rules: - host: foo.bar.com http: paths: - path: / pathType: Prefix backend: service: name: http-svc port: number: 80 \" | kubectl create -f - Use curl to confirm authorization is required by the ingress \u00b6 $ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' * Trying 10.2.29.4... * Connected to 10.2.29.4 (10.2.29.4) port 80 (#0) > GET / HTTP/1.1 > Host: foo.bar.com > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 401 Unauthorized < Server: nginx/1.10.0 < Date: Wed, 11 May 2016 05:27:23 GMT < Content-Type: text/html < Content-Length: 195 < Connection: keep-alive < WWW-Authenticate: Basic realm=\"Authentication Required - foo\" < <html> <head><title>401 Authorization Required</title></head> <body bgcolor=\"white\"> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx/1.10.0</center> </body> </html> * Connection #0 to host 10.2.29.4 left intact Use curl with the correct credentials to connect to the ingress \u00b6 $ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' -u 'foo:bar' * Trying 10.2.29.4... * Connected to 10.2.29.4 (10.2.29.4) port 80 (#0) * Server auth using Basic with user 'foo' > GET / HTTP/1.1 > Host: foo.bar.com > Authorization: Basic Zm9vOmJhcg== > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.10.0 < Date: Wed, 11 May 2016 06:05:26 GMT < Content-Type: text/plain < Transfer-Encoding: chunked < Connection: keep-alive < Vary: Accept-Encoding < CLIENT VALUES: client_address=10.2.29.4 command=GET real path=/ query=nil request_version=1.1 request_uri=http://foo.bar.com:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=close host=foo.bar.com user-agent=curl/7.43.0 x-request-id=e426c7829ef9f3b18d40730857c3eddb x-forwarded-for=10.2.29.1 x-forwarded-host=foo.bar.com x-forwarded-port=80 x-forwarded-proto=http x-real-ip=10.2.29.1 x-scheme=http BODY: * Connection #0 to host 10.2.29.4 left intact -no body in request-","title":"Basic Authentication"},{"location":"examples/auth/basic/#basic-authentication","text":"This example shows how to add authentication in a Ingress rule using a secret that contains a file generated with htpasswd . It's important the file generated is named auth (actually - that the secret has a key data.auth ), otherwise the ingress-controller returns a 503.","title":"Basic Authentication"},{"location":"examples/auth/basic/#create-htpasswd-file","text":"$ htpasswd -c auth foo New password: <bar> New password: Re-type new password: Adding password for user foo","title":"Create htpasswd file"},{"location":"examples/auth/basic/#convert-htpasswd-into-a-secret","text":"$ kubectl create secret generic basic-auth --from-file = auth secret \"basic-auth\" created","title":"Convert htpasswd into a secret"},{"location":"examples/auth/basic/#examine-secret","text":"$ kubectl get secret basic-auth -o yaml apiVersion: v1 data: auth: Zm9vOiRhcHIxJE9GRzNYeWJwJGNrTDBGSERBa29YWUlsSDkuY3lzVDAK kind: Secret metadata: name: basic-auth namespace: default type: Opaque","title":"Examine secret"},{"location":"examples/auth/basic/#using-kubectl-create-an-ingress-tied-to-the-basic-auth-secret","text":"$ echo \" apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-with-auth annotations: # type of authentication nginx.ingress.kubernetes.io/auth-type: basic # name of the secret that contains the user/password definitions nginx.ingress.kubernetes.io/auth-secret: basic-auth # message to display with an appropriate context why the authentication is required nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - foo' spec: ingressClassName: nginx rules: - host: foo.bar.com http: paths: - path: / pathType: Prefix backend: service: name: http-svc port: number: 80 \" | kubectl create -f -","title":"Using kubectl, create an ingress tied to the basic-auth secret"},{"location":"examples/auth/basic/#use-curl-to-confirm-authorization-is-required-by-the-ingress","text":"$ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' * Trying 10.2.29.4... * Connected to 10.2.29.4 (10.2.29.4) port 80 (#0) > GET / HTTP/1.1 > Host: foo.bar.com > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 401 Unauthorized < Server: nginx/1.10.0 < Date: Wed, 11 May 2016 05:27:23 GMT < Content-Type: text/html < Content-Length: 195 < Connection: keep-alive < WWW-Authenticate: Basic realm=\"Authentication Required - foo\" < <html> <head><title>401 Authorization Required</title></head> <body bgcolor=\"white\"> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx/1.10.0</center> </body> </html> * Connection #0 to host 10.2.29.4 left intact","title":"Use curl to confirm authorization is required by the ingress"},{"location":"examples/auth/basic/#use-curl-with-the-correct-credentials-to-connect-to-the-ingress","text":"$ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' -u 'foo:bar' * Trying 10.2.29.4... * Connected to 10.2.29.4 (10.2.29.4) port 80 (#0) * Server auth using Basic with user 'foo' > GET / HTTP/1.1 > Host: foo.bar.com > Authorization: Basic Zm9vOmJhcg== > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.10.0 < Date: Wed, 11 May 2016 06:05:26 GMT < Content-Type: text/plain < Transfer-Encoding: chunked < Connection: keep-alive < Vary: Accept-Encoding < CLIENT VALUES: client_address=10.2.29.4 command=GET real path=/ query=nil request_version=1.1 request_uri=http://foo.bar.com:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=close host=foo.bar.com user-agent=curl/7.43.0 x-request-id=e426c7829ef9f3b18d40730857c3eddb x-forwarded-for=10.2.29.1 x-forwarded-host=foo.bar.com x-forwarded-port=80 x-forwarded-proto=http x-real-ip=10.2.29.1 x-scheme=http BODY: * Connection #0 to host 10.2.29.4 left intact -no body in request-","title":"Use curl with the correct credentials to connect to the ingress"},{"location":"examples/auth/client-certs/","text":"Client Certificate Authentication \u00b6 It is possible to enable Client-Certificate Authentication by adding additional annotations to your Ingress Resource. Before getting started you must have the following Certificates configured: CA certificate and Key (Intermediate Certs need to be in CA) Server Certificate (Signed by CA) and Key (CN should be equal the hostname you will use) Client Certificate (Signed by CA) and Key For more details on the generation process, checkout the Prerequisite docs . You can have as many certificates as you want. If they're in the binary DER format, you can convert them as the following: openssl x509 -in certificate.der -inform der -out certificate.crt -outform pem Then, you can concatenate them all into one file, named 'ca.crt' with the following: cat certificate1.crt certificate2.crt certificate3.crt >> ca.crt Note: Make sure that the Key Size is greater than 1024 and Hashing Algorithm (Digest) is something better than md5 for each certificate generated. Otherwise you will receive an error. Creating Certificate Secrets \u00b6 There are many different ways of configuring your secrets to enable Client-Certificate Authentication to work properly. You can create a secret containing just the CA certificate and another Secret containing the Server Certificate which is Signed by the CA. kubectl create secret generic ca-secret --from-file = ca.crt = ca.crt kubectl create secret generic tls-secret --from-file = tls.crt = server.crt --from-file = tls.key = server.key You can create a secret containing CA certificate along with the Server Certificate that can be used for both TLS and Client Auth. kubectl create secret generic ca-secret --from-file = tls.crt = server.crt --from-file = tls.key = server.key --from-file = ca.crt = ca.crt If you want to also enable Certificate Revocation List verification you can create the secret also containing the CRL file in PEM format: kubectl create secret generic ca-secret --from-file = ca.crt = ca.crt --from-file = ca.crl = ca.crl Note: The CA Certificate must contain the trusted certificate authority chain to verify client certificates. Setup Instructions \u00b6 Add the annotations as provided in the ingress.yaml example to your own ingress resources as required. Test by performing a curl against the Ingress Path without the Client Cert and expect a Status Code 400. Test by performing a curl against the Ingress Path with the Client Cert and expect a Status Code 200.","title":"Client Certificate Authentication"},{"location":"examples/auth/client-certs/#client-certificate-authentication","text":"It is possible to enable Client-Certificate Authentication by adding additional annotations to your Ingress Resource. Before getting started you must have the following Certificates configured: CA certificate and Key (Intermediate Certs need to be in CA) Server Certificate (Signed by CA) and Key (CN should be equal the hostname you will use) Client Certificate (Signed by CA) and Key For more details on the generation process, checkout the Prerequisite docs . You can have as many certificates as you want. If they're in the binary DER format, you can convert them as the following: openssl x509 -in certificate.der -inform der -out certificate.crt -outform pem Then, you can concatenate them all into one file, named 'ca.crt' with the following: cat certificate1.crt certificate2.crt certificate3.crt >> ca.crt Note: Make sure that the Key Size is greater than 1024 and Hashing Algorithm (Digest) is something better than md5 for each certificate generated. Otherwise you will receive an error.","title":"Client Certificate Authentication"},{"location":"examples/auth/client-certs/#creating-certificate-secrets","text":"There are many different ways of configuring your secrets to enable Client-Certificate Authentication to work properly. You can create a secret containing just the CA certificate and another Secret containing the Server Certificate which is Signed by the CA. kubectl create secret generic ca-secret --from-file = ca.crt = ca.crt kubectl create secret generic tls-secret --from-file = tls.crt = server.crt --from-file = tls.key = server.key You can create a secret containing CA certificate along with the Server Certificate that can be used for both TLS and Client Auth. kubectl create secret generic ca-secret --from-file = tls.crt = server.crt --from-file = tls.key = server.key --from-file = ca.crt = ca.crt If you want to also enable Certificate Revocation List verification you can create the secret also containing the CRL file in PEM format: kubectl create secret generic ca-secret --from-file = ca.crt = ca.crt --from-file = ca.crl = ca.crl Note: The CA Certificate must contain the trusted certificate authority chain to verify client certificates.","title":"Creating Certificate Secrets"},{"location":"examples/auth/client-certs/#setup-instructions","text":"Add the annotations as provided in the ingress.yaml example to your own ingress resources as required. Test by performing a curl against the Ingress Path without the Client Cert and expect a Status Code 400. Test by performing a curl against the Ingress Path with the Client Cert and expect a Status Code 200.","title":"Setup Instructions"},{"location":"examples/auth/external-auth/","text":"External Basic Authentication \u00b6 Example 1 \u00b6 Use an external service (Basic Auth) located in https://httpbin.org $ kubectl create -f ingress.yaml ingress \"external-auth\" created $ kubectl get ing external-auth NAME HOSTS ADDRESS PORTS AGE external-auth external-auth-01.sample.com 172.17.4.99 80 13s $ kubectl get ing external-auth -o yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/auth-url: https://httpbin.org/basic-auth/user/passwd creationTimestamp: 2016-10-03T13:50:35Z generation: 1 name: external-auth namespace: default resourceVersion: \"2068378\" selfLink: /apis/networking/v1/namespaces/default/ingresses/external-auth uid: 5c388f1d-8970-11e6-9004-080027d2dc94 spec: rules: - host: external-auth-01.sample.com http: paths: - path: / pathType: Prefix backend: service: name: http-svc port: number: 80 status: loadBalancer: ingress: - ip: 172.17.4.99 $ Test 1: no username/password (expect code 401) \u00b6 $ curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) > GET / HTTP/1.1 > Host: external-auth-01.sample.com > User-Agent: curl/7.50.1 > Accept: */* > < HTTP/1.1 401 Unauthorized < Server: nginx/1.11.3 < Date: Mon, 03 Oct 2016 14:52:08 GMT < Content-Type: text/html < Content-Length: 195 < Connection: keep-alive < WWW-Authenticate: Basic realm=\"Fake Realm\" < <html> <head><title>401 Authorization Required</title></head> <body bgcolor=\"white\"> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx/1.11.3</center> </body> </html> * Connection #0 to host 172.17.4.99 left intact Test 2: valid username/password (expect code 200) \u00b6 $ curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' -u 'user:passwd' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) * Server auth using Basic with user 'user' > GET / HTTP/1.1 > Host: external-auth-01.sample.com > Authorization: Basic dXNlcjpwYXNzd2Q= > User-Agent: curl/7.50.1 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.11.3 < Date: Mon, 03 Oct 2016 14:52:50 GMT < Content-Type: text/plain < Transfer-Encoding: chunked < Connection: keep-alive < CLIENT VALUES: client_address=10.2.60.2 command=GET real path=/ query=nil request_version=1.1 request_uri=http://external-auth-01.sample.com:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* authorization=Basic dXNlcjpwYXNzd2Q= connection=close host=external-auth-01.sample.com user-agent=curl/7.50.1 x-forwarded-for=10.2.60.1 x-forwarded-host=external-auth-01.sample.com x-forwarded-port=80 x-forwarded-proto=http x-real-ip=10.2.60.1 BODY: * Connection #0 to host 172.17.4.99 left intact -no body in request- Test 3: invalid username/password (expect code 401) \u00b6 curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' -u 'user:user' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) * Server auth using Basic with user 'user' > GET / HTTP/1.1 > Host: external-auth-01.sample.com > Authorization: Basic dXNlcjp1c2Vy > User-Agent: curl/7.50.1 > Accept: */* > < HTTP/1.1 401 Unauthorized < Server: nginx/1.11.3 < Date: Mon, 03 Oct 2016 14:53:04 GMT < Content-Type: text/html < Content-Length: 195 < Connection: keep-alive * Authentication problem. Ignoring this. < WWW-Authenticate: Basic realm=\"Fake Realm\" < <html> <head><title>401 Authorization Required</title></head> <body bgcolor=\"white\"> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx/1.11.3</center> </body> </html> * Connection #0 to host 172.17.4.99 left intact","title":"External Basic Authentication"},{"location":"examples/auth/external-auth/#external-basic-authentication","text":"","title":"External Basic Authentication"},{"location":"examples/auth/external-auth/#example-1","text":"Use an external service (Basic Auth) located in https://httpbin.org $ kubectl create -f ingress.yaml ingress \"external-auth\" created $ kubectl get ing external-auth NAME HOSTS ADDRESS PORTS AGE external-auth external-auth-01.sample.com 172.17.4.99 80 13s $ kubectl get ing external-auth -o yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/auth-url: https://httpbin.org/basic-auth/user/passwd creationTimestamp: 2016-10-03T13:50:35Z generation: 1 name: external-auth namespace: default resourceVersion: \"2068378\" selfLink: /apis/networking/v1/namespaces/default/ingresses/external-auth uid: 5c388f1d-8970-11e6-9004-080027d2dc94 spec: rules: - host: external-auth-01.sample.com http: paths: - path: / pathType: Prefix backend: service: name: http-svc port: number: 80 status: loadBalancer: ingress: - ip: 172.17.4.99 $","title":"Example 1"},{"location":"examples/auth/external-auth/#test-1-no-usernamepassword-expect-code-401","text":"$ curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) > GET / HTTP/1.1 > Host: external-auth-01.sample.com > User-Agent: curl/7.50.1 > Accept: */* > < HTTP/1.1 401 Unauthorized < Server: nginx/1.11.3 < Date: Mon, 03 Oct 2016 14:52:08 GMT < Content-Type: text/html < Content-Length: 195 < Connection: keep-alive < WWW-Authenticate: Basic realm=\"Fake Realm\" < <html> <head><title>401 Authorization Required</title></head> <body bgcolor=\"white\"> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx/1.11.3</center> </body> </html> * Connection #0 to host 172.17.4.99 left intact","title":"Test 1: no username/password (expect code 401)"},{"location":"examples/auth/external-auth/#test-2-valid-usernamepassword-expect-code-200","text":"$ curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' -u 'user:passwd' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) * Server auth using Basic with user 'user' > GET / HTTP/1.1 > Host: external-auth-01.sample.com > Authorization: Basic dXNlcjpwYXNzd2Q= > User-Agent: curl/7.50.1 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.11.3 < Date: Mon, 03 Oct 2016 14:52:50 GMT < Content-Type: text/plain < Transfer-Encoding: chunked < Connection: keep-alive < CLIENT VALUES: client_address=10.2.60.2 command=GET real path=/ query=nil request_version=1.1 request_uri=http://external-auth-01.sample.com:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* authorization=Basic dXNlcjpwYXNzd2Q= connection=close host=external-auth-01.sample.com user-agent=curl/7.50.1 x-forwarded-for=10.2.60.1 x-forwarded-host=external-auth-01.sample.com x-forwarded-port=80 x-forwarded-proto=http x-real-ip=10.2.60.1 BODY: * Connection #0 to host 172.17.4.99 left intact -no body in request-","title":"Test 2: valid username/password (expect code 200)"},{"location":"examples/auth/external-auth/#test-3-invalid-usernamepassword-expect-code-401","text":"curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' -u 'user:user' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) * Server auth using Basic with user 'user' > GET / HTTP/1.1 > Host: external-auth-01.sample.com > Authorization: Basic dXNlcjp1c2Vy > User-Agent: curl/7.50.1 > Accept: */* > < HTTP/1.1 401 Unauthorized < Server: nginx/1.11.3 < Date: Mon, 03 Oct 2016 14:53:04 GMT < Content-Type: text/html < Content-Length: 195 < Connection: keep-alive * Authentication problem. Ignoring this. < WWW-Authenticate: Basic realm=\"Fake Realm\" < <html> <head><title>401 Authorization Required</title></head> <body bgcolor=\"white\"> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx/1.11.3</center> </body> </html> * Connection #0 to host 172.17.4.99 left intact","title":"Test 3: invalid username/password (expect code 401)"},{"location":"examples/auth/oauth-external-auth/","text":"External OAUTH Authentication \u00b6 Overview \u00b6 The auth-url and auth-signin annotations allow you to use an external authentication provider to protect your Ingress resources. Important This annotation requires ingress-nginx-controller v0.9.0 or greater. Key Detail \u00b6 This functionality is enabled by deploying multiple Ingress objects for a single host. One Ingress object has no special annotations and handles authentication. Other Ingress objects can then be annotated in such a way that require the user to authenticate against the first Ingress's endpoint, and can redirect 401 s to the same endpoint. Sample: ... metadata : name : application annotations : nginx.ingress.kubernetes.io/auth-url : \"https://$host/oauth2/auth\" nginx.ingress.kubernetes.io/auth-signin : \"https://$host/oauth2/start?rd=$escaped_request_uri\" ... Example: OAuth2 Proxy + Kubernetes-Dashboard \u00b6 This example will show you how to deploy oauth2_proxy into a Kubernetes cluster and use it to protect the Kubernetes Dashboard using GitHub as the OAuth2 provider. Prepare \u00b6 Install the kubernetes dashboard kubectl create -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/kubernetes-dashboard/v1.10.1.yaml Create a custom GitHub OAuth application Homepage URL is the FQDN in the Ingress rule, like https://foo.bar.com Authorization callback URL is the same as the base FQDN plus /oauth2/callback , like https://foo.bar.com/oauth2/callback Configure oauth2_proxy values in the file oauth2-proxy.yaml with the values: OAUTH2_PROXY_CLIENT_ID with the github <Client ID> OAUTH2_PROXY_CLIENT_SECRET with the github <Client Secret> OAUTH2_PROXY_COOKIE_SECRET with value of python -c 'import os,base64; print(base64.b64encode(os.urandom(16)).decode(\"ascii\"))' Customize the contents of the file dashboard-ingress.yaml : Replace __INGRESS_HOST__ with a valid FQDN and __INGRESS_SECRET__ with a Secret with a valid SSL certificate. Deploy the oauth2 proxy and the ingress rules running: $ kubectl create -f oauth2-proxy.yaml,dashboard-ingress.yaml Test \u00b6 Test the oauth integration accessing the configured URL, e.g. https://foo.bar.com","title":"External OAUTH Authentication"},{"location":"examples/auth/oauth-external-auth/#external-oauth-authentication","text":"","title":"External OAUTH Authentication"},{"location":"examples/auth/oauth-external-auth/#overview","text":"The auth-url and auth-signin annotations allow you to use an external authentication provider to protect your Ingress resources. Important This annotation requires ingress-nginx-controller v0.9.0 or greater.","title":"Overview"},{"location":"examples/auth/oauth-external-auth/#key-detail","text":"This functionality is enabled by deploying multiple Ingress objects for a single host. One Ingress object has no special annotations and handles authentication. Other Ingress objects can then be annotated in such a way that require the user to authenticate against the first Ingress's endpoint, and can redirect 401 s to the same endpoint. Sample: ... metadata : name : application annotations : nginx.ingress.kubernetes.io/auth-url : \"https://$host/oauth2/auth\" nginx.ingress.kubernetes.io/auth-signin : \"https://$host/oauth2/start?rd=$escaped_request_uri\" ...","title":"Key Detail"},{"location":"examples/auth/oauth-external-auth/#example-oauth2-proxy-kubernetes-dashboard","text":"This example will show you how to deploy oauth2_proxy into a Kubernetes cluster and use it to protect the Kubernetes Dashboard using GitHub as the OAuth2 provider.","title":"Example: OAuth2 Proxy + Kubernetes-Dashboard"},{"location":"examples/auth/oauth-external-auth/#prepare","text":"Install the kubernetes dashboard kubectl create -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/kubernetes-dashboard/v1.10.1.yaml Create a custom GitHub OAuth application Homepage URL is the FQDN in the Ingress rule, like https://foo.bar.com Authorization callback URL is the same as the base FQDN plus /oauth2/callback , like https://foo.bar.com/oauth2/callback Configure oauth2_proxy values in the file oauth2-proxy.yaml with the values: OAUTH2_PROXY_CLIENT_ID with the github <Client ID> OAUTH2_PROXY_CLIENT_SECRET with the github <Client Secret> OAUTH2_PROXY_COOKIE_SECRET with value of python -c 'import os,base64; print(base64.b64encode(os.urandom(16)).decode(\"ascii\"))' Customize the contents of the file dashboard-ingress.yaml : Replace __INGRESS_HOST__ with a valid FQDN and __INGRESS_SECRET__ with a Secret with a valid SSL certificate. Deploy the oauth2 proxy and the ingress rules running: $ kubectl create -f oauth2-proxy.yaml,dashboard-ingress.yaml","title":"Prepare"},{"location":"examples/auth/oauth-external-auth/#test","text":"Test the oauth integration accessing the configured URL, e.g. https://foo.bar.com","title":"Test"},{"location":"examples/customization/configuration-snippets/","text":"Configuration Snippets \u00b6 Ingress \u00b6 The Ingress in this example adds a custom header to Nginx configuration that only applies to that specific Ingress. If you want to add headers that apply globally to all Ingresses, please have a look at an example of specifying custom headers . kubectl apply -f ingress.yaml Test \u00b6 Check if the contents of the annotation are present in the nginx.conf file using: kubectl exec ingress-nginx-controller-873061567-4n3k2 -n kube-system -- cat /etc/nginx/nginx.conf","title":"Configuration Snippets"},{"location":"examples/customization/configuration-snippets/#configuration-snippets","text":"","title":"Configuration Snippets"},{"location":"examples/customization/configuration-snippets/#ingress","text":"The Ingress in this example adds a custom header to Nginx configuration that only applies to that specific Ingress. If you want to add headers that apply globally to all Ingresses, please have a look at an example of specifying custom headers . kubectl apply -f ingress.yaml","title":"Ingress"},{"location":"examples/customization/configuration-snippets/#test","text":"Check if the contents of the annotation are present in the nginx.conf file using: kubectl exec ingress-nginx-controller-873061567-4n3k2 -n kube-system -- cat /etc/nginx/nginx.conf","title":"Test"},{"location":"examples/customization/custom-configuration/","text":"Custom Configuration \u00b6 Using a ConfigMap is possible to customize the NGINX configuration For example, if we want to change the timeouts we need to create a ConfigMap: $ cat configmap.yaml apiVersion: v1 data: proxy-connect-timeout: \"10\" proxy-read-timeout: \"120\" proxy-send-timeout: \"120\" kind: ConfigMap metadata: name: ingress-nginx-controller curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/custom-configuration/configmap.yaml \\ | kubectl apply -f - If the Configmap is updated, NGINX will be reloaded with the new configuration.","title":"Custom Configuration"},{"location":"examples/customization/custom-configuration/#custom-configuration","text":"Using a ConfigMap is possible to customize the NGINX configuration For example, if we want to change the timeouts we need to create a ConfigMap: $ cat configmap.yaml apiVersion: v1 data: proxy-connect-timeout: \"10\" proxy-read-timeout: \"120\" proxy-send-timeout: \"120\" kind: ConfigMap metadata: name: ingress-nginx-controller curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/custom-configuration/configmap.yaml \\ | kubectl apply -f - If the Configmap is updated, NGINX will be reloaded with the new configuration.","title":"Custom Configuration"},{"location":"examples/customization/custom-errors/","text":"Custom Errors \u00b6 This example demonstrates how to use a custom backend to render custom error pages. If you are using Helm Chart, look at example values and don't forget to add configMap to your deployment, otherwise continue with Customized default backend manual deployment. Customized default backend \u00b6 First, create the custom default-backend . It will be used by the Ingress controller later on. To do that, you can take a look at the example manifest in this project's GitHub repository. $ kubectl create -f custom-default-backend.yaml service \"nginx-errors\" created deployment.apps \"nginx-errors\" created This should have created a Deployment and a Service with the name nginx-errors . $ kubectl get deploy,svc NAME DESIRED CURRENT READY AGE deployment.apps/nginx-errors 1 1 1 10s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nginx-errors ClusterIP 10.0.0.12 <none> 80/TCP 10s Ingress controller configuration \u00b6 If you do not already have an instance of the NGINX Ingress controller running, deploy it according to the deployment guide , then follow these steps: Edit the ingress-nginx-controller Deployment and set the value of the --default-backend-service flag to the name of the newly created error backend. Edit the ingress-nginx-controller ConfigMap and create the key custom-http-errors with a value of 404,503 . Take note of the IP address assigned to the NGINX Ingress controller Service. $ kubectl get svc ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx ClusterIP 10.0.0.13 <none> 80/TCP,443/TCP 10m Note The ingress-nginx Service is of type ClusterIP in this example. This may vary depending on your environment. Make sure you can use the Service to reach NGINX before proceeding with the rest of this example. Testing error pages \u00b6 Let us send a couple of HTTP requests using cURL and validate everything is working as expected. A request to the default backend returns a 404 error with a custom message: $ curl -D- http://10.0.0.13/ HTTP/1.1 404 Not Found Server: nginx/1.13.12 Date: Tue, 12 Jun 2018 19:11:24 GMT Content-Type: */* Transfer-Encoding: chunked Connection: keep-alive <span>The page you're looking for could not be found.</span> A request with a custom Accept header returns the corresponding document type (JSON): $ curl -D- -H 'Accept: application/json' http://10.0.0.13/ HTTP/1.1 404 Not Found Server: nginx/1.13.12 Date: Tue, 12 Jun 2018 19:12:36 GMT Content-Type: application/json Transfer-Encoding: chunked Connection: keep-alive Vary: Accept-Encoding { \"message\": \"The page you're looking for could not be found\" } To go further with this example, feel free to deploy your own applications and Ingress objects, and validate that the responses are still in the correct format when a backend returns 503 (eg. if you scale a Deployment down to 0 replica).","title":"Custom Errors"},{"location":"examples/customization/custom-errors/#custom-errors","text":"This example demonstrates how to use a custom backend to render custom error pages. If you are using Helm Chart, look at example values and don't forget to add configMap to your deployment, otherwise continue with Customized default backend manual deployment.","title":"Custom Errors"},{"location":"examples/customization/custom-errors/#customized-default-backend","text":"First, create the custom default-backend . It will be used by the Ingress controller later on. To do that, you can take a look at the example manifest in this project's GitHub repository. $ kubectl create -f custom-default-backend.yaml service \"nginx-errors\" created deployment.apps \"nginx-errors\" created This should have created a Deployment and a Service with the name nginx-errors . $ kubectl get deploy,svc NAME DESIRED CURRENT READY AGE deployment.apps/nginx-errors 1 1 1 10s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nginx-errors ClusterIP 10.0.0.12 <none> 80/TCP 10s","title":"Customized default backend"},{"location":"examples/customization/custom-errors/#ingress-controller-configuration","text":"If you do not already have an instance of the NGINX Ingress controller running, deploy it according to the deployment guide , then follow these steps: Edit the ingress-nginx-controller Deployment and set the value of the --default-backend-service flag to the name of the newly created error backend. Edit the ingress-nginx-controller ConfigMap and create the key custom-http-errors with a value of 404,503 . Take note of the IP address assigned to the NGINX Ingress controller Service. $ kubectl get svc ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx ClusterIP 10.0.0.13 <none> 80/TCP,443/TCP 10m Note The ingress-nginx Service is of type ClusterIP in this example. This may vary depending on your environment. Make sure you can use the Service to reach NGINX before proceeding with the rest of this example.","title":"Ingress controller configuration"},{"location":"examples/customization/custom-errors/#testing-error-pages","text":"Let us send a couple of HTTP requests using cURL and validate everything is working as expected. A request to the default backend returns a 404 error with a custom message: $ curl -D- http://10.0.0.13/ HTTP/1.1 404 Not Found Server: nginx/1.13.12 Date: Tue, 12 Jun 2018 19:11:24 GMT Content-Type: */* Transfer-Encoding: chunked Connection: keep-alive <span>The page you're looking for could not be found.</span> A request with a custom Accept header returns the corresponding document type (JSON): $ curl -D- -H 'Accept: application/json' http://10.0.0.13/ HTTP/1.1 404 Not Found Server: nginx/1.13.12 Date: Tue, 12 Jun 2018 19:12:36 GMT Content-Type: application/json Transfer-Encoding: chunked Connection: keep-alive Vary: Accept-Encoding { \"message\": \"The page you're looking for could not be found\" } To go further with this example, feel free to deploy your own applications and Ingress objects, and validate that the responses are still in the correct format when a backend returns 503 (eg. if you scale a Deployment down to 0 replica).","title":"Testing error pages"},{"location":"examples/customization/custom-headers/","text":"Custom Headers \u00b6 Caveats \u00b6 Changes to the custom header config maps do not force a reload of the ingress-nginx-controllers. Workaround \u00b6 To work around this limitation, perform a rolling restart of the deployment. Example \u00b6 This example demonstrates configuration of the nginx ingress controller via a ConfigMap to pass a custom list of headers to the upstream server. custom-headers.yaml defines a ConfigMap in the ingress-nginx namespace named custom-headers , holding several custom X-prefixed HTTP headers. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/custom-headers/custom-headers.yaml configmap.yaml defines a ConfigMap in the ingress-nginx namespace named ingress-nginx-controller . This controls the global configuration of the ingress controller, and already exists in a standard installation. The key proxy-set-headers is set to cite the previously-created ingress-nginx/custom-headers ConfigMap. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/custom-headers/configmap.yaml The nginx ingress controller will read the ingress-nginx/ingress-nginx-controller ConfigMap, find the proxy-set-headers key, read HTTP headers from the ingress-nginx/custom-headers ConfigMap, and include those HTTP headers in all requests flowing from nginx to the backends. The above example was for passing a custom list of headers to the upstream server. To pass the custom headers before sending response traffic to the client, use the add-headers key: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/custom-headers/configmap-client-response.yaml Test \u00b6 Check the contents of the ConfigMaps are present in the nginx.conf file using: kubectl exec ingress-nginx-controller-873061567-4n3k2 -n ingress-nginx -- cat /etc/nginx/nginx.conf","title":"Custom Headers"},{"location":"examples/customization/custom-headers/#custom-headers","text":"","title":"Custom Headers"},{"location":"examples/customization/custom-headers/#caveats","text":"Changes to the custom header config maps do not force a reload of the ingress-nginx-controllers.","title":"Caveats"},{"location":"examples/customization/custom-headers/#workaround","text":"To work around this limitation, perform a rolling restart of the deployment.","title":"Workaround"},{"location":"examples/customization/custom-headers/#example","text":"This example demonstrates configuration of the nginx ingress controller via a ConfigMap to pass a custom list of headers to the upstream server. custom-headers.yaml defines a ConfigMap in the ingress-nginx namespace named custom-headers , holding several custom X-prefixed HTTP headers. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/custom-headers/custom-headers.yaml configmap.yaml defines a ConfigMap in the ingress-nginx namespace named ingress-nginx-controller . This controls the global configuration of the ingress controller, and already exists in a standard installation. The key proxy-set-headers is set to cite the previously-created ingress-nginx/custom-headers ConfigMap. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/custom-headers/configmap.yaml The nginx ingress controller will read the ingress-nginx/ingress-nginx-controller ConfigMap, find the proxy-set-headers key, read HTTP headers from the ingress-nginx/custom-headers ConfigMap, and include those HTTP headers in all requests flowing from nginx to the backends. The above example was for passing a custom list of headers to the upstream server. To pass the custom headers before sending response traffic to the client, use the add-headers key: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/custom-headers/configmap-client-response.yaml","title":"Example"},{"location":"examples/customization/custom-headers/#test","text":"Check the contents of the ConfigMaps are present in the nginx.conf file using: kubectl exec ingress-nginx-controller-873061567-4n3k2 -n ingress-nginx -- cat /etc/nginx/nginx.conf","title":"Test"},{"location":"examples/customization/external-auth-headers/","text":"External authentication, authentication service response headers propagation \u00b6 This example demonstrates propagation of selected authentication service response headers to a backend service. Sample configuration includes: Sample authentication service producing several response headers Authentication logic is based on HTTP header: requests with header User containing string internal are considered authenticated After successful authentication service generates response headers UserID and UserRole Sample echo service displaying header information Two ingress objects pointing to echo service Public, which allows access from unauthenticated users Private, which allows access from authenticated users only You can deploy the controller as follows: $ kubectl create -f deploy/ deployment \"demo-auth-service\" created service \"demo-auth-service\" created ingress \"demo-auth-service\" created deployment \"demo-echo-service\" created service \"demo-echo-service\" created ingress \"public-demo-echo-service\" created ingress \"secure-demo-echo-service\" created $ kubectl get po NAME READY STATUS RESTARTS AGE demo-auth-service-2769076528-7g9mh 1/1 Running 0 30s demo-echo-service-3636052215-3vw8c 1/1 Running 0 29s kubectl get ing NAME HOSTS ADDRESS PORTS AGE public-demo-echo-service public-demo-echo-service.kube.local 80 1m secure-demo-echo-service secure-demo-echo-service.kube.local 80 1m Test 1: public service with no auth header \u00b6 $ curl -H 'Host: public-demo-echo-service.kube.local' -v 192 .168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) > GET / HTTP/1.1 > Host: public-demo-echo-service.kube.local > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.11.10 < Date: Mon, 13 Mar 2017 20:19:21 GMT < Content-Type: text/plain; charset=utf-8 < Content-Length: 20 < Connection: keep-alive < * Connection #0 to host 192.168.99.100 left intact UserID: , UserRole: Test 2: secure service with no auth header \u00b6 $ curl -H 'Host: secure-demo-echo-service.kube.local' -v 192 .168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) > GET / HTTP/1.1 > Host: secure-demo-echo-service.kube.local > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 403 Forbidden < Server: nginx/1.11.10 < Date: Mon, 13 Mar 2017 20:18:48 GMT < Content-Type: text/html < Content-Length: 170 < Connection: keep-alive < <html> <head><title>403 Forbidden</title></head> <body bgcolor=\"white\"> <center><h1>403 Forbidden</h1></center> <hr><center>nginx/1.11.10</center> </body> </html> * Connection #0 to host 192.168.99.100 left intact Test 3: public service with valid auth header \u00b6 $ curl -H 'Host: public-demo-echo-service.kube.local' -H 'User:internal' -v 192 .168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) > GET / HTTP/1.1 > Host: public-demo-echo-service.kube.local > User-Agent: curl/7.43.0 > Accept: */* > User:internal > < HTTP/1.1 200 OK < Server: nginx/1.11.10 < Date: Mon, 13 Mar 2017 20:19:59 GMT < Content-Type: text/plain; charset=utf-8 < Content-Length: 44 < Connection: keep-alive < * Connection #0 to host 192.168.99.100 left intact UserID: 1443635317331776148, UserRole: admin Test 4: secure service with valid auth header \u00b6 $ curl -H 'Host: secure-demo-echo-service.kube.local' -H 'User:internal' -v 192 .168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) > GET / HTTP/1.1 > Host: secure-demo-echo-service.kube.local > User-Agent: curl/7.43.0 > Accept: */* > User:internal > < HTTP/1.1 200 OK < Server: nginx/1.11.10 < Date: Mon, 13 Mar 2017 20:17:23 GMT < Content-Type: text/plain; charset=utf-8 < Content-Length: 43 < Connection: keep-alive < * Connection #0 to host 192.168.99.100 left intact UserID: 605394647632969758, UserRole: admin","title":"External authentication"},{"location":"examples/customization/external-auth-headers/#external-authentication-authentication-service-response-headers-propagation","text":"This example demonstrates propagation of selected authentication service response headers to a backend service. Sample configuration includes: Sample authentication service producing several response headers Authentication logic is based on HTTP header: requests with header User containing string internal are considered authenticated After successful authentication service generates response headers UserID and UserRole Sample echo service displaying header information Two ingress objects pointing to echo service Public, which allows access from unauthenticated users Private, which allows access from authenticated users only You can deploy the controller as follows: $ kubectl create -f deploy/ deployment \"demo-auth-service\" created service \"demo-auth-service\" created ingress \"demo-auth-service\" created deployment \"demo-echo-service\" created service \"demo-echo-service\" created ingress \"public-demo-echo-service\" created ingress \"secure-demo-echo-service\" created $ kubectl get po NAME READY STATUS RESTARTS AGE demo-auth-service-2769076528-7g9mh 1/1 Running 0 30s demo-echo-service-3636052215-3vw8c 1/1 Running 0 29s kubectl get ing NAME HOSTS ADDRESS PORTS AGE public-demo-echo-service public-demo-echo-service.kube.local 80 1m secure-demo-echo-service secure-demo-echo-service.kube.local 80 1m","title":"External authentication, authentication service response headers propagation"},{"location":"examples/customization/external-auth-headers/#test-1-public-service-with-no-auth-header","text":"$ curl -H 'Host: public-demo-echo-service.kube.local' -v 192 .168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) > GET / HTTP/1.1 > Host: public-demo-echo-service.kube.local > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.11.10 < Date: Mon, 13 Mar 2017 20:19:21 GMT < Content-Type: text/plain; charset=utf-8 < Content-Length: 20 < Connection: keep-alive < * Connection #0 to host 192.168.99.100 left intact UserID: , UserRole:","title":"Test 1: public service with no auth header"},{"location":"examples/customization/external-auth-headers/#test-2-secure-service-with-no-auth-header","text":"$ curl -H 'Host: secure-demo-echo-service.kube.local' -v 192 .168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) > GET / HTTP/1.1 > Host: secure-demo-echo-service.kube.local > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 403 Forbidden < Server: nginx/1.11.10 < Date: Mon, 13 Mar 2017 20:18:48 GMT < Content-Type: text/html < Content-Length: 170 < Connection: keep-alive < <html> <head><title>403 Forbidden</title></head> <body bgcolor=\"white\"> <center><h1>403 Forbidden</h1></center> <hr><center>nginx/1.11.10</center> </body> </html> * Connection #0 to host 192.168.99.100 left intact","title":"Test 2: secure service with no auth header"},{"location":"examples/customization/external-auth-headers/#test-3-public-service-with-valid-auth-header","text":"$ curl -H 'Host: public-demo-echo-service.kube.local' -H 'User:internal' -v 192 .168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) > GET / HTTP/1.1 > Host: public-demo-echo-service.kube.local > User-Agent: curl/7.43.0 > Accept: */* > User:internal > < HTTP/1.1 200 OK < Server: nginx/1.11.10 < Date: Mon, 13 Mar 2017 20:19:59 GMT < Content-Type: text/plain; charset=utf-8 < Content-Length: 44 < Connection: keep-alive < * Connection #0 to host 192.168.99.100 left intact UserID: 1443635317331776148, UserRole: admin","title":"Test 3: public service with valid auth header"},{"location":"examples/customization/external-auth-headers/#test-4-secure-service-with-valid-auth-header","text":"$ curl -H 'Host: secure-demo-echo-service.kube.local' -H 'User:internal' -v 192 .168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) > GET / HTTP/1.1 > Host: secure-demo-echo-service.kube.local > User-Agent: curl/7.43.0 > Accept: */* > User:internal > < HTTP/1.1 200 OK < Server: nginx/1.11.10 < Date: Mon, 13 Mar 2017 20:17:23 GMT < Content-Type: text/plain; charset=utf-8 < Content-Length: 43 < Connection: keep-alive < * Connection #0 to host 192.168.99.100 left intact UserID: 605394647632969758, UserRole: admin","title":"Test 4: secure service with valid auth header"},{"location":"examples/customization/jwt/","text":"Accommodation for JWT \u00b6 JWT (short for Json Web Token) is an authentication method widely used. Basically an authentication server generates a JWT and you then use this token in every request you make to a backend service. The JWT can be quite big and is present in every http headers. This means you may have to adapt the max-header size of your nginx-ingress in order to support it. Symptoms \u00b6 If you use JWT and you get http 502 error from your ingress, it may be a sign that the buffer size is not big enough. To be 100% sure look at the logs of the ingress-nginx-controller pod, you should see something like this: upstream sent too big header while reading response header from upstream... Increase buffer size for headers \u00b6 In nginx, we want to modify the property proxy-buffer-size . The size is arbitrary. It depends on your needs. Be aware that a high value can lower the performance of your ingress proxy. In general a value of 16k should get you covered. Using helm \u00b6 If you're using helm you can simply use the config properties . # -- Will add custom configuration options to Nginx https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/ config : proxy-buffer-size : 16k Manually in kubernetes config files \u00b6 If you use an already generated config from for a provider, you will have to change the controller-configmap.yaml --- # Source: ingress-nginx/templates/controller-configmap.yaml apiVersion : v1 kind : ConfigMap # ... data : #... proxy-buffer-size : \"16k\" References: * Custom Configuration","title":"Accommodation for JWT"},{"location":"examples/customization/jwt/#accommodation-for-jwt","text":"JWT (short for Json Web Token) is an authentication method widely used. Basically an authentication server generates a JWT and you then use this token in every request you make to a backend service. The JWT can be quite big and is present in every http headers. This means you may have to adapt the max-header size of your nginx-ingress in order to support it.","title":"Accommodation for JWT"},{"location":"examples/customization/jwt/#symptoms","text":"If you use JWT and you get http 502 error from your ingress, it may be a sign that the buffer size is not big enough. To be 100% sure look at the logs of the ingress-nginx-controller pod, you should see something like this: upstream sent too big header while reading response header from upstream...","title":"Symptoms"},{"location":"examples/customization/jwt/#increase-buffer-size-for-headers","text":"In nginx, we want to modify the property proxy-buffer-size . The size is arbitrary. It depends on your needs. Be aware that a high value can lower the performance of your ingress proxy. In general a value of 16k should get you covered.","title":"Increase buffer size for headers"},{"location":"examples/customization/jwt/#using-helm","text":"If you're using helm you can simply use the config properties . # -- Will add custom configuration options to Nginx https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/ config : proxy-buffer-size : 16k","title":"Using helm"},{"location":"examples/customization/jwt/#manually-in-kubernetes-config-files","text":"If you use an already generated config from for a provider, you will have to change the controller-configmap.yaml --- # Source: ingress-nginx/templates/controller-configmap.yaml apiVersion : v1 kind : ConfigMap # ... data : #... proxy-buffer-size : \"16k\" References: * Custom Configuration","title":"Manually in kubernetes config files"},{"location":"examples/customization/ssl-dh-param/","text":"Custom DH parameters for perfect forward secrecy \u00b6 This example aims to demonstrate the deployment of an nginx ingress controller and use a ConfigMap to configure a custom Diffie-Hellman parameters file to help with \"Perfect Forward Secrecy\". Custom configuration \u00b6 $ cat configmap.yaml apiVersion: v1 data: ssl-dh-param: \"ingress-nginx/lb-dhparam\" kind: ConfigMap metadata: name: ingress-nginx-controller namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx $ kubectl create -f configmap.yaml Custom DH parameters secret \u00b6 $ openssl dhparam 4096 2 > /dev/null | base64 LS0tLS1CRUdJTiBESCBQQVJBTUVURVJ... $ cat ssl-dh-param.yaml apiVersion: v1 data: dhparam.pem: \"LS0tLS1CRUdJTiBESCBQQVJBTUVURVJ...\" kind: Secret metadata: name: lb-dhparam namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx $ kubectl create -f ssl-dh-param.yaml Test \u00b6 Check the contents of the configmap is present in the nginx.conf file using: $ kubectl exec ingress-nginx-controller-873061567-4n3k2 -n kube-system -- cat /etc/nginx/nginx.conf","title":"Custom DH parameters for perfect forward secrecy"},{"location":"examples/customization/ssl-dh-param/#custom-dh-parameters-for-perfect-forward-secrecy","text":"This example aims to demonstrate the deployment of an nginx ingress controller and use a ConfigMap to configure a custom Diffie-Hellman parameters file to help with \"Perfect Forward Secrecy\".","title":"Custom DH parameters for perfect forward secrecy"},{"location":"examples/customization/ssl-dh-param/#custom-configuration","text":"$ cat configmap.yaml apiVersion: v1 data: ssl-dh-param: \"ingress-nginx/lb-dhparam\" kind: ConfigMap metadata: name: ingress-nginx-controller namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx $ kubectl create -f configmap.yaml","title":"Custom configuration"},{"location":"examples/customization/ssl-dh-param/#custom-dh-parameters-secret","text":"$ openssl dhparam 4096 2 > /dev/null | base64 LS0tLS1CRUdJTiBESCBQQVJBTUVURVJ... $ cat ssl-dh-param.yaml apiVersion: v1 data: dhparam.pem: \"LS0tLS1CRUdJTiBESCBQQVJBTUVURVJ...\" kind: Secret metadata: name: lb-dhparam namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx $ kubectl create -f ssl-dh-param.yaml","title":"Custom DH parameters secret"},{"location":"examples/customization/ssl-dh-param/#test","text":"Check the contents of the configmap is present in the nginx.conf file using: $ kubectl exec ingress-nginx-controller-873061567-4n3k2 -n kube-system -- cat /etc/nginx/nginx.conf","title":"Test"},{"location":"examples/customization/sysctl/","text":"Sysctl tuning \u00b6 This example aims to demonstrate the use of an Init Container to adjust sysctl default values using kubectl patch . kubectl patch deployment -n ingress-nginx ingress-nginx-controller \\ --patch=\"$(curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/sysctl/patch.json)\" Changes: Backlog Queue setting net.core.somaxconn from 128 to 32768 Ephemeral Ports setting net.ipv4.ip_local_port_range from 32768 60999 to 1024 65000 In a post from the NGINX blog , it is possible to see an explanation for the changes.","title":"Sysctl tuning"},{"location":"examples/customization/sysctl/#sysctl-tuning","text":"This example aims to demonstrate the use of an Init Container to adjust sysctl default values using kubectl patch . kubectl patch deployment -n ingress-nginx ingress-nginx-controller \\ --patch=\"$(curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/sysctl/patch.json)\" Changes: Backlog Queue setting net.core.somaxconn from 128 to 32768 Ephemeral Ports setting net.ipv4.ip_local_port_range from 32768 60999 to 1024 65000 In a post from the NGINX blog , it is possible to see an explanation for the changes.","title":"Sysctl tuning"},{"location":"examples/docker-registry/","text":"Docker registry \u00b6 This example demonstrates how to deploy a docker registry in the cluster and configure Ingress to enable access from the Internet. Deployment \u00b6 First we deploy the docker registry in the cluster: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/docker-registry/deployment.yaml Important DO NOT RUN THIS IN PRODUCTION This deployment uses emptyDir in the volumeMount which means the contents of the registry will be deleted when the pod dies. The next required step is creation of the ingress rules. To do this we have two options: with and without TLS Without TLS \u00b6 Download and edit the yaml deployment replacing registry.<your domain> with a valid DNS name pointing to the ingress controller: wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/docker-registry/ingress-without-tls.yaml Important Running a docker registry without TLS requires we configure our local docker daemon with the insecure registry flag. Please check deploy a plain http registry With TLS \u00b6 Download and edit the yaml deployment replacing registry.<your domain> with a valid DNS name pointing to the ingress controller: wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/docker-registry/ingress-with-tls.yaml Deploy kube lego use Let's Encrypt certificates or edit the ingress rule to use a secret with an existing SSL certificate. Testing \u00b6 To test the registry is working correctly we download a known image from docker hub , create a tag pointing to the new registry and upload the image: docker pull ubuntu:16.04 docker tag ubuntu:16.04 `registry.<your domain>/ubuntu:16.04` docker push `registry.<your domain>/ubuntu:16.04` Please replace registry.<your domain> with your domain.","title":"Docker registry"},{"location":"examples/docker-registry/#docker-registry","text":"This example demonstrates how to deploy a docker registry in the cluster and configure Ingress to enable access from the Internet.","title":"Docker registry"},{"location":"examples/docker-registry/#deployment","text":"First we deploy the docker registry in the cluster: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/docker-registry/deployment.yaml Important DO NOT RUN THIS IN PRODUCTION This deployment uses emptyDir in the volumeMount which means the contents of the registry will be deleted when the pod dies. The next required step is creation of the ingress rules. To do this we have two options: with and without TLS","title":"Deployment"},{"location":"examples/docker-registry/#without-tls","text":"Download and edit the yaml deployment replacing registry.<your domain> with a valid DNS name pointing to the ingress controller: wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/docker-registry/ingress-without-tls.yaml Important Running a docker registry without TLS requires we configure our local docker daemon with the insecure registry flag. Please check deploy a plain http registry","title":"Without TLS"},{"location":"examples/docker-registry/#with-tls","text":"Download and edit the yaml deployment replacing registry.<your domain> with a valid DNS name pointing to the ingress controller: wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/docker-registry/ingress-with-tls.yaml Deploy kube lego use Let's Encrypt certificates or edit the ingress rule to use a secret with an existing SSL certificate.","title":"With TLS"},{"location":"examples/docker-registry/#testing","text":"To test the registry is working correctly we download a known image from docker hub , create a tag pointing to the new registry and upload the image: docker pull ubuntu:16.04 docker tag ubuntu:16.04 `registry.<your domain>/ubuntu:16.04` docker push `registry.<your domain>/ubuntu:16.04` Please replace registry.<your domain> with your domain.","title":"Testing"},{"location":"examples/grpc/","text":"gRPC \u00b6 This example demonstrates how to route traffic to a gRPC service through the Ingress-NGINX controller. Prerequisites \u00b6 You have a kubernetes cluster running. You have a domain name such as example.com that is configured to route traffic to the Ingress-NGINX controller. You have the ingress-nginx-controller installed as per docs. You have a backend application running a gRPC server listening for TCP traffic. If you want, you can use https://github.com/grpc/grpc-go/blob/91e0aeb192456225adf27966d04ada4cf8599915/examples/features/reflection/server/main.go as an example. You're also responsible for provisioning an SSL certificate for the ingress. So you need to have a valid SSL certificate, deployed as a Kubernetes secret of type tls , in the same namespace as the gRPC application. Step 1: Create a Kubernetes Deployment for gRPC app \u00b6 Make sure your gRPC application pod is running and listening for connections. For example you can try a kubectl command like this below: $ kubectl get po -A -o wide | grep go-grpc-greeter-server If you have a gRPC app deployed in your cluster, then skip further notes in this Step 1, and continue from Step 2 below. As an example gRPC application, we can use this app https://github.com/grpc/grpc-go/blob/91e0aeb192456225adf27966d04ada4cf8599915/examples/features/reflection/server/main.go . To create a container image for this app, you can use this Dockerfile . If you use the Dockerfile mentioned above, to create a image, then you can use the following example Kubernetes manifest to create a deployment resource that uses that image. If necessary edit this manifest to suit your needs. cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: labels: app: go-grpc-greeter-server name: go-grpc-greeter-server spec: replicas: 1 selector: matchLabels: app: go-grpc-greeter-server template: metadata: labels: app: go-grpc-greeter-server spec: containers: - image: <reponame>/go-grpc-greeter-server # Edit this for your reponame resources: limits: cpu: 100m memory: 100Mi requests: cpu: 50m memory: 50Mi name: go-grpc-greeter-server ports: - containerPort: 50051 EOF Step 2: Create the Kubernetes Service for the gRPC app \u00b6 You can use the following example manifest to create a service of type ClusterIP. Edit the name/namespace/label/port to match your deployment/pod. cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Service metadata: labels: app: go-grpc-greeter-server name: go-grpc-greeter-server spec: ports: - port: 80 protocol: TCP targetPort: 50051 selector: app: go-grpc-greeter-server type: ClusterIP EOF You can save the above example manifest to a file with name service.go-grpc-greeter-server.yaml and edit it to match your deployment/pod, if required. You can create the service resource with a kubectl command like this: $ kubectl create -f service.go-grpc-greeter-server.yaml Step 3: Create the Kubernetes Ingress resource for the gRPC app \u00b6 Use the following example manifest of a ingress resource to create a ingress for your grpc app. If required, edit it to match your app's details like name, namespace, service, secret etc. Make sure you have the required SSL-Certificate, existing in your Kubernetes cluster in the same namespace where the gRPC app is. The certificate must be available as a kubernetes secret resource, of type \"kubernetes.io/tls\" https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets. This is because we are terminating TLS on the ingress. cat <<EOF | kubectl apply -f - apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/ssl-redirect: \"true\" nginx.ingress.kubernetes.io/backend-protocol: \"GRPC\" name: fortune-ingress namespace: default spec: ingressClassName: nginx rules: - host: grpctest.dev.mydomain.com http: paths: - path: / pathType: Prefix backend: service: name: go-grpc-greeter-server port: number: 80 tls: # This secret must exist beforehand # The cert must also contain the subj-name grpctest.dev.mydomain.com # https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/PREREQUISITES.md#tls-certificates - secretName: wildcard.dev.mydomain.com hosts: - grpctest.dev.mydomain.com EOF If you save the above example manifest as a file named ingress.go-grpc-greeter-server.yaml and edit it to match your deployment and service, you can create the ingress like this: $ kubectl create -f ingress.go-grpc-greeter-server.yaml The takeaway is that we are not doing any TLS configuration on the server (as we are terminating TLS at the ingress level, gRPC traffic will travel unencrypted inside the cluster and arrive \"insecure\"). For your own application you may or may not want to do this. If you prefer to forward encrypted traffic to your POD and terminate TLS at the gRPC server itself, add the ingress annotation nginx.ingress.kubernetes.io/backend-protocol: \"GRPCS\" . A few more things to note: We've tagged the ingress with the annotation nginx.ingress.kubernetes.io/backend-protocol: \"GRPC\" . This is the magic ingredient that sets up the appropriate nginx configuration to route http/2 traffic to our service. We're terminating TLS at the ingress and have configured an SSL certificate wildcard.dev.mydomain.com . The ingress matches traffic arriving as https://grpctest.dev.mydomain.com:443 and routes unencrypted messages to the backend Kubernetes service. Step 4: test the connection \u00b6 Once we've applied our configuration to Kubernetes, it's time to test that we can actually talk to the backend. To do this, we'll use the grpcurl utility: $ grpcurl grpctest.dev.mydomain.com:443 helloworld.Greeter/SayHello { \"message\": \"Hello \" } Debugging Hints \u00b6 Obviously, watch the logs on your app. Watch the logs for the ingress-nginx-controller (increasing verbosity as needed). Double-check your address and ports. Set the GODEBUG=http2debug=2 environment variable to get detailed http/2 logging on the client and/or server. Study RFC 7540 (http/2) https://tools.ietf.org/html/rfc7540 . If you are developing public gRPC endpoints, check out https://proto.stack.build, a protocol buffer / gRPC build service that can use to help make it easier for your users to consume your API. See also the specific gRPC settings of NGINX: https://nginx.org/en/docs/http/ngx_http_grpc_module.html Notes on using response/request streams \u00b6 If your server only does response streaming and you expect a stream to be open longer than 60 seconds, you will have to change the grpc_read_timeout to accommodate this. If your service only does request streaming and you expect a stream to be open longer than 60 seconds, you have to change the grpc_send_timeout and the client_body_timeout . If you do both response and request streaming with an open stream longer than 60 seconds, you have to change all three timeouts: grpc_read_timeout , grpc_send_timeout and client_body_timeout . Values for the timeouts must be specified as e.g. \"1200s\" . On the most recent versions of ingress-nginx, changing these timeouts requires using the nginx.ingress.kubernetes.io/server-snippet annotation. There are plans for future releases to allow using the Kubernetes annotations to define each timeout separately.","title":"gRPC"},{"location":"examples/grpc/#grpc","text":"This example demonstrates how to route traffic to a gRPC service through the Ingress-NGINX controller.","title":"gRPC"},{"location":"examples/grpc/#prerequisites","text":"You have a kubernetes cluster running. You have a domain name such as example.com that is configured to route traffic to the Ingress-NGINX controller. You have the ingress-nginx-controller installed as per docs. You have a backend application running a gRPC server listening for TCP traffic. If you want, you can use https://github.com/grpc/grpc-go/blob/91e0aeb192456225adf27966d04ada4cf8599915/examples/features/reflection/server/main.go as an example. You're also responsible for provisioning an SSL certificate for the ingress. So you need to have a valid SSL certificate, deployed as a Kubernetes secret of type tls , in the same namespace as the gRPC application.","title":"Prerequisites"},{"location":"examples/grpc/#step-1-create-a-kubernetes-deployment-for-grpc-app","text":"Make sure your gRPC application pod is running and listening for connections. For example you can try a kubectl command like this below: $ kubectl get po -A -o wide | grep go-grpc-greeter-server If you have a gRPC app deployed in your cluster, then skip further notes in this Step 1, and continue from Step 2 below. As an example gRPC application, we can use this app https://github.com/grpc/grpc-go/blob/91e0aeb192456225adf27966d04ada4cf8599915/examples/features/reflection/server/main.go . To create a container image for this app, you can use this Dockerfile . If you use the Dockerfile mentioned above, to create a image, then you can use the following example Kubernetes manifest to create a deployment resource that uses that image. If necessary edit this manifest to suit your needs. cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: labels: app: go-grpc-greeter-server name: go-grpc-greeter-server spec: replicas: 1 selector: matchLabels: app: go-grpc-greeter-server template: metadata: labels: app: go-grpc-greeter-server spec: containers: - image: <reponame>/go-grpc-greeter-server # Edit this for your reponame resources: limits: cpu: 100m memory: 100Mi requests: cpu: 50m memory: 50Mi name: go-grpc-greeter-server ports: - containerPort: 50051 EOF","title":"Step 1: Create a Kubernetes Deployment for gRPC app"},{"location":"examples/grpc/#step-2-create-the-kubernetes-service-for-the-grpc-app","text":"You can use the following example manifest to create a service of type ClusterIP. Edit the name/namespace/label/port to match your deployment/pod. cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Service metadata: labels: app: go-grpc-greeter-server name: go-grpc-greeter-server spec: ports: - port: 80 protocol: TCP targetPort: 50051 selector: app: go-grpc-greeter-server type: ClusterIP EOF You can save the above example manifest to a file with name service.go-grpc-greeter-server.yaml and edit it to match your deployment/pod, if required. You can create the service resource with a kubectl command like this: $ kubectl create -f service.go-grpc-greeter-server.yaml","title":"Step 2: Create the Kubernetes Service for the gRPC app"},{"location":"examples/grpc/#step-3-create-the-kubernetes-ingress-resource-for-the-grpc-app","text":"Use the following example manifest of a ingress resource to create a ingress for your grpc app. If required, edit it to match your app's details like name, namespace, service, secret etc. Make sure you have the required SSL-Certificate, existing in your Kubernetes cluster in the same namespace where the gRPC app is. The certificate must be available as a kubernetes secret resource, of type \"kubernetes.io/tls\" https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets. This is because we are terminating TLS on the ingress. cat <<EOF | kubectl apply -f - apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/ssl-redirect: \"true\" nginx.ingress.kubernetes.io/backend-protocol: \"GRPC\" name: fortune-ingress namespace: default spec: ingressClassName: nginx rules: - host: grpctest.dev.mydomain.com http: paths: - path: / pathType: Prefix backend: service: name: go-grpc-greeter-server port: number: 80 tls: # This secret must exist beforehand # The cert must also contain the subj-name grpctest.dev.mydomain.com # https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/PREREQUISITES.md#tls-certificates - secretName: wildcard.dev.mydomain.com hosts: - grpctest.dev.mydomain.com EOF If you save the above example manifest as a file named ingress.go-grpc-greeter-server.yaml and edit it to match your deployment and service, you can create the ingress like this: $ kubectl create -f ingress.go-grpc-greeter-server.yaml The takeaway is that we are not doing any TLS configuration on the server (as we are terminating TLS at the ingress level, gRPC traffic will travel unencrypted inside the cluster and arrive \"insecure\"). For your own application you may or may not want to do this. If you prefer to forward encrypted traffic to your POD and terminate TLS at the gRPC server itself, add the ingress annotation nginx.ingress.kubernetes.io/backend-protocol: \"GRPCS\" . A few more things to note: We've tagged the ingress with the annotation nginx.ingress.kubernetes.io/backend-protocol: \"GRPC\" . This is the magic ingredient that sets up the appropriate nginx configuration to route http/2 traffic to our service. We're terminating TLS at the ingress and have configured an SSL certificate wildcard.dev.mydomain.com . The ingress matches traffic arriving as https://grpctest.dev.mydomain.com:443 and routes unencrypted messages to the backend Kubernetes service.","title":"Step 3: Create the Kubernetes Ingress resource for the gRPC app"},{"location":"examples/grpc/#step-4-test-the-connection","text":"Once we've applied our configuration to Kubernetes, it's time to test that we can actually talk to the backend. To do this, we'll use the grpcurl utility: $ grpcurl grpctest.dev.mydomain.com:443 helloworld.Greeter/SayHello { \"message\": \"Hello \" }","title":"Step 4: test the connection"},{"location":"examples/grpc/#debugging-hints","text":"Obviously, watch the logs on your app. Watch the logs for the ingress-nginx-controller (increasing verbosity as needed). Double-check your address and ports. Set the GODEBUG=http2debug=2 environment variable to get detailed http/2 logging on the client and/or server. Study RFC 7540 (http/2) https://tools.ietf.org/html/rfc7540 . If you are developing public gRPC endpoints, check out https://proto.stack.build, a protocol buffer / gRPC build service that can use to help make it easier for your users to consume your API. See also the specific gRPC settings of NGINX: https://nginx.org/en/docs/http/ngx_http_grpc_module.html","title":"Debugging Hints"},{"location":"examples/grpc/#notes-on-using-responserequest-streams","text":"If your server only does response streaming and you expect a stream to be open longer than 60 seconds, you will have to change the grpc_read_timeout to accommodate this. If your service only does request streaming and you expect a stream to be open longer than 60 seconds, you have to change the grpc_send_timeout and the client_body_timeout . If you do both response and request streaming with an open stream longer than 60 seconds, you have to change all three timeouts: grpc_read_timeout , grpc_send_timeout and client_body_timeout . Values for the timeouts must be specified as e.g. \"1200s\" . On the most recent versions of ingress-nginx, changing these timeouts requires using the nginx.ingress.kubernetes.io/server-snippet annotation. There are plans for future releases to allow using the Kubernetes annotations to define each timeout separately.","title":"Notes on using response/request streams"},{"location":"examples/multi-tls/","text":"Multi TLS certificate termination \u00b6 This example uses 2 different certificates to terminate SSL for 2 hostnames. Create tls secrets for foo.bar.com and bar.baz.com as indicated in the yaml Create multi-tls.yaml This should generate a segment like: $ kubectl exec -it ingress-nginx-controller-6vwd1 -- cat /etc/nginx/nginx.conf | grep \"foo.bar.com\" -B 7 -A 35 server { listen 80; listen 443 ssl http2; ssl_certificate /etc/nginx-ssl/default-foobar.pem; ssl_certificate_key /etc/nginx-ssl/default-foobar.pem; server_name foo.bar.com; if ($scheme = http) { return 301 https://$host$request_uri; } location / { proxy_set_header Host $host; # Pass Real IP proxy_set_header X-Real-IP $remote_addr; # Allow websocket connections proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Proto $pass_access_scheme; proxy_connect_timeout 5s; proxy_send_timeout 60s; proxy_read_timeout 60s; proxy_redirect off; proxy_buffering off; proxy_http_version 1.1; proxy_pass http://default-http-svc-80; } And you should be able to reach your nginx service or http-svc service using a hostname switch: $ kubectl get ing NAME RULE BACKEND ADDRESS AGE foo-tls - 104.154.30.67 13m foo.bar.com / http-svc:80 bar.baz.com / nginx:80 $ curl https://104.154.30.67 -H 'Host:foo.bar.com' -k CLIENT VALUES: client_address=10.245.0.6 command=GET real path=/ query=nil request_version=1.1 request_uri=http://foo.bar.com:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=close host=foo.bar.com user-agent=curl/7.35.0 x-forwarded-for=10.245.0.1 x-forwarded-host=foo.bar.com x-forwarded-proto=https $ curl https://104.154.30.67 -H 'Host:bar.baz.com' -k <!DOCTYPE html> <html> <head> <title>Welcome to nginx on Debian!</title> $ curl 104 .154.30.67 default backend - 404","title":"Multi TLS certificate termination"},{"location":"examples/multi-tls/#multi-tls-certificate-termination","text":"This example uses 2 different certificates to terminate SSL for 2 hostnames. Create tls secrets for foo.bar.com and bar.baz.com as indicated in the yaml Create multi-tls.yaml This should generate a segment like: $ kubectl exec -it ingress-nginx-controller-6vwd1 -- cat /etc/nginx/nginx.conf | grep \"foo.bar.com\" -B 7 -A 35 server { listen 80; listen 443 ssl http2; ssl_certificate /etc/nginx-ssl/default-foobar.pem; ssl_certificate_key /etc/nginx-ssl/default-foobar.pem; server_name foo.bar.com; if ($scheme = http) { return 301 https://$host$request_uri; } location / { proxy_set_header Host $host; # Pass Real IP proxy_set_header X-Real-IP $remote_addr; # Allow websocket connections proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Proto $pass_access_scheme; proxy_connect_timeout 5s; proxy_send_timeout 60s; proxy_read_timeout 60s; proxy_redirect off; proxy_buffering off; proxy_http_version 1.1; proxy_pass http://default-http-svc-80; } And you should be able to reach your nginx service or http-svc service using a hostname switch: $ kubectl get ing NAME RULE BACKEND ADDRESS AGE foo-tls - 104.154.30.67 13m foo.bar.com / http-svc:80 bar.baz.com / nginx:80 $ curl https://104.154.30.67 -H 'Host:foo.bar.com' -k CLIENT VALUES: client_address=10.245.0.6 command=GET real path=/ query=nil request_version=1.1 request_uri=http://foo.bar.com:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=close host=foo.bar.com user-agent=curl/7.35.0 x-forwarded-for=10.245.0.1 x-forwarded-host=foo.bar.com x-forwarded-proto=https $ curl https://104.154.30.67 -H 'Host:bar.baz.com' -k <!DOCTYPE html> <html> <head> <title>Welcome to nginx on Debian!</title> $ curl 104 .154.30.67 default backend - 404","title":"Multi TLS certificate termination"},{"location":"examples/psp/","text":"Pod Security Policy (PSP) \u00b6 In most clusters today, by default, all resources (e.g. Deployments and ReplicatSets ) have permissions to create pods. Kubernetes however provides a more fine-grained authorization policy called Pod Security Policy (PSP) . PSP allows the cluster owner to define the permission of each object, for example creating a pod. If you have PSP enabled on the cluster, and you deploy ingress-nginx, you will need to provide the Deployment with the permissions to create pods. Before applying any objects, first apply the PSP permissions by running: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/psp/psp.yaml Note: PSP permissions must be granted before the creation of the Deployment and the ReplicaSet .","title":"Pod Security Policy (PSP)"},{"location":"examples/psp/#pod-security-policy-psp","text":"In most clusters today, by default, all resources (e.g. Deployments and ReplicatSets ) have permissions to create pods. Kubernetes however provides a more fine-grained authorization policy called Pod Security Policy (PSP) . PSP allows the cluster owner to define the permission of each object, for example creating a pod. If you have PSP enabled on the cluster, and you deploy ingress-nginx, you will need to provide the Deployment with the permissions to create pods. Before applying any objects, first apply the PSP permissions by running: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/psp/psp.yaml Note: PSP permissions must be granted before the creation of the Deployment and the ReplicaSet .","title":"Pod Security Policy (PSP)"},{"location":"examples/rewrite/","text":"Rewrite \u00b6 This example demonstrates how to use Rewrite annotations. Prerequisites \u00b6 You will need to make sure your Ingress targets exactly one Ingress controller by specifying the ingress.class annotation , and that you have an ingress controller running in your cluster. Deployment \u00b6 Rewriting can be controlled using the following annotations: Name Description Values nginx.ingress.kubernetes.io/rewrite-target Target URI where the traffic must be redirected string nginx.ingress.kubernetes.io/ssl-redirect Indicates if the location section is only accessible via SSL (defaults to True when Ingress contains a Certificate) bool nginx.ingress.kubernetes.io/force-ssl-redirect Forces the redirection to HTTPS even if the Ingress is not TLS Enabled bool nginx.ingress.kubernetes.io/app-root Defines the Application Root that the Controller must redirect if it's in / context string nginx.ingress.kubernetes.io/use-regex Indicates if the paths defined on an Ingress use regular expressions bool Examples \u00b6 Rewrite Target \u00b6 Attention Starting in Version 0.22.0, ingress definitions using the annotation nginx.ingress.kubernetes.io/rewrite-target are not backwards compatible with previous versions. In Version 0.22.0 and beyond, any substrings within the request URI that need to be passed to the rewritten path must explicitly be defined in a capture group . Note Captured groups are saved in numbered placeholders, chronologically, in the form $1 , $2 ... $n . These placeholders can be used as parameters in the rewrite-target annotation. Create an Ingress rule with a rewrite annotation: $ echo ' apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/rewrite-target: /$2 name: rewrite namespace: default spec: ingressClassName: nginx rules: - host: rewrite.bar.com http: paths: - path: /something(/|$)(.*) pathType: Prefix backend: service: name: http-svc port: number: 80 ' | kubectl create -f - In this ingress definition, any characters captured by (.*) will be assigned to the placeholder $2 , which is then used as a parameter in the rewrite-target annotation. For example, the ingress definition above will result in the following rewrites: rewrite.bar.com/something rewrites to rewrite.bar.com/ rewrite.bar.com/something/ rewrites to rewrite.bar.com/ rewrite.bar.com/something/new rewrites to rewrite.bar.com/new App Root \u00b6 Create an Ingress rule with an app-root annotation: $ echo \" apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/app-root: /app1 name: approot namespace: default spec: ingressClassName: nginx rules: - host: approot.bar.com http: paths: - path: / pathType: Prefix backend: service: name: http-svc port: number: 80 \" | kubectl create -f - Check the rewrite is working $ curl -I -k http://approot.bar.com/ HTTP/1.1 302 Moved Temporarily Server: nginx/1.11.10 Date: Mon, 13 Mar 2017 14:57:15 GMT Content-Type: text/html Content-Length: 162 Location: http://approot.bar.com/app1 Connection: keep-alive","title":"Rewrite"},{"location":"examples/rewrite/#rewrite","text":"This example demonstrates how to use Rewrite annotations.","title":"Rewrite"},{"location":"examples/rewrite/#prerequisites","text":"You will need to make sure your Ingress targets exactly one Ingress controller by specifying the ingress.class annotation , and that you have an ingress controller running in your cluster.","title":"Prerequisites"},{"location":"examples/rewrite/#deployment","text":"Rewriting can be controlled using the following annotations: Name Description Values nginx.ingress.kubernetes.io/rewrite-target Target URI where the traffic must be redirected string nginx.ingress.kubernetes.io/ssl-redirect Indicates if the location section is only accessible via SSL (defaults to True when Ingress contains a Certificate) bool nginx.ingress.kubernetes.io/force-ssl-redirect Forces the redirection to HTTPS even if the Ingress is not TLS Enabled bool nginx.ingress.kubernetes.io/app-root Defines the Application Root that the Controller must redirect if it's in / context string nginx.ingress.kubernetes.io/use-regex Indicates if the paths defined on an Ingress use regular expressions bool","title":"Deployment"},{"location":"examples/rewrite/#examples","text":"","title":"Examples"},{"location":"examples/rewrite/#rewrite-target","text":"Attention Starting in Version 0.22.0, ingress definitions using the annotation nginx.ingress.kubernetes.io/rewrite-target are not backwards compatible with previous versions. In Version 0.22.0 and beyond, any substrings within the request URI that need to be passed to the rewritten path must explicitly be defined in a capture group . Note Captured groups are saved in numbered placeholders, chronologically, in the form $1 , $2 ... $n . These placeholders can be used as parameters in the rewrite-target annotation. Create an Ingress rule with a rewrite annotation: $ echo ' apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/rewrite-target: /$2 name: rewrite namespace: default spec: ingressClassName: nginx rules: - host: rewrite.bar.com http: paths: - path: /something(/|$)(.*) pathType: Prefix backend: service: name: http-svc port: number: 80 ' | kubectl create -f - In this ingress definition, any characters captured by (.*) will be assigned to the placeholder $2 , which is then used as a parameter in the rewrite-target annotation. For example, the ingress definition above will result in the following rewrites: rewrite.bar.com/something rewrites to rewrite.bar.com/ rewrite.bar.com/something/ rewrites to rewrite.bar.com/ rewrite.bar.com/something/new rewrites to rewrite.bar.com/new","title":"Rewrite Target"},{"location":"examples/rewrite/#app-root","text":"Create an Ingress rule with an app-root annotation: $ echo \" apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/app-root: /app1 name: approot namespace: default spec: ingressClassName: nginx rules: - host: approot.bar.com http: paths: - path: / pathType: Prefix backend: service: name: http-svc port: number: 80 \" | kubectl create -f - Check the rewrite is working $ curl -I -k http://approot.bar.com/ HTTP/1.1 302 Moved Temporarily Server: nginx/1.11.10 Date: Mon, 13 Mar 2017 14:57:15 GMT Content-Type: text/html Content-Length: 162 Location: http://approot.bar.com/app1 Connection: keep-alive","title":"App Root"},{"location":"examples/static-ip/","text":"Static IPs \u00b6 This example demonstrates how to assign a static-ip to an Ingress on through the Ingress-NGINX controller. Prerequisites \u00b6 You need a TLS cert and a test HTTP service for this example. You will also need to make sure your Ingress targets exactly one Ingress controller by specifying the ingress.class annotation , and that you have an ingress controller running in your cluster. Acquiring an IP \u00b6 Since instances of the ingress nginx controller actually run on nodes in your cluster, by default nginx Ingresses will only get static IPs if your cloudprovider supports static IP assignments to nodes. On GKE/GCE for example, even though nodes get static IPs, the IPs are not retained across upgrades. To acquire a static IP for the ingress-nginx-controller, simply put it behind a Service of Type=LoadBalancer . First, create a loadbalancer Service and wait for it to acquire an IP: $ kubectl create -f static-ip-svc.yaml service \"ingress-nginx-lb\" created $ kubectl get svc ingress-nginx-lb NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-lb 10.0.138.113 104.154.109.191 80:31457/TCP,443:32240/TCP 15m Then, update the ingress controller so it adopts the static IP of the Service by passing the --publish-service flag (the example yaml used in the next step already has it set to \"ingress-nginx-lb\"). $ kubectl create -f ingress-nginx-controller.yaml deployment \"ingress-nginx-controller\" created Assigning the IP to an Ingress \u00b6 From here on every Ingress created with the ingress.class annotation set to nginx will get the IP allocated in the previous step. $ kubectl create -f ingress-nginx.yaml ingress \"ingress-nginx\" created $ kubectl get ing ingress-nginx NAME HOSTS ADDRESS PORTS AGE ingress-nginx * 104.154.109.191 80, 443 13m $ curl 104 .154.109.191 -kL CLIENT VALUES: client_address=10.180.1.25 command=GET real path=/ query=nil request_version=1.1 request_uri=http://104.154.109.191:8080/ ... Retaining the IP \u00b6 You can test retention by deleting the Ingress: $ kubectl delete ing ingress-nginx ingress \"ingress-nginx\" deleted $ kubectl create -f ingress-nginx.yaml ingress \"ingress-nginx\" created $ kubectl get ing ingress-nginx NAME HOSTS ADDRESS PORTS AGE ingress-nginx * 104.154.109.191 80, 443 13m Note that unlike the GCE Ingress, the same loadbalancer IP is shared amongst all Ingresses, because all requests are proxied through the same set of nginx controllers. Promote ephemeral to static IP \u00b6 To promote the allocated IP to static, you can update the Service manifest: $ kubectl patch svc ingress-nginx-lb -p '{\"spec\": {\"loadBalancerIP\": \"104.154.109.191\"}}' \"ingress-nginx-lb\" patched ... and promote the IP to static (promotion works differently for cloudproviders, provided example is for GKE/GCE): $ gcloud compute addresses create ingress-nginx-lb --addresses 104 .154.109.191 --region us-central1 Created [https://www.googleapis.com/compute/v1/projects/kubernetesdev/regions/us-central1/addresses/ingress-nginx-lb]. --- address: 104.154.109.191 creationTimestamp: '2017-01-31T16:34:50.089-08:00' description: '' id: '5208037144487826373' kind: compute#address name: ingress-nginx-lb region: us-central1 selfLink: https://www.googleapis.com/compute/v1/projects/kubernetesdev/regions/us-central1/addresses/ingress-nginx-lb status: IN_USE users: - us-central1/forwardingRules/a09f6913ae80e11e6a8c542010af0000 Now even if the Service is deleted, the IP will persist, so you can recreate the Service with spec.loadBalancerIP set to 104.154.109.191 .","title":"Static IPs"},{"location":"examples/static-ip/#static-ips","text":"This example demonstrates how to assign a static-ip to an Ingress on through the Ingress-NGINX controller.","title":"Static IPs"},{"location":"examples/static-ip/#prerequisites","text":"You need a TLS cert and a test HTTP service for this example. You will also need to make sure your Ingress targets exactly one Ingress controller by specifying the ingress.class annotation , and that you have an ingress controller running in your cluster.","title":"Prerequisites"},{"location":"examples/static-ip/#acquiring-an-ip","text":"Since instances of the ingress nginx controller actually run on nodes in your cluster, by default nginx Ingresses will only get static IPs if your cloudprovider supports static IP assignments to nodes. On GKE/GCE for example, even though nodes get static IPs, the IPs are not retained across upgrades. To acquire a static IP for the ingress-nginx-controller, simply put it behind a Service of Type=LoadBalancer . First, create a loadbalancer Service and wait for it to acquire an IP: $ kubectl create -f static-ip-svc.yaml service \"ingress-nginx-lb\" created $ kubectl get svc ingress-nginx-lb NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-lb 10.0.138.113 104.154.109.191 80:31457/TCP,443:32240/TCP 15m Then, update the ingress controller so it adopts the static IP of the Service by passing the --publish-service flag (the example yaml used in the next step already has it set to \"ingress-nginx-lb\"). $ kubectl create -f ingress-nginx-controller.yaml deployment \"ingress-nginx-controller\" created","title":"Acquiring an IP"},{"location":"examples/static-ip/#assigning-the-ip-to-an-ingress","text":"From here on every Ingress created with the ingress.class annotation set to nginx will get the IP allocated in the previous step. $ kubectl create -f ingress-nginx.yaml ingress \"ingress-nginx\" created $ kubectl get ing ingress-nginx NAME HOSTS ADDRESS PORTS AGE ingress-nginx * 104.154.109.191 80, 443 13m $ curl 104 .154.109.191 -kL CLIENT VALUES: client_address=10.180.1.25 command=GET real path=/ query=nil request_version=1.1 request_uri=http://104.154.109.191:8080/ ...","title":"Assigning the IP to an Ingress"},{"location":"examples/static-ip/#retaining-the-ip","text":"You can test retention by deleting the Ingress: $ kubectl delete ing ingress-nginx ingress \"ingress-nginx\" deleted $ kubectl create -f ingress-nginx.yaml ingress \"ingress-nginx\" created $ kubectl get ing ingress-nginx NAME HOSTS ADDRESS PORTS AGE ingress-nginx * 104.154.109.191 80, 443 13m Note that unlike the GCE Ingress, the same loadbalancer IP is shared amongst all Ingresses, because all requests are proxied through the same set of nginx controllers.","title":"Retaining the IP"},{"location":"examples/static-ip/#promote-ephemeral-to-static-ip","text":"To promote the allocated IP to static, you can update the Service manifest: $ kubectl patch svc ingress-nginx-lb -p '{\"spec\": {\"loadBalancerIP\": \"104.154.109.191\"}}' \"ingress-nginx-lb\" patched ... and promote the IP to static (promotion works differently for cloudproviders, provided example is for GKE/GCE): $ gcloud compute addresses create ingress-nginx-lb --addresses 104 .154.109.191 --region us-central1 Created [https://www.googleapis.com/compute/v1/projects/kubernetesdev/regions/us-central1/addresses/ingress-nginx-lb]. --- address: 104.154.109.191 creationTimestamp: '2017-01-31T16:34:50.089-08:00' description: '' id: '5208037144487826373' kind: compute#address name: ingress-nginx-lb region: us-central1 selfLink: https://www.googleapis.com/compute/v1/projects/kubernetesdev/regions/us-central1/addresses/ingress-nginx-lb status: IN_USE users: - us-central1/forwardingRules/a09f6913ae80e11e6a8c542010af0000 Now even if the Service is deleted, the IP will persist, so you can recreate the Service with spec.loadBalancerIP set to 104.154.109.191 .","title":"Promote ephemeral to static IP"},{"location":"examples/tls-termination/","text":"TLS termination \u00b6 This example demonstrates how to terminate TLS through the nginx Ingress controller. Prerequisites \u00b6 You need a TLS cert and a test HTTP service for this example. Deployment \u00b6 Create a ingress.yaml file. apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : nginx-test spec : tls : - hosts : - foo.bar.com # This assumes tls-secret exists and the SSL # certificate contains a CN for foo.bar.com secretName : tls-secret ingressClassName : nginx rules : - host : foo.bar.com http : paths : - path : / pathType : Prefix backend : # This assumes http-svc exists and routes to healthy endpoints service : name : http-svc port : number : 80 The following command instructs the controller to terminate traffic using the provided TLS cert, and forward un-encrypted HTTP traffic to the test HTTP service. kubectl apply -f ingress.yaml Validation \u00b6 You can confirm that the Ingress works. $ kubectl describe ing nginx-test Name: nginx-test Namespace: default Address: 104.198.183.6 Default backend: default-http-backend:80 (10.180.0.4:8080,10.240.0.2:8080) TLS: tls-secret terminates Rules: Host Path Backends ---- ---- -------- * http-svc:80 (<none>) Annotations: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 7s 7s 1 {ingress-nginx-controller } Normal CREATE default/nginx-test 7s 7s 1 {ingress-nginx-controller } Normal UPDATE default/nginx-test 7s 7s 1 {ingress-nginx-controller } Normal CREATE ip: 104.198.183.6 7s 7s 1 {ingress-nginx-controller } Warning MAPPING Ingress rule 'default/nginx-test' contains no path definition. Assuming / $ curl 104 .198.183.6 -L curl: (60) SSL certificate problem: self signed certificate More details here: http://curl.haxx.se/docs/sslcerts.html $ curl 104 .198.183.6 -Lk CLIENT VALUES: client_address=10.240.0.4 command=GET real path=/ query=nil request_version=1.1 request_uri=http://35.186.221.137:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=Keep-Alive host=35.186.221.137 user-agent=curl/7.46.0 via=1.1 google x-cloud-trace-context=f708ea7e369d4514fc90d51d7e27e91d/13322322294276298106 x-forwarded-for=104.132.0.80, 35.186.221.137 x-forwarded-proto=https BODY:","title":"TLS termination"},{"location":"examples/tls-termination/#tls-termination","text":"This example demonstrates how to terminate TLS through the nginx Ingress controller.","title":"TLS termination"},{"location":"examples/tls-termination/#prerequisites","text":"You need a TLS cert and a test HTTP service for this example.","title":"Prerequisites"},{"location":"examples/tls-termination/#deployment","text":"Create a ingress.yaml file. apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : nginx-test spec : tls : - hosts : - foo.bar.com # This assumes tls-secret exists and the SSL # certificate contains a CN for foo.bar.com secretName : tls-secret ingressClassName : nginx rules : - host : foo.bar.com http : paths : - path : / pathType : Prefix backend : # This assumes http-svc exists and routes to healthy endpoints service : name : http-svc port : number : 80 The following command instructs the controller to terminate traffic using the provided TLS cert, and forward un-encrypted HTTP traffic to the test HTTP service. kubectl apply -f ingress.yaml","title":"Deployment"},{"location":"examples/tls-termination/#validation","text":"You can confirm that the Ingress works. $ kubectl describe ing nginx-test Name: nginx-test Namespace: default Address: 104.198.183.6 Default backend: default-http-backend:80 (10.180.0.4:8080,10.240.0.2:8080) TLS: tls-secret terminates Rules: Host Path Backends ---- ---- -------- * http-svc:80 (<none>) Annotations: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 7s 7s 1 {ingress-nginx-controller } Normal CREATE default/nginx-test 7s 7s 1 {ingress-nginx-controller } Normal UPDATE default/nginx-test 7s 7s 1 {ingress-nginx-controller } Normal CREATE ip: 104.198.183.6 7s 7s 1 {ingress-nginx-controller } Warning MAPPING Ingress rule 'default/nginx-test' contains no path definition. Assuming / $ curl 104 .198.183.6 -L curl: (60) SSL certificate problem: self signed certificate More details here: http://curl.haxx.se/docs/sslcerts.html $ curl 104 .198.183.6 -Lk CLIENT VALUES: client_address=10.240.0.4 command=GET real path=/ query=nil request_version=1.1 request_uri=http://35.186.221.137:8080/ SERVER VALUES: server_version=nginx: 1.9.11 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=Keep-Alive host=35.186.221.137 user-agent=curl/7.46.0 via=1.1 google x-cloud-trace-context=f708ea7e369d4514fc90d51d7e27e91d/13322322294276298106 x-forwarded-for=104.132.0.80, 35.186.221.137 x-forwarded-proto=https BODY:","title":"Validation"},{"location":"user-guide/basic-usage/","text":"Basic usage - host based routing \u00b6 ingress-nginx can be used for many use cases, inside various cloud provider and supports a lot of configurations. In this section you can find a common usage scenario where a single load balancer powered by ingress-nginx will route traffic to 2 different HTTP backend services based on the host name. First of all follow the instructions to install ingress-nginx. Then imagine that you need to expose 2 HTTP services already installed, myServiceA , myServiceB , and configured as type: ClusterIP . Let's say that you want to expose the first at myServiceA.foo.org and the second at myServiceB.foo.org . If the cluster version is < 1.19, you can create two ingress resources like this: apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-myservicea spec: ingressClassName: nginx rules: - host: myservicea.foo.org http: paths: - path: / backend: serviceName: myservicea servicePort: 80 --- apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-myserviceb annotations: # use the shared ingress-nginx kubernetes.io/ingress.class: \"nginx\" spec: rules: - host: myserviceb.foo.org http: paths: - path: / backend: serviceName: myserviceb servicePort: 80 If the cluster uses Kubernetes version >= 1.19.x, then its suggested to create 2 ingress resources, using yaml examples shown below. These examples are in conformity with the networking.kubernetes.io/v1 api. apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-myservicea spec: rules: - host: myservicea.foo.org http: paths: - path: / pathType: Prefix backend: service: name: myservicea port: number: 80 ingressClassName: nginx --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-myserviceb spec: rules: - host: myserviceb.foo.org http: paths: - path: / pathType: Prefix backend: service: name: myserviceb port: number: 80 ingressClassName: nginx When you apply this yaml, 2 ingress resources will be created managed by the ingress-nginx instance. Nginx is configured to automatically discover all ingress with the kubernetes.io/ingress.class: \"nginx\" annotation or where ingressClassName: nginx is present. Please note that the ingress resource should be placed inside the same namespace of the backend resource. On many cloud providers ingress-nginx will also create the corresponding Load Balancer resource. All you have to do is get the external IP and add a DNS A record inside your DNS provider that point myservicea.foo.org and myserviceb.foo.org to the nginx external IP. Get the external IP by running: kubectl get services -n ingress-nginx To test inside minikube refer to this documentation: Set up Ingress on Minikube with the NGINX Ingress Controller","title":"Basic usage"},{"location":"user-guide/basic-usage/#basic-usage-host-based-routing","text":"ingress-nginx can be used for many use cases, inside various cloud provider and supports a lot of configurations. In this section you can find a common usage scenario where a single load balancer powered by ingress-nginx will route traffic to 2 different HTTP backend services based on the host name. First of all follow the instructions to install ingress-nginx. Then imagine that you need to expose 2 HTTP services already installed, myServiceA , myServiceB , and configured as type: ClusterIP . Let's say that you want to expose the first at myServiceA.foo.org and the second at myServiceB.foo.org . If the cluster version is < 1.19, you can create two ingress resources like this: apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-myservicea spec: ingressClassName: nginx rules: - host: myservicea.foo.org http: paths: - path: / backend: serviceName: myservicea servicePort: 80 --- apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: ingress-myserviceb annotations: # use the shared ingress-nginx kubernetes.io/ingress.class: \"nginx\" spec: rules: - host: myserviceb.foo.org http: paths: - path: / backend: serviceName: myserviceb servicePort: 80 If the cluster uses Kubernetes version >= 1.19.x, then its suggested to create 2 ingress resources, using yaml examples shown below. These examples are in conformity with the networking.kubernetes.io/v1 api. apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-myservicea spec: rules: - host: myservicea.foo.org http: paths: - path: / pathType: Prefix backend: service: name: myservicea port: number: 80 ingressClassName: nginx --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-myserviceb spec: rules: - host: myserviceb.foo.org http: paths: - path: / pathType: Prefix backend: service: name: myserviceb port: number: 80 ingressClassName: nginx When you apply this yaml, 2 ingress resources will be created managed by the ingress-nginx instance. Nginx is configured to automatically discover all ingress with the kubernetes.io/ingress.class: \"nginx\" annotation or where ingressClassName: nginx is present. Please note that the ingress resource should be placed inside the same namespace of the backend resource. On many cloud providers ingress-nginx will also create the corresponding Load Balancer resource. All you have to do is get the external IP and add a DNS A record inside your DNS provider that point myservicea.foo.org and myserviceb.foo.org to the nginx external IP. Get the external IP by running: kubectl get services -n ingress-nginx To test inside minikube refer to this documentation: Set up Ingress on Minikube with the NGINX Ingress Controller","title":"Basic usage - host based routing"},{"location":"user-guide/cli-arguments/","text":"Command line arguments \u00b6 The following command line arguments are accepted by the Ingress controller executable. They are set in the container spec of the ingress-nginx-controller Deployment manifest Argument Description --add_dir_header If true, adds the file directory to the header --alsologtostderr log to standard error as well as files --annotations-prefix Prefix of the Ingress annotations specific to the NGINX controller. (default \"nginx.ingress.kubernetes.io\") --apiserver-host Address of the Kubernetes API server. Takes the form \"protocol://address:port\". If not specified, it is assumed the program runs inside a Kubernetes cluster and local discovery is attempted. --certificate-authority Path to a cert file for the certificate authority. This certificate is used only when the flag --apiserver-host is specified. --configmap Name of the ConfigMap containing custom global configurations for the controller. --deep-inspect Enables ingress object security deep inspector. (default true) --default-backend-service Service used to serve HTTP requests not matching any known server name (catch-all). Takes the form \"namespace/name\". The controller configures NGINX to forward requests to the first port of this Service. --default-server-port Port to use for exposing the default server (catch-all). (default 8181) --default-ssl-certificate Secret containing a SSL certificate to be used by the default HTTPS server (catch-all). Takes the form \"namespace/name\". --disable-catch-all Disable support for catch-all Ingresses --disable-full-test Disable full test of all merged ingresses at the admission stage and tests the template of the ingress being created or updated (full test of all ingresses is enabled by default) --election-id Election id to use for Ingress status updates. (default \"ingress-controller-leader\") --enable-metrics Enables the collection of NGINX metrics (default true) --enable-ssl-chain-completion Autocomplete SSL certificate chains with missing intermediate CA certificates. Certificates uploaded to Kubernetes must have the \"Authority Information Access\" X.509 v3 extension for this to succeed. --enable-ssl-passthrough Enable SSL Passthrough. --health-check-path URL path of the health check endpoint. Configured inside the NGINX status server. All requests received on the port defined by the healthz-port parameter are forwarded internally to this path. (default \"/healthz\") --health-check-timeout Time limit, in seconds, for a probe to health-check-path to succeed. (default 10) --healthz-port Port to use for the healthz endpoint. (default 10254) --healthz-host Address to bind the healthz endpoint. --http-port Port to use for servicing HTTP traffic. (default 80) --https-port Port to use for servicing HTTPS traffic. (default 443) --ingress-class Name of the ingress class this controller satisfies. The class of an Ingress object is set using the field IngressClassName in Kubernetes clusters version v1.18.0 or higher or the annotation \"kubernetes.io/ingress.class\" (deprecated). If this parameter is not set, or set to the default value of \"nginx\", it will handle ingresses with either an empty or \"nginx\" class name. --ingress-class-by-name Define if Ingress Controller should watch for Ingress Class by Name together with Controller Class. (default false) --internal-logger-address Define the internal logger address to use when chroot images is used. (default 127.0.0.1:11514) --kubeconfig Path to a kubeconfig file containing authorization and API server information. --length-buckets Set of buckets which will be used for prometheus histogram metrics such as RequestLength, ResponseLength. (default [10, 20, 30, 40, 50, 60, 70, 80, 90, 100] ) --log_backtrace_at when logging hits line file:N, emit a stack trace (default :0) --log_dir If non-empty, write log files in this directory --log_file If non-empty, use this log file --log_file_max_size Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800) --logtostderr log to standard error instead of files (default true) --maxmind-edition-ids Maxmind edition ids to download GeoLite2 Databases. (default \"GeoLite2-City,GeoLite2-ASN\") --maxmind-retries-timeout Maxmind downloading delay between 1 st and 2 nd attempt, 0s - do not retry to download if something went wrong. (default 0s) --maxmind-retries-count Number of attempts to download the GeoIP DB. (default 1) --maxmind-license-key Maxmind license key to download GeoLite2 Databases. https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases --metrics-per-host Export metrics per-host (default true) --post-shutdown-grace-period Additional delay in seconds before controller container exits. (default 10) --profiler-port Port to use for expose the ingress controller Go profiler when it is enabled. (default 10245) --profiling Enable profiling via web interface host:port/debug/pprof/ (default true) --publish-service Service fronting the Ingress controller. Takes the form \"namespace/name\". When used together with update-status, the controller mirrors the address of this service's endpoints to the load-balancer status of all Ingress objects it satisfies. --publish-status-address Customized address (or addresses, separated by comma) to set as the load-balancer status of Ingress objects this controller satisfies. Requires the update-status parameter. --report-node-internal-ip-address Set the load-balancer status of Ingress objects to internal Node addresses instead of external. Requires the update-status parameter. --report-status-classes If true, report status classes in metrics (2xx, 3xx, 4xx and 5xx) instead of full status codes. (default false) --skip_headers If true, avoid header prefixes in the log messages --skip_log_headers If true, avoid headers when opening log files --ssl-passthrough-proxy-port Port to use internally for SSL Passthrough. (default 442) --size-buckets Set of buckets which will be used for prometheus histogram metrics such as BytesSent. (default [10, 100, 1000, 10000, 100000, 1e+06, 1e+07] ) --status-port Port to use for the lua HTTP endpoint configuration. (default 10246) --status-update-interval Time interval in seconds in which the status should check if an update is required. Default is 60 seconds (default 60) --stderrthreshold logs at or above this threshold go to stderr (default 2) --stream-port Port to use for the lua TCP/UDP endpoint configuration. (default 10247) --sync-period Period at which the controller forces the repopulation of its local object stores. Disabled by default. --sync-rate-limit Define the sync frequency upper limit (default 0.3) --tcp-services-configmap Name of the ConfigMap containing the definition of the TCP services to expose. The key in the map indicates the external port to be used. The value is a reference to a Service in the form \"namespace/name:port\", where \"port\" can either be a port number or name. TCP ports 80 and 443 are reserved by the controller for servicing HTTP traffic. --time-buckets Set of buckets which will be used for prometheus histogram metrics such as RequestTime, ResponseTime. (default [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] ) --udp-services-configmap Name of the ConfigMap containing the definition of the UDP services to expose. The key in the map indicates the external port to be used. The value is a reference to a Service in the form \"namespace/name:port\", where \"port\" can either be a port name or number. --update-status Update the load-balancer status of Ingress objects this controller satisfies. Requires setting the publish-service parameter to a valid Service reference. (default true) --update-status-on-shutdown Update the load-balancer status of Ingress objects when the controller shuts down. Requires the update-status parameter. (default true) --shutdown-grace-period Seconds to wait after receiving the shutdown signal, before stopping the nginx process. -v, --v Level number for the log level verbosity --validating-webhook The address to start an admission controller on to validate incoming ingresses. Takes the form \" :port\". If not provided, no admission controller is started. --validating-webhook-certificate The path of the validating webhook certificate PEM. --validating-webhook-key The path of the validating webhook key PEM. --version Show release information about the NGINX Ingress controller and exit. --vmodule comma-separated list of pattern=N settings for file-filtered logging --watch-namespace Namespace the controller watches for updates to Kubernetes objects. This includes Ingresses, Services and all configuration resources. All namespaces are watched if this parameter is left empty. --watch-namespace-selector The controller will watch namespaces whose labels match the given selector. This flag only takes effective when --watch-namespace is empty.","title":"Command line arguments"},{"location":"user-guide/cli-arguments/#command-line-arguments","text":"The following command line arguments are accepted by the Ingress controller executable. They are set in the container spec of the ingress-nginx-controller Deployment manifest Argument Description --add_dir_header If true, adds the file directory to the header --alsologtostderr log to standard error as well as files --annotations-prefix Prefix of the Ingress annotations specific to the NGINX controller. (default \"nginx.ingress.kubernetes.io\") --apiserver-host Address of the Kubernetes API server. Takes the form \"protocol://address:port\". If not specified, it is assumed the program runs inside a Kubernetes cluster and local discovery is attempted. --certificate-authority Path to a cert file for the certificate authority. This certificate is used only when the flag --apiserver-host is specified. --configmap Name of the ConfigMap containing custom global configurations for the controller. --deep-inspect Enables ingress object security deep inspector. (default true) --default-backend-service Service used to serve HTTP requests not matching any known server name (catch-all). Takes the form \"namespace/name\". The controller configures NGINX to forward requests to the first port of this Service. --default-server-port Port to use for exposing the default server (catch-all). (default 8181) --default-ssl-certificate Secret containing a SSL certificate to be used by the default HTTPS server (catch-all). Takes the form \"namespace/name\". --disable-catch-all Disable support for catch-all Ingresses --disable-full-test Disable full test of all merged ingresses at the admission stage and tests the template of the ingress being created or updated (full test of all ingresses is enabled by default) --election-id Election id to use for Ingress status updates. (default \"ingress-controller-leader\") --enable-metrics Enables the collection of NGINX metrics (default true) --enable-ssl-chain-completion Autocomplete SSL certificate chains with missing intermediate CA certificates. Certificates uploaded to Kubernetes must have the \"Authority Information Access\" X.509 v3 extension for this to succeed. --enable-ssl-passthrough Enable SSL Passthrough. --health-check-path URL path of the health check endpoint. Configured inside the NGINX status server. All requests received on the port defined by the healthz-port parameter are forwarded internally to this path. (default \"/healthz\") --health-check-timeout Time limit, in seconds, for a probe to health-check-path to succeed. (default 10) --healthz-port Port to use for the healthz endpoint. (default 10254) --healthz-host Address to bind the healthz endpoint. --http-port Port to use for servicing HTTP traffic. (default 80) --https-port Port to use for servicing HTTPS traffic. (default 443) --ingress-class Name of the ingress class this controller satisfies. The class of an Ingress object is set using the field IngressClassName in Kubernetes clusters version v1.18.0 or higher or the annotation \"kubernetes.io/ingress.class\" (deprecated). If this parameter is not set, or set to the default value of \"nginx\", it will handle ingresses with either an empty or \"nginx\" class name. --ingress-class-by-name Define if Ingress Controller should watch for Ingress Class by Name together with Controller Class. (default false) --internal-logger-address Define the internal logger address to use when chroot images is used. (default 127.0.0.1:11514) --kubeconfig Path to a kubeconfig file containing authorization and API server information. --length-buckets Set of buckets which will be used for prometheus histogram metrics such as RequestLength, ResponseLength. (default [10, 20, 30, 40, 50, 60, 70, 80, 90, 100] ) --log_backtrace_at when logging hits line file:N, emit a stack trace (default :0) --log_dir If non-empty, write log files in this directory --log_file If non-empty, use this log file --log_file_max_size Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800) --logtostderr log to standard error instead of files (default true) --maxmind-edition-ids Maxmind edition ids to download GeoLite2 Databases. (default \"GeoLite2-City,GeoLite2-ASN\") --maxmind-retries-timeout Maxmind downloading delay between 1 st and 2 nd attempt, 0s - do not retry to download if something went wrong. (default 0s) --maxmind-retries-count Number of attempts to download the GeoIP DB. (default 1) --maxmind-license-key Maxmind license key to download GeoLite2 Databases. https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases --metrics-per-host Export metrics per-host (default true) --post-shutdown-grace-period Additional delay in seconds before controller container exits. (default 10) --profiler-port Port to use for expose the ingress controller Go profiler when it is enabled. (default 10245) --profiling Enable profiling via web interface host:port/debug/pprof/ (default true) --publish-service Service fronting the Ingress controller. Takes the form \"namespace/name\". When used together with update-status, the controller mirrors the address of this service's endpoints to the load-balancer status of all Ingress objects it satisfies. --publish-status-address Customized address (or addresses, separated by comma) to set as the load-balancer status of Ingress objects this controller satisfies. Requires the update-status parameter. --report-node-internal-ip-address Set the load-balancer status of Ingress objects to internal Node addresses instead of external. Requires the update-status parameter. --report-status-classes If true, report status classes in metrics (2xx, 3xx, 4xx and 5xx) instead of full status codes. (default false) --skip_headers If true, avoid header prefixes in the log messages --skip_log_headers If true, avoid headers when opening log files --ssl-passthrough-proxy-port Port to use internally for SSL Passthrough. (default 442) --size-buckets Set of buckets which will be used for prometheus histogram metrics such as BytesSent. (default [10, 100, 1000, 10000, 100000, 1e+06, 1e+07] ) --status-port Port to use for the lua HTTP endpoint configuration. (default 10246) --status-update-interval Time interval in seconds in which the status should check if an update is required. Default is 60 seconds (default 60) --stderrthreshold logs at or above this threshold go to stderr (default 2) --stream-port Port to use for the lua TCP/UDP endpoint configuration. (default 10247) --sync-period Period at which the controller forces the repopulation of its local object stores. Disabled by default. --sync-rate-limit Define the sync frequency upper limit (default 0.3) --tcp-services-configmap Name of the ConfigMap containing the definition of the TCP services to expose. The key in the map indicates the external port to be used. The value is a reference to a Service in the form \"namespace/name:port\", where \"port\" can either be a port number or name. TCP ports 80 and 443 are reserved by the controller for servicing HTTP traffic. --time-buckets Set of buckets which will be used for prometheus histogram metrics such as RequestTime, ResponseTime. (default [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] ) --udp-services-configmap Name of the ConfigMap containing the definition of the UDP services to expose. The key in the map indicates the external port to be used. The value is a reference to a Service in the form \"namespace/name:port\", where \"port\" can either be a port name or number. --update-status Update the load-balancer status of Ingress objects this controller satisfies. Requires setting the publish-service parameter to a valid Service reference. (default true) --update-status-on-shutdown Update the load-balancer status of Ingress objects when the controller shuts down. Requires the update-status parameter. (default true) --shutdown-grace-period Seconds to wait after receiving the shutdown signal, before stopping the nginx process. -v, --v Level number for the log level verbosity --validating-webhook The address to start an admission controller on to validate incoming ingresses. Takes the form \" :port\". If not provided, no admission controller is started. --validating-webhook-certificate The path of the validating webhook certificate PEM. --validating-webhook-key The path of the validating webhook key PEM. --version Show release information about the NGINX Ingress controller and exit. --vmodule comma-separated list of pattern=N settings for file-filtered logging --watch-namespace Namespace the controller watches for updates to Kubernetes objects. This includes Ingresses, Services and all configuration resources. All namespaces are watched if this parameter is left empty. --watch-namespace-selector The controller will watch namespaces whose labels match the given selector. This flag only takes effective when --watch-namespace is empty.","title":"Command line arguments"},{"location":"user-guide/custom-errors/","text":"Custom errors \u00b6 When the custom-http-errors option is enabled, the Ingress controller configures NGINX so that it passes several HTTP headers down to its default-backend in case of error: Header Value X-Code HTTP status code returned by the request X-Format Value of the Accept header sent by the client X-Original-URI URI that caused the error X-Namespace Namespace where the backend Service is located X-Ingress-Name Name of the Ingress where the backend is defined X-Service-Name Name of the Service backing the backend X-Service-Port Port number of the Service backing the backend X-Request-ID Unique ID that identifies the request - same as for backend service A custom error backend can use this information to return the best possible representation of an error page. For example, if the value of the Accept header send by the client was application/json , a carefully crafted backend could decide to return the error payload as a JSON document instead of HTML. Important The custom backend is expected to return the correct HTTP status code instead of 200 . NGINX does not change the response from the custom default backend. An example of such custom backend is available inside the source repository at images/custom-error-pages . See also the Custom errors example.","title":"Custom errors"},{"location":"user-guide/custom-errors/#custom-errors","text":"When the custom-http-errors option is enabled, the Ingress controller configures NGINX so that it passes several HTTP headers down to its default-backend in case of error: Header Value X-Code HTTP status code returned by the request X-Format Value of the Accept header sent by the client X-Original-URI URI that caused the error X-Namespace Namespace where the backend Service is located X-Ingress-Name Name of the Ingress where the backend is defined X-Service-Name Name of the Service backing the backend X-Service-Port Port number of the Service backing the backend X-Request-ID Unique ID that identifies the request - same as for backend service A custom error backend can use this information to return the best possible representation of an error page. For example, if the value of the Accept header send by the client was application/json , a carefully crafted backend could decide to return the error payload as a JSON document instead of HTML. Important The custom backend is expected to return the correct HTTP status code instead of 200 . NGINX does not change the response from the custom default backend. An example of such custom backend is available inside the source repository at images/custom-error-pages . See also the Custom errors example.","title":"Custom errors"},{"location":"user-guide/default-backend/","text":"Default backend \u00b6 The default backend is a service which handles all URL paths and hosts the Ingress-NGINX controller doesn't understand (i.e., all the requests that are not mapped with an Ingress). Basically a default backend exposes two URLs: /healthz that returns 200 / that returns 404 Example The sub-directory /images/custom-error-pages provides an additional service for the purpose of customizing the error pages served via the default backend.","title":"Default backend"},{"location":"user-guide/default-backend/#default-backend","text":"The default backend is a service which handles all URL paths and hosts the Ingress-NGINX controller doesn't understand (i.e., all the requests that are not mapped with an Ingress). Basically a default backend exposes two URLs: /healthz that returns 200 / that returns 404 Example The sub-directory /images/custom-error-pages provides an additional service for the purpose of customizing the error pages served via the default backend.","title":"Default backend"},{"location":"user-guide/exposing-tcp-udp-services/","text":"Exposing TCP and UDP services \u00b6 Ingress does not support TCP or UDP services. For this reason this Ingress controller uses the flags --tcp-services-configmap and --udp-services-configmap to point to an existing config map where the key is the external port to use and the value indicates the service to expose using the format: <namespace/service name>:<service port>:[PROXY]:[PROXY] It is also possible to use a number or the name of the port. The two last fields are optional. Adding PROXY in either or both of the two last fields we can use Proxy Protocol decoding (listen) and/or encoding (proxy_pass) in a TCP service. The first PROXY controls the decode of the proxy protocol and the second PROXY controls the encoding using proxy protocol. This allows an incoming connection to be decoded or an outgoing connection to be encoded. It is also possible to arbitrate between two different proxies by turning on the decode and encode on a TCP service. The next example shows how to expose the service example-go running in the namespace default in the port 8080 using the port 9000 apiVersion : v1 kind : ConfigMap metadata : name : tcp-services namespace : ingress-nginx data : 9000 : \"default/example-go:8080\" Since 1.9.13 NGINX provides UDP Load Balancing . The next example shows how to expose the service kube-dns running in the namespace kube-system in the port 53 using the port 53 apiVersion : v1 kind : ConfigMap metadata : name : udp-services namespace : ingress-nginx data : 53 : \"kube-system/kube-dns:53\" If TCP/UDP proxy support is used, then those ports need to be exposed in the Service defined for the Ingress. apiVersion : v1 kind : Service metadata : name : ingress-nginx namespace : ingress-nginx labels : app.kubernetes.io/name : ingress-nginx app.kubernetes.io/part-of : ingress-nginx spec : type : LoadBalancer ports : - name : http port : 80 targetPort : 80 protocol : TCP - name : https port : 443 targetPort : 443 protocol : TCP - name : proxied-tcp-9000 port : 9000 targetPort : 9000 protocol : TCP selector : app.kubernetes.io/name : ingress-nginx app.kubernetes.io/part-of : ingress-nginx","title":"Exposing TCP and UDP services"},{"location":"user-guide/exposing-tcp-udp-services/#exposing-tcp-and-udp-services","text":"Ingress does not support TCP or UDP services. For this reason this Ingress controller uses the flags --tcp-services-configmap and --udp-services-configmap to point to an existing config map where the key is the external port to use and the value indicates the service to expose using the format: <namespace/service name>:<service port>:[PROXY]:[PROXY] It is also possible to use a number or the name of the port. The two last fields are optional. Adding PROXY in either or both of the two last fields we can use Proxy Protocol decoding (listen) and/or encoding (proxy_pass) in a TCP service. The first PROXY controls the decode of the proxy protocol and the second PROXY controls the encoding using proxy protocol. This allows an incoming connection to be decoded or an outgoing connection to be encoded. It is also possible to arbitrate between two different proxies by turning on the decode and encode on a TCP service. The next example shows how to expose the service example-go running in the namespace default in the port 8080 using the port 9000 apiVersion : v1 kind : ConfigMap metadata : name : tcp-services namespace : ingress-nginx data : 9000 : \"default/example-go:8080\" Since 1.9.13 NGINX provides UDP Load Balancing . The next example shows how to expose the service kube-dns running in the namespace kube-system in the port 53 using the port 53 apiVersion : v1 kind : ConfigMap metadata : name : udp-services namespace : ingress-nginx data : 53 : \"kube-system/kube-dns:53\" If TCP/UDP proxy support is used, then those ports need to be exposed in the Service defined for the Ingress. apiVersion : v1 kind : Service metadata : name : ingress-nginx namespace : ingress-nginx labels : app.kubernetes.io/name : ingress-nginx app.kubernetes.io/part-of : ingress-nginx spec : type : LoadBalancer ports : - name : http port : 80 targetPort : 80 protocol : TCP - name : https port : 443 targetPort : 443 protocol : TCP - name : proxied-tcp-9000 port : 9000 targetPort : 9000 protocol : TCP selector : app.kubernetes.io/name : ingress-nginx app.kubernetes.io/part-of : ingress-nginx","title":"Exposing TCP and UDP services"},{"location":"user-guide/external-articles/","text":"External Articles \u00b6 Pain(less) NGINX Ingress Accessing Kubernetes Pods from Outside of the Cluster Kubernetes - Redirect HTTP to HTTPS with ELB and the nginx ingress controller Configure Nginx Ingress Controller for TLS termination on Kubernetes on Azure","title":"External Articles"},{"location":"user-guide/external-articles/#external-articles","text":"Pain(less) NGINX Ingress Accessing Kubernetes Pods from Outside of the Cluster Kubernetes - Redirect HTTP to HTTPS with ELB and the nginx ingress controller Configure Nginx Ingress Controller for TLS termination on Kubernetes on Azure","title":"External Articles"},{"location":"user-guide/fcgi-services/","text":"Exposing FastCGI Servers \u00b6 FastCGI is a binary protocol for interfacing interactive programs with a web server . [...] (It's) aim is to reduce the overhead related to interfacing between web server and CGI programs, allowing a server to handle more web page requests per unit of time. \u2014 Wikipedia The ingress-nginx ingress controller can be used to directly expose FastCGI servers. Enabling FastCGI in your Ingress only requires setting the backend-protocol annotation to FCGI , and with a couple more annotations you can customize the way ingress-nginx handles the communication with your FastCGI server . Example Objects to Expose a FastCGI Pod \u00b6 The Pod example object below exposes port 9000 , which is the conventional FastCGI port. apiVersion : v1 kind : Pod metadata : name : example-app labels : app : example-app spec : containers : - name : example-app image : example-app:1.0 ports : - containerPort : 9000 name : fastcgi The Service object example below matches port 9000 from the Pod object above. apiVersion : v1 kind : Service metadata : name : example-service spec : selector : app : example-app ports : - port : 9000 targetPort : 9000 name : fastcgi And the Ingress and ConfigMap objects below demonstrates the supported FastCGI specific annotations (NGINX actually has 50 FastCGI directives, all of which have not been exposed in the ingress yet), and matches the service example-service , and the port named fastcgi from above. The ConfigMap must be created first for the Ingress Controller to be able to find it when the Ingress object is created, otherwise you will need to restart the Ingress Controller pods. # The ConfigMap MUST be created first for the ingress controller to be able to # find it when the Ingress object is created. apiVersion : v1 kind : ConfigMap metadata : name : example-cm data : SCRIPT_FILENAME : \"/example/index.php\" --- apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : nginx.ingress.kubernetes.io/backend-protocol : \"FCGI\" nginx.ingress.kubernetes.io/fastcgi-index : \"index.php\" nginx.ingress.kubernetes.io/fastcgi-params-configmap : \"example-cm\" name : example-app spec : ingressClassName : nginx rules : - host : app.example.com http : paths : - path : / pathType : Prefix backend : service : name : example-service port : name : fastcgi FastCGI Ingress Annotations \u00b6 To enable FastCGI, the nginx.ingress.kubernetes.io/backend-protocol annotation needs to be set to FCGI , which overrides the default HTTP value. nginx.ingress.kubernetes.io/backend-protocol: \"FCGI\" This enables the FastCGI mode for all paths defined in the Ingress object The nginx.ingress.kubernetes.io/fastcgi-index Annotation \u00b6 To specify an index file, the fastcgi-index annotation value can optionally be set. In the example below, the value is set to index.php . This annotation corresponds to the NGINX fastcgi_index directive . nginx.ingress.kubernetes.io/fastcgi-index: \"index.php\" The nginx.ingress.kubernetes.io/fastcgi-params-configmap Annotation \u00b6 To specify NGINX fastcgi_param directives , the fastcgi-params-configmap annotation is used, which in turn must lead to a ConfigMap object containing the NGINX fastcgi_param directives as key/values. nginx.ingress.kubernetes.io/fastcgi-params-configmap: \"example-configmap\" And the ConfigMap object to specify the SCRIPT_FILENAME and HTTP_PROXY NGINX's fastcgi_param directives will look like the following: apiVersion : v1 kind : ConfigMap metadata : name : example-configmap data : SCRIPT_FILENAME : \"/example/index.php\" HTTP_PROXY : \"\" Using the namespace/ prefix is also supported, for example: nginx.ingress.kubernetes.io/fastcgi-params-configmap: \"example-namespace/example-configmap\"","title":"Exposing FCGI services"},{"location":"user-guide/fcgi-services/#exposing-fastcgi-servers","text":"FastCGI is a binary protocol for interfacing interactive programs with a web server . [...] (It's) aim is to reduce the overhead related to interfacing between web server and CGI programs, allowing a server to handle more web page requests per unit of time. \u2014 Wikipedia The ingress-nginx ingress controller can be used to directly expose FastCGI servers. Enabling FastCGI in your Ingress only requires setting the backend-protocol annotation to FCGI , and with a couple more annotations you can customize the way ingress-nginx handles the communication with your FastCGI server .","title":"Exposing FastCGI Servers"},{"location":"user-guide/fcgi-services/#example-objects-to-expose-a-fastcgi-pod","text":"The Pod example object below exposes port 9000 , which is the conventional FastCGI port. apiVersion : v1 kind : Pod metadata : name : example-app labels : app : example-app spec : containers : - name : example-app image : example-app:1.0 ports : - containerPort : 9000 name : fastcgi The Service object example below matches port 9000 from the Pod object above. apiVersion : v1 kind : Service metadata : name : example-service spec : selector : app : example-app ports : - port : 9000 targetPort : 9000 name : fastcgi And the Ingress and ConfigMap objects below demonstrates the supported FastCGI specific annotations (NGINX actually has 50 FastCGI directives, all of which have not been exposed in the ingress yet), and matches the service example-service , and the port named fastcgi from above. The ConfigMap must be created first for the Ingress Controller to be able to find it when the Ingress object is created, otherwise you will need to restart the Ingress Controller pods. # The ConfigMap MUST be created first for the ingress controller to be able to # find it when the Ingress object is created. apiVersion : v1 kind : ConfigMap metadata : name : example-cm data : SCRIPT_FILENAME : \"/example/index.php\" --- apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : nginx.ingress.kubernetes.io/backend-protocol : \"FCGI\" nginx.ingress.kubernetes.io/fastcgi-index : \"index.php\" nginx.ingress.kubernetes.io/fastcgi-params-configmap : \"example-cm\" name : example-app spec : ingressClassName : nginx rules : - host : app.example.com http : paths : - path : / pathType : Prefix backend : service : name : example-service port : name : fastcgi","title":"Example Objects to Expose a FastCGI Pod"},{"location":"user-guide/fcgi-services/#fastcgi-ingress-annotations","text":"To enable FastCGI, the nginx.ingress.kubernetes.io/backend-protocol annotation needs to be set to FCGI , which overrides the default HTTP value. nginx.ingress.kubernetes.io/backend-protocol: \"FCGI\" This enables the FastCGI mode for all paths defined in the Ingress object","title":"FastCGI Ingress Annotations"},{"location":"user-guide/fcgi-services/#the-nginxingresskubernetesiofastcgi-index-annotation","text":"To specify an index file, the fastcgi-index annotation value can optionally be set. In the example below, the value is set to index.php . This annotation corresponds to the NGINX fastcgi_index directive . nginx.ingress.kubernetes.io/fastcgi-index: \"index.php\"","title":"The nginx.ingress.kubernetes.io/fastcgi-index Annotation"},{"location":"user-guide/fcgi-services/#the-nginxingresskubernetesiofastcgi-params-configmap-annotation","text":"To specify NGINX fastcgi_param directives , the fastcgi-params-configmap annotation is used, which in turn must lead to a ConfigMap object containing the NGINX fastcgi_param directives as key/values. nginx.ingress.kubernetes.io/fastcgi-params-configmap: \"example-configmap\" And the ConfigMap object to specify the SCRIPT_FILENAME and HTTP_PROXY NGINX's fastcgi_param directives will look like the following: apiVersion : v1 kind : ConfigMap metadata : name : example-configmap data : SCRIPT_FILENAME : \"/example/index.php\" HTTP_PROXY : \"\" Using the namespace/ prefix is also supported, for example: nginx.ingress.kubernetes.io/fastcgi-params-configmap: \"example-namespace/example-configmap\"","title":"The nginx.ingress.kubernetes.io/fastcgi-params-configmap Annotation"},{"location":"user-guide/ingress-path-matching/","text":"Ingress Path Matching \u00b6 Regular Expression Support \u00b6 Important Regular expressions and wild cards are not supported in the spec.rules.host field. Full hostnames must be used. The ingress controller supports case insensitive regular expressions in the spec.rules.http.paths.path field. This can be enabled by setting the nginx.ingress.kubernetes.io/use-regex annotation to true (the default is false). Hint Kubernetes only accept expressions that comply with the RE2 engine syntax. It is possible that valid expressions accepted by NGINX cannot be used with ingress-nginx, because the PCRE library (used in NGINX) supports a wider syntax than RE2. See the RE2 Syntax documentation for differences. See the description of the use-regex annotation for more details. apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : test-ingress annotations : nginx.ingress.kubernetes.io/use-regex : \"true\" spec : ingressClassName : nginx rules : - host : test.com http : paths : - path : /foo/.* pathType : Prefix backend : service : name : test port : number : 80 The preceding ingress definition would translate to the following location block within the NGINX configuration for the test.com server: location ~* \"^/foo/.*\" { ... } Path Priority \u00b6 In NGINX, regular expressions follow a first match policy. In order to enable more accurate path matching, ingress-nginx first orders the paths by descending length before writing them to the NGINX template as location blocks. Please read the warning before using regular expressions in your ingress definitions. Example \u00b6 Let the following two ingress definitions be created: apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : test-ingress-1 spec : ingressClassName : nginx rules : - host : test.com http : paths : - path : /foo/bar pathType : Prefix backend : service : name : service1 port : number : 80 - path : /foo/bar/ pathType : Prefix backend : service : name : service2 port : number : 80 apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : test-ingress-2 annotations : nginx.ingress.kubernetes.io/rewrite-target : /$1 spec : ingressClassName : nginx rules : - host : test.com http : paths : - path : /foo/bar/(.+) pathType : Prefix backend : service : name : service3 port : number : 80 The ingress controller would define the following location blocks, in order of descending length, within the NGINX template for the test.com server: location ~* ^/foo/bar/.+ { ... } location ~* \"^/foo/bar/\" { ... } location ~* \"^/foo/bar\" { ... } The following request URI's would match the corresponding location blocks: test.com/foo/bar/1 matches ~* ^/foo/bar/.+ and will go to service 3. test.com/foo/bar/ matches ~* ^/foo/bar/ and will go to service 2. test.com/foo/bar matches ~* ^/foo/bar and will go to service 1. IMPORTANT NOTES : If the use-regex OR rewrite-target annotation is used on any Ingress for a given host, then the case insensitive regular expression location modifier will be enforced on ALL paths for a given host regardless of what Ingress they are defined on. Warning \u00b6 The following example describes a case that may inflict unwanted path matching behavior. This case is expected and a result of NGINX's a first match policy for paths that use the regular expression location modifier . For more information about how a path is chosen, please read the following article: \"Understanding Nginx Server and Location Block Selection Algorithms\" . Example \u00b6 Let the following ingress be defined: apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : test-ingress-3 annotations : nginx.ingress.kubernetes.io/use-regex : \"true\" spec : ingressClassName : nginx rules : - host : test.com http : paths : - path : /foo/bar/bar pathType : Prefix backend : service : name : test port : number : 80 - path : /foo/bar/[A-Z0-9]{3} pathType : Prefix backend : service : name : test port : number : 80 The ingress controller would define the following location blocks (in this order) within the NGINX template for the test.com server: location ~* \"^/foo/bar/[A-Z0-9]{3}\" { ... } location ~* \"^/foo/bar/bar\" { ... } A request to test.com/foo/bar/bar would match the ^/foo/bar/[A-Z0-9]{3} location block instead of the longest EXACT matching path.","title":"Regular expressions in paths"},{"location":"user-guide/ingress-path-matching/#ingress-path-matching","text":"","title":"Ingress Path Matching"},{"location":"user-guide/ingress-path-matching/#regular-expression-support","text":"Important Regular expressions and wild cards are not supported in the spec.rules.host field. Full hostnames must be used. The ingress controller supports case insensitive regular expressions in the spec.rules.http.paths.path field. This can be enabled by setting the nginx.ingress.kubernetes.io/use-regex annotation to true (the default is false). Hint Kubernetes only accept expressions that comply with the RE2 engine syntax. It is possible that valid expressions accepted by NGINX cannot be used with ingress-nginx, because the PCRE library (used in NGINX) supports a wider syntax than RE2. See the RE2 Syntax documentation for differences. See the description of the use-regex annotation for more details. apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : test-ingress annotations : nginx.ingress.kubernetes.io/use-regex : \"true\" spec : ingressClassName : nginx rules : - host : test.com http : paths : - path : /foo/.* pathType : Prefix backend : service : name : test port : number : 80 The preceding ingress definition would translate to the following location block within the NGINX configuration for the test.com server: location ~* \"^/foo/.*\" { ... }","title":"Regular Expression Support"},{"location":"user-guide/ingress-path-matching/#path-priority","text":"In NGINX, regular expressions follow a first match policy. In order to enable more accurate path matching, ingress-nginx first orders the paths by descending length before writing them to the NGINX template as location blocks. Please read the warning before using regular expressions in your ingress definitions.","title":"Path Priority"},{"location":"user-guide/ingress-path-matching/#example","text":"Let the following two ingress definitions be created: apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : test-ingress-1 spec : ingressClassName : nginx rules : - host : test.com http : paths : - path : /foo/bar pathType : Prefix backend : service : name : service1 port : number : 80 - path : /foo/bar/ pathType : Prefix backend : service : name : service2 port : number : 80 apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : test-ingress-2 annotations : nginx.ingress.kubernetes.io/rewrite-target : /$1 spec : ingressClassName : nginx rules : - host : test.com http : paths : - path : /foo/bar/(.+) pathType : Prefix backend : service : name : service3 port : number : 80 The ingress controller would define the following location blocks, in order of descending length, within the NGINX template for the test.com server: location ~* ^/foo/bar/.+ { ... } location ~* \"^/foo/bar/\" { ... } location ~* \"^/foo/bar\" { ... } The following request URI's would match the corresponding location blocks: test.com/foo/bar/1 matches ~* ^/foo/bar/.+ and will go to service 3. test.com/foo/bar/ matches ~* ^/foo/bar/ and will go to service 2. test.com/foo/bar matches ~* ^/foo/bar and will go to service 1. IMPORTANT NOTES : If the use-regex OR rewrite-target annotation is used on any Ingress for a given host, then the case insensitive regular expression location modifier will be enforced on ALL paths for a given host regardless of what Ingress they are defined on.","title":"Example"},{"location":"user-guide/ingress-path-matching/#warning","text":"The following example describes a case that may inflict unwanted path matching behavior. This case is expected and a result of NGINX's a first match policy for paths that use the regular expression location modifier . For more information about how a path is chosen, please read the following article: \"Understanding Nginx Server and Location Block Selection Algorithms\" .","title":"Warning"},{"location":"user-guide/ingress-path-matching/#example_1","text":"Let the following ingress be defined: apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : test-ingress-3 annotations : nginx.ingress.kubernetes.io/use-regex : \"true\" spec : ingressClassName : nginx rules : - host : test.com http : paths : - path : /foo/bar/bar pathType : Prefix backend : service : name : test port : number : 80 - path : /foo/bar/[A-Z0-9]{3} pathType : Prefix backend : service : name : test port : number : 80 The ingress controller would define the following location blocks (in this order) within the NGINX template for the test.com server: location ~* \"^/foo/bar/[A-Z0-9]{3}\" { ... } location ~* \"^/foo/bar/bar\" { ... } A request to test.com/foo/bar/bar would match the ^/foo/bar/[A-Z0-9]{3} location block instead of the longest EXACT matching path.","title":"Example"},{"location":"user-guide/miscellaneous/","text":"Miscellaneous \u00b6 Source IP address \u00b6 By default NGINX uses the content of the header X-Forwarded-For as the source of truth to get information about the client IP address. This works without issues in L7 if we configure the setting proxy-real-ip-cidr with the correct information of the IP/network address of trusted external load balancer. If the ingress controller is running in AWS we need to use the VPC IPv4 CIDR. Another option is to enable proxy protocol using use-proxy-protocol: \"true\" . In this mode NGINX does not use the content of the header to get the source IP address of the connection. Path types \u00b6 Each path in an Ingress is required to have a corresponding path type. Paths that do not include an explicit pathType will fail validation. By default NGINX path type is Prefix to not break existing definitions Proxy Protocol \u00b6 If you are using a L4 proxy to forward the traffic to the NGINX pods and terminate HTTP/HTTPS there, you will lose the remote endpoint's IP address. To prevent this you could use the Proxy Protocol for forwarding traffic, this will send the connection details before forwarding the actual TCP connection itself. Amongst others ELBs in AWS and HAProxy support Proxy Protocol. Websockets \u00b6 Support for websockets is provided by NGINX out of the box. No special configuration required. The only requirement to avoid the close of connections is the increase of the values of proxy-read-timeout and proxy-send-timeout . The default value of this settings is 60 seconds . A more adequate value to support websockets is a value higher than one hour ( 3600 ). Important If the NGINX ingress controller is exposed with a service type=LoadBalancer make sure the protocol between the loadbalancer and NGINX is TCP. Optimizing TLS Time To First Byte (TTTFB) \u00b6 NGINX provides the configuration option ssl_buffer_size to allow the optimization of the TLS record size. This improves the TLS Time To First Byte (TTTFB). The default value in the Ingress controller is 4k (NGINX default is 16k ). Retries in non-idempotent methods \u00b6 Since 1.9.13 NGINX will not retry non-idempotent requests (POST, LOCK, PATCH) in case of an error. The previous behavior can be restored using retry-non-idempotent=true in the configuration ConfigMap. Limitations \u00b6 Ingress rules for TLS require the definition of the field host Why endpoints and not services \u00b6 The NGINX ingress controller does not use Services to route traffic to the pods. Instead it uses the Endpoints API in order to bypass kube-proxy to allow NGINX features like session affinity and custom load balancing algorithms. It also removes some overhead, such as conntrack entries for iptables DNAT.","title":"Miscellaneous"},{"location":"user-guide/miscellaneous/#miscellaneous","text":"","title":"Miscellaneous"},{"location":"user-guide/miscellaneous/#source-ip-address","text":"By default NGINX uses the content of the header X-Forwarded-For as the source of truth to get information about the client IP address. This works without issues in L7 if we configure the setting proxy-real-ip-cidr with the correct information of the IP/network address of trusted external load balancer. If the ingress controller is running in AWS we need to use the VPC IPv4 CIDR. Another option is to enable proxy protocol using use-proxy-protocol: \"true\" . In this mode NGINX does not use the content of the header to get the source IP address of the connection.","title":"Source IP address"},{"location":"user-guide/miscellaneous/#path-types","text":"Each path in an Ingress is required to have a corresponding path type. Paths that do not include an explicit pathType will fail validation. By default NGINX path type is Prefix to not break existing definitions","title":"Path types"},{"location":"user-guide/miscellaneous/#proxy-protocol","text":"If you are using a L4 proxy to forward the traffic to the NGINX pods and terminate HTTP/HTTPS there, you will lose the remote endpoint's IP address. To prevent this you could use the Proxy Protocol for forwarding traffic, this will send the connection details before forwarding the actual TCP connection itself. Amongst others ELBs in AWS and HAProxy support Proxy Protocol.","title":"Proxy Protocol"},{"location":"user-guide/miscellaneous/#websockets","text":"Support for websockets is provided by NGINX out of the box. No special configuration required. The only requirement to avoid the close of connections is the increase of the values of proxy-read-timeout and proxy-send-timeout . The default value of this settings is 60 seconds . A more adequate value to support websockets is a value higher than one hour ( 3600 ). Important If the NGINX ingress controller is exposed with a service type=LoadBalancer make sure the protocol between the loadbalancer and NGINX is TCP.","title":"Websockets"},{"location":"user-guide/miscellaneous/#optimizing-tls-time-to-first-byte-tttfb","text":"NGINX provides the configuration option ssl_buffer_size to allow the optimization of the TLS record size. This improves the TLS Time To First Byte (TTTFB). The default value in the Ingress controller is 4k (NGINX default is 16k ).","title":"Optimizing TLS Time To First Byte (TTTFB)"},{"location":"user-guide/miscellaneous/#retries-in-non-idempotent-methods","text":"Since 1.9.13 NGINX will not retry non-idempotent requests (POST, LOCK, PATCH) in case of an error. The previous behavior can be restored using retry-non-idempotent=true in the configuration ConfigMap.","title":"Retries in non-idempotent methods"},{"location":"user-guide/miscellaneous/#limitations","text":"Ingress rules for TLS require the definition of the field host","title":"Limitations"},{"location":"user-guide/miscellaneous/#why-endpoints-and-not-services","text":"The NGINX ingress controller does not use Services to route traffic to the pods. Instead it uses the Endpoints API in order to bypass kube-proxy to allow NGINX features like session affinity and custom load balancing algorithms. It also removes some overhead, such as conntrack entries for iptables DNAT.","title":"Why endpoints and not services"},{"location":"user-guide/monitoring/","text":"Monitoring \u00b6 Two different methods to install and configure Prometheus and Grafana are described in this doc. * Prometheus and Grafana installation using Pod Annotations. This installs Prometheus and Grafana in the same namespace as NGINX Ingress * Prometheus and Grafana installation using Service Monitors. This installs Prometheus and Grafana in two different namespaces. This is the preferred method, and helm charts supports this by default. Prometheus and Grafana installation using Pod Annotations \u00b6 This tutorial will show you how to install Prometheus and Grafana for scraping the metrics of the NGINX Ingress controller. Important This example uses emptyDir volumes for Prometheus and Grafana. This means once the pod gets terminated you will lose all the data. Before You Begin \u00b6 The NGINX Ingress controller should already be deployed according to the deployment instructions here . The controller should be configured for exporting metrics. This requires 3 configurations to the controller. These configurations are : controller.metrics.enabled=true controller.podAnnotations.\"prometheus.io/scrape\"=\"true\" controller.podAnnotations.\"prometheus.io/port\"=\"10254\" The easiest way to configure the controller for metrics is via helm upgrade. Assuming you have installed the ingress-nginx controller as a helm release named ingress-nginx, then you can simply type the command shown below : helm upgrade ingress-nginx ingress-nginx \\ --repo https://kubernetes.github.io/ingress-nginx \\ --namespace ingress-nginx \\ --set controller.metrics.enabled=true \\ --set-string controller.podAnnotations.\"prometheus\\.io/scrape\"=\"true\" \\ --set-string controller.podAnnotations.\"prometheus\\.io/port\"=\"10254\" You can validate that the controller is configured for metrics by looking at the values of the installed release, like this: helm get values ingress-nginx --namespace ingress-nginx You should be able to see the values shown below: .. controller: metrics: enabled: true service: annotations: prometheus.io/port: \"10254\" prometheus.io/scrape: \"true\" .. If you are not using helm , you will have to edit your manifests like this: Service manifest: apiVersion: v1 kind: Service metadata: annotations: prometheus.io/scrape: \"true\" prometheus.io/port: \"10254\" .. spec: ports: - name: prometheus port: 10254 targetPort: prometheus .. Deployment manifest: apiVersion: v1 kind: Deployment metadata: annotations: prometheus.io/scrape: \"true\" prometheus.io/port: \"10254\" .. spec: ports: - name: prometheus containerPort: 10254 .. Deploy and configure Prometheus Server \u00b6 Note that the kustomize bases used in this tutorial are stored in the deploy folder of the GitHub repository kubernetes/ingress-nginx . The Prometheus server must be configured so that it can discover endpoints of services. If a Prometheus server is already running in the cluster and if it is configured in a way that it can find the ingress controller pods, no extra configuration is needed. If there is no existing Prometheus server running, the rest of this tutorial will guide you through the steps needed to deploy a properly configured Prometheus server. Running the following command deploys prometheus in Kubernetes: kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/prometheus/ Prometheus Dashboard \u00b6 Open Prometheus dashboard in a web browser: kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-http-backend ClusterIP 10.103.59.201 <none> 80/TCP 3d ingress-nginx NodePort 10.97.44.72 <none> 80:30100/TCP,443:30154/TCP,10254:32049/TCP 5h prometheus-server NodePort 10.98.233.86 <none> 9090:32630/TCP 1m Obtain the IP address of the nodes in the running cluster: kubectl get nodes -o wide In some cases where the node only have internal IP addresses we need to execute: kubectl get nodes --selector=kubernetes.io/role!=master -o jsonpath={.items[*].status.addresses[?\\(@.type==\\\"InternalIP\\\"\\)].address} 10.192.0.2 10.192.0.3 10.192.0.4 Open your browser and visit the following URL: http://{node IP address}:{prometheus-svc-nodeport} to load the Prometheus Dashboard. According to the above example, this URL will be http://10.192.0.3:32630 Grafana \u00b6 Install grafana using the below command kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/grafana/ Look at the services kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-http-backend ClusterIP 10.103.59.201 <none> 80/TCP 3d ingress-nginx NodePort 10.97.44.72 <none> 80:30100/TCP,443:30154/TCP,10254:32049/TCP 5h prometheus-server NodePort 10.98.233.86 <none> 9090:32630/TCP 10m grafana NodePort 10.98.233.87 <none> 3000:31086/TCP 10m Open your browser and visit the following URL: http://{node IP address}:{grafana-svc-nodeport} to load the Grafana Dashboard. According to the above example, this URL will be http://10.192.0.3:31086 The username and password is admin After the login you can import the Grafana dashboard from official dashboards , by following steps given below : Navigate to lefthand panel of grafana Hover on the gearwheel icon for Configuration and click \"Data Sources\" Click \"Add data source\" Select \"Prometheus\" Enter the details (note: I used http://CLUSTER_IP_PROMETHEUS_SVC:9090) Left menu (hover over +) -> Dashboard Click \"Import\" Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/grafana/dashboards/nginx.json Click Import JSON Select the Prometheus data source Click \"Import\" Caveats \u00b6 Wildcard ingresses \u00b6 By default request metrics are labeled with the hostname. When you have a wildcard domain ingress, then there will be no metrics for that ingress (to prevent the metrics from exploding in cardinality). To get metrics in this case you need to run the ingress controller with --metrics-per-host=false (you will lose labeling by hostname, but still have labeling by ingress). Grafana dashboard using ingress resource \u00b6 If you want to expose the dashboard for grafana using a ingress resource, then you can : change the service type of the prometheus-server service and the grafana service to \"ClusterIP\" like this : kubectl -n ingress-nginx edit svc grafana This will open the currently deployed service grafana in the default editor configured in your shell (vi/nvim/nano/other) scroll down to line 34 that looks like \"type: NodePort\" change it to look like \"type: ClusterIP\". Save and exit. create a ingress resource with backend as \"grafana\" and port as \"3000\" Similarly, you can edit the service \"prometheus-server\" and add a ingress resource. Prometheus and Grafana installation using Service Monitors \u00b6 This document assumes you're using helm and using the kube-prometheus-stack package to install Prometheus and Grafana. Verify NGINX Ingress controller is installed \u00b6 The NGINX Ingress controller should already be deployed according to the deployment instructions here . To check if Ingress controller is deployed, kubectl get pods -n ingress-nginx The result should look something like: NAME READY STATUS RESTARTS AGE ingress-nginx-controller-7c489dc7b7-ccrf6 1/1 Running 0 19h Verify Prometheus is installed \u00b6 To check if Prometheus is already deployed, run the following command: helm ls -A NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION ingress-nginx ingress-nginx 10 2022-01-20 18:08:55.267373 -0800 PST deployed ingress-nginx-4.0.16 1.1.1 prometheus prometheus 1 2022-01-20 16:07:25.086828 -0800 PST deployed kube-prometheus-stack-30.1.0 0.53.1 - Notice that prometheus is installed in a differenet namespace than ingress-nginx If prometheus is not installed, then you can install from here Re-configure NGINX Ingress controller \u00b6 The Ingress NGINX controller needs to be reconfigured for exporting metrics. This requires 3 additional configurations to the controller. These configurations are : controller.metrics.enabled=true controller.metrics.serviceMonitor.enabled=true controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" The easiest way of doing this is to helm upgrade helm upgrade ingress-nginx ingress-nginx/ingress-nginx \\ --namespace ingress-nginx \\ --set controller.metrics.enabled=true \\ --set controller.metrics.serviceMonitor.enabled=true \\ --set controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" Here controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" should match the name of the helm release of the kube-prometheus-stack You can validate that the controller has been successfully reconfigured to export metrics by looking at the values of the installed release, like this: helm get values ingress-nginx --namespace ingress-nginx controller: metrics: enabled: true serviceMonitor: additionalLabels: release: prometheus enabled: true Configure Prometheus \u00b6 Since Prometheus is running in a different namespace and not in the ingress-nginx namespace, it would not be able to discover ServiceMonitors in other namespaces when installed. Reconfigure your kube-prometheus-stack Helm installation to set serviceMonitorSelectorNilUsesHelmValues flag to false. By default, Prometheus only discovers PodMonitors within its own namespace. This should be disabled by setting podMonitorSelectorNilUsesHelmValues to false The configurations required are: prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false The easiest way of doing this is to use helm upgrade ... helm upgrade prometheus prometheus-community/kube-prometheus-stack \\ --namespace prometheus \\ --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \\ --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false You can validate that Prometheus has been reconfigured by looking at the values of the installed release, like this: helm get values prometheus --namespace prometheus You should be able to see the values shown below: prometheus: prometheusSpec: podMonitorSelectorNilUsesHelmValues: false serviceMonitorSelectorNilUsesHelmValues: false Connect and view Prometheus dashboard \u00b6 Port forward to Prometheus service. Find out the name of the prometheus service by using the following command: kubectl get svc -n prometheus The result of this command would look like: NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 7h46m prometheus-grafana ClusterIP 10.106.28.162 <none> 80/TCP 7h46m prometheus-kube-prometheus-alertmanager ClusterIP 10.108.125.245 <none> 9093/TCP 7h46m prometheus-kube-prometheus-operator ClusterIP 10.110.220.1 <none> 443/TCP 7h46m prometheus-kube-prometheus-prometheus ClusterIP 10.102.72.134 <none> 9090/TCP 7h46m prometheus-kube-state-metrics ClusterIP 10.104.231.181 <none> 8080/TCP 7h46m prometheus-operated ClusterIP None <none> 9090/TCP 7h46m prometheus-prometheus-node-exporter ClusterIP 10.96.247.128 <none> 9100/TCP 7h46m prometheus-kube-prometheus-prometheus is the service we want to port forward to. We can do so using the following command: kubectl port-forward svc/prometheus-kube-prometheus-prometheus -n prometheus 9090:9090 When you run the above command, you should see something like: Forwarding from 127.0.0.1:9090 -> 9090 Forwarding from [::1]:9090 -> 9090 - Open your browser and visit the following URL http://localhost:{port-forwarded-port} according to the above example it would be, http://localhost:9090 Connect and view Grafana dashboard \u00b6 Port forward to Grafana service. Find out the name of the Grafana service by using the following command: kubectl get svc -n prometheus The result of this command would look like: NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 7h46m prometheus-grafana ClusterIP 10.106.28.162 <none> 80/TCP 7h46m prometheus-kube-prometheus-alertmanager ClusterIP 10.108.125.245 <none> 9093/TCP 7h46m prometheus-kube-prometheus-operator ClusterIP 10.110.220.1 <none> 443/TCP 7h46m prometheus-kube-prometheus-prometheus ClusterIP 10.102.72.134 <none> 9090/TCP 7h46m prometheus-kube-state-metrics ClusterIP 10.104.231.181 <none> 8080/TCP 7h46m prometheus-operated ClusterIP None <none> 9090/TCP 7h46m prometheus-prometheus-node-exporter ClusterIP 10.96.247.128 <none> 9100/TCP 7h46m prometheus-grafana is the service we want to port forward to. We can do so using the following command: kubectl port-forward svc/prometheus-grafana 3000:80 -n prometheus When you run the above command, you should see something like: Forwarding from 127.0.0.1:3000 -> 3000 Forwarding from [::1]:3000 -> 3000 - Open your browser and visit the following URL http://localhost:{port-forwarded-port} according to the above example it would be, http://localhost:3000 The default username/ password is admin/prom-operator - After the login you can import the Grafana dashboard from official dashboards , by following steps given below : Navigate to lefthand panel of grafana Hover on the gearwheel icon for Configuration and click \"Data Sources\" Click \"Add data source\" Select \"Prometheus\" Enter the details (note: I used http://10.102.72.134:9090 which is the CLUSTER-IP for Prometheus service) Left menu (hover over +) -> Dashboard Click \"Import\" Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/grafana/dashboards/nginx.json Click Import JSON Select the Prometheus data source Click \"Import\" Exposed metrics \u00b6 Prometheus metrics are exposed on port 10254. Request metrics \u00b6 nginx_ingress_controller_request_duration_seconds Histogram The request processing time in milliseconds (affected by client speed) nginx var: request_time nginx_ingress_controller_response_duration_seconds Histogram The time spent on receiving the response from the upstream server (affected by client speed) nginx var: upstream_response_time nginx_ingress_controller_header_duration_seconds Histogram The time spent on receiving first header from the upstream server nginx var: upstream_header_time nginx_ingress_controller_connect_duration_seconds Histogram The time spent on establishing a connection with the upstream server nginx var: upstream_connect_time nginx_ingress_controller_response_size Histogram The response length (including request line, header, and request body) nginx var: bytes_sent nginx_ingress_controller_request_size Histogram The request length (including request line, header, and request body) nginx var: request_length nginx_ingress_controller_requests Counter The total number of client requests nginx_ingress_controller_bytes_sent Histogram The number of bytes sent to a client. Deprecated , use nginx_ingress_controller_response_size nginx var: bytes_sent nginx_ingress_controller_ingress_upstream_latency_seconds Summary Upstream service latency per Ingress. Deprecated , use nginx_ingress_controller_connect_duration_seconds nginx var: upstream_connect_time # HELP nginx_ingress_controller_bytes_sent The number of bytes sent to a client. DEPRECATED! Use nginx_ingress_controller_response_size # TYPE nginx_ingress_controller_bytes_sent histogram # HELP nginx_ingress_controller_connect_duration_seconds The time spent on establishing a connection with the upstream server # TYPE nginx_ingress_controller_connect_duration_seconds nginx_ingress_controller_connect_duration_seconds * HELP nginx_ingress_controller_header_duration_seconds The time spent on receiving first header from the upstream server # TYPE nginx_ingress_controller_header_duration_seconds histogram # HELP nginx_ingress_controller_ingress_upstream_latency_seconds Upstream service latency per Ingress DEPRECATED! Use nginx_ingress_controller_connect_duration_seconds # TYPE nginx_ingress_controller_ingress_upstream_latency_seconds summary # HELP nginx_ingress_controller_request_duration_seconds The request processing time in milliseconds # TYPE nginx_ingress_controller_request_duration_seconds histogram # HELP nginx_ingress_controller_request_size The request length (including request line, header, and request body) # TYPE nginx_ingress_controller_request_size histogram # HELP nginx_ingress_controller_requests The total number of client requests. # TYPE nginx_ingress_controller_requests counter # HELP nginx_ingress_controller_response_duration_seconds The time spent on receiving the response from the upstream server # TYPE nginx_ingress_controller_response_duration_seconds histogram # HELP nginx_ingress_controller_response_size The response length (including request line, header, and request body) # TYPE nginx_ingress_controller_response_size histogram Nginx process metrics \u00b6 # HELP nginx_ingress_controller_nginx_process_connections current number of client connections with state {active, reading, writing, waiting} # TYPE nginx_ingress_controller_nginx_process_connections gauge # HELP nginx_ingress_controller_nginx_process_connections_total total number of connections with state {accepted, handled} # TYPE nginx_ingress_controller_nginx_process_connections_total counter # HELP nginx_ingress_controller_nginx_process_cpu_seconds_total Cpu usage in seconds # TYPE nginx_ingress_controller_nginx_process_cpu_seconds_total counter # HELP nginx_ingress_controller_nginx_process_num_procs number of processes # TYPE nginx_ingress_controller_nginx_process_num_procs gauge # HELP nginx_ingress_controller_nginx_process_oldest_start_time_seconds start time in seconds since 1970/01/01 # TYPE nginx_ingress_controller_nginx_process_oldest_start_time_seconds gauge # HELP nginx_ingress_controller_nginx_process_read_bytes_total number of bytes read # TYPE nginx_ingress_controller_nginx_process_read_bytes_total counter # HELP nginx_ingress_controller_nginx_process_requests_total total number of client requests # TYPE nginx_ingress_controller_nginx_process_requests_total counter # HELP nginx_ingress_controller_nginx_process_resident_memory_bytes number of bytes of memory in use # TYPE nginx_ingress_controller_nginx_process_resident_memory_bytes gauge # HELP nginx_ingress_controller_nginx_process_virtual_memory_bytes number of bytes of memory in use # TYPE nginx_ingress_controller_nginx_process_virtual_memory_bytes gauge # HELP nginx_ingress_controller_nginx_process_write_bytes_total number of bytes written # TYPE nginx_ingress_controller_nginx_process_write_bytes_total counter Controller metrics \u00b6 # HELP nginx_ingress_controller_build_info A metric with a constant '1' labeled with information about the build. # TYPE nginx_ingress_controller_build_info gauge # HELP nginx_ingress_controller_check_success Cumulative number of Ingress controller syntax check operations # TYPE nginx_ingress_controller_check_success counter # HELP nginx_ingress_controller_config_hash Running configuration hash actually running # TYPE nginx_ingress_controller_config_hash gauge # HELP nginx_ingress_controller_config_last_reload_successful Whether the last configuration reload attempt was successful # TYPE nginx_ingress_controller_config_last_reload_successful gauge # HELP nginx_ingress_controller_config_last_reload_successful_timestamp_seconds Timestamp of the last successful configuration reload. # TYPE nginx_ingress_controller_config_last_reload_successful_timestamp_seconds gauge # HELP nginx_ingress_controller_ssl_certificate_info Hold all labels associated to a certificate # TYPE nginx_ingress_controller_ssl_certificate_info gauge # HELP nginx_ingress_controller_success Cumulative number of Ingress controller reload operations # TYPE nginx_ingress_controller_success counter Admission metrics \u00b6 # HELP nginx_ingress_controller_admission_config_size The size of the tested configuration # TYPE nginx_ingress_controller_admission_config_size gauge # HELP nginx_ingress_controller_admission_render_duration The processing duration of ingresses rendering by the admission controller (float seconds) # TYPE nginx_ingress_controller_admission_render_duration gauge # HELP nginx_ingress_controller_admission_render_ingresses The length of ingresses rendered by the admission controller # TYPE nginx_ingress_controller_admission_render_ingresses gauge # HELP nginx_ingress_controller_admission_roundtrip_duration The complete duration of the admission controller at the time to process a new event (float seconds) # TYPE nginx_ingress_controller_admission_roundtrip_duration gauge # HELP nginx_ingress_controller_admission_tested_duration The processing duration of the admission controller tests (float seconds) # TYPE nginx_ingress_controller_admission_tested_duration gauge # HELP nginx_ingress_controller_admission_tested_ingresses The length of ingresses processed by the admission controller # TYPE nginx_ingress_controller_admission_tested_ingresses gauge Histogram buckets \u00b6 You can configure buckets for histogram metrics using these command line options (here are their default values): * --time-buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] * --length-buckets=[10, 20, 30, 40, 50, 60, 70, 80, 90, 100] * --size-buckets=[10, 100, 1000, 10000, 100000, 1e+06, 1e+07]","title":"Prometheus and Grafana installation"},{"location":"user-guide/monitoring/#monitoring","text":"Two different methods to install and configure Prometheus and Grafana are described in this doc. * Prometheus and Grafana installation using Pod Annotations. This installs Prometheus and Grafana in the same namespace as NGINX Ingress * Prometheus and Grafana installation using Service Monitors. This installs Prometheus and Grafana in two different namespaces. This is the preferred method, and helm charts supports this by default.","title":"Monitoring"},{"location":"user-guide/monitoring/#prometheus-and-grafana-installation-using-pod-annotations","text":"This tutorial will show you how to install Prometheus and Grafana for scraping the metrics of the NGINX Ingress controller. Important This example uses emptyDir volumes for Prometheus and Grafana. This means once the pod gets terminated you will lose all the data.","title":"Prometheus and Grafana installation using Pod Annotations"},{"location":"user-guide/monitoring/#before-you-begin","text":"The NGINX Ingress controller should already be deployed according to the deployment instructions here . The controller should be configured for exporting metrics. This requires 3 configurations to the controller. These configurations are : controller.metrics.enabled=true controller.podAnnotations.\"prometheus.io/scrape\"=\"true\" controller.podAnnotations.\"prometheus.io/port\"=\"10254\" The easiest way to configure the controller for metrics is via helm upgrade. Assuming you have installed the ingress-nginx controller as a helm release named ingress-nginx, then you can simply type the command shown below : helm upgrade ingress-nginx ingress-nginx \\ --repo https://kubernetes.github.io/ingress-nginx \\ --namespace ingress-nginx \\ --set controller.metrics.enabled=true \\ --set-string controller.podAnnotations.\"prometheus\\.io/scrape\"=\"true\" \\ --set-string controller.podAnnotations.\"prometheus\\.io/port\"=\"10254\" You can validate that the controller is configured for metrics by looking at the values of the installed release, like this: helm get values ingress-nginx --namespace ingress-nginx You should be able to see the values shown below: .. controller: metrics: enabled: true service: annotations: prometheus.io/port: \"10254\" prometheus.io/scrape: \"true\" .. If you are not using helm , you will have to edit your manifests like this: Service manifest: apiVersion: v1 kind: Service metadata: annotations: prometheus.io/scrape: \"true\" prometheus.io/port: \"10254\" .. spec: ports: - name: prometheus port: 10254 targetPort: prometheus .. Deployment manifest: apiVersion: v1 kind: Deployment metadata: annotations: prometheus.io/scrape: \"true\" prometheus.io/port: \"10254\" .. spec: ports: - name: prometheus containerPort: 10254 ..","title":"Before You Begin"},{"location":"user-guide/monitoring/#deploy-and-configure-prometheus-server","text":"Note that the kustomize bases used in this tutorial are stored in the deploy folder of the GitHub repository kubernetes/ingress-nginx . The Prometheus server must be configured so that it can discover endpoints of services. If a Prometheus server is already running in the cluster and if it is configured in a way that it can find the ingress controller pods, no extra configuration is needed. If there is no existing Prometheus server running, the rest of this tutorial will guide you through the steps needed to deploy a properly configured Prometheus server. Running the following command deploys prometheus in Kubernetes: kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/prometheus/","title":"Deploy and configure Prometheus Server"},{"location":"user-guide/monitoring/#prometheus-dashboard","text":"Open Prometheus dashboard in a web browser: kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-http-backend ClusterIP 10.103.59.201 <none> 80/TCP 3d ingress-nginx NodePort 10.97.44.72 <none> 80:30100/TCP,443:30154/TCP,10254:32049/TCP 5h prometheus-server NodePort 10.98.233.86 <none> 9090:32630/TCP 1m Obtain the IP address of the nodes in the running cluster: kubectl get nodes -o wide In some cases where the node only have internal IP addresses we need to execute: kubectl get nodes --selector=kubernetes.io/role!=master -o jsonpath={.items[*].status.addresses[?\\(@.type==\\\"InternalIP\\\"\\)].address} 10.192.0.2 10.192.0.3 10.192.0.4 Open your browser and visit the following URL: http://{node IP address}:{prometheus-svc-nodeport} to load the Prometheus Dashboard. According to the above example, this URL will be http://10.192.0.3:32630","title":"Prometheus Dashboard"},{"location":"user-guide/monitoring/#grafana","text":"Install grafana using the below command kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/grafana/ Look at the services kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-http-backend ClusterIP 10.103.59.201 <none> 80/TCP 3d ingress-nginx NodePort 10.97.44.72 <none> 80:30100/TCP,443:30154/TCP,10254:32049/TCP 5h prometheus-server NodePort 10.98.233.86 <none> 9090:32630/TCP 10m grafana NodePort 10.98.233.87 <none> 3000:31086/TCP 10m Open your browser and visit the following URL: http://{node IP address}:{grafana-svc-nodeport} to load the Grafana Dashboard. According to the above example, this URL will be http://10.192.0.3:31086 The username and password is admin After the login you can import the Grafana dashboard from official dashboards , by following steps given below : Navigate to lefthand panel of grafana Hover on the gearwheel icon for Configuration and click \"Data Sources\" Click \"Add data source\" Select \"Prometheus\" Enter the details (note: I used http://CLUSTER_IP_PROMETHEUS_SVC:9090) Left menu (hover over +) -> Dashboard Click \"Import\" Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/grafana/dashboards/nginx.json Click Import JSON Select the Prometheus data source Click \"Import\"","title":"Grafana"},{"location":"user-guide/monitoring/#caveats","text":"","title":"Caveats"},{"location":"user-guide/monitoring/#wildcard-ingresses","text":"By default request metrics are labeled with the hostname. When you have a wildcard domain ingress, then there will be no metrics for that ingress (to prevent the metrics from exploding in cardinality). To get metrics in this case you need to run the ingress controller with --metrics-per-host=false (you will lose labeling by hostname, but still have labeling by ingress).","title":"Wildcard ingresses"},{"location":"user-guide/monitoring/#grafana-dashboard-using-ingress-resource","text":"If you want to expose the dashboard for grafana using a ingress resource, then you can : change the service type of the prometheus-server service and the grafana service to \"ClusterIP\" like this : kubectl -n ingress-nginx edit svc grafana This will open the currently deployed service grafana in the default editor configured in your shell (vi/nvim/nano/other) scroll down to line 34 that looks like \"type: NodePort\" change it to look like \"type: ClusterIP\". Save and exit. create a ingress resource with backend as \"grafana\" and port as \"3000\" Similarly, you can edit the service \"prometheus-server\" and add a ingress resource.","title":"Grafana dashboard using ingress resource"},{"location":"user-guide/monitoring/#prometheus-and-grafana-installation-using-service-monitors","text":"This document assumes you're using helm and using the kube-prometheus-stack package to install Prometheus and Grafana.","title":"Prometheus and Grafana installation using Service Monitors"},{"location":"user-guide/monitoring/#verify-nginx-ingress-controller-is-installed","text":"The NGINX Ingress controller should already be deployed according to the deployment instructions here . To check if Ingress controller is deployed, kubectl get pods -n ingress-nginx The result should look something like: NAME READY STATUS RESTARTS AGE ingress-nginx-controller-7c489dc7b7-ccrf6 1/1 Running 0 19h","title":"Verify NGINX Ingress controller is installed"},{"location":"user-guide/monitoring/#verify-prometheus-is-installed","text":"To check if Prometheus is already deployed, run the following command: helm ls -A NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION ingress-nginx ingress-nginx 10 2022-01-20 18:08:55.267373 -0800 PST deployed ingress-nginx-4.0.16 1.1.1 prometheus prometheus 1 2022-01-20 16:07:25.086828 -0800 PST deployed kube-prometheus-stack-30.1.0 0.53.1 - Notice that prometheus is installed in a differenet namespace than ingress-nginx If prometheus is not installed, then you can install from here","title":"Verify Prometheus is installed"},{"location":"user-guide/monitoring/#re-configure-nginx-ingress-controller","text":"The Ingress NGINX controller needs to be reconfigured for exporting metrics. This requires 3 additional configurations to the controller. These configurations are : controller.metrics.enabled=true controller.metrics.serviceMonitor.enabled=true controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" The easiest way of doing this is to helm upgrade helm upgrade ingress-nginx ingress-nginx/ingress-nginx \\ --namespace ingress-nginx \\ --set controller.metrics.enabled=true \\ --set controller.metrics.serviceMonitor.enabled=true \\ --set controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" Here controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" should match the name of the helm release of the kube-prometheus-stack You can validate that the controller has been successfully reconfigured to export metrics by looking at the values of the installed release, like this: helm get values ingress-nginx --namespace ingress-nginx controller: metrics: enabled: true serviceMonitor: additionalLabels: release: prometheus enabled: true","title":"Re-configure NGINX Ingress controller"},{"location":"user-guide/monitoring/#configure-prometheus","text":"Since Prometheus is running in a different namespace and not in the ingress-nginx namespace, it would not be able to discover ServiceMonitors in other namespaces when installed. Reconfigure your kube-prometheus-stack Helm installation to set serviceMonitorSelectorNilUsesHelmValues flag to false. By default, Prometheus only discovers PodMonitors within its own namespace. This should be disabled by setting podMonitorSelectorNilUsesHelmValues to false The configurations required are: prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false The easiest way of doing this is to use helm upgrade ... helm upgrade prometheus prometheus-community/kube-prometheus-stack \\ --namespace prometheus \\ --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \\ --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false You can validate that Prometheus has been reconfigured by looking at the values of the installed release, like this: helm get values prometheus --namespace prometheus You should be able to see the values shown below: prometheus: prometheusSpec: podMonitorSelectorNilUsesHelmValues: false serviceMonitorSelectorNilUsesHelmValues: false","title":"Configure Prometheus"},{"location":"user-guide/monitoring/#connect-and-view-prometheus-dashboard","text":"Port forward to Prometheus service. Find out the name of the prometheus service by using the following command: kubectl get svc -n prometheus The result of this command would look like: NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 7h46m prometheus-grafana ClusterIP 10.106.28.162 <none> 80/TCP 7h46m prometheus-kube-prometheus-alertmanager ClusterIP 10.108.125.245 <none> 9093/TCP 7h46m prometheus-kube-prometheus-operator ClusterIP 10.110.220.1 <none> 443/TCP 7h46m prometheus-kube-prometheus-prometheus ClusterIP 10.102.72.134 <none> 9090/TCP 7h46m prometheus-kube-state-metrics ClusterIP 10.104.231.181 <none> 8080/TCP 7h46m prometheus-operated ClusterIP None <none> 9090/TCP 7h46m prometheus-prometheus-node-exporter ClusterIP 10.96.247.128 <none> 9100/TCP 7h46m prometheus-kube-prometheus-prometheus is the service we want to port forward to. We can do so using the following command: kubectl port-forward svc/prometheus-kube-prometheus-prometheus -n prometheus 9090:9090 When you run the above command, you should see something like: Forwarding from 127.0.0.1:9090 -> 9090 Forwarding from [::1]:9090 -> 9090 - Open your browser and visit the following URL http://localhost:{port-forwarded-port} according to the above example it would be, http://localhost:9090","title":"Connect and view Prometheus dashboard"},{"location":"user-guide/monitoring/#connect-and-view-grafana-dashboard","text":"Port forward to Grafana service. Find out the name of the Grafana service by using the following command: kubectl get svc -n prometheus The result of this command would look like: NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 7h46m prometheus-grafana ClusterIP 10.106.28.162 <none> 80/TCP 7h46m prometheus-kube-prometheus-alertmanager ClusterIP 10.108.125.245 <none> 9093/TCP 7h46m prometheus-kube-prometheus-operator ClusterIP 10.110.220.1 <none> 443/TCP 7h46m prometheus-kube-prometheus-prometheus ClusterIP 10.102.72.134 <none> 9090/TCP 7h46m prometheus-kube-state-metrics ClusterIP 10.104.231.181 <none> 8080/TCP 7h46m prometheus-operated ClusterIP None <none> 9090/TCP 7h46m prometheus-prometheus-node-exporter ClusterIP 10.96.247.128 <none> 9100/TCP 7h46m prometheus-grafana is the service we want to port forward to. We can do so using the following command: kubectl port-forward svc/prometheus-grafana 3000:80 -n prometheus When you run the above command, you should see something like: Forwarding from 127.0.0.1:3000 -> 3000 Forwarding from [::1]:3000 -> 3000 - Open your browser and visit the following URL http://localhost:{port-forwarded-port} according to the above example it would be, http://localhost:3000 The default username/ password is admin/prom-operator - After the login you can import the Grafana dashboard from official dashboards , by following steps given below : Navigate to lefthand panel of grafana Hover on the gearwheel icon for Configuration and click \"Data Sources\" Click \"Add data source\" Select \"Prometheus\" Enter the details (note: I used http://10.102.72.134:9090 which is the CLUSTER-IP for Prometheus service) Left menu (hover over +) -> Dashboard Click \"Import\" Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/grafana/dashboards/nginx.json Click Import JSON Select the Prometheus data source Click \"Import\"","title":"Connect and view Grafana dashboard"},{"location":"user-guide/monitoring/#exposed-metrics","text":"Prometheus metrics are exposed on port 10254.","title":"Exposed metrics"},{"location":"user-guide/monitoring/#request-metrics","text":"nginx_ingress_controller_request_duration_seconds Histogram The request processing time in milliseconds (affected by client speed) nginx var: request_time nginx_ingress_controller_response_duration_seconds Histogram The time spent on receiving the response from the upstream server (affected by client speed) nginx var: upstream_response_time nginx_ingress_controller_header_duration_seconds Histogram The time spent on receiving first header from the upstream server nginx var: upstream_header_time nginx_ingress_controller_connect_duration_seconds Histogram The time spent on establishing a connection with the upstream server nginx var: upstream_connect_time nginx_ingress_controller_response_size Histogram The response length (including request line, header, and request body) nginx var: bytes_sent nginx_ingress_controller_request_size Histogram The request length (including request line, header, and request body) nginx var: request_length nginx_ingress_controller_requests Counter The total number of client requests nginx_ingress_controller_bytes_sent Histogram The number of bytes sent to a client. Deprecated , use nginx_ingress_controller_response_size nginx var: bytes_sent nginx_ingress_controller_ingress_upstream_latency_seconds Summary Upstream service latency per Ingress. Deprecated , use nginx_ingress_controller_connect_duration_seconds nginx var: upstream_connect_time # HELP nginx_ingress_controller_bytes_sent The number of bytes sent to a client. DEPRECATED! Use nginx_ingress_controller_response_size # TYPE nginx_ingress_controller_bytes_sent histogram # HELP nginx_ingress_controller_connect_duration_seconds The time spent on establishing a connection with the upstream server # TYPE nginx_ingress_controller_connect_duration_seconds nginx_ingress_controller_connect_duration_seconds * HELP nginx_ingress_controller_header_duration_seconds The time spent on receiving first header from the upstream server # TYPE nginx_ingress_controller_header_duration_seconds histogram # HELP nginx_ingress_controller_ingress_upstream_latency_seconds Upstream service latency per Ingress DEPRECATED! Use nginx_ingress_controller_connect_duration_seconds # TYPE nginx_ingress_controller_ingress_upstream_latency_seconds summary # HELP nginx_ingress_controller_request_duration_seconds The request processing time in milliseconds # TYPE nginx_ingress_controller_request_duration_seconds histogram # HELP nginx_ingress_controller_request_size The request length (including request line, header, and request body) # TYPE nginx_ingress_controller_request_size histogram # HELP nginx_ingress_controller_requests The total number of client requests. # TYPE nginx_ingress_controller_requests counter # HELP nginx_ingress_controller_response_duration_seconds The time spent on receiving the response from the upstream server # TYPE nginx_ingress_controller_response_duration_seconds histogram # HELP nginx_ingress_controller_response_size The response length (including request line, header, and request body) # TYPE nginx_ingress_controller_response_size histogram","title":"Request metrics"},{"location":"user-guide/monitoring/#nginx-process-metrics","text":"# HELP nginx_ingress_controller_nginx_process_connections current number of client connections with state {active, reading, writing, waiting} # TYPE nginx_ingress_controller_nginx_process_connections gauge # HELP nginx_ingress_controller_nginx_process_connections_total total number of connections with state {accepted, handled} # TYPE nginx_ingress_controller_nginx_process_connections_total counter # HELP nginx_ingress_controller_nginx_process_cpu_seconds_total Cpu usage in seconds # TYPE nginx_ingress_controller_nginx_process_cpu_seconds_total counter # HELP nginx_ingress_controller_nginx_process_num_procs number of processes # TYPE nginx_ingress_controller_nginx_process_num_procs gauge # HELP nginx_ingress_controller_nginx_process_oldest_start_time_seconds start time in seconds since 1970/01/01 # TYPE nginx_ingress_controller_nginx_process_oldest_start_time_seconds gauge # HELP nginx_ingress_controller_nginx_process_read_bytes_total number of bytes read # TYPE nginx_ingress_controller_nginx_process_read_bytes_total counter # HELP nginx_ingress_controller_nginx_process_requests_total total number of client requests # TYPE nginx_ingress_controller_nginx_process_requests_total counter # HELP nginx_ingress_controller_nginx_process_resident_memory_bytes number of bytes of memory in use # TYPE nginx_ingress_controller_nginx_process_resident_memory_bytes gauge # HELP nginx_ingress_controller_nginx_process_virtual_memory_bytes number of bytes of memory in use # TYPE nginx_ingress_controller_nginx_process_virtual_memory_bytes gauge # HELP nginx_ingress_controller_nginx_process_write_bytes_total number of bytes written # TYPE nginx_ingress_controller_nginx_process_write_bytes_total counter","title":"Nginx process metrics"},{"location":"user-guide/monitoring/#controller-metrics","text":"# HELP nginx_ingress_controller_build_info A metric with a constant '1' labeled with information about the build. # TYPE nginx_ingress_controller_build_info gauge # HELP nginx_ingress_controller_check_success Cumulative number of Ingress controller syntax check operations # TYPE nginx_ingress_controller_check_success counter # HELP nginx_ingress_controller_config_hash Running configuration hash actually running # TYPE nginx_ingress_controller_config_hash gauge # HELP nginx_ingress_controller_config_last_reload_successful Whether the last configuration reload attempt was successful # TYPE nginx_ingress_controller_config_last_reload_successful gauge # HELP nginx_ingress_controller_config_last_reload_successful_timestamp_seconds Timestamp of the last successful configuration reload. # TYPE nginx_ingress_controller_config_last_reload_successful_timestamp_seconds gauge # HELP nginx_ingress_controller_ssl_certificate_info Hold all labels associated to a certificate # TYPE nginx_ingress_controller_ssl_certificate_info gauge # HELP nginx_ingress_controller_success Cumulative number of Ingress controller reload operations # TYPE nginx_ingress_controller_success counter","title":"Controller metrics"},{"location":"user-guide/monitoring/#admission-metrics","text":"# HELP nginx_ingress_controller_admission_config_size The size of the tested configuration # TYPE nginx_ingress_controller_admission_config_size gauge # HELP nginx_ingress_controller_admission_render_duration The processing duration of ingresses rendering by the admission controller (float seconds) # TYPE nginx_ingress_controller_admission_render_duration gauge # HELP nginx_ingress_controller_admission_render_ingresses The length of ingresses rendered by the admission controller # TYPE nginx_ingress_controller_admission_render_ingresses gauge # HELP nginx_ingress_controller_admission_roundtrip_duration The complete duration of the admission controller at the time to process a new event (float seconds) # TYPE nginx_ingress_controller_admission_roundtrip_duration gauge # HELP nginx_ingress_controller_admission_tested_duration The processing duration of the admission controller tests (float seconds) # TYPE nginx_ingress_controller_admission_tested_duration gauge # HELP nginx_ingress_controller_admission_tested_ingresses The length of ingresses processed by the admission controller # TYPE nginx_ingress_controller_admission_tested_ingresses gauge","title":"Admission metrics"},{"location":"user-guide/monitoring/#histogram-buckets","text":"You can configure buckets for histogram metrics using these command line options (here are their default values): * --time-buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] * --length-buckets=[10, 20, 30, 40, 50, 60, 70, 80, 90, 100] * --size-buckets=[10, 100, 1000, 10000, 100000, 1e+06, 1e+07]","title":"Histogram buckets"},{"location":"user-guide/multiple-ingress/","text":"Multiple Ingress controllers \u00b6 By default, deploying multiple Ingress controllers (e.g., ingress-nginx & gce ) will result in all controllers simultaneously racing to update Ingress status fields in confusing ways. To fix this problem, use IngressClasses . The kubernetes.io/ingress.class annotation is deprecated from kubernetes v1.22+. Using IngressClasses \u00b6 If all ingress controllers respect IngressClasses (e.g. multiple instances of ingress-nginx v1.0), you can deploy two Ingress controllers by granting them control over two different IngressClasses, then selecting one of the two IngressClasses with ingressClassName . First, ensure the --controller-class= and --ingress-class are set to something different on each ingress controller, If your additional ingress controller is to be installed in a namespace, where there is/are one/more-than-one ingress-nginx-controller(s) already installed, then you need to specify a different unique --election-id for the new instance of the controller. # ingress-nginx Deployment/Statefulset spec : template : spec : containers : - name : ingress-nginx-internal-controller args : - /nginx-ingress-controller - '--election-id=ingress-controller-leader' - '--controller-class=k8s.io/internal-ingress-nginx' - '--ingress-class=k8s.io/internal-nginx' ... Then use the same value in the IngressClass: # ingress-nginx IngressClass apiVersion : networking.k8s.io/v1 kind : IngressClass metadata : name : internal-nginx spec : controller : k8s.io/internal-ingress-nginx ... And refer to that IngressClass in your Ingress: apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : my-ingress spec : ingressClassName : internal-nginx ... or if installing with Helm: controller : electionID : ingress-controller-leader ingressClassResource : name : internal-nginx # default: nginx enabled : true default : false controllerValue : \"k8s.io/internal-ingress-nginx\" # default: k8s.io/ingress-nginx Important When running multiple ingress-nginx controllers, it will only process an unset class annotation if one of the controllers uses the default --controller-class value (see IsValid method in internal/ingress/annotations/class/main.go ), otherwise the class annotation becomes required. If --controller-class is set to the default value of k8s.io/ingress-nginx , the controller will monitor Ingresses with no class annotation and Ingresses with annotation class set to nginx . Use a non-default value for --controller-class , to ensure that the controller only satisfied the specific class of Ingresses. Using the kubernetes.io/ingress.class annotation (in deprecation) \u00b6 If you're running multiple ingress controllers where one or more do not support IngressClasses, you must specify the annotation kubernetes.io/ingress.class: \"nginx\" in all ingresses that you would like ingress-nginx to claim. For instance, metadata : name : foo annotations : kubernetes.io/ingress.class : \"gce\" will target the GCE controller, forcing the Ingress-NGINX controller to ignore it, while an annotation like: metadata : name : foo annotations : kubernetes.io/ingress.class : \"nginx\" will target the Ingress-NGINX controller, forcing the GCE controller to ignore it. You can change the value \"nginx\" to something else by setting the --ingress-class flag: spec : template : spec : containers : - name : ingress-nginx-internal-controller args : - /nginx-ingress-controller - --ingress-class=internal-nginx then setting the corresponding kubernetes.io/ingress.class: \"internal-nginx\" annotation on your Ingresses. To reiterate, setting the annotation to any value which does not match a valid ingress class will force the NGINX Ingress controller to ignore your Ingress. If you are only running a single NGINX ingress controller, this can be achieved by setting the annotation to any value except \"nginx\" or an empty string. Do this if you wish to use one of the other Ingress controllers at the same time as the NGINX controller.","title":"Multiple Ingress controllers"},{"location":"user-guide/multiple-ingress/#multiple-ingress-controllers","text":"By default, deploying multiple Ingress controllers (e.g., ingress-nginx & gce ) will result in all controllers simultaneously racing to update Ingress status fields in confusing ways. To fix this problem, use IngressClasses . The kubernetes.io/ingress.class annotation is deprecated from kubernetes v1.22+.","title":"Multiple Ingress controllers"},{"location":"user-guide/multiple-ingress/#using-ingressclasses","text":"If all ingress controllers respect IngressClasses (e.g. multiple instances of ingress-nginx v1.0), you can deploy two Ingress controllers by granting them control over two different IngressClasses, then selecting one of the two IngressClasses with ingressClassName . First, ensure the --controller-class= and --ingress-class are set to something different on each ingress controller, If your additional ingress controller is to be installed in a namespace, where there is/are one/more-than-one ingress-nginx-controller(s) already installed, then you need to specify a different unique --election-id for the new instance of the controller. # ingress-nginx Deployment/Statefulset spec : template : spec : containers : - name : ingress-nginx-internal-controller args : - /nginx-ingress-controller - '--election-id=ingress-controller-leader' - '--controller-class=k8s.io/internal-ingress-nginx' - '--ingress-class=k8s.io/internal-nginx' ... Then use the same value in the IngressClass: # ingress-nginx IngressClass apiVersion : networking.k8s.io/v1 kind : IngressClass metadata : name : internal-nginx spec : controller : k8s.io/internal-ingress-nginx ... And refer to that IngressClass in your Ingress: apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : my-ingress spec : ingressClassName : internal-nginx ... or if installing with Helm: controller : electionID : ingress-controller-leader ingressClassResource : name : internal-nginx # default: nginx enabled : true default : false controllerValue : \"k8s.io/internal-ingress-nginx\" # default: k8s.io/ingress-nginx Important When running multiple ingress-nginx controllers, it will only process an unset class annotation if one of the controllers uses the default --controller-class value (see IsValid method in internal/ingress/annotations/class/main.go ), otherwise the class annotation becomes required. If --controller-class is set to the default value of k8s.io/ingress-nginx , the controller will monitor Ingresses with no class annotation and Ingresses with annotation class set to nginx . Use a non-default value for --controller-class , to ensure that the controller only satisfied the specific class of Ingresses.","title":"Using IngressClasses"},{"location":"user-guide/multiple-ingress/#using-the-kubernetesioingressclass-annotation-in-deprecation","text":"If you're running multiple ingress controllers where one or more do not support IngressClasses, you must specify the annotation kubernetes.io/ingress.class: \"nginx\" in all ingresses that you would like ingress-nginx to claim. For instance, metadata : name : foo annotations : kubernetes.io/ingress.class : \"gce\" will target the GCE controller, forcing the Ingress-NGINX controller to ignore it, while an annotation like: metadata : name : foo annotations : kubernetes.io/ingress.class : \"nginx\" will target the Ingress-NGINX controller, forcing the GCE controller to ignore it. You can change the value \"nginx\" to something else by setting the --ingress-class flag: spec : template : spec : containers : - name : ingress-nginx-internal-controller args : - /nginx-ingress-controller - --ingress-class=internal-nginx then setting the corresponding kubernetes.io/ingress.class: \"internal-nginx\" annotation on your Ingresses. To reiterate, setting the annotation to any value which does not match a valid ingress class will force the NGINX Ingress controller to ignore your Ingress. If you are only running a single NGINX ingress controller, this can be achieved by setting the annotation to any value except \"nginx\" or an empty string. Do this if you wish to use one of the other Ingress controllers at the same time as the NGINX controller.","title":"Using the kubernetes.io/ingress.class annotation (in deprecation)"},{"location":"user-guide/tls/","text":"TLS/HTTPS \u00b6 TLS Secrets \u00b6 Anytime we reference a TLS secret, we mean a PEM-encoded X.509, RSA (2048) secret. Warning Ensure that the certificate order is leaf->intermediate->root, otherwise the controller will not be able to import the certificate, and you'll see this error in the logs W1012 09:15:45.920000 6 backend_ssl.go:46] Error obtaining X.509 certificate: unexpected error creating SSL Cert: certificate and private key does not have a matching public key: tls: private key does not match public key You can generate a self-signed certificate and private key with: $ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ${ KEY_FILE } -out ${ CERT_FILE } -subj \"/CN= ${ HOST } /O= ${ HOST } \" Then create the secret in the cluster via: kubectl create secret tls ${ CERT_NAME } --key ${ KEY_FILE } --cert ${ CERT_FILE } The resulting secret will be of type kubernetes.io/tls . Host names \u00b6 Ensure that the relevant ingress rules specify a matching host name . Default SSL Certificate \u00b6 NGINX provides the option to configure a server as a catch-all with server_name for requests that do not match any of the configured server names. This configuration works out-of-the-box for HTTP traffic. For HTTPS, a certificate is naturally required. For this reason the Ingress controller provides the flag --default-ssl-certificate . The secret referred to by this flag contains the default certificate to be used when accessing the catch-all server. If this flag is not provided NGINX will use a self-signed certificate. For instance, if you have a TLS secret foo-tls in the default namespace, add --default-ssl-certificate=default/foo-tls in the nginx-controller deployment. The default certificate will also be used for ingress tls: sections that do not have a secretName option. To force redirects for Ingresses that do not specify a TLS-block at all, take a look at force-ssl-redirect in ConfigMap . SSL Passthrough \u00b6 The --enable-ssl-passthrough flag enables the SSL Passthrough feature, which is disabled by default. This is required to enable passthrough backends in Ingress objects. Warning This feature is implemented by intercepting all traffic on the configured HTTPS port (default: 443) and handing it over to a local TCP proxy. This bypasses NGINX completely and introduces a non-negligible performance penalty. SSL Passthrough leverages SNI and reads the virtual domain from the TLS negotiation, which requires compatible clients. After a connection has been accepted by the TLS listener, it is handled by the controller itself and piped back and forth between the backend and the client. If there is no hostname matching the requested host name, the request is handed over to NGINX on the configured passthrough proxy port (default: 442), which proxies the request to the default backend. Note Unlike HTTP backends, traffic to Passthrough backends is sent to the clusterIP of the backing Service instead of individual Endpoints. HTTP Strict Transport Security \u00b6 HTTP Strict Transport Security (HSTS) is an opt-in security enhancement specified through the use of a special response header. Once a supported browser receives this header that browser will prevent any communications from being sent over HTTP to the specified domain and will instead send all communications over HTTPS. HSTS is enabled by default. To disable this behavior use hsts: \"false\" in the configuration ConfigMap . Server-side HTTPS enforcement through redirect \u00b6 By default the controller redirects HTTP clients to the HTTPS port 443 using a 308 Permanent Redirect response if TLS is enabled for that Ingress. This can be disabled globally using ssl-redirect: \"false\" in the NGINX config map , or per-Ingress with the nginx.ingress.kubernetes.io/ssl-redirect: \"false\" annotation in the particular resource. Tip When using SSL offloading outside of cluster (e.g. AWS ELB) it may be useful to enforce a redirect to HTTPS even when there is no TLS certificate available. This can be achieved by using the nginx.ingress.kubernetes.io/force-ssl-redirect: \"true\" annotation in the particular resource. Automated Certificate Management with cert-manager \u00b6 cert-manager automatically requests missing or expired certificates from a range of supported issuers (including Let's Encrypt ) by monitoring ingress resources. To set up cert-manager you should take a look at this full example . To enable it for an ingress resource you have to deploy cert-manager, configure a certificate issuer update the manifest: apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : ingress-demo annotations : cert-manager.io/issuer : \"letsencrypt-staging\" # Replace this with a production issuer once you've tested it [ .. ] spec : tls : - hosts : - ingress-demo.example.com secretName : ingress-demo-tls [ ... ] Default TLS Version and Ciphers \u00b6 To provide the most secure baseline configuration possible, ingress-nginx defaults to using TLS 1.2 and 1.3 only, with a secure set of TLS ciphers . Legacy TLS \u00b6 The default configuration, though secure, does not support some older browsers and operating systems. For instance, TLS 1.1+ is only enabled by default from Android 5.0 on. At the time of writing, May 2018, approximately 15% of Android devices are not compatible with ingress-nginx's default configuration. To change this default behavior, use a ConfigMap . A sample ConfigMap fragment to allow these older clients to connect could look something like the following (generated using the Mozilla SSL Configuration Generator) mozilla-ssl-config-old : kind: ConfigMap apiVersion: v1 metadata: name: nginx-config data: ssl-ciphers: \"ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA\" ssl-protocols: \"TLSv1 TLSv1.1 TLSv1.2 TLSv1.3\"","title":"TLS/HTTPS"},{"location":"user-guide/tls/#tlshttps","text":"","title":"TLS/HTTPS"},{"location":"user-guide/tls/#tls-secrets","text":"Anytime we reference a TLS secret, we mean a PEM-encoded X.509, RSA (2048) secret. Warning Ensure that the certificate order is leaf->intermediate->root, otherwise the controller will not be able to import the certificate, and you'll see this error in the logs W1012 09:15:45.920000 6 backend_ssl.go:46] Error obtaining X.509 certificate: unexpected error creating SSL Cert: certificate and private key does not have a matching public key: tls: private key does not match public key You can generate a self-signed certificate and private key with: $ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ${ KEY_FILE } -out ${ CERT_FILE } -subj \"/CN= ${ HOST } /O= ${ HOST } \" Then create the secret in the cluster via: kubectl create secret tls ${ CERT_NAME } --key ${ KEY_FILE } --cert ${ CERT_FILE } The resulting secret will be of type kubernetes.io/tls .","title":"TLS Secrets"},{"location":"user-guide/tls/#host-names","text":"Ensure that the relevant ingress rules specify a matching host name .","title":"Host names"},{"location":"user-guide/tls/#default-ssl-certificate","text":"NGINX provides the option to configure a server as a catch-all with server_name for requests that do not match any of the configured server names. This configuration works out-of-the-box for HTTP traffic. For HTTPS, a certificate is naturally required. For this reason the Ingress controller provides the flag --default-ssl-certificate . The secret referred to by this flag contains the default certificate to be used when accessing the catch-all server. If this flag is not provided NGINX will use a self-signed certificate. For instance, if you have a TLS secret foo-tls in the default namespace, add --default-ssl-certificate=default/foo-tls in the nginx-controller deployment. The default certificate will also be used for ingress tls: sections that do not have a secretName option. To force redirects for Ingresses that do not specify a TLS-block at all, take a look at force-ssl-redirect in ConfigMap .","title":"Default SSL Certificate"},{"location":"user-guide/tls/#ssl-passthrough","text":"The --enable-ssl-passthrough flag enables the SSL Passthrough feature, which is disabled by default. This is required to enable passthrough backends in Ingress objects. Warning This feature is implemented by intercepting all traffic on the configured HTTPS port (default: 443) and handing it over to a local TCP proxy. This bypasses NGINX completely and introduces a non-negligible performance penalty. SSL Passthrough leverages SNI and reads the virtual domain from the TLS negotiation, which requires compatible clients. After a connection has been accepted by the TLS listener, it is handled by the controller itself and piped back and forth between the backend and the client. If there is no hostname matching the requested host name, the request is handed over to NGINX on the configured passthrough proxy port (default: 442), which proxies the request to the default backend. Note Unlike HTTP backends, traffic to Passthrough backends is sent to the clusterIP of the backing Service instead of individual Endpoints.","title":"SSL Passthrough"},{"location":"user-guide/tls/#http-strict-transport-security","text":"HTTP Strict Transport Security (HSTS) is an opt-in security enhancement specified through the use of a special response header. Once a supported browser receives this header that browser will prevent any communications from being sent over HTTP to the specified domain and will instead send all communications over HTTPS. HSTS is enabled by default. To disable this behavior use hsts: \"false\" in the configuration ConfigMap .","title":"HTTP Strict Transport Security"},{"location":"user-guide/tls/#server-side-https-enforcement-through-redirect","text":"By default the controller redirects HTTP clients to the HTTPS port 443 using a 308 Permanent Redirect response if TLS is enabled for that Ingress. This can be disabled globally using ssl-redirect: \"false\" in the NGINX config map , or per-Ingress with the nginx.ingress.kubernetes.io/ssl-redirect: \"false\" annotation in the particular resource. Tip When using SSL offloading outside of cluster (e.g. AWS ELB) it may be useful to enforce a redirect to HTTPS even when there is no TLS certificate available. This can be achieved by using the nginx.ingress.kubernetes.io/force-ssl-redirect: \"true\" annotation in the particular resource.","title":"Server-side HTTPS enforcement through redirect"},{"location":"user-guide/tls/#automated-certificate-management-with-cert-manager","text":"cert-manager automatically requests missing or expired certificates from a range of supported issuers (including Let's Encrypt ) by monitoring ingress resources. To set up cert-manager you should take a look at this full example . To enable it for an ingress resource you have to deploy cert-manager, configure a certificate issuer update the manifest: apiVersion : networking.k8s.io/v1 kind : Ingress metadata : name : ingress-demo annotations : cert-manager.io/issuer : \"letsencrypt-staging\" # Replace this with a production issuer once you've tested it [ .. ] spec : tls : - hosts : - ingress-demo.example.com secretName : ingress-demo-tls [ ... ]","title":"Automated Certificate Management with cert-manager"},{"location":"user-guide/tls/#default-tls-version-and-ciphers","text":"To provide the most secure baseline configuration possible, ingress-nginx defaults to using TLS 1.2 and 1.3 only, with a secure set of TLS ciphers .","title":"Default TLS Version and Ciphers"},{"location":"user-guide/tls/#legacy-tls","text":"The default configuration, though secure, does not support some older browsers and operating systems. For instance, TLS 1.1+ is only enabled by default from Android 5.0 on. At the time of writing, May 2018, approximately 15% of Android devices are not compatible with ingress-nginx's default configuration. To change this default behavior, use a ConfigMap . A sample ConfigMap fragment to allow these older clients to connect could look something like the following (generated using the Mozilla SSL Configuration Generator) mozilla-ssl-config-old : kind: ConfigMap apiVersion: v1 metadata: name: nginx-config data: ssl-ciphers: \"ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA\" ssl-protocols: \"TLSv1 TLSv1.1 TLSv1.2 TLSv1.3\"","title":"Legacy TLS"},{"location":"user-guide/nginx-configuration/","text":"NGINX Configuration \u00b6 There are three ways to customize NGINX: ConfigMap : using a Configmap to set global configurations in NGINX. Annotations : use this if you want a specific configuration for a particular Ingress rule. Custom template : when more specific settings are required, like open_file_cache , adjust listen options as rcvbuf or when is not possible to change the configuration through the ConfigMap.","title":"Introduction"},{"location":"user-guide/nginx-configuration/#nginx-configuration","text":"There are three ways to customize NGINX: ConfigMap : using a Configmap to set global configurations in NGINX. Annotations : use this if you want a specific configuration for a particular Ingress rule. Custom template : when more specific settings are required, like open_file_cache , adjust listen options as rcvbuf or when is not possible to change the configuration through the ConfigMap.","title":"NGINX Configuration"},{"location":"user-guide/nginx-configuration/annotations/","text":"Annotations \u00b6 You can add these Kubernetes annotations to specific Ingress objects to customize their behavior. Tip Annotation keys and values can only be strings. Other types, such as boolean or numeric values must be quoted, i.e. \"true\" , \"false\" , \"100\" . Note The annotation prefix can be changed using the --annotations-prefix command line argument , but the default is nginx.ingress.kubernetes.io , as described in the table below. Name type nginx.ingress.kubernetes.io/app-root string nginx.ingress.kubernetes.io/affinity cookie nginx.ingress.kubernetes.io/affinity-mode \"balanced\" or \"persistent\" nginx.ingress.kubernetes.io/affinity-canary-behavior \"sticky\" or \"legacy\" nginx.ingress.kubernetes.io/auth-realm string nginx.ingress.kubernetes.io/auth-secret string nginx.ingress.kubernetes.io/auth-secret-type string nginx.ingress.kubernetes.io/auth-type basic or digest nginx.ingress.kubernetes.io/auth-tls-secret string nginx.ingress.kubernetes.io/auth-tls-verify-depth number nginx.ingress.kubernetes.io/auth-tls-verify-client string nginx.ingress.kubernetes.io/auth-tls-error-page string nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream \"true\" or \"false\" nginx.ingress.kubernetes.io/auth-tls-match-cn string nginx.ingress.kubernetes.io/auth-url string nginx.ingress.kubernetes.io/auth-cache-key string nginx.ingress.kubernetes.io/auth-cache-duration string nginx.ingress.kubernetes.io/auth-keepalive number nginx.ingress.kubernetes.io/auth-keepalive-requests number nginx.ingress.kubernetes.io/auth-keepalive-timeout number nginx.ingress.kubernetes.io/auth-proxy-set-headers string nginx.ingress.kubernetes.io/auth-snippet string nginx.ingress.kubernetes.io/enable-global-auth \"true\" or \"false\" nginx.ingress.kubernetes.io/backend-protocol string nginx.ingress.kubernetes.io/canary \"true\" or \"false\" nginx.ingress.kubernetes.io/canary-by-header string nginx.ingress.kubernetes.io/canary-by-header-value string nginx.ingress.kubernetes.io/canary-by-header-pattern string nginx.ingress.kubernetes.io/canary-by-cookie string nginx.ingress.kubernetes.io/canary-weight number nginx.ingress.kubernetes.io/canary-weight-total number nginx.ingress.kubernetes.io/client-body-buffer-size string nginx.ingress.kubernetes.io/configuration-snippet string nginx.ingress.kubernetes.io/custom-http-errors []int nginx.ingress.kubernetes.io/default-backend string nginx.ingress.kubernetes.io/enable-cors \"true\" or \"false\" nginx.ingress.kubernetes.io/cors-allow-origin string nginx.ingress.kubernetes.io/cors-allow-methods string nginx.ingress.kubernetes.io/cors-allow-headers string nginx.ingress.kubernetes.io/cors-expose-headers string nginx.ingress.kubernetes.io/cors-allow-credentials \"true\" or \"false\" nginx.ingress.kubernetes.io/cors-max-age number nginx.ingress.kubernetes.io/force-ssl-redirect \"true\" or \"false\" nginx.ingress.kubernetes.io/from-to-www-redirect \"true\" or \"false\" nginx.ingress.kubernetes.io/http2-push-preload \"true\" or \"false\" nginx.ingress.kubernetes.io/limit-connections number nginx.ingress.kubernetes.io/limit-rps number nginx.ingress.kubernetes.io/global-rate-limit number nginx.ingress.kubernetes.io/global-rate-limit-window duration nginx.ingress.kubernetes.io/global-rate-limit-key string nginx.ingress.kubernetes.io/global-rate-limit-ignored-cidrs string nginx.ingress.kubernetes.io/permanent-redirect string nginx.ingress.kubernetes.io/permanent-redirect-code number nginx.ingress.kubernetes.io/temporal-redirect string nginx.ingress.kubernetes.io/preserve-trailing-slash \"true\" or \"false\" nginx.ingress.kubernetes.io/proxy-body-size string nginx.ingress.kubernetes.io/proxy-cookie-domain string nginx.ingress.kubernetes.io/proxy-cookie-path string nginx.ingress.kubernetes.io/proxy-connect-timeout number nginx.ingress.kubernetes.io/proxy-send-timeout number nginx.ingress.kubernetes.io/proxy-read-timeout number nginx.ingress.kubernetes.io/proxy-next-upstream string nginx.ingress.kubernetes.io/proxy-next-upstream-timeout number nginx.ingress.kubernetes.io/proxy-next-upstream-tries number nginx.ingress.kubernetes.io/proxy-request-buffering string nginx.ingress.kubernetes.io/proxy-redirect-from string nginx.ingress.kubernetes.io/proxy-redirect-to string nginx.ingress.kubernetes.io/proxy-http-version \"1.0\" or \"1.1\" nginx.ingress.kubernetes.io/proxy-ssl-secret string nginx.ingress.kubernetes.io/proxy-ssl-ciphers string nginx.ingress.kubernetes.io/proxy-ssl-name string nginx.ingress.kubernetes.io/proxy-ssl-protocols string nginx.ingress.kubernetes.io/proxy-ssl-verify string nginx.ingress.kubernetes.io/proxy-ssl-verify-depth number nginx.ingress.kubernetes.io/proxy-ssl-server-name string nginx.ingress.kubernetes.io/enable-rewrite-log \"true\" or \"false\" nginx.ingress.kubernetes.io/rewrite-target URI nginx.ingress.kubernetes.io/satisfy string nginx.ingress.kubernetes.io/server-alias string nginx.ingress.kubernetes.io/server-snippet string nginx.ingress.kubernetes.io/service-upstream \"true\" or \"false\" nginx.ingress.kubernetes.io/session-cookie-name string nginx.ingress.kubernetes.io/session-cookie-path string nginx.ingress.kubernetes.io/session-cookie-domain string nginx.ingress.kubernetes.io/session-cookie-change-on-failure \"true\" or \"false\" nginx.ingress.kubernetes.io/session-cookie-samesite string nginx.ingress.kubernetes.io/session-cookie-conditional-samesite-none \"true\" or \"false\" nginx.ingress.kubernetes.io/ssl-redirect \"true\" or \"false\" nginx.ingress.kubernetes.io/ssl-passthrough \"true\" or \"false\" nginx.ingress.kubernetes.io/stream-snippet string nginx.ingress.kubernetes.io/upstream-hash-by string nginx.ingress.kubernetes.io/x-forwarded-prefix string nginx.ingress.kubernetes.io/load-balance string nginx.ingress.kubernetes.io/upstream-vhost string nginx.ingress.kubernetes.io/whitelist-source-range CIDR nginx.ingress.kubernetes.io/proxy-buffering string nginx.ingress.kubernetes.io/proxy-buffers-number number nginx.ingress.kubernetes.io/proxy-buffer-size string nginx.ingress.kubernetes.io/proxy-max-temp-file-size string nginx.ingress.kubernetes.io/ssl-ciphers string nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers \"true\" or \"false\" nginx.ingress.kubernetes.io/connection-proxy-header string nginx.ingress.kubernetes.io/enable-access-log \"true\" or \"false\" nginx.ingress.kubernetes.io/enable-opentracing \"true\" or \"false\" nginx.ingress.kubernetes.io/opentracing-trust-incoming-span \"true\" or \"false\" nginx.ingress.kubernetes.io/enable-influxdb \"true\" or \"false\" nginx.ingress.kubernetes.io/influxdb-measurement string nginx.ingress.kubernetes.io/influxdb-port string nginx.ingress.kubernetes.io/influxdb-host string nginx.ingress.kubernetes.io/influxdb-server-name string nginx.ingress.kubernetes.io/use-regex bool nginx.ingress.kubernetes.io/enable-modsecurity bool nginx.ingress.kubernetes.io/enable-owasp-core-rules bool nginx.ingress.kubernetes.io/modsecurity-transaction-id string nginx.ingress.kubernetes.io/modsecurity-snippet string nginx.ingress.kubernetes.io/mirror-request-body string nginx.ingress.kubernetes.io/mirror-target string nginx.ingress.kubernetes.io/mirror-host string Canary \u00b6 In some cases, you may want to \"canary\" a new set of changes by sending a small number of requests to a different service than the production service. The canary annotation enables the Ingress spec to act as an alternative service for requests to route to depending on the rules applied. The following annotations to configure canary can be enabled after nginx.ingress.kubernetes.io/canary: \"true\" is set: nginx.ingress.kubernetes.io/canary-by-header : The header to use for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the request header is set to always , it will be routed to the canary. When the header is set to never , it will never be routed to the canary. For any other value, the header will be ignored and the request compared against the other canary rules by precedence. nginx.ingress.kubernetes.io/canary-by-header-value : The header value to match for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the request header is set to this value, it will be routed to the canary. For any other header value, the header will be ignored and the request compared against the other canary rules by precedence. This annotation has to be used together with nginx.ingress.kubernetes.io/canary-by-header . The annotation is an extension of the nginx.ingress.kubernetes.io/canary-by-header to allow customizing the header value instead of using hardcoded values. It doesn't have any effect if the nginx.ingress.kubernetes.io/canary-by-header annotation is not defined. nginx.ingress.kubernetes.io/canary-by-header-pattern : This works the same way as canary-by-header-value except it does PCRE Regex matching. Note that when canary-by-header-value is set this annotation will be ignored. When the given Regex causes error during request processing, the request will be considered as not matching. nginx.ingress.kubernetes.io/canary-by-cookie : The cookie to use for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the cookie value is set to always , it will be routed to the canary. When the cookie is set to never , it will never be routed to the canary. For any other value, the cookie will be ignored and the request compared against the other canary rules by precedence. nginx.ingress.kubernetes.io/canary-weight : The integer based (0 - ) percent of random requests that should be routed to the service specified in the canary Ingress. A weight of 0 implies that no requests will be sent to the service in the Canary ingress by this canary rule. A weight of means implies all requests will be sent to the alternative service specified in the Ingress. <weight-total> defaults to 100, and can be increased via nginx.ingress.kubernetes.io/canary-weight-total . nginx.ingress.kubernetes.io/canary-weight-total : The total weight of traffic. If unspecified, it defaults to 100. Canary rules are evaluated in order of precedence. Precedence is as follows: canary-by-header -> canary-by-cookie -> canary-weight Note that when you mark an ingress as canary, then all the other non-canary annotations will be ignored (inherited from the corresponding main ingress) except nginx.ingress.kubernetes.io/load-balance , nginx.ingress.kubernetes.io/upstream-hash-by , and annotations related to session affinity . If you want to restore the original behavior of canaries when session affinity was ignored, set nginx.ingress.kubernetes.io/affinity-canary-behavior annotation with value legacy on the canary ingress definition. Known Limitations Currently a maximum of one canary ingress can be applied per Ingress rule. Rewrite \u00b6 In some scenarios the exposed URL in the backend service differs from the specified path in the Ingress rule. Without a rewrite any request will return 404. Set the annotation nginx.ingress.kubernetes.io/rewrite-target to the path expected by the service. If the Application Root is exposed in a different path and needs to be redirected, set the annotation nginx.ingress.kubernetes.io/app-root to redirect requests for / . Example Please check the rewrite example. Session Affinity \u00b6 The annotation nginx.ingress.kubernetes.io/affinity enables and sets the affinity type in all Upstreams of an Ingress. This way, a request will always be directed to the same upstream server. The only affinity type available for NGINX is cookie . The annotation nginx.ingress.kubernetes.io/affinity-mode defines the stickiness of a session. Setting this to balanced (default) will redistribute some sessions if a deployment gets scaled up, therefore rebalancing the load on the servers. Setting this to persistent will not rebalance sessions to new servers, therefore providing maximum stickiness. The annotation nginx.ingress.kubernetes.io/affinity-canary-behavior defines the behavior of canaries when session affinity is enabled. Setting this to sticky (default) will ensure that users that were served by canaries, will continue to be served by canaries. Setting this to legacy will restore original canary behavior, when session affinity was ignored. Attention If more than one Ingress is defined for a host and at least one Ingress uses nginx.ingress.kubernetes.io/affinity: cookie , then only paths on the Ingress using nginx.ingress.kubernetes.io/affinity will use session cookie affinity. All paths defined on other Ingresses for the host will be load balanced through the random selection of a backend server. Example Please check the affinity example. Cookie affinity \u00b6 If you use the cookie affinity type you can also specify the name of the cookie that will be used to route the requests with the annotation nginx.ingress.kubernetes.io/session-cookie-name . The default is to create a cookie named 'INGRESSCOOKIE'. The NGINX annotation nginx.ingress.kubernetes.io/session-cookie-path defines the path that will be set on the cookie. This is optional unless the annotation nginx.ingress.kubernetes.io/use-regex is set to true; Session cookie paths do not support regex. Use nginx.ingress.kubernetes.io/session-cookie-domain to set the Domain attribute of the sticky cookie. Use nginx.ingress.kubernetes.io/session-cookie-samesite to apply a SameSite attribute to the sticky cookie. Browser accepted values are None , Lax , and Strict . Some browsers reject cookies with SameSite=None , including those created before the SameSite=None specification (e.g. Chrome 5X). Other browsers mistakenly treat SameSite=None cookies as SameSite=Strict (e.g. Safari running on OSX 14). To omit SameSite=None from browsers with these incompatibilities, add the annotation nginx.ingress.kubernetes.io/session-cookie-conditional-samesite-none: \"true\" . Authentication \u00b6 It is possible to add authentication by adding additional annotations in the Ingress rule. The source of the authentication is a secret that contains usernames and passwords. The annotations are: nginx.ingress.kubernetes.io/auth-type: [basic|digest] Indicates the HTTP Authentication Type: Basic or Digest Access Authentication . nginx.ingress.kubernetes.io/auth-secret: secretName The name of the Secret that contains the usernames and passwords which are granted access to the path s defined in the Ingress rules. This annotation also accepts the alternative form \"namespace/secretName\", in which case the Secret lookup is performed in the referenced namespace instead of the Ingress namespace. nginx.ingress.kubernetes.io/auth-secret-type: [auth-file|auth-map] The auth-secret can have two forms: auth-file - default, an htpasswd file in the key auth within the secret auth-map - the keys of the secret are the usernames, and the values are the hashed passwords nginx.ingress.kubernetes.io/auth-realm: \"realm string\" Example Please check the auth example. Custom NGINX upstream hashing \u00b6 NGINX supports load balancing by client-server mapping based on consistent hashing for a given key. The key can contain text, variables or any combination thereof. This feature allows for request stickiness other than client IP or cookies. The ketama consistent hashing method will be used which ensures only a few keys would be remapped to different servers on upstream group changes. There is a special mode of upstream hashing called subset. In this mode, upstream servers are grouped into subsets, and stickiness works by mapping keys to a subset instead of individual upstream servers. Specific server is chosen uniformly at random from the selected sticky subset. It provides a balance between stickiness and load distribution. To enable consistent hashing for a backend: nginx.ingress.kubernetes.io/upstream-hash-by : the nginx variable, text value or any combination thereof to use for consistent hashing. For example: nginx.ingress.kubernetes.io/upstream-hash-by: \"$request_uri\" or nginx.ingress.kubernetes.io/upstream-hash-by: \"$request_uri$host\" or nginx.ingress.kubernetes.io/upstream-hash-by: \"${request_uri}-text-value\" to consistently hash upstream requests by the current request URI. \"subset\" hashing can be enabled setting nginx.ingress.kubernetes.io/upstream-hash-by-subset : \"true\". This maps requests to subset of nodes instead of a single one. upstream-hash-by-subset-size determines the size of each subset (default 3). Please check the chashsubset example. Custom NGINX load balancing \u00b6 This is similar to load-balance in ConfigMap , but configures load balancing algorithm per ingress. Note that nginx.ingress.kubernetes.io/upstream-hash-by takes preference over this. If this and nginx.ingress.kubernetes.io/upstream-hash-by are not set then we fallback to using globally configured load balancing algorithm. Custom NGINX upstream vhost \u00b6 This configuration setting allows you to control the value for host in the following statement: proxy_set_header Host $host , which forms part of the location block. This is useful if you need to call the upstream server by something other than $host . Client Certificate Authentication \u00b6 It is possible to enable Client Certificate Authentication using additional annotations in Ingress Rule. Client Certificate Authentication is applied per host and it is not possible to specify rules that differ for individual paths. To enable, add the annotation nginx.ingress.kubernetes.io/auth-tls-secret: namespace/secretName . This secret must have a file named ca.crt containing the full Certificate Authority chain ca.crt that is enabled to authenticate against this Ingress. You can further customize client certificate authentication and behavior with these annotations: nginx.ingress.kubernetes.io/auth-tls-verify-depth : The validation depth between the provided client certificate and the Certification Authority chain. (default: 1) nginx.ingress.kubernetes.io/auth-tls-verify-client : Enables verification of client certificates. Possible values are: on : Request a client certificate that must be signed by a certificate that is included in the secret key ca.crt of the secret specified by nginx.ingress.kubernetes.io/auth-tls-secret: namespace/secretName . Failed certificate verification will result in a status code 400 (Bad Request) (default) off : Don't request client certificates and don't do client certificate verification. optional : Do optional client certificate validation against the CAs from auth-tls-secret . The request fails with status code 400 (Bad Request) when a certificate is provided that is not signed by the CA. When no or an otherwise invalid certificate is provided, the request does not fail, but instead the verification result is sent to the upstream service. optional_no_ca : Do optional client certificate validation, but do not fail the request when the client certificate is not signed by the CAs from auth-tls-secret . Certificate verification result is sent to the upstream service. nginx.ingress.kubernetes.io/auth-tls-error-page : The URL/Page that user should be redirected in case of a Certificate Authentication Error nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream : Indicates if the received certificates should be passed or not to the upstream server in the header ssl-client-cert . Possible values are \"true\" or \"false\" (default). nginx.ingress.kubernetes.io/auth-tls-match-cn : Adds a sanity check for the CN of the client certificate that is sent over using a string / regex starting with \"CN=\", example: \"CN=myvalidclient\" . If the certificate CN sent during mTLS does not match your string / regex it will fail with status code 403. Another way of using this is by adding multiple options in your regex, example: \"CN=(option1|option2|myvalidclient)\" . In this case, as long as one of the options in the brackets matches the certificate CN then you will receive a 200 status code. The following headers are sent to the upstream service according to the auth-tls-* annotations: ssl-client-issuer-dn : The issuer information of the client certificate. Example: \"CN=My CA\" ssl-client-subject-dn : The subject information of the client certificate. Example: \"CN=My Client\" ssl-client-verify : The result of the client verification. Possible values: \"SUCCESS\", \"FAILED: \" ssl-client-cert : The full client certificate in PEM format. Will only be sent when nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream is set to \"true\". Example: -----BEGIN%20CERTIFICATE-----%0A...---END%20CERTIFICATE-----%0A Example Please check the client-certs example. Attention TLS with Client Authentication is not possible in Cloudflare and might result in unexpected behavior. Cloudflare only allows Authenticated Origin Pulls and is required to use their own certificate: https://blog.cloudflare.com/protecting-the-origin-with-tls-authenticated-origin-pulls/ Only Authenticated Origin Pulls are allowed and can be configured by following their tutorial: https://support.cloudflare.com/hc/en-us/articles/204494148-Setting-up-NGINX-to-use-TLS-Authenticated-Origin-Pulls Backend Certificate Authentication \u00b6 It is possible to authenticate to a proxied HTTPS backend with certificate using additional annotations in Ingress Rule. nginx.ingress.kubernetes.io/proxy-ssl-secret: secretName : Specifies a Secret with the certificate tls.crt , key tls.key in PEM format used for authentication to a proxied HTTPS server. It should also contain trusted CA certificates ca.crt in PEM format used to verify the certificate of the proxied HTTPS server. This annotation expects the Secret name in the form \"namespace/secretName\". nginx.ingress.kubernetes.io/proxy-ssl-verify : Enables or disables verification of the proxied HTTPS server certificate. (default: off) nginx.ingress.kubernetes.io/proxy-ssl-verify-depth : Sets the verification depth in the proxied HTTPS server certificates chain. (default: 1) nginx.ingress.kubernetes.io/proxy-ssl-ciphers : Specifies the enabled ciphers for requests to a proxied HTTPS server. The ciphers are specified in the format understood by the OpenSSL library. nginx.ingress.kubernetes.io/proxy-ssl-name : Allows to set proxy_ssl_name . This allows overriding the server name used to verify the certificate of the proxied HTTPS server. This value is also passed through SNI when a connection is established to the proxied HTTPS server. nginx.ingress.kubernetes.io/proxy-ssl-protocols : Enables the specified protocols for requests to a proxied HTTPS server. nginx.ingress.kubernetes.io/proxy-ssl-server-name : Enables passing of the server name through TLS Server Name Indication extension (SNI, RFC 6066) when establishing a connection with the proxied HTTPS server. Configuration snippet \u00b6 Using this annotation you can add additional configuration to the NGINX location. For example: nginx.ingress.kubernetes.io/configuration-snippet : | more_set_headers \"Request-Id: $req_id\"; Be aware this can be dangerous in multi-tenant clusters, as it can lead to people with otherwise limited permissions being able to retrieve all secrets on the cluster. The recommended mitigation for this threat is to disable this feature, so it may not work for you. See CVE-2021-25742 and the related issue on github for more information. Custom HTTP Errors \u00b6 Like the custom-http-errors value in the ConfigMap, this annotation will set NGINX proxy-intercept-errors , but only for the NGINX location associated with this ingress. If a default backend annotation is specified on the ingress, the errors will be routed to that annotation's default backend service (instead of the global default backend). Different ingresses can specify different sets of error codes. Even if multiple ingress objects share the same hostname, this annotation can be used to intercept different error codes for each ingress (for example, different error codes to be intercepted for different paths on the same hostname, if each path is on a different ingress). If custom-http-errors is also specified globally, the error values specified in this annotation will override the global value for the given ingress' hostname and path. Example usage: nginx.ingress.kubernetes.io/custom-http-errors: \"404,415\" Default Backend \u00b6 This annotation is of the form nginx.ingress.kubernetes.io/default-backend: <svc name> to specify a custom default backend. This <svc name> is a reference to a service inside of the same namespace in which you are applying this annotation. This annotation overrides the global default backend. In case the service has multiple ports , the first one is the one which will receive the backend traffic. This service will be used to handle the response when the configured service in the Ingress rule does not have any active endpoints. It will also be used to handle the error responses if both this annotation and the custom-http-errors annotation are set. Enable CORS \u00b6 To enable Cross-Origin Resource Sharing (CORS) in an Ingress rule, add the annotation nginx.ingress.kubernetes.io/enable-cors: \"true\" . This will add a section in the server location enabling this functionality. CORS can be controlled with the following annotations: nginx.ingress.kubernetes.io/cors-allow-methods : Controls which methods are accepted. This is a multi-valued field, separated by ',' and accepts only letters (upper and lower case). Default: GET, PUT, POST, DELETE, PATCH, OPTIONS Example: nginx.ingress.kubernetes.io/cors-allow-methods: \"PUT, GET, POST, OPTIONS\" nginx.ingress.kubernetes.io/cors-allow-headers : Controls which headers are accepted. This is a multi-valued field, separated by ',' and accepts letters, numbers, _ and -. Default: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization Example: nginx.ingress.kubernetes.io/cors-allow-headers: \"X-Forwarded-For, X-app123-XPTO\" nginx.ingress.kubernetes.io/cors-expose-headers : Controls which headers are exposed to response. This is a multi-valued field, separated by ',' and accepts letters, numbers, _, - and *. Default: empty Example: nginx.ingress.kubernetes.io/cors-expose-headers: \"*, X-CustomResponseHeader\" nginx.ingress.kubernetes.io/cors-allow-origin : Controls what's the accepted Origin for CORS. This is a multi-valued field, separated by ','. It must follow this format: http(s)://origin-site.com or http(s)://origin-site.com:port Default: * Example: nginx.ingress.kubernetes.io/cors-allow-origin: \"https://origin-site.com:4443, http://origin-site.com, https://example.org:1199\" It also supports single level wildcard subdomains and follows this format: http(s)://*.foo.bar , http(s)://*.bar.foo:8080 or http(s)://*.abc.bar.foo:9000 - Example: nginx.ingress.kubernetes.io/cors-allow-origin: \"https://*.origin-site.com:4443, http://*.origin-site.com, https://example.org:1199\" nginx.ingress.kubernetes.io/cors-allow-credentials : Controls if credentials can be passed during CORS operations. Default: true Example: nginx.ingress.kubernetes.io/cors-allow-credentials: \"false\" nginx.ingress.kubernetes.io/cors-max-age : Controls how long preflight requests can be cached. Default: 1728000 Example: nginx.ingress.kubernetes.io/cors-max-age: 600 Note For more information please see https://enable-cors.org HTTP2 Push Preload. \u00b6 Enables automatic conversion of preload links specified in the \u201cLink\u201d response header fields into push requests. Example nginx.ingress.kubernetes.io/http2-push-preload: \"true\" Server Alias \u00b6 Allows the definition of one or more aliases in the server definition of the NGINX configuration using the annotation nginx.ingress.kubernetes.io/server-alias: \"<alias 1>,<alias 2>\" . This will create a server with the same configuration, but adding new values to the server_name directive. Note A server-alias name cannot conflict with the hostname of an existing server. If it does, the server-alias annotation will be ignored. If a server-alias is created and later a new server with the same hostname is created, the new server configuration will take place over the alias configuration. For more information please see the server_name documentation . Server snippet \u00b6 Using the annotation nginx.ingress.kubernetes.io/server-snippet it is possible to add custom configuration in the server configuration block. apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : nginx.ingress.kubernetes.io/server-snippet : | set $agentflag 0; if ($http_user_agent ~* \"(Mobile)\" ){ set $agentflag 1; } if ( $agentflag = 1 ) { return 301 https://m.example.com; } Attention This annotation can be used only once per host. Client Body Buffer Size \u00b6 Sets buffer size for reading client request body per location. In case the request body is larger than the buffer, the whole body or only its part is written to a temporary file. By default, buffer size is equal to two memory pages. This is 8K on x86, other 32-bit platforms, and x86-64. It is usually 16K on other 64-bit platforms. This annotation is applied to each location provided in the ingress rule. Note The annotation value must be given in a format understood by Nginx. Example nginx.ingress.kubernetes.io/client-body-buffer-size: \"1000\" # 1000 bytes nginx.ingress.kubernetes.io/client-body-buffer-size: 1k # 1 kilobyte nginx.ingress.kubernetes.io/client-body-buffer-size: 1K # 1 kilobyte nginx.ingress.kubernetes.io/client-body-buffer-size: 1m # 1 megabyte nginx.ingress.kubernetes.io/client-body-buffer-size: 1M # 1 megabyte For more information please see https://nginx.org External Authentication \u00b6 To use an existing service that provides authentication the Ingress rule can be annotated with nginx.ingress.kubernetes.io/auth-url to indicate the URL where the HTTP request should be sent. nginx.ingress.kubernetes.io/auth-url : \"URL to the authentication service\" Additionally it is possible to set: nginx.ingress.kubernetes.io/auth-keepalive : <Connections> to specify the maximum number of keepalive connections to auth-url . Only takes effect when no variables are used in the host part of the URL. Defaults to 0 (keepalive disabled). Note: does not work with HTTP/2 listener because of a limitation in Lua subrequests . UseHTTP2 configuration should be disabled! nginx.ingress.kubernetes.io/auth-keepalive-requests : <Requests> to specify the maximum number of requests that can be served through one keepalive connection. Defaults to 1000 and only applied if auth-keepalive is set to higher than 0 . nginx.ingress.kubernetes.io/auth-keepalive-timeout : <Timeout> to specify a duration in seconds which an idle keepalive connection to an upstream server will stay open. Defaults to 60 and only applied if auth-keepalive is set to higher than 0 . nginx.ingress.kubernetes.io/auth-method : <Method> to specify the HTTP method to use. nginx.ingress.kubernetes.io/auth-signin : <SignIn_URL> to specify the location of the error page. nginx.ingress.kubernetes.io/auth-signin-redirect-param : <SignIn_URL> to specify the URL parameter in the error page which should contain the original URL for a failed signin request. nginx.ingress.kubernetes.io/auth-response-headers : <Response_Header_1, ..., Response_Header_n> to specify headers to pass to backend once authentication request completes. nginx.ingress.kubernetes.io/auth-proxy-set-headers : <ConfigMap> the name of a ConfigMap that specifies headers to pass to the authentication service nginx.ingress.kubernetes.io/auth-request-redirect : <Request_Redirect_URL> to specify the X-Auth-Request-Redirect header value. nginx.ingress.kubernetes.io/auth-cache-key : <Cache_Key> this enables caching for auth requests. specify a lookup key for auth responses. e.g. $remote_user$http_authorization . Each server and location has it's own keyspace. Hence a cached response is only valid on a per-server and per-location basis. nginx.ingress.kubernetes.io/auth-cache-duration : <Cache_duration> to specify a caching time for auth responses based on their response codes, e.g. 200 202 30m . See proxy_cache_valid for details. You may specify multiple, comma-separated values: 200 202 10m, 401 5m . defaults to 200 202 401 5m . nginx.ingress.kubernetes.io/auth-always-set-cookie : <Boolean_Flag> to set a cookie returned by auth request. By default, the cookie will be set only if an upstream reports with the code 200, 201, 204, 206, 301, 302, 303, 304, 307, or 308. nginx.ingress.kubernetes.io/auth-snippet : <Auth_Snippet> to specify a custom snippet to use with external authentication, e.g. nginx.ingress.kubernetes.io/auth-url : http://foo.com/external-auth nginx.ingress.kubernetes.io/auth-snippet : | proxy_set_header Foo-Header 42; Note: nginx.ingress.kubernetes.io/auth-snippet is an optional annotation. However, it may only be used in conjunction with nginx.ingress.kubernetes.io/auth-url and will be ignored if nginx.ingress.kubernetes.io/auth-url is not set Example Please check the external-auth example. Global External Authentication \u00b6 By default the controller redirects all requests to an existing service that provides authentication if global-auth-url is set in the NGINX ConfigMap. If you want to disable this behavior for that ingress, you can use enable-global-auth: \"false\" in the NGINX ConfigMap. nginx.ingress.kubernetes.io/enable-global-auth : indicates if GlobalExternalAuth configuration should be applied or not to this Ingress rule. Default values is set to \"true\" . Note For more information please see global-auth-url . Rate Limiting \u00b6 These annotations define limits on connections and transmission rates. These can be used to mitigate DDoS Attacks . nginx.ingress.kubernetes.io/limit-connections : number of concurrent connections allowed from a single IP address. A 503 error is returned when exceeding this limit. nginx.ingress.kubernetes.io/limit-rps : number of requests accepted from a given IP each second. The burst limit is set to this limit multiplied by the burst multiplier, the default multiplier is 5. When clients exceed this limit, limit-req-status-code default: 503 is returned. nginx.ingress.kubernetes.io/limit-rpm : number of requests accepted from a given IP each minute. The burst limit is set to this limit multiplied by the burst multiplier, the default multiplier is 5. When clients exceed this limit, limit-req-status-code default: 503 is returned. nginx.ingress.kubernetes.io/limit-burst-multiplier : multiplier of the limit rate for burst size. The default burst multiplier is 5, this annotation override the default multiplier. When clients exceed this limit, limit-req-status-code default: 503 is returned. nginx.ingress.kubernetes.io/limit-rate-after : initial number of kilobytes after which the further transmission of a response to a given connection will be rate limited. This feature must be used with proxy-buffering enabled. nginx.ingress.kubernetes.io/limit-rate : number of kilobytes per second allowed to send to a given connection. The zero value disables rate limiting. This feature must be used with proxy-buffering enabled. nginx.ingress.kubernetes.io/limit-whitelist : client IP source ranges to be excluded from rate-limiting. The value is a comma separated list of CIDRs. If you specify multiple annotations in a single Ingress rule, limits are applied in the order limit-connections , limit-rpm , limit-rps . To configure settings globally for all Ingress rules, the limit-rate-after and limit-rate values may be set in the NGINX ConfigMap . The value set in an Ingress annotation will override the global setting. The client IP address will be set based on the use of PROXY protocol or from the X-Forwarded-For header value when use-forwarded-headers is enabled. Global Rate Limiting \u00b6 Note: Be careful when configuring both (Local) Rate Limiting and Global Rate Limiting at the same time. They are two completely different rate limiting implementations. Whichever limit exceeds first will reject the requests. It might be a good idea to configure both of them to ease load on Global Rate Limiting backend in cases of spike in traffic. The stock NGINX rate limiting does not share its counters among different NGINX instances. Given that most ingress-nginx deployments are elastic and number of replicas can change any day it is impossible to configure a proper rate limit using stock NGINX functionalities. Global Rate Limiting overcome this by using lua-resty-global-throttle . lua-resty-global-throttle shares its counters via a central store such as memcached . The obvious shortcoming of this is users have to deploy and operate a memcached instance in order to benefit from this functionality. Configure the memcached using these configmap settings . Here are a few remarks for ingress-nginx integration of lua-resty-global-throttle : We minimize memcached access by caching exceeding limit decisions. The expiry of cache entry is the desired delay lua-resty-global-throttle calculates for us. The Lua Shared Dictionary used for that is global_throttle_cache . Currently its size defaults to 10M. Customize it as per your needs using lua-shared-dicts . When we fail to cache the exceeding limit decision then we log an NGINX error. You can monitor for that error to decide if you need to bump the cache size. Without cache the cost of processing a request is two memcached commands: GET , and INCR . With the cache it is only INCR . Log NGINX variable $global_rate_limit_exceeding 's value to have some visibility into what portion of requests are rejected (value y ), whether they are rejected using cached decision (value c ), or if they are not rejected (default value n ). You can use log-format-upstream to include that in access logs. In case of an error it will log the error message and fail open . The annotations below creates Global Rate Limiting instance per ingress. That means if there are multiple paths configured under the same ingress, the Global Rate Limiting will count requests to all the paths under the same counter. Extract a path out into its own ingress if you need to isolate a certain path. nginx.ingress.kubernetes.io/global-rate-limit : Configures maximum allowed number of requests per window. Required. nginx.ingress.kubernetes.io/global-rate-limit-window : Configures a time window (i.e 1m ) that the limit is applied. Required. nginx.ingress.kubernetes.io/global-rate-limit-key : Configures a key for counting the samples. Defaults to $remote_addr . You can also combine multiple NGINX variables here, like ${remote_addr}-${http_x_api_client} which would mean the limit will be applied to requests coming from the same API client (indicated by X-API-Client HTTP request header) with the same source IP address. nginx.ingress.kubernetes.io/global-rate-limit-ignored-cidrs : comma separated list of IPs and CIDRs to match client IP against. When there's a match request is not considered for rate limiting. Permanent Redirect \u00b6 This annotation allows to return a permanent redirect (Return Code 301) instead of sending data to the upstream. For example nginx.ingress.kubernetes.io/permanent-redirect: https://www.google.com would redirect everything to Google. Permanent Redirect Code \u00b6 This annotation allows you to modify the status code used for permanent redirects. For example nginx.ingress.kubernetes.io/permanent-redirect-code: '308' would return your permanent-redirect with a 308. Temporal Redirect \u00b6 This annotation allows you to return a temporal redirect (Return Code 302) instead of sending data to the upstream. For example nginx.ingress.kubernetes.io/temporal-redirect: https://www.google.com would redirect everything to Google with a Return Code of 302 (Moved Temporarily) SSL Passthrough \u00b6 The annotation nginx.ingress.kubernetes.io/ssl-passthrough instructs the controller to send TLS connections directly to the backend instead of letting NGINX decrypt the communication. See also TLS/HTTPS in the User guide. Note SSL Passthrough is disabled by default and requires starting the controller with the --enable-ssl-passthrough flag. Attention Because SSL Passthrough works on layer 4 of the OSI model (TCP) and not on the layer 7 (HTTP), using SSL Passthrough invalidates all the other annotations set on an Ingress object. Service Upstream \u00b6 By default the NGINX ingress controller uses a list of all endpoints (Pod IP/port) in the NGINX upstream configuration. The nginx.ingress.kubernetes.io/service-upstream annotation disables that behavior and instead uses a single upstream in NGINX, the service's Cluster IP and port. This can be desirable for things like zero-downtime deployments . See issue #257 . Known Issues \u00b6 If the service-upstream annotation is specified the following things should be taken into consideration: Sticky Sessions will not work as only round-robin load balancing is supported. The proxy_next_upstream directive will not have any effect meaning on error the request will not be dispatched to another upstream. Server-side HTTPS enforcement through redirect \u00b6 By default the controller redirects (308) to HTTPS if TLS is enabled for that ingress. If you want to disable this behavior globally, you can use ssl-redirect: \"false\" in the NGINX ConfigMap . To configure this feature for specific ingress resources, you can use the nginx.ingress.kubernetes.io/ssl-redirect: \"false\" annotation in the particular resource. When using SSL offloading outside of cluster (e.g. AWS ELB) it may be useful to enforce a redirect to HTTPS even when there is no TLS certificate available. This can be achieved by using the nginx.ingress.kubernetes.io/force-ssl-redirect: \"true\" annotation in the particular resource. To preserve the trailing slash in the URI with ssl-redirect , set nginx.ingress.kubernetes.io/preserve-trailing-slash: \"true\" annotation for that particular resource. Redirect from/to www \u00b6 In some scenarios is required to redirect from www.domain.com to domain.com or vice versa. To enable this feature use the annotation nginx.ingress.kubernetes.io/from-to-www-redirect: \"true\" Attention If at some point a new Ingress is created with a host equal to one of the options (like domain.com ) the annotation will be omitted. Attention For HTTPS to HTTPS redirects is mandatory the SSL Certificate defined in the Secret, located in the TLS section of Ingress, contains both FQDN in the common name of the certificate. Whitelist source range \u00b6 You can specify allowed client IP source ranges through the nginx.ingress.kubernetes.io/whitelist-source-range annotation. The value is a comma separated list of CIDRs , e.g. 10.0.0.0/24,172.10.0.1 . To configure this setting globally for all Ingress rules, the whitelist-source-range value may be set in the NGINX ConfigMap . Note Adding an annotation to an Ingress rule overrides any global restriction. Custom timeouts \u00b6 Using the configuration configmap it is possible to set the default global timeout for connections to the upstream servers. In some scenarios is required to have different values. To allow this we provide annotations that allows this customization: nginx.ingress.kubernetes.io/proxy-connect-timeout nginx.ingress.kubernetes.io/proxy-send-timeout nginx.ingress.kubernetes.io/proxy-read-timeout nginx.ingress.kubernetes.io/proxy-next-upstream nginx.ingress.kubernetes.io/proxy-next-upstream-timeout nginx.ingress.kubernetes.io/proxy-next-upstream-tries nginx.ingress.kubernetes.io/proxy-request-buffering Note: All timeout values are unitless and in seconds e.g. nginx.ingress.kubernetes.io/proxy-read-timeout: \"120\" sets a valid 120 seconds proxy read timeout. Proxy redirect \u00b6 The annotations nginx.ingress.kubernetes.io/proxy-redirect-from and nginx.ingress.kubernetes.io/proxy-redirect-to will set the first and second parameters of NGINX's proxy_redirect directive respectively. It is possible to set the text that should be changed in the Location and Refresh header fields of a proxied server response Setting \"off\" or \"default\" in the annotation nginx.ingress.kubernetes.io/proxy-redirect-from disables nginx.ingress.kubernetes.io/proxy-redirect-to , otherwise, both annotations must be used in unison. Note that each annotation must be a string without spaces. By default the value of each annotation is \"off\". Custom max body size \u00b6 For NGINX, an 413 error will be returned to the client when the size in a request exceeds the maximum allowed size of the client request body. This size can be configured by the parameter client_max_body_size . To configure this setting globally for all Ingress rules, the proxy-body-size value may be set in the NGINX ConfigMap . To use custom values in an Ingress rule define these annotation: nginx.ingress.kubernetes.io/proxy-body-size : 8m Proxy cookie domain \u00b6 Sets a text that should be changed in the domain attribute of the \"Set-Cookie\" header fields of a proxied server response. To configure this setting globally for all Ingress rules, the proxy-cookie-domain value may be set in the NGINX ConfigMap . Proxy cookie path \u00b6 Sets a text that should be changed in the path attribute of the \"Set-Cookie\" header fields of a proxied server response. To configure this setting globally for all Ingress rules, the proxy-cookie-path value may be set in the NGINX ConfigMap . Proxy buffering \u00b6 Enable or disable proxy buffering proxy_buffering . By default proxy buffering is disabled in the NGINX config. To configure this setting globally for all Ingress rules, the proxy-buffering value may be set in the NGINX ConfigMap . To use custom values in an Ingress rule define these annotation: nginx.ingress.kubernetes.io/proxy-buffering : \"on\" Proxy buffers Number \u00b6 Sets the number of the buffers in proxy_buffers used for reading the first part of the response received from the proxied server. By default proxy buffers number is set as 4 To configure this setting globally, set proxy-buffers-number in NGINX ConfigMap . To use custom values in an Ingress rule, define this annotation: nginx.ingress.kubernetes.io/proxy-buffers-number : \"4\" Proxy buffer size \u00b6 Sets the size of the buffer proxy_buffer_size used for reading the first part of the response received from the proxied server. By default proxy buffer size is set as \"4k\" To configure this setting globally, set proxy-buffer-size in NGINX ConfigMap . To use custom values in an Ingress rule, define this annotation: nginx.ingress.kubernetes.io/proxy-buffer-size : \"8k\" Proxy max temp file size \u00b6 When buffering of responses from the proxied server is enabled, and the whole response does not fit into the buffers set by the proxy_buffer_size and proxy_buffers directives, a part of the response can be saved to a temporary file. This directive sets the maximum size of the temporary file setting the proxy_max_temp_file_size . The size of data written to the temporary file at a time is set by the proxy_temp_file_write_size directive. The zero value disables buffering of responses to temporary files. To use custom values in an Ingress rule, define this annotation: nginx.ingress.kubernetes.io/proxy-max-temp-file-size : \"1024m\" Proxy HTTP version \u00b6 Using this annotation sets the proxy_http_version that the Nginx reverse proxy will use to communicate with the backend. By default this is set to \"1.1\". nginx.ingress.kubernetes.io/proxy-http-version : \"1.0\" SSL ciphers \u00b6 Specifies the enabled ciphers . Using this annotation will set the ssl_ciphers directive at the server level. This configuration is active for all the paths in the host. nginx.ingress.kubernetes.io/ssl-ciphers : \"ALL:!aNULL:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP\" The following annotation will set the ssl_prefer_server_ciphers directive at the server level. This configuration specifies that server ciphers should be preferred over client ciphers when using the SSLv3 and TLS protocols. nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers : \"true\" Connection proxy header \u00b6 Using this annotation will override the default connection header set by NGINX. To use custom values in an Ingress rule, define the annotation: nginx.ingress.kubernetes.io/connection-proxy-header : \"keep-alive\" Enable Access Log \u00b6 Access logs are enabled by default, but in some scenarios access logs might be required to be disabled for a given ingress. To do this, use the annotation: nginx.ingress.kubernetes.io/enable-access-log : \"false\" Enable Rewrite Log \u00b6 Rewrite logs are not enabled by default. In some scenarios it could be required to enable NGINX rewrite logs. Note that rewrite logs are sent to the error_log file at the notice level. To enable this feature use the annotation: nginx.ingress.kubernetes.io/enable-rewrite-log : \"true\" Enable Opentracing \u00b6 Opentracing can be enabled or disabled globally through the ConfigMap but this will sometimes need to be overridden to enable it or disable it for a specific ingress (e.g. to turn off tracing of external health check endpoints) nginx.ingress.kubernetes.io/enable-opentracing : \"true\" Opentracing Trust Incoming Span \u00b6 The option to trust incoming trace spans can be enabled or disabled globally through the ConfigMap but this will sometimes need to be overridden to enable it or disable it for a specific ingress (e.g. only enable on a private endpoint) nginx.ingress.kubernetes.io/opentracing-trust-incoming-span : \"true\" X-Forwarded-Prefix Header \u00b6 To add the non-standard X-Forwarded-Prefix header to the upstream request with a string value, the following annotation can be used: nginx.ingress.kubernetes.io/x-forwarded-prefix : \"/path\" ModSecurity \u00b6 ModSecurity is an OpenSource Web Application firewall. It can be enabled for a particular set of ingress locations. The ModSecurity module must first be enabled by enabling ModSecurity in the ConfigMap . Note this will enable ModSecurity for all paths, and each path must be disabled manually. It can be enabled using the following annotation: nginx.ingress.kubernetes.io/enable-modsecurity : \"true\" ModSecurity will run in \"Detection-Only\" mode using the recommended configuration . You can enable the OWASP Core Rule Set by setting the following annotation: nginx.ingress.kubernetes.io/enable-owasp-core-rules : \"true\" You can pass transactionIDs from nginx by setting up the following: nginx.ingress.kubernetes.io/modsecurity-transaction-id : \"$request_id\" You can also add your own set of modsecurity rules via a snippet: nginx.ingress.kubernetes.io/modsecurity-snippet : | SecRuleEngine On SecDebugLog /tmp/modsec_debug.log Note: If you use both enable-owasp-core-rules and modsecurity-snippet annotations together, only the modsecurity-snippet will take effect. If you wish to include the OWASP Core Rule Set or recommended configuration simply use the include statement: nginx 0.24.1 and below nginx.ingress.kubernetes.io/modsecurity-snippet : | Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf Include /etc/nginx/modsecurity/modsecurity.conf nginx 0.25.0 and above nginx.ingress.kubernetes.io/modsecurity-snippet : | Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf InfluxDB \u00b6 Using influxdb-* annotations we can monitor requests passing through a Location by sending them to an InfluxDB backend exposing the UDP socket using the nginx-influxdb-module . nginx.ingress.kubernetes.io/enable-influxdb : \"true\" nginx.ingress.kubernetes.io/influxdb-measurement : \"nginx-reqs\" nginx.ingress.kubernetes.io/influxdb-port : \"8089\" nginx.ingress.kubernetes.io/influxdb-host : \"127.0.0.1\" nginx.ingress.kubernetes.io/influxdb-server-name : \"nginx-ingress\" For the influxdb-host parameter you have two options: Use an InfluxDB server configured with the UDP protocol enabled. Deploy Telegraf as a sidecar proxy to the Ingress controller configured to listen UDP with the socket listener input and to write using anyone of the outputs plugins like InfluxDB, Apache Kafka, Prometheus, etc.. (recommended) It's important to remember that there's no DNS resolver at this stage so you will have to configure an ip address to nginx.ingress.kubernetes.io/influxdb-host . If you deploy Influx or Telegraf as sidecar (another container in the same pod) this becomes straightforward since you can directly use 127.0.0.1 . Backend Protocol \u00b6 Using backend-protocol annotations is possible to indicate how NGINX should communicate with the backend service. (Replaces secure-backends in older versions) Valid Values: HTTP, HTTPS, GRPC, GRPCS, AJP and FCGI By default NGINX uses HTTP . Example: nginx.ingress.kubernetes.io/backend-protocol : \"HTTPS\" Use Regex \u00b6 Attention When using this annotation with the NGINX annotation nginx.ingress.kubernetes.io/affinity of type cookie , nginx.ingress.kubernetes.io/session-cookie-path must be also set; Session cookie paths do not support regex. Using the nginx.ingress.kubernetes.io/use-regex annotation will indicate whether or not the paths defined on an Ingress use regular expressions. The default value is false . The following will indicate that regular expression paths are being used: nginx.ingress.kubernetes.io/use-regex : \"true\" The following will indicate that regular expression paths are not being used: nginx.ingress.kubernetes.io/use-regex : \"false\" When this annotation is set to true , the case insensitive regular expression location modifier will be enforced on ALL paths for a given host regardless of what Ingress they are defined on. Additionally, if the rewrite-target annotation is used on any Ingress for a given host, then the case insensitive regular expression location modifier will be enforced on ALL paths for a given host regardless of what Ingress they are defined on. Please read about ingress path matching before using this modifier. Satisfy \u00b6 By default, a request would need to satisfy all authentication requirements in order to be allowed. By using this annotation, requests that satisfy either any or all authentication requirements are allowed, based on the configuration value. nginx.ingress.kubernetes.io/satisfy : \"any\" Mirror \u00b6 Enables a request to be mirrored to a mirror backend. Responses by mirror backends are ignored. This feature is useful, to see how requests will react in \"test\" backends. The mirror backend can be set by applying: nginx.ingress.kubernetes.io/mirror-target : https://test.env.com/$request_uri By default the request-body is sent to the mirror backend, but can be turned off by applying: nginx.ingress.kubernetes.io/mirror-request-body : \"off\" Also by default header Host for mirrored requests will be set the same as a host part of uri in the \"mirror-target\" annotation. You can override it by \"mirror-host\" annotation: nginx.ingress.kubernetes.io/mirror-target : https://1.2.3.4/$request_uri nginx.ingress.kubernetes.io/mirror-host : \"test.env.com\" Note: The mirror directive will be applied to all paths within the ingress resource. The request sent to the mirror is linked to the original request. If you have a slow mirror backend, then the original request will throttle. For more information on the mirror module see ngx_http_mirror_module Stream snippet \u00b6 Using the annotation nginx.ingress.kubernetes.io/stream-snippet it is possible to add custom stream configuration. apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : nginx.ingress.kubernetes.io/stream-snippet : | server { listen 8000; proxy_pass 127.0.0.1:80; }","title":"Annotations"},{"location":"user-guide/nginx-configuration/annotations/#annotations","text":"You can add these Kubernetes annotations to specific Ingress objects to customize their behavior. Tip Annotation keys and values can only be strings. Other types, such as boolean or numeric values must be quoted, i.e. \"true\" , \"false\" , \"100\" . Note The annotation prefix can be changed using the --annotations-prefix command line argument , but the default is nginx.ingress.kubernetes.io , as described in the table below. Name type nginx.ingress.kubernetes.io/app-root string nginx.ingress.kubernetes.io/affinity cookie nginx.ingress.kubernetes.io/affinity-mode \"balanced\" or \"persistent\" nginx.ingress.kubernetes.io/affinity-canary-behavior \"sticky\" or \"legacy\" nginx.ingress.kubernetes.io/auth-realm string nginx.ingress.kubernetes.io/auth-secret string nginx.ingress.kubernetes.io/auth-secret-type string nginx.ingress.kubernetes.io/auth-type basic or digest nginx.ingress.kubernetes.io/auth-tls-secret string nginx.ingress.kubernetes.io/auth-tls-verify-depth number nginx.ingress.kubernetes.io/auth-tls-verify-client string nginx.ingress.kubernetes.io/auth-tls-error-page string nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream \"true\" or \"false\" nginx.ingress.kubernetes.io/auth-tls-match-cn string nginx.ingress.kubernetes.io/auth-url string nginx.ingress.kubernetes.io/auth-cache-key string nginx.ingress.kubernetes.io/auth-cache-duration string nginx.ingress.kubernetes.io/auth-keepalive number nginx.ingress.kubernetes.io/auth-keepalive-requests number nginx.ingress.kubernetes.io/auth-keepalive-timeout number nginx.ingress.kubernetes.io/auth-proxy-set-headers string nginx.ingress.kubernetes.io/auth-snippet string nginx.ingress.kubernetes.io/enable-global-auth \"true\" or \"false\" nginx.ingress.kubernetes.io/backend-protocol string nginx.ingress.kubernetes.io/canary \"true\" or \"false\" nginx.ingress.kubernetes.io/canary-by-header string nginx.ingress.kubernetes.io/canary-by-header-value string nginx.ingress.kubernetes.io/canary-by-header-pattern string nginx.ingress.kubernetes.io/canary-by-cookie string nginx.ingress.kubernetes.io/canary-weight number nginx.ingress.kubernetes.io/canary-weight-total number nginx.ingress.kubernetes.io/client-body-buffer-size string nginx.ingress.kubernetes.io/configuration-snippet string nginx.ingress.kubernetes.io/custom-http-errors []int nginx.ingress.kubernetes.io/default-backend string nginx.ingress.kubernetes.io/enable-cors \"true\" or \"false\" nginx.ingress.kubernetes.io/cors-allow-origin string nginx.ingress.kubernetes.io/cors-allow-methods string nginx.ingress.kubernetes.io/cors-allow-headers string nginx.ingress.kubernetes.io/cors-expose-headers string nginx.ingress.kubernetes.io/cors-allow-credentials \"true\" or \"false\" nginx.ingress.kubernetes.io/cors-max-age number nginx.ingress.kubernetes.io/force-ssl-redirect \"true\" or \"false\" nginx.ingress.kubernetes.io/from-to-www-redirect \"true\" or \"false\" nginx.ingress.kubernetes.io/http2-push-preload \"true\" or \"false\" nginx.ingress.kubernetes.io/limit-connections number nginx.ingress.kubernetes.io/limit-rps number nginx.ingress.kubernetes.io/global-rate-limit number nginx.ingress.kubernetes.io/global-rate-limit-window duration nginx.ingress.kubernetes.io/global-rate-limit-key string nginx.ingress.kubernetes.io/global-rate-limit-ignored-cidrs string nginx.ingress.kubernetes.io/permanent-redirect string nginx.ingress.kubernetes.io/permanent-redirect-code number nginx.ingress.kubernetes.io/temporal-redirect string nginx.ingress.kubernetes.io/preserve-trailing-slash \"true\" or \"false\" nginx.ingress.kubernetes.io/proxy-body-size string nginx.ingress.kubernetes.io/proxy-cookie-domain string nginx.ingress.kubernetes.io/proxy-cookie-path string nginx.ingress.kubernetes.io/proxy-connect-timeout number nginx.ingress.kubernetes.io/proxy-send-timeout number nginx.ingress.kubernetes.io/proxy-read-timeout number nginx.ingress.kubernetes.io/proxy-next-upstream string nginx.ingress.kubernetes.io/proxy-next-upstream-timeout number nginx.ingress.kubernetes.io/proxy-next-upstream-tries number nginx.ingress.kubernetes.io/proxy-request-buffering string nginx.ingress.kubernetes.io/proxy-redirect-from string nginx.ingress.kubernetes.io/proxy-redirect-to string nginx.ingress.kubernetes.io/proxy-http-version \"1.0\" or \"1.1\" nginx.ingress.kubernetes.io/proxy-ssl-secret string nginx.ingress.kubernetes.io/proxy-ssl-ciphers string nginx.ingress.kubernetes.io/proxy-ssl-name string nginx.ingress.kubernetes.io/proxy-ssl-protocols string nginx.ingress.kubernetes.io/proxy-ssl-verify string nginx.ingress.kubernetes.io/proxy-ssl-verify-depth number nginx.ingress.kubernetes.io/proxy-ssl-server-name string nginx.ingress.kubernetes.io/enable-rewrite-log \"true\" or \"false\" nginx.ingress.kubernetes.io/rewrite-target URI nginx.ingress.kubernetes.io/satisfy string nginx.ingress.kubernetes.io/server-alias string nginx.ingress.kubernetes.io/server-snippet string nginx.ingress.kubernetes.io/service-upstream \"true\" or \"false\" nginx.ingress.kubernetes.io/session-cookie-name string nginx.ingress.kubernetes.io/session-cookie-path string nginx.ingress.kubernetes.io/session-cookie-domain string nginx.ingress.kubernetes.io/session-cookie-change-on-failure \"true\" or \"false\" nginx.ingress.kubernetes.io/session-cookie-samesite string nginx.ingress.kubernetes.io/session-cookie-conditional-samesite-none \"true\" or \"false\" nginx.ingress.kubernetes.io/ssl-redirect \"true\" or \"false\" nginx.ingress.kubernetes.io/ssl-passthrough \"true\" or \"false\" nginx.ingress.kubernetes.io/stream-snippet string nginx.ingress.kubernetes.io/upstream-hash-by string nginx.ingress.kubernetes.io/x-forwarded-prefix string nginx.ingress.kubernetes.io/load-balance string nginx.ingress.kubernetes.io/upstream-vhost string nginx.ingress.kubernetes.io/whitelist-source-range CIDR nginx.ingress.kubernetes.io/proxy-buffering string nginx.ingress.kubernetes.io/proxy-buffers-number number nginx.ingress.kubernetes.io/proxy-buffer-size string nginx.ingress.kubernetes.io/proxy-max-temp-file-size string nginx.ingress.kubernetes.io/ssl-ciphers string nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers \"true\" or \"false\" nginx.ingress.kubernetes.io/connection-proxy-header string nginx.ingress.kubernetes.io/enable-access-log \"true\" or \"false\" nginx.ingress.kubernetes.io/enable-opentracing \"true\" or \"false\" nginx.ingress.kubernetes.io/opentracing-trust-incoming-span \"true\" or \"false\" nginx.ingress.kubernetes.io/enable-influxdb \"true\" or \"false\" nginx.ingress.kubernetes.io/influxdb-measurement string nginx.ingress.kubernetes.io/influxdb-port string nginx.ingress.kubernetes.io/influxdb-host string nginx.ingress.kubernetes.io/influxdb-server-name string nginx.ingress.kubernetes.io/use-regex bool nginx.ingress.kubernetes.io/enable-modsecurity bool nginx.ingress.kubernetes.io/enable-owasp-core-rules bool nginx.ingress.kubernetes.io/modsecurity-transaction-id string nginx.ingress.kubernetes.io/modsecurity-snippet string nginx.ingress.kubernetes.io/mirror-request-body string nginx.ingress.kubernetes.io/mirror-target string nginx.ingress.kubernetes.io/mirror-host string","title":"Annotations"},{"location":"user-guide/nginx-configuration/annotations/#canary","text":"In some cases, you may want to \"canary\" a new set of changes by sending a small number of requests to a different service than the production service. The canary annotation enables the Ingress spec to act as an alternative service for requests to route to depending on the rules applied. The following annotations to configure canary can be enabled after nginx.ingress.kubernetes.io/canary: \"true\" is set: nginx.ingress.kubernetes.io/canary-by-header : The header to use for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the request header is set to always , it will be routed to the canary. When the header is set to never , it will never be routed to the canary. For any other value, the header will be ignored and the request compared against the other canary rules by precedence. nginx.ingress.kubernetes.io/canary-by-header-value : The header value to match for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the request header is set to this value, it will be routed to the canary. For any other header value, the header will be ignored and the request compared against the other canary rules by precedence. This annotation has to be used together with nginx.ingress.kubernetes.io/canary-by-header . The annotation is an extension of the nginx.ingress.kubernetes.io/canary-by-header to allow customizing the header value instead of using hardcoded values. It doesn't have any effect if the nginx.ingress.kubernetes.io/canary-by-header annotation is not defined. nginx.ingress.kubernetes.io/canary-by-header-pattern : This works the same way as canary-by-header-value except it does PCRE Regex matching. Note that when canary-by-header-value is set this annotation will be ignored. When the given Regex causes error during request processing, the request will be considered as not matching. nginx.ingress.kubernetes.io/canary-by-cookie : The cookie to use for notifying the Ingress to route the request to the service specified in the Canary Ingress. When the cookie value is set to always , it will be routed to the canary. When the cookie is set to never , it will never be routed to the canary. For any other value, the cookie will be ignored and the request compared against the other canary rules by precedence. nginx.ingress.kubernetes.io/canary-weight : The integer based (0 - ) percent of random requests that should be routed to the service specified in the canary Ingress. A weight of 0 implies that no requests will be sent to the service in the Canary ingress by this canary rule. A weight of means implies all requests will be sent to the alternative service specified in the Ingress. <weight-total> defaults to 100, and can be increased via nginx.ingress.kubernetes.io/canary-weight-total . nginx.ingress.kubernetes.io/canary-weight-total : The total weight of traffic. If unspecified, it defaults to 100. Canary rules are evaluated in order of precedence. Precedence is as follows: canary-by-header -> canary-by-cookie -> canary-weight Note that when you mark an ingress as canary, then all the other non-canary annotations will be ignored (inherited from the corresponding main ingress) except nginx.ingress.kubernetes.io/load-balance , nginx.ingress.kubernetes.io/upstream-hash-by , and annotations related to session affinity . If you want to restore the original behavior of canaries when session affinity was ignored, set nginx.ingress.kubernetes.io/affinity-canary-behavior annotation with value legacy on the canary ingress definition. Known Limitations Currently a maximum of one canary ingress can be applied per Ingress rule.","title":"Canary"},{"location":"user-guide/nginx-configuration/annotations/#rewrite","text":"In some scenarios the exposed URL in the backend service differs from the specified path in the Ingress rule. Without a rewrite any request will return 404. Set the annotation nginx.ingress.kubernetes.io/rewrite-target to the path expected by the service. If the Application Root is exposed in a different path and needs to be redirected, set the annotation nginx.ingress.kubernetes.io/app-root to redirect requests for / . Example Please check the rewrite example.","title":"Rewrite"},{"location":"user-guide/nginx-configuration/annotations/#session-affinity","text":"The annotation nginx.ingress.kubernetes.io/affinity enables and sets the affinity type in all Upstreams of an Ingress. This way, a request will always be directed to the same upstream server. The only affinity type available for NGINX is cookie . The annotation nginx.ingress.kubernetes.io/affinity-mode defines the stickiness of a session. Setting this to balanced (default) will redistribute some sessions if a deployment gets scaled up, therefore rebalancing the load on the servers. Setting this to persistent will not rebalance sessions to new servers, therefore providing maximum stickiness. The annotation nginx.ingress.kubernetes.io/affinity-canary-behavior defines the behavior of canaries when session affinity is enabled. Setting this to sticky (default) will ensure that users that were served by canaries, will continue to be served by canaries. Setting this to legacy will restore original canary behavior, when session affinity was ignored. Attention If more than one Ingress is defined for a host and at least one Ingress uses nginx.ingress.kubernetes.io/affinity: cookie , then only paths on the Ingress using nginx.ingress.kubernetes.io/affinity will use session cookie affinity. All paths defined on other Ingresses for the host will be load balanced through the random selection of a backend server. Example Please check the affinity example.","title":"Session Affinity"},{"location":"user-guide/nginx-configuration/annotations/#cookie-affinity","text":"If you use the cookie affinity type you can also specify the name of the cookie that will be used to route the requests with the annotation nginx.ingress.kubernetes.io/session-cookie-name . The default is to create a cookie named 'INGRESSCOOKIE'. The NGINX annotation nginx.ingress.kubernetes.io/session-cookie-path defines the path that will be set on the cookie. This is optional unless the annotation nginx.ingress.kubernetes.io/use-regex is set to true; Session cookie paths do not support regex. Use nginx.ingress.kubernetes.io/session-cookie-domain to set the Domain attribute of the sticky cookie. Use nginx.ingress.kubernetes.io/session-cookie-samesite to apply a SameSite attribute to the sticky cookie. Browser accepted values are None , Lax , and Strict . Some browsers reject cookies with SameSite=None , including those created before the SameSite=None specification (e.g. Chrome 5X). Other browsers mistakenly treat SameSite=None cookies as SameSite=Strict (e.g. Safari running on OSX 14). To omit SameSite=None from browsers with these incompatibilities, add the annotation nginx.ingress.kubernetes.io/session-cookie-conditional-samesite-none: \"true\" .","title":"Cookie affinity"},{"location":"user-guide/nginx-configuration/annotations/#authentication","text":"It is possible to add authentication by adding additional annotations in the Ingress rule. The source of the authentication is a secret that contains usernames and passwords. The annotations are: nginx.ingress.kubernetes.io/auth-type: [basic|digest] Indicates the HTTP Authentication Type: Basic or Digest Access Authentication . nginx.ingress.kubernetes.io/auth-secret: secretName The name of the Secret that contains the usernames and passwords which are granted access to the path s defined in the Ingress rules. This annotation also accepts the alternative form \"namespace/secretName\", in which case the Secret lookup is performed in the referenced namespace instead of the Ingress namespace. nginx.ingress.kubernetes.io/auth-secret-type: [auth-file|auth-map] The auth-secret can have two forms: auth-file - default, an htpasswd file in the key auth within the secret auth-map - the keys of the secret are the usernames, and the values are the hashed passwords nginx.ingress.kubernetes.io/auth-realm: \"realm string\" Example Please check the auth example.","title":"Authentication"},{"location":"user-guide/nginx-configuration/annotations/#custom-nginx-upstream-hashing","text":"NGINX supports load balancing by client-server mapping based on consistent hashing for a given key. The key can contain text, variables or any combination thereof. This feature allows for request stickiness other than client IP or cookies. The ketama consistent hashing method will be used which ensures only a few keys would be remapped to different servers on upstream group changes. There is a special mode of upstream hashing called subset. In this mode, upstream servers are grouped into subsets, and stickiness works by mapping keys to a subset instead of individual upstream servers. Specific server is chosen uniformly at random from the selected sticky subset. It provides a balance between stickiness and load distribution. To enable consistent hashing for a backend: nginx.ingress.kubernetes.io/upstream-hash-by : the nginx variable, text value or any combination thereof to use for consistent hashing. For example: nginx.ingress.kubernetes.io/upstream-hash-by: \"$request_uri\" or nginx.ingress.kubernetes.io/upstream-hash-by: \"$request_uri$host\" or nginx.ingress.kubernetes.io/upstream-hash-by: \"${request_uri}-text-value\" to consistently hash upstream requests by the current request URI. \"subset\" hashing can be enabled setting nginx.ingress.kubernetes.io/upstream-hash-by-subset : \"true\". This maps requests to subset of nodes instead of a single one. upstream-hash-by-subset-size determines the size of each subset (default 3). Please check the chashsubset example.","title":"Custom NGINX upstream hashing"},{"location":"user-guide/nginx-configuration/annotations/#custom-nginx-load-balancing","text":"This is similar to load-balance in ConfigMap , but configures load balancing algorithm per ingress. Note that nginx.ingress.kubernetes.io/upstream-hash-by takes preference over this. If this and nginx.ingress.kubernetes.io/upstream-hash-by are not set then we fallback to using globally configured load balancing algorithm.","title":"Custom NGINX load balancing"},{"location":"user-guide/nginx-configuration/annotations/#custom-nginx-upstream-vhost","text":"This configuration setting allows you to control the value for host in the following statement: proxy_set_header Host $host , which forms part of the location block. This is useful if you need to call the upstream server by something other than $host .","title":"Custom NGINX upstream vhost"},{"location":"user-guide/nginx-configuration/annotations/#client-certificate-authentication","text":"It is possible to enable Client Certificate Authentication using additional annotations in Ingress Rule. Client Certificate Authentication is applied per host and it is not possible to specify rules that differ for individual paths. To enable, add the annotation nginx.ingress.kubernetes.io/auth-tls-secret: namespace/secretName . This secret must have a file named ca.crt containing the full Certificate Authority chain ca.crt that is enabled to authenticate against this Ingress. You can further customize client certificate authentication and behavior with these annotations: nginx.ingress.kubernetes.io/auth-tls-verify-depth : The validation depth between the provided client certificate and the Certification Authority chain. (default: 1) nginx.ingress.kubernetes.io/auth-tls-verify-client : Enables verification of client certificates. Possible values are: on : Request a client certificate that must be signed by a certificate that is included in the secret key ca.crt of the secret specified by nginx.ingress.kubernetes.io/auth-tls-secret: namespace/secretName . Failed certificate verification will result in a status code 400 (Bad Request) (default) off : Don't request client certificates and don't do client certificate verification. optional : Do optional client certificate validation against the CAs from auth-tls-secret . The request fails with status code 400 (Bad Request) when a certificate is provided that is not signed by the CA. When no or an otherwise invalid certificate is provided, the request does not fail, but instead the verification result is sent to the upstream service. optional_no_ca : Do optional client certificate validation, but do not fail the request when the client certificate is not signed by the CAs from auth-tls-secret . Certificate verification result is sent to the upstream service. nginx.ingress.kubernetes.io/auth-tls-error-page : The URL/Page that user should be redirected in case of a Certificate Authentication Error nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream : Indicates if the received certificates should be passed or not to the upstream server in the header ssl-client-cert . Possible values are \"true\" or \"false\" (default). nginx.ingress.kubernetes.io/auth-tls-match-cn : Adds a sanity check for the CN of the client certificate that is sent over using a string / regex starting with \"CN=\", example: \"CN=myvalidclient\" . If the certificate CN sent during mTLS does not match your string / regex it will fail with status code 403. Another way of using this is by adding multiple options in your regex, example: \"CN=(option1|option2|myvalidclient)\" . In this case, as long as one of the options in the brackets matches the certificate CN then you will receive a 200 status code. The following headers are sent to the upstream service according to the auth-tls-* annotations: ssl-client-issuer-dn : The issuer information of the client certificate. Example: \"CN=My CA\" ssl-client-subject-dn : The subject information of the client certificate. Example: \"CN=My Client\" ssl-client-verify : The result of the client verification. Possible values: \"SUCCESS\", \"FAILED: \" ssl-client-cert : The full client certificate in PEM format. Will only be sent when nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream is set to \"true\". Example: -----BEGIN%20CERTIFICATE-----%0A...---END%20CERTIFICATE-----%0A Example Please check the client-certs example. Attention TLS with Client Authentication is not possible in Cloudflare and might result in unexpected behavior. Cloudflare only allows Authenticated Origin Pulls and is required to use their own certificate: https://blog.cloudflare.com/protecting-the-origin-with-tls-authenticated-origin-pulls/ Only Authenticated Origin Pulls are allowed and can be configured by following their tutorial: https://support.cloudflare.com/hc/en-us/articles/204494148-Setting-up-NGINX-to-use-TLS-Authenticated-Origin-Pulls","title":"Client Certificate Authentication"},{"location":"user-guide/nginx-configuration/annotations/#backend-certificate-authentication","text":"It is possible to authenticate to a proxied HTTPS backend with certificate using additional annotations in Ingress Rule. nginx.ingress.kubernetes.io/proxy-ssl-secret: secretName : Specifies a Secret with the certificate tls.crt , key tls.key in PEM format used for authentication to a proxied HTTPS server. It should also contain trusted CA certificates ca.crt in PEM format used to verify the certificate of the proxied HTTPS server. This annotation expects the Secret name in the form \"namespace/secretName\". nginx.ingress.kubernetes.io/proxy-ssl-verify : Enables or disables verification of the proxied HTTPS server certificate. (default: off) nginx.ingress.kubernetes.io/proxy-ssl-verify-depth : Sets the verification depth in the proxied HTTPS server certificates chain. (default: 1) nginx.ingress.kubernetes.io/proxy-ssl-ciphers : Specifies the enabled ciphers for requests to a proxied HTTPS server. The ciphers are specified in the format understood by the OpenSSL library. nginx.ingress.kubernetes.io/proxy-ssl-name : Allows to set proxy_ssl_name . This allows overriding the server name used to verify the certificate of the proxied HTTPS server. This value is also passed through SNI when a connection is established to the proxied HTTPS server. nginx.ingress.kubernetes.io/proxy-ssl-protocols : Enables the specified protocols for requests to a proxied HTTPS server. nginx.ingress.kubernetes.io/proxy-ssl-server-name : Enables passing of the server name through TLS Server Name Indication extension (SNI, RFC 6066) when establishing a connection with the proxied HTTPS server.","title":"Backend Certificate Authentication"},{"location":"user-guide/nginx-configuration/annotations/#configuration-snippet","text":"Using this annotation you can add additional configuration to the NGINX location. For example: nginx.ingress.kubernetes.io/configuration-snippet : | more_set_headers \"Request-Id: $req_id\"; Be aware this can be dangerous in multi-tenant clusters, as it can lead to people with otherwise limited permissions being able to retrieve all secrets on the cluster. The recommended mitigation for this threat is to disable this feature, so it may not work for you. See CVE-2021-25742 and the related issue on github for more information.","title":"Configuration snippet"},{"location":"user-guide/nginx-configuration/annotations/#custom-http-errors","text":"Like the custom-http-errors value in the ConfigMap, this annotation will set NGINX proxy-intercept-errors , but only for the NGINX location associated with this ingress. If a default backend annotation is specified on the ingress, the errors will be routed to that annotation's default backend service (instead of the global default backend). Different ingresses can specify different sets of error codes. Even if multiple ingress objects share the same hostname, this annotation can be used to intercept different error codes for each ingress (for example, different error codes to be intercepted for different paths on the same hostname, if each path is on a different ingress). If custom-http-errors is also specified globally, the error values specified in this annotation will override the global value for the given ingress' hostname and path. Example usage: nginx.ingress.kubernetes.io/custom-http-errors: \"404,415\"","title":"Custom HTTP Errors"},{"location":"user-guide/nginx-configuration/annotations/#default-backend","text":"This annotation is of the form nginx.ingress.kubernetes.io/default-backend: <svc name> to specify a custom default backend. This <svc name> is a reference to a service inside of the same namespace in which you are applying this annotation. This annotation overrides the global default backend. In case the service has multiple ports , the first one is the one which will receive the backend traffic. This service will be used to handle the response when the configured service in the Ingress rule does not have any active endpoints. It will also be used to handle the error responses if both this annotation and the custom-http-errors annotation are set.","title":"Default Backend"},{"location":"user-guide/nginx-configuration/annotations/#enable-cors","text":"To enable Cross-Origin Resource Sharing (CORS) in an Ingress rule, add the annotation nginx.ingress.kubernetes.io/enable-cors: \"true\" . This will add a section in the server location enabling this functionality. CORS can be controlled with the following annotations: nginx.ingress.kubernetes.io/cors-allow-methods : Controls which methods are accepted. This is a multi-valued field, separated by ',' and accepts only letters (upper and lower case). Default: GET, PUT, POST, DELETE, PATCH, OPTIONS Example: nginx.ingress.kubernetes.io/cors-allow-methods: \"PUT, GET, POST, OPTIONS\" nginx.ingress.kubernetes.io/cors-allow-headers : Controls which headers are accepted. This is a multi-valued field, separated by ',' and accepts letters, numbers, _ and -. Default: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization Example: nginx.ingress.kubernetes.io/cors-allow-headers: \"X-Forwarded-For, X-app123-XPTO\" nginx.ingress.kubernetes.io/cors-expose-headers : Controls which headers are exposed to response. This is a multi-valued field, separated by ',' and accepts letters, numbers, _, - and *. Default: empty Example: nginx.ingress.kubernetes.io/cors-expose-headers: \"*, X-CustomResponseHeader\" nginx.ingress.kubernetes.io/cors-allow-origin : Controls what's the accepted Origin for CORS. This is a multi-valued field, separated by ','. It must follow this format: http(s)://origin-site.com or http(s)://origin-site.com:port Default: * Example: nginx.ingress.kubernetes.io/cors-allow-origin: \"https://origin-site.com:4443, http://origin-site.com, https://example.org:1199\" It also supports single level wildcard subdomains and follows this format: http(s)://*.foo.bar , http(s)://*.bar.foo:8080 or http(s)://*.abc.bar.foo:9000 - Example: nginx.ingress.kubernetes.io/cors-allow-origin: \"https://*.origin-site.com:4443, http://*.origin-site.com, https://example.org:1199\" nginx.ingress.kubernetes.io/cors-allow-credentials : Controls if credentials can be passed during CORS operations. Default: true Example: nginx.ingress.kubernetes.io/cors-allow-credentials: \"false\" nginx.ingress.kubernetes.io/cors-max-age : Controls how long preflight requests can be cached. Default: 1728000 Example: nginx.ingress.kubernetes.io/cors-max-age: 600 Note For more information please see https://enable-cors.org","title":"Enable CORS"},{"location":"user-guide/nginx-configuration/annotations/#http2-push-preload","text":"Enables automatic conversion of preload links specified in the \u201cLink\u201d response header fields into push requests. Example nginx.ingress.kubernetes.io/http2-push-preload: \"true\"","title":"HTTP2 Push Preload."},{"location":"user-guide/nginx-configuration/annotations/#server-alias","text":"Allows the definition of one or more aliases in the server definition of the NGINX configuration using the annotation nginx.ingress.kubernetes.io/server-alias: \"<alias 1>,<alias 2>\" . This will create a server with the same configuration, but adding new values to the server_name directive. Note A server-alias name cannot conflict with the hostname of an existing server. If it does, the server-alias annotation will be ignored. If a server-alias is created and later a new server with the same hostname is created, the new server configuration will take place over the alias configuration. For more information please see the server_name documentation .","title":"Server Alias"},{"location":"user-guide/nginx-configuration/annotations/#server-snippet","text":"Using the annotation nginx.ingress.kubernetes.io/server-snippet it is possible to add custom configuration in the server configuration block. apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : nginx.ingress.kubernetes.io/server-snippet : | set $agentflag 0; if ($http_user_agent ~* \"(Mobile)\" ){ set $agentflag 1; } if ( $agentflag = 1 ) { return 301 https://m.example.com; } Attention This annotation can be used only once per host.","title":"Server snippet"},{"location":"user-guide/nginx-configuration/annotations/#client-body-buffer-size","text":"Sets buffer size for reading client request body per location. In case the request body is larger than the buffer, the whole body or only its part is written to a temporary file. By default, buffer size is equal to two memory pages. This is 8K on x86, other 32-bit platforms, and x86-64. It is usually 16K on other 64-bit platforms. This annotation is applied to each location provided in the ingress rule. Note The annotation value must be given in a format understood by Nginx. Example nginx.ingress.kubernetes.io/client-body-buffer-size: \"1000\" # 1000 bytes nginx.ingress.kubernetes.io/client-body-buffer-size: 1k # 1 kilobyte nginx.ingress.kubernetes.io/client-body-buffer-size: 1K # 1 kilobyte nginx.ingress.kubernetes.io/client-body-buffer-size: 1m # 1 megabyte nginx.ingress.kubernetes.io/client-body-buffer-size: 1M # 1 megabyte For more information please see https://nginx.org","title":"Client Body Buffer Size"},{"location":"user-guide/nginx-configuration/annotations/#external-authentication","text":"To use an existing service that provides authentication the Ingress rule can be annotated with nginx.ingress.kubernetes.io/auth-url to indicate the URL where the HTTP request should be sent. nginx.ingress.kubernetes.io/auth-url : \"URL to the authentication service\" Additionally it is possible to set: nginx.ingress.kubernetes.io/auth-keepalive : <Connections> to specify the maximum number of keepalive connections to auth-url . Only takes effect when no variables are used in the host part of the URL. Defaults to 0 (keepalive disabled). Note: does not work with HTTP/2 listener because of a limitation in Lua subrequests . UseHTTP2 configuration should be disabled! nginx.ingress.kubernetes.io/auth-keepalive-requests : <Requests> to specify the maximum number of requests that can be served through one keepalive connection. Defaults to 1000 and only applied if auth-keepalive is set to higher than 0 . nginx.ingress.kubernetes.io/auth-keepalive-timeout : <Timeout> to specify a duration in seconds which an idle keepalive connection to an upstream server will stay open. Defaults to 60 and only applied if auth-keepalive is set to higher than 0 . nginx.ingress.kubernetes.io/auth-method : <Method> to specify the HTTP method to use. nginx.ingress.kubernetes.io/auth-signin : <SignIn_URL> to specify the location of the error page. nginx.ingress.kubernetes.io/auth-signin-redirect-param : <SignIn_URL> to specify the URL parameter in the error page which should contain the original URL for a failed signin request. nginx.ingress.kubernetes.io/auth-response-headers : <Response_Header_1, ..., Response_Header_n> to specify headers to pass to backend once authentication request completes. nginx.ingress.kubernetes.io/auth-proxy-set-headers : <ConfigMap> the name of a ConfigMap that specifies headers to pass to the authentication service nginx.ingress.kubernetes.io/auth-request-redirect : <Request_Redirect_URL> to specify the X-Auth-Request-Redirect header value. nginx.ingress.kubernetes.io/auth-cache-key : <Cache_Key> this enables caching for auth requests. specify a lookup key for auth responses. e.g. $remote_user$http_authorization . Each server and location has it's own keyspace. Hence a cached response is only valid on a per-server and per-location basis. nginx.ingress.kubernetes.io/auth-cache-duration : <Cache_duration> to specify a caching time for auth responses based on their response codes, e.g. 200 202 30m . See proxy_cache_valid for details. You may specify multiple, comma-separated values: 200 202 10m, 401 5m . defaults to 200 202 401 5m . nginx.ingress.kubernetes.io/auth-always-set-cookie : <Boolean_Flag> to set a cookie returned by auth request. By default, the cookie will be set only if an upstream reports with the code 200, 201, 204, 206, 301, 302, 303, 304, 307, or 308. nginx.ingress.kubernetes.io/auth-snippet : <Auth_Snippet> to specify a custom snippet to use with external authentication, e.g. nginx.ingress.kubernetes.io/auth-url : http://foo.com/external-auth nginx.ingress.kubernetes.io/auth-snippet : | proxy_set_header Foo-Header 42; Note: nginx.ingress.kubernetes.io/auth-snippet is an optional annotation. However, it may only be used in conjunction with nginx.ingress.kubernetes.io/auth-url and will be ignored if nginx.ingress.kubernetes.io/auth-url is not set Example Please check the external-auth example.","title":"External Authentication"},{"location":"user-guide/nginx-configuration/annotations/#global-external-authentication","text":"By default the controller redirects all requests to an existing service that provides authentication if global-auth-url is set in the NGINX ConfigMap. If you want to disable this behavior for that ingress, you can use enable-global-auth: \"false\" in the NGINX ConfigMap. nginx.ingress.kubernetes.io/enable-global-auth : indicates if GlobalExternalAuth configuration should be applied or not to this Ingress rule. Default values is set to \"true\" . Note For more information please see global-auth-url .","title":"Global External Authentication"},{"location":"user-guide/nginx-configuration/annotations/#rate-limiting","text":"These annotations define limits on connections and transmission rates. These can be used to mitigate DDoS Attacks . nginx.ingress.kubernetes.io/limit-connections : number of concurrent connections allowed from a single IP address. A 503 error is returned when exceeding this limit. nginx.ingress.kubernetes.io/limit-rps : number of requests accepted from a given IP each second. The burst limit is set to this limit multiplied by the burst multiplier, the default multiplier is 5. When clients exceed this limit, limit-req-status-code default: 503 is returned. nginx.ingress.kubernetes.io/limit-rpm : number of requests accepted from a given IP each minute. The burst limit is set to this limit multiplied by the burst multiplier, the default multiplier is 5. When clients exceed this limit, limit-req-status-code default: 503 is returned. nginx.ingress.kubernetes.io/limit-burst-multiplier : multiplier of the limit rate for burst size. The default burst multiplier is 5, this annotation override the default multiplier. When clients exceed this limit, limit-req-status-code default: 503 is returned. nginx.ingress.kubernetes.io/limit-rate-after : initial number of kilobytes after which the further transmission of a response to a given connection will be rate limited. This feature must be used with proxy-buffering enabled. nginx.ingress.kubernetes.io/limit-rate : number of kilobytes per second allowed to send to a given connection. The zero value disables rate limiting. This feature must be used with proxy-buffering enabled. nginx.ingress.kubernetes.io/limit-whitelist : client IP source ranges to be excluded from rate-limiting. The value is a comma separated list of CIDRs. If you specify multiple annotations in a single Ingress rule, limits are applied in the order limit-connections , limit-rpm , limit-rps . To configure settings globally for all Ingress rules, the limit-rate-after and limit-rate values may be set in the NGINX ConfigMap . The value set in an Ingress annotation will override the global setting. The client IP address will be set based on the use of PROXY protocol or from the X-Forwarded-For header value when use-forwarded-headers is enabled.","title":"Rate Limiting"},{"location":"user-guide/nginx-configuration/annotations/#global-rate-limiting","text":"Note: Be careful when configuring both (Local) Rate Limiting and Global Rate Limiting at the same time. They are two completely different rate limiting implementations. Whichever limit exceeds first will reject the requests. It might be a good idea to configure both of them to ease load on Global Rate Limiting backend in cases of spike in traffic. The stock NGINX rate limiting does not share its counters among different NGINX instances. Given that most ingress-nginx deployments are elastic and number of replicas can change any day it is impossible to configure a proper rate limit using stock NGINX functionalities. Global Rate Limiting overcome this by using lua-resty-global-throttle . lua-resty-global-throttle shares its counters via a central store such as memcached . The obvious shortcoming of this is users have to deploy and operate a memcached instance in order to benefit from this functionality. Configure the memcached using these configmap settings . Here are a few remarks for ingress-nginx integration of lua-resty-global-throttle : We minimize memcached access by caching exceeding limit decisions. The expiry of cache entry is the desired delay lua-resty-global-throttle calculates for us. The Lua Shared Dictionary used for that is global_throttle_cache . Currently its size defaults to 10M. Customize it as per your needs using lua-shared-dicts . When we fail to cache the exceeding limit decision then we log an NGINX error. You can monitor for that error to decide if you need to bump the cache size. Without cache the cost of processing a request is two memcached commands: GET , and INCR . With the cache it is only INCR . Log NGINX variable $global_rate_limit_exceeding 's value to have some visibility into what portion of requests are rejected (value y ), whether they are rejected using cached decision (value c ), or if they are not rejected (default value n ). You can use log-format-upstream to include that in access logs. In case of an error it will log the error message and fail open . The annotations below creates Global Rate Limiting instance per ingress. That means if there are multiple paths configured under the same ingress, the Global Rate Limiting will count requests to all the paths under the same counter. Extract a path out into its own ingress if you need to isolate a certain path. nginx.ingress.kubernetes.io/global-rate-limit : Configures maximum allowed number of requests per window. Required. nginx.ingress.kubernetes.io/global-rate-limit-window : Configures a time window (i.e 1m ) that the limit is applied. Required. nginx.ingress.kubernetes.io/global-rate-limit-key : Configures a key for counting the samples. Defaults to $remote_addr . You can also combine multiple NGINX variables here, like ${remote_addr}-${http_x_api_client} which would mean the limit will be applied to requests coming from the same API client (indicated by X-API-Client HTTP request header) with the same source IP address. nginx.ingress.kubernetes.io/global-rate-limit-ignored-cidrs : comma separated list of IPs and CIDRs to match client IP against. When there's a match request is not considered for rate limiting.","title":"Global Rate Limiting"},{"location":"user-guide/nginx-configuration/annotations/#permanent-redirect","text":"This annotation allows to return a permanent redirect (Return Code 301) instead of sending data to the upstream. For example nginx.ingress.kubernetes.io/permanent-redirect: https://www.google.com would redirect everything to Google.","title":"Permanent Redirect"},{"location":"user-guide/nginx-configuration/annotations/#permanent-redirect-code","text":"This annotation allows you to modify the status code used for permanent redirects. For example nginx.ingress.kubernetes.io/permanent-redirect-code: '308' would return your permanent-redirect with a 308.","title":"Permanent Redirect Code"},{"location":"user-guide/nginx-configuration/annotations/#temporal-redirect","text":"This annotation allows you to return a temporal redirect (Return Code 302) instead of sending data to the upstream. For example nginx.ingress.kubernetes.io/temporal-redirect: https://www.google.com would redirect everything to Google with a Return Code of 302 (Moved Temporarily)","title":"Temporal Redirect"},{"location":"user-guide/nginx-configuration/annotations/#ssl-passthrough","text":"The annotation nginx.ingress.kubernetes.io/ssl-passthrough instructs the controller to send TLS connections directly to the backend instead of letting NGINX decrypt the communication. See also TLS/HTTPS in the User guide. Note SSL Passthrough is disabled by default and requires starting the controller with the --enable-ssl-passthrough flag. Attention Because SSL Passthrough works on layer 4 of the OSI model (TCP) and not on the layer 7 (HTTP), using SSL Passthrough invalidates all the other annotations set on an Ingress object.","title":"SSL Passthrough"},{"location":"user-guide/nginx-configuration/annotations/#service-upstream","text":"By default the NGINX ingress controller uses a list of all endpoints (Pod IP/port) in the NGINX upstream configuration. The nginx.ingress.kubernetes.io/service-upstream annotation disables that behavior and instead uses a single upstream in NGINX, the service's Cluster IP and port. This can be desirable for things like zero-downtime deployments . See issue #257 .","title":"Service Upstream"},{"location":"user-guide/nginx-configuration/annotations/#known-issues","text":"If the service-upstream annotation is specified the following things should be taken into consideration: Sticky Sessions will not work as only round-robin load balancing is supported. The proxy_next_upstream directive will not have any effect meaning on error the request will not be dispatched to another upstream.","title":"Known Issues"},{"location":"user-guide/nginx-configuration/annotations/#server-side-https-enforcement-through-redirect","text":"By default the controller redirects (308) to HTTPS if TLS is enabled for that ingress. If you want to disable this behavior globally, you can use ssl-redirect: \"false\" in the NGINX ConfigMap . To configure this feature for specific ingress resources, you can use the nginx.ingress.kubernetes.io/ssl-redirect: \"false\" annotation in the particular resource. When using SSL offloading outside of cluster (e.g. AWS ELB) it may be useful to enforce a redirect to HTTPS even when there is no TLS certificate available. This can be achieved by using the nginx.ingress.kubernetes.io/force-ssl-redirect: \"true\" annotation in the particular resource. To preserve the trailing slash in the URI with ssl-redirect , set nginx.ingress.kubernetes.io/preserve-trailing-slash: \"true\" annotation for that particular resource.","title":"Server-side HTTPS enforcement through redirect"},{"location":"user-guide/nginx-configuration/annotations/#redirect-fromto-www","text":"In some scenarios is required to redirect from www.domain.com to domain.com or vice versa. To enable this feature use the annotation nginx.ingress.kubernetes.io/from-to-www-redirect: \"true\" Attention If at some point a new Ingress is created with a host equal to one of the options (like domain.com ) the annotation will be omitted. Attention For HTTPS to HTTPS redirects is mandatory the SSL Certificate defined in the Secret, located in the TLS section of Ingress, contains both FQDN in the common name of the certificate.","title":"Redirect from/to www"},{"location":"user-guide/nginx-configuration/annotations/#whitelist-source-range","text":"You can specify allowed client IP source ranges through the nginx.ingress.kubernetes.io/whitelist-source-range annotation. The value is a comma separated list of CIDRs , e.g. 10.0.0.0/24,172.10.0.1 . To configure this setting globally for all Ingress rules, the whitelist-source-range value may be set in the NGINX ConfigMap . Note Adding an annotation to an Ingress rule overrides any global restriction.","title":"Whitelist source range"},{"location":"user-guide/nginx-configuration/annotations/#custom-timeouts","text":"Using the configuration configmap it is possible to set the default global timeout for connections to the upstream servers. In some scenarios is required to have different values. To allow this we provide annotations that allows this customization: nginx.ingress.kubernetes.io/proxy-connect-timeout nginx.ingress.kubernetes.io/proxy-send-timeout nginx.ingress.kubernetes.io/proxy-read-timeout nginx.ingress.kubernetes.io/proxy-next-upstream nginx.ingress.kubernetes.io/proxy-next-upstream-timeout nginx.ingress.kubernetes.io/proxy-next-upstream-tries nginx.ingress.kubernetes.io/proxy-request-buffering Note: All timeout values are unitless and in seconds e.g. nginx.ingress.kubernetes.io/proxy-read-timeout: \"120\" sets a valid 120 seconds proxy read timeout.","title":"Custom timeouts"},{"location":"user-guide/nginx-configuration/annotations/#proxy-redirect","text":"The annotations nginx.ingress.kubernetes.io/proxy-redirect-from and nginx.ingress.kubernetes.io/proxy-redirect-to will set the first and second parameters of NGINX's proxy_redirect directive respectively. It is possible to set the text that should be changed in the Location and Refresh header fields of a proxied server response Setting \"off\" or \"default\" in the annotation nginx.ingress.kubernetes.io/proxy-redirect-from disables nginx.ingress.kubernetes.io/proxy-redirect-to , otherwise, both annotations must be used in unison. Note that each annotation must be a string without spaces. By default the value of each annotation is \"off\".","title":"Proxy redirect"},{"location":"user-guide/nginx-configuration/annotations/#custom-max-body-size","text":"For NGINX, an 413 error will be returned to the client when the size in a request exceeds the maximum allowed size of the client request body. This size can be configured by the parameter client_max_body_size . To configure this setting globally for all Ingress rules, the proxy-body-size value may be set in the NGINX ConfigMap . To use custom values in an Ingress rule define these annotation: nginx.ingress.kubernetes.io/proxy-body-size : 8m","title":"Custom max body size"},{"location":"user-guide/nginx-configuration/annotations/#proxy-cookie-domain","text":"Sets a text that should be changed in the domain attribute of the \"Set-Cookie\" header fields of a proxied server response. To configure this setting globally for all Ingress rules, the proxy-cookie-domain value may be set in the NGINX ConfigMap .","title":"Proxy cookie domain"},{"location":"user-guide/nginx-configuration/annotations/#proxy-cookie-path","text":"Sets a text that should be changed in the path attribute of the \"Set-Cookie\" header fields of a proxied server response. To configure this setting globally for all Ingress rules, the proxy-cookie-path value may be set in the NGINX ConfigMap .","title":"Proxy cookie path"},{"location":"user-guide/nginx-configuration/annotations/#proxy-buffering","text":"Enable or disable proxy buffering proxy_buffering . By default proxy buffering is disabled in the NGINX config. To configure this setting globally for all Ingress rules, the proxy-buffering value may be set in the NGINX ConfigMap . To use custom values in an Ingress rule define these annotation: nginx.ingress.kubernetes.io/proxy-buffering : \"on\"","title":"Proxy buffering"},{"location":"user-guide/nginx-configuration/annotations/#proxy-buffers-number","text":"Sets the number of the buffers in proxy_buffers used for reading the first part of the response received from the proxied server. By default proxy buffers number is set as 4 To configure this setting globally, set proxy-buffers-number in NGINX ConfigMap . To use custom values in an Ingress rule, define this annotation: nginx.ingress.kubernetes.io/proxy-buffers-number : \"4\"","title":"Proxy buffers Number"},{"location":"user-guide/nginx-configuration/annotations/#proxy-buffer-size","text":"Sets the size of the buffer proxy_buffer_size used for reading the first part of the response received from the proxied server. By default proxy buffer size is set as \"4k\" To configure this setting globally, set proxy-buffer-size in NGINX ConfigMap . To use custom values in an Ingress rule, define this annotation: nginx.ingress.kubernetes.io/proxy-buffer-size : \"8k\"","title":"Proxy buffer size"},{"location":"user-guide/nginx-configuration/annotations/#proxy-max-temp-file-size","text":"When buffering of responses from the proxied server is enabled, and the whole response does not fit into the buffers set by the proxy_buffer_size and proxy_buffers directives, a part of the response can be saved to a temporary file. This directive sets the maximum size of the temporary file setting the proxy_max_temp_file_size . The size of data written to the temporary file at a time is set by the proxy_temp_file_write_size directive. The zero value disables buffering of responses to temporary files. To use custom values in an Ingress rule, define this annotation: nginx.ingress.kubernetes.io/proxy-max-temp-file-size : \"1024m\"","title":"Proxy max temp file size"},{"location":"user-guide/nginx-configuration/annotations/#proxy-http-version","text":"Using this annotation sets the proxy_http_version that the Nginx reverse proxy will use to communicate with the backend. By default this is set to \"1.1\". nginx.ingress.kubernetes.io/proxy-http-version : \"1.0\"","title":"Proxy HTTP version"},{"location":"user-guide/nginx-configuration/annotations/#ssl-ciphers","text":"Specifies the enabled ciphers . Using this annotation will set the ssl_ciphers directive at the server level. This configuration is active for all the paths in the host. nginx.ingress.kubernetes.io/ssl-ciphers : \"ALL:!aNULL:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP\" The following annotation will set the ssl_prefer_server_ciphers directive at the server level. This configuration specifies that server ciphers should be preferred over client ciphers when using the SSLv3 and TLS protocols. nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers : \"true\"","title":"SSL ciphers"},{"location":"user-guide/nginx-configuration/annotations/#connection-proxy-header","text":"Using this annotation will override the default connection header set by NGINX. To use custom values in an Ingress rule, define the annotation: nginx.ingress.kubernetes.io/connection-proxy-header : \"keep-alive\"","title":"Connection proxy header"},{"location":"user-guide/nginx-configuration/annotations/#enable-access-log","text":"Access logs are enabled by default, but in some scenarios access logs might be required to be disabled for a given ingress. To do this, use the annotation: nginx.ingress.kubernetes.io/enable-access-log : \"false\"","title":"Enable Access Log"},{"location":"user-guide/nginx-configuration/annotations/#enable-rewrite-log","text":"Rewrite logs are not enabled by default. In some scenarios it could be required to enable NGINX rewrite logs. Note that rewrite logs are sent to the error_log file at the notice level. To enable this feature use the annotation: nginx.ingress.kubernetes.io/enable-rewrite-log : \"true\"","title":"Enable Rewrite Log"},{"location":"user-guide/nginx-configuration/annotations/#enable-opentracing","text":"Opentracing can be enabled or disabled globally through the ConfigMap but this will sometimes need to be overridden to enable it or disable it for a specific ingress (e.g. to turn off tracing of external health check endpoints) nginx.ingress.kubernetes.io/enable-opentracing : \"true\"","title":"Enable Opentracing"},{"location":"user-guide/nginx-configuration/annotations/#opentracing-trust-incoming-span","text":"The option to trust incoming trace spans can be enabled or disabled globally through the ConfigMap but this will sometimes need to be overridden to enable it or disable it for a specific ingress (e.g. only enable on a private endpoint) nginx.ingress.kubernetes.io/opentracing-trust-incoming-span : \"true\"","title":"Opentracing Trust Incoming Span"},{"location":"user-guide/nginx-configuration/annotations/#x-forwarded-prefix-header","text":"To add the non-standard X-Forwarded-Prefix header to the upstream request with a string value, the following annotation can be used: nginx.ingress.kubernetes.io/x-forwarded-prefix : \"/path\"","title":"X-Forwarded-Prefix Header"},{"location":"user-guide/nginx-configuration/annotations/#modsecurity","text":"ModSecurity is an OpenSource Web Application firewall. It can be enabled for a particular set of ingress locations. The ModSecurity module must first be enabled by enabling ModSecurity in the ConfigMap . Note this will enable ModSecurity for all paths, and each path must be disabled manually. It can be enabled using the following annotation: nginx.ingress.kubernetes.io/enable-modsecurity : \"true\" ModSecurity will run in \"Detection-Only\" mode using the recommended configuration . You can enable the OWASP Core Rule Set by setting the following annotation: nginx.ingress.kubernetes.io/enable-owasp-core-rules : \"true\" You can pass transactionIDs from nginx by setting up the following: nginx.ingress.kubernetes.io/modsecurity-transaction-id : \"$request_id\" You can also add your own set of modsecurity rules via a snippet: nginx.ingress.kubernetes.io/modsecurity-snippet : | SecRuleEngine On SecDebugLog /tmp/modsec_debug.log Note: If you use both enable-owasp-core-rules and modsecurity-snippet annotations together, only the modsecurity-snippet will take effect. If you wish to include the OWASP Core Rule Set or recommended configuration simply use the include statement: nginx 0.24.1 and below nginx.ingress.kubernetes.io/modsecurity-snippet : | Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf Include /etc/nginx/modsecurity/modsecurity.conf nginx 0.25.0 and above nginx.ingress.kubernetes.io/modsecurity-snippet : | Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf","title":"ModSecurity"},{"location":"user-guide/nginx-configuration/annotations/#influxdb","text":"Using influxdb-* annotations we can monitor requests passing through a Location by sending them to an InfluxDB backend exposing the UDP socket using the nginx-influxdb-module . nginx.ingress.kubernetes.io/enable-influxdb : \"true\" nginx.ingress.kubernetes.io/influxdb-measurement : \"nginx-reqs\" nginx.ingress.kubernetes.io/influxdb-port : \"8089\" nginx.ingress.kubernetes.io/influxdb-host : \"127.0.0.1\" nginx.ingress.kubernetes.io/influxdb-server-name : \"nginx-ingress\" For the influxdb-host parameter you have two options: Use an InfluxDB server configured with the UDP protocol enabled. Deploy Telegraf as a sidecar proxy to the Ingress controller configured to listen UDP with the socket listener input and to write using anyone of the outputs plugins like InfluxDB, Apache Kafka, Prometheus, etc.. (recommended) It's important to remember that there's no DNS resolver at this stage so you will have to configure an ip address to nginx.ingress.kubernetes.io/influxdb-host . If you deploy Influx or Telegraf as sidecar (another container in the same pod) this becomes straightforward since you can directly use 127.0.0.1 .","title":"InfluxDB"},{"location":"user-guide/nginx-configuration/annotations/#backend-protocol","text":"Using backend-protocol annotations is possible to indicate how NGINX should communicate with the backend service. (Replaces secure-backends in older versions) Valid Values: HTTP, HTTPS, GRPC, GRPCS, AJP and FCGI By default NGINX uses HTTP . Example: nginx.ingress.kubernetes.io/backend-protocol : \"HTTPS\"","title":"Backend Protocol"},{"location":"user-guide/nginx-configuration/annotations/#use-regex","text":"Attention When using this annotation with the NGINX annotation nginx.ingress.kubernetes.io/affinity of type cookie , nginx.ingress.kubernetes.io/session-cookie-path must be also set; Session cookie paths do not support regex. Using the nginx.ingress.kubernetes.io/use-regex annotation will indicate whether or not the paths defined on an Ingress use regular expressions. The default value is false . The following will indicate that regular expression paths are being used: nginx.ingress.kubernetes.io/use-regex : \"true\" The following will indicate that regular expression paths are not being used: nginx.ingress.kubernetes.io/use-regex : \"false\" When this annotation is set to true , the case insensitive regular expression location modifier will be enforced on ALL paths for a given host regardless of what Ingress they are defined on. Additionally, if the rewrite-target annotation is used on any Ingress for a given host, then the case insensitive regular expression location modifier will be enforced on ALL paths for a given host regardless of what Ingress they are defined on. Please read about ingress path matching before using this modifier.","title":"Use Regex"},{"location":"user-guide/nginx-configuration/annotations/#satisfy","text":"By default, a request would need to satisfy all authentication requirements in order to be allowed. By using this annotation, requests that satisfy either any or all authentication requirements are allowed, based on the configuration value. nginx.ingress.kubernetes.io/satisfy : \"any\"","title":"Satisfy"},{"location":"user-guide/nginx-configuration/annotations/#mirror","text":"Enables a request to be mirrored to a mirror backend. Responses by mirror backends are ignored. This feature is useful, to see how requests will react in \"test\" backends. The mirror backend can be set by applying: nginx.ingress.kubernetes.io/mirror-target : https://test.env.com/$request_uri By default the request-body is sent to the mirror backend, but can be turned off by applying: nginx.ingress.kubernetes.io/mirror-request-body : \"off\" Also by default header Host for mirrored requests will be set the same as a host part of uri in the \"mirror-target\" annotation. You can override it by \"mirror-host\" annotation: nginx.ingress.kubernetes.io/mirror-target : https://1.2.3.4/$request_uri nginx.ingress.kubernetes.io/mirror-host : \"test.env.com\" Note: The mirror directive will be applied to all paths within the ingress resource. The request sent to the mirror is linked to the original request. If you have a slow mirror backend, then the original request will throttle. For more information on the mirror module see ngx_http_mirror_module","title":"Mirror"},{"location":"user-guide/nginx-configuration/annotations/#stream-snippet","text":"Using the annotation nginx.ingress.kubernetes.io/stream-snippet it is possible to add custom stream configuration. apiVersion : networking.k8s.io/v1 kind : Ingress metadata : annotations : nginx.ingress.kubernetes.io/stream-snippet : | server { listen 8000; proxy_pass 127.0.0.1:80; }","title":"Stream snippet"},{"location":"user-guide/nginx-configuration/configmap/","text":"ConfigMaps \u00b6 ConfigMaps allow you to decouple configuration artifacts from image content to keep containerized applications portable. The ConfigMap API resource stores configuration data as key-value pairs. The data provides the configurations for system components for the nginx-controller. In order to overwrite nginx-controller configuration values as seen in config.go , you can add key-value pairs to the data section of the config-map. For Example: data : map-hash-bucket-size : \"128\" ssl-protocols : SSLv2 Important The key and values in a ConfigMap can only be strings. This means that we want a value with boolean values we need to quote the values, like \"true\" or \"false\". Same for numbers, like \"100\". \"Slice\" types (defined below as []string or []int ) can be provided as a comma-delimited string. Configuration options \u00b6 The following table shows a configuration option's name, type, and the default value: name type default add-headers string \"\" allow-backend-server-header bool \"false\" allow-snippet-annotations bool true annotation-value-word-blocklist string array \"\" hide-headers string array empty access-log-params string \"\" access-log-path string \"/var/log/nginx/access.log\" http-access-log-path string \"\" stream-access-log-path string \"\" enable-access-log-for-default-backend bool \"false\" error-log-path string \"/var/log/nginx/error.log\" enable-modsecurity bool \"false\" modsecurity-snippet string \"\" enable-owasp-modsecurity-crs bool \"false\" client-header-buffer-size string \"1k\" client-header-timeout int 60 client-body-buffer-size string \"8k\" client-body-timeout int 60 disable-access-log bool false disable-ipv6 bool false disable-ipv6-dns bool false enable-underscores-in-headers bool false enable-ocsp bool false ignore-invalid-headers bool true retry-non-idempotent bool \"false\" error-log-level string \"notice\" http2-max-field-size string \"4k\" http2-max-header-size string \"16k\" http2-max-requests int 1000 http2-max-concurrent-streams int 128 hsts bool \"true\" hsts-include-subdomains bool \"true\" hsts-max-age string \"15724800\" hsts-preload bool \"false\" keep-alive int 75 keep-alive-requests int 100 large-client-header-buffers string \"4 8k\" log-format-escape-none bool \"false\" log-format-escape-json bool \"false\" log-format-upstream string $remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id log-format-stream string [$remote_addr] [$time_local] $protocol $status $bytes_sent $bytes_received $session_time enable-multi-accept bool \"true\" max-worker-connections int 16384 max-worker-open-files int 0 map-hash-bucket-size int 64 nginx-status-ipv4-whitelist []string \"127.0.0.1\" nginx-status-ipv6-whitelist []string \"::1\" proxy-real-ip-cidr []string \"0.0.0.0/0\" proxy-set-headers string \"\" server-name-hash-max-size int 1024 server-name-hash-bucket-size int <size of the processor\u2019s cache line> proxy-headers-hash-max-size int 512 proxy-headers-hash-bucket-size int 64 plugins []string reuse-port bool \"true\" server-tokens bool \"false\" ssl-ciphers string \"ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384\" ssl-ecdh-curve string \"auto\" ssl-dh-param string \"\" ssl-protocols string \"TLSv1.2 TLSv1.3\" ssl-session-cache bool \"true\" ssl-session-cache-size string \"10m\" ssl-session-tickets bool \"false\" ssl-session-ticket-key string <Randomly Generated> ssl-session-timeout string \"10m\" ssl-buffer-size string \"4k\" use-proxy-protocol bool \"false\" proxy-protocol-header-timeout string \"5s\" use-gzip bool \"false\" use-geoip bool \"true\" use-geoip2 bool \"false\" enable-brotli bool \"false\" brotli-level int 4 brotli-min-length int 20 brotli-types string \"application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/javascript text/plain text/x-component\" use-http2 bool \"true\" gzip-level int 1 gzip-types string \"application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/javascript text/plain text/x-component\" worker-processes string <Number of CPUs> worker-cpu-affinity string \"\" worker-shutdown-timeout string \"240s\" load-balance string \"round_robin\" variables-hash-bucket-size int 128 variables-hash-max-size int 2048 upstream-keepalive-connections int 320 upstream-keepalive-time string \"1h\" upstream-keepalive-timeout int 60 upstream-keepalive-requests int 10000 limit-conn-zone-variable string \"$binary_remote_addr\" proxy-stream-timeout string \"600s\" proxy-stream-next-upstream bool \"true\" proxy-stream-next-upstream-timeout string \"600s\" proxy-stream-next-upstream-tries int 3 proxy-stream-responses int 1 bind-address []string \"\" use-forwarded-headers bool \"false\" enable-real-ip bool \"false\" forwarded-for-header string \"X-Forwarded-For\" compute-full-forwarded-for bool \"false\" proxy-add-original-uri-header bool \"false\" generate-request-id bool \"true\" enable-opentracing bool \"false\" opentracing-operation-name string \"\" opentracing-location-operation-name string \"\" zipkin-collector-host string \"\" zipkin-collector-port int 9411 zipkin-service-name string \"nginx\" zipkin-sample-rate float 1.0 jaeger-collector-host string \"\" jaeger-collector-port int 6831 jaeger-endpoint string \"\" jaeger-service-name string \"nginx\" jaeger-propagation-format string \"jaeger\" jaeger-sampler-type string \"const\" jaeger-sampler-param string \"1\" jaeger-sampler-host string \"http://127.0.0.1\" jaeger-sampler-port int 5778 jaeger-trace-context-header-name string uber-trace-id jaeger-debug-header string uber-debug-id jaeger-baggage-header string jaeger-baggage jaeger-trace-baggage-header-prefix string uberctx- datadog-collector-host string \"\" datadog-collector-port int 8126 datadog-service-name string \"nginx\" datadog-environment string \"prod\" datadog-operation-name-override string \"nginx.handle\" datadog-priority-sampling bool \"true\" datadog-sample-rate float 1.0 main-snippet string \"\" http-snippet string \"\" server-snippet string \"\" stream-snippet string \"\" location-snippet string \"\" custom-http-errors []int []int{} proxy-body-size string \"1m\" proxy-connect-timeout int 5 proxy-read-timeout int 60 proxy-send-timeout int 60 proxy-buffers-number int 4 proxy-buffer-size string \"4k\" proxy-cookie-path string \"off\" proxy-cookie-domain string \"off\" proxy-next-upstream string \"error timeout\" proxy-next-upstream-timeout int 0 proxy-next-upstream-tries int 3 proxy-redirect-from string \"off\" proxy-request-buffering string \"on\" ssl-redirect bool \"true\" force-ssl-redirect bool \"false\" whitelist-source-range []string []string{} skip-access-log-urls []string []string{} limit-rate int 0 limit-rate-after int 0 lua-shared-dicts string \"\" http-redirect-code int 308 proxy-buffering string \"off\" limit-req-status-code int 503 limit-conn-status-code int 503 enable-syslog bool false syslog-host string \"\" syslog-port int 514 no-tls-redirect-locations string \"/.well-known/acme-challenge\" global-auth-url string \"\" global-auth-method string \"\" global-auth-signin string \"\" global-auth-signin-redirect-param string \"rd\" global-auth-response-headers string \"\" global-auth-request-redirect string \"\" global-auth-snippet string \"\" global-auth-cache-key string \"\" global-auth-cache-duration string \"200 202 401 5m\" no-auth-locations string \"/.well-known/acme-challenge\" block-cidrs []string \"\" block-user-agents []string \"\" block-referers []string \"\" proxy-ssl-location-only bool \"false\" default-type string \"text/html\" global-rate-limit-memcached-host string \"\" global-rate-limit-memcached-port int 11211 global-rate-limit-memcached-connect-timeout int 50 global-rate-limit-memcached-max-idle-timeout int 10000 global-rate-limit-memcached-pool-size int 50 global-rate-limit-status-code int 429 service-upstream bool \"false\" ssl-reject-handshake bool \"false\" debug-connections []string \"127.0.0.1,1.1.1.1/24\" add-headers \u00b6 Sets custom headers from named configmap before sending traffic to the client. See proxy-set-headers . example allow-backend-server-header \u00b6 Enables the return of the header Server from the backend instead of the generic nginx string. default: is disabled allow-snippet-annotations \u00b6 Enables Ingress to parse and add -snippet annotations/directives created by the user. _**default:* _ true Warning: We recommend enabling this option only if you TRUST users with permission to create Ingress objects, as this may allow a user to add restricted configurations to the final nginx.conf file annotation-value-word-blocklist \u00b6 Contains a comma-separated value of chars/words that are well known of being used to abuse Ingress configuration and must be blocked. Related to CVE-2021-25742 When an annotation is detected with a value that matches one of the blocked bad words, the whole Ingress won't be configured. default: \"\" When doing this, the default blocklist is override, which means that the Ingress admin should add all the words that should be blocked, here is a suggested block list. suggested: \"load_module,lua_package,_by_lua,location,root,proxy_pass,serviceaccount,{,},',\\\"\" hide-headers \u00b6 Sets additional header that will not be passed from the upstream server to the client response. default: empty References: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header access-log-params \u00b6 Additional params for access_log. For example, buffer=16k, gzip, flush=1m References: https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log access-log-path \u00b6 Access log path for both http and stream context. Goes to /var/log/nginx/access.log by default. Note: the file /var/log/nginx/access.log is a symlink to /dev/stdout http-access-log-path \u00b6 Access log path for http context globally. default: \"\" Note: If not specified, the access-log-path will be used. stream-access-log-path \u00b6 Access log path for stream context globally. default: \"\" Note: If not specified, the access-log-path will be used. enable-access-log-for-default-backend \u00b6 Enables logging access to default backend. default: is disabled. error-log-path \u00b6 Error log path. Goes to /var/log/nginx/error.log by default. Note: the file /var/log/nginx/error.log is a symlink to /dev/stderr References: https://nginx.org/en/docs/ngx_core_module.html#error_log enable-modsecurity \u00b6 Enables the modsecurity module for NGINX. default: is disabled enable-owasp-modsecurity-crs \u00b6 Enables the OWASP ModSecurity Core Rule Set (CRS). default: is disabled modsecurity-snippet \u00b6 Adds custom rules to modsecurity section of nginx configuration client-header-buffer-size \u00b6 Allows to configure a custom buffer size for reading client request header. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_buffer_size client-header-timeout \u00b6 Defines a timeout for reading client request header, in seconds. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_timeout client-body-buffer-size \u00b6 Sets buffer size for reading client request body. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size client-body-timeout \u00b6 Defines a timeout for reading client request body, in seconds. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_timeout disable-access-log \u00b6 Disables the Access Log from the entire Ingress Controller. default: false References: https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log disable-ipv6 \u00b6 Disable listening on IPV6. default: false ; IPv6 listening is enabled disable-ipv6-dns \u00b6 Disable IPV6 for nginx DNS resolver. default: false ; IPv6 resolving enabled. enable-underscores-in-headers \u00b6 Enables underscores in header names. default: is disabled enable-ocsp \u00b6 Enables Online Certificate Status Protocol stapling (OCSP) support. default: is disabled ignore-invalid-headers \u00b6 Set if header fields with invalid names should be ignored. default: is enabled retry-non-idempotent \u00b6 Since 1.9.13 NGINX will not retry non-idempotent requests (POST, LOCK, PATCH) in case of an error in the upstream server. The previous behavior can be restored using the value \"true\". error-log-level \u00b6 Configures the logging level of errors. Log levels above are listed in the order of increasing severity. References: https://nginx.org/en/docs/ngx_core_module.html#error_log http2-max-field-size \u00b6 Warning This feature was deprecated in 1.1.3 and will be removed in 1.3.0. Use large-client-header-buffers instead. Limits the maximum size of an HPACK-compressed request header field. References: https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_field_size http2-max-header-size \u00b6 Warning This feature was deprecated in 1.1.3 and will be removed in 1.3.0. Use large-client-header-buffers instead. Limits the maximum size of the entire request header list after HPACK decompression. References: https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_header_size http2-max-requests \u00b6 Warning This feature was deprecated in 1.1.3 and will be removed in 1.3.0. Use upstream-keepalive-requests instead. Sets the maximum number of requests (including push requests) that can be served through one HTTP/2 connection, after which the next client request will lead to connection closing and the need of establishing a new connection. References: https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_requests http2-max-concurrent-streams \u00b6 Sets the maximum number of concurrent HTTP/2 streams in a connection. References: https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_concurrent_streams hsts \u00b6 Enables or disables the header HSTS in servers running SSL. HTTP Strict Transport Security (often abbreviated as HSTS) is a security feature (HTTP header) that tell browsers that it should only be communicated with using HTTPS, instead of using HTTP. It provides protection against protocol downgrade attacks and cookie theft. References: https://developer.mozilla.org/en-US/docs/Web/Security/HTTP_strict_transport_security https://blog.qualys.com/securitylabs/2016/03/28/the-importance-of-a-proper-http-strict-transport-security-implementation-on-your-web-server hsts-include-subdomains \u00b6 Enables or disables the use of HSTS in all the subdomains of the server-name. hsts-max-age \u00b6 Sets the time, in seconds, that the browser should remember that this site is only to be accessed using HTTPS. hsts-preload \u00b6 Enables or disables the preload attribute in the HSTS feature (when it is enabled). keep-alive \u00b6 Sets the time, in seconds, during which a keep-alive client connection will stay open on the server side. The zero value disables keep-alive client connections. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout Important Setting keep-alive: '0' will most likely break concurrent http/2 requests due to changes introduced with nginx 1.19.7 Changes with nginx 1.19.7 16 Feb 2021 *) Change: connections handling in HTTP/2 has been changed to better match HTTP/1.x; the \"http2_recv_timeout\", \"http2_idle_timeout\", and \"http2_max_requests\" directives have been removed, the \"keepalive_timeout\" and \"keepalive_requests\" directives should be used instead. References: nginx change log nginx issue tracker nginx mailing list keep-alive-requests \u00b6 Sets the maximum number of requests that can be served through one keep-alive connection. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests large-client-header-buffers \u00b6 Sets the maximum number and size of buffers used for reading large client request header. default: 4 8k References: https://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers log-format-escape-none \u00b6 Sets if the escape parameter is disabled entirely for character escaping in variables (\"true\") or controlled by log-format-escape-json (\"false\") Sets the nginx log format . log-format-escape-json \u00b6 Sets if the escape parameter allows JSON (\"true\") or default characters escaping in variables (\"false\") Sets the nginx log format . log-format-upstream \u00b6 Sets the nginx log format . Example for json output: log - f orma t - ups trea m : ' { \"time\" : \"$time_iso8601\" , \"remote_addr\" : \"$proxy_protocol_addr\" , \"x_forwarded_for\" : \"$proxy_add_x_forwarded_for\" , \"request_id\" : \"$req_id\" , \"remote_user\" : \"$remote_user\" , \"bytes_sent\" : $by tes _se nt , \"request_time\" : $reques t _ t ime , \"status\" : $s tatus , \"vhost\" : \"$host\" , \"request_proto\" : \"$server_protocol\" , \"path\" : \"$uri\" , \"request_query\" : \"$args\" , \"request_length\" : $reques t _le n g t h , \"duration\" : $reques t _ t ime , \"method\" : \"$request_method\" , \"http_referrer\" : \"$http_referer\" , \"http_user_agent\" : \"$http_user_agent\" } ' Please check the log-format for definition of each field. log-format-stream \u00b6 Sets the nginx stream format . enable-multi-accept \u00b6 If disabled, a worker process will accept one new connection at a time. Otherwise, a worker process will accept all new connections at a time. default: true References: https://nginx.org/en/docs/ngx_core_module.html#multi_accept max-worker-connections \u00b6 Sets the maximum number of simultaneous connections that can be opened by each worker process. 0 will use the value of max-worker-open-files . default: 16384 Tip Using 0 in scenarios of high load improves performance at the cost of increasing RAM utilization (even on idle). max-worker-open-files \u00b6 Sets the maximum number of files that can be opened by each worker process. The default of 0 means \"max open files (system's limit) - 1024\". default: 0 map-hash-bucket-size \u00b6 Sets the bucket size for the map variables hash tables . The details of setting up hash tables are provided in a separate document . proxy-real-ip-cidr \u00b6 If use-forwarded-headers or use-proxy-protocol is enabled, proxy-real-ip-cidr defines the default IP/network address of your external load balancer. Can be a comma-separated list of CIDR blocks. default: \"0.0.0.0/0\" proxy-set-headers \u00b6 Sets custom headers from named configmap before sending traffic to backends. The value format is namespace/name. See example server-name-hash-max-size \u00b6 Sets the maximum size of the server names hash tables used in server names,map directive\u2019s values, MIME types, names of request header strings, etc. References: https://nginx.org/en/docs/hash.html server-name-hash-bucket-size \u00b6 Sets the size of the bucket for the server names hash tables. References: https://nginx.org/en/docs/hash.html https://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_bucket_size proxy-headers-hash-max-size \u00b6 Sets the maximum size of the proxy headers hash tables. References: https://nginx.org/en/docs/hash.html https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_headers_hash_max_size reuse-port \u00b6 Instructs NGINX to create an individual listening socket for each worker process (using the SO_REUSEPORT socket option), allowing a kernel to distribute incoming connections between worker processes default: true proxy-headers-hash-bucket-size \u00b6 Sets the size of the bucket for the proxy headers hash tables. References: https://nginx.org/en/docs/hash.html https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_headers_hash_bucket_size plugins \u00b6 Activates plugins installed in /etc/nginx/lua/plugins . Refer to ingress-nginx plugins README for more information on how to write and install a plugin. server-tokens \u00b6 Send NGINX Server header in responses and display NGINX version in error pages. default: is disabled ssl-ciphers \u00b6 Sets the ciphers list to enable. The ciphers are specified in the format understood by the OpenSSL library. The default cipher list is: ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 . The ordering of a ciphersuite is very important because it decides which algorithms are going to be selected in priority. The recommendation above prioritizes algorithms that provide perfect forward secrecy . DHE-based cyphers will not be available until DH parameter is configured Custom DH parameters for perfect forward secrecy Please check the Mozilla SSL Configuration Generator . Note: ssl_prefer_server_ciphers directive will be enabled by default for http context. ssl-ecdh-curve \u00b6 Specifies a curve for ECDHE ciphers. References: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ecdh_curve ssl-dh-param \u00b6 Sets the name of the secret that contains Diffie-Hellman key to help with \"Perfect Forward Secrecy\". References: https://wiki.openssl.org/index.php/Diffie-Hellman_parameters https://wiki.mozilla.org/Security/Server_Side_TLS#DHE_handshake_and_dhparam https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_dhparam ssl-protocols \u00b6 Sets the SSL protocols to use. The default is: TLSv1.2 TLSv1.3 . Please check the result of the configuration using https://ssllabs.com/ssltest/analyze.html or https://testssl.sh . ssl-early-data \u00b6 Enables or disables TLS 1.3 early data , also known as Zero Round Trip Time Resumption (0-RTT). This requires ssl-protocols to have TLSv1.3 enabled. Enable this with caution, because requests sent within early data are subject to replay attacks . ssl_early_data . The default is: false . ssl-session-cache \u00b6 Enables or disables the use of shared SSL cache among worker processes. ssl-session-cache-size \u00b6 Sets the size of the SSL shared session cache between all worker processes. ssl-session-tickets \u00b6 Enables or disables session resumption through TLS session tickets . ssl-session-ticket-key \u00b6 Sets the secret key used to encrypt and decrypt TLS session tickets. The value must be a valid base64 string. To create a ticket: openssl rand 80 | openssl enc -A -base64 TLS session ticket-key , by default, a randomly generated key is used. ssl-session-timeout \u00b6 Sets the time during which a client may reuse the session parameters stored in a cache. ssl-buffer-size \u00b6 Sets the size of the SSL buffer used for sending data. The default of 4k helps NGINX to improve TLS Time To First Byte (TTTFB). References: https://www.igvita.com/2013/12/16/optimizing-nginx-tls-time-to-first-byte/ use-proxy-protocol \u00b6 Enables or disables the PROXY protocol to receive client connection (real IP address) information passed through proxy servers and load balancers such as HAProxy and Amazon Elastic Load Balancer (ELB). proxy-protocol-header-timeout \u00b6 Sets the timeout value for receiving the proxy-protocol headers. The default of 5 seconds prevents the TLS passthrough handler from waiting indefinitely on a dropped connection. default: 5s use-gzip \u00b6 Enables or disables compression of HTTP responses using the \"gzip\" module . MIME types to compress are controlled by gzip-types . default: false use-geoip \u00b6 Enables or disables \"geoip\" module that creates variables with values depending on the client IP address, using the precompiled MaxMind databases. default: true Note: MaxMind legacy databases are discontinued and will not receive updates after 2019-01-02, cf. discontinuation notice . Consider use-geoip2 below. use-geoip2 \u00b6 Enables the geoip2 module for NGINX. Since 0.27.0 and due to a change in the MaxMind databases a license is required to have access to the databases. For this reason, it is required to define a new flag --maxmind-license-key in the ingress controller deployment to download the databases needed during the initialization of the ingress controller. Alternatively, it is possible to use a volume to mount the files /etc/nginx/geoip/GeoLite2-City.mmdb and /etc/nginx/geoip/GeoLite2-ASN.mmdb , avoiding the overhead of the download. Important If the feature is enabled but the files are missing, GeoIP2 will not be enabled. default: false enable-brotli \u00b6 Enables or disables compression of HTTP responses using the \"brotli\" module . The default mime type list to compress is: application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component . default: is disabled Note: Brotli does not works in Safari < 11. For more information see https://caniuse.com/#feat=brotli brotli-level \u00b6 Sets the Brotli Compression Level that will be used. default: 4 brotli-min-length \u00b6 Minimum length of responses, in bytes, that will be eligible for brotli compression. default: 20 brotli-types \u00b6 Sets the MIME Types that will be compressed on-the-fly by brotli. default: application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component use-http2 \u00b6 Enables or disables HTTP/2 support in secure connections. gzip-level \u00b6 Sets the gzip Compression Level that will be used. default: 1 gzip-min-length \u00b6 Minimum length of responses to be returned to the client before it is eligible for gzip compression, in bytes. default: 256 gzip-types \u00b6 Sets the MIME types in addition to \"text/html\" to compress. The special value \"*\" matches any MIME type. Responses with the \"text/html\" type are always compressed if use-gzip is enabled. default: application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component . worker-processes \u00b6 Sets the number of worker processes . The default of \"auto\" means number of available CPU cores. worker-cpu-affinity \u00b6 Binds worker processes to the sets of CPUs. worker_cpu_affinity . By default worker processes are not bound to any specific CPUs. The value can be: \"\": empty string indicate no affinity is applied. cpumask: e.g. 0001 0010 0100 1000 to bind processes to specific cpus. auto: binding worker processes automatically to available CPUs. worker-shutdown-timeout \u00b6 Sets a timeout for Nginx to wait for worker to gracefully shutdown . default: \"240s\" load-balance \u00b6 Sets the algorithm to use for load balancing. The value can either be: round_robin: to use the default round robin loadbalancer ewma: to use the Peak EWMA method for routing ( implementation ) The default is round_robin . To load balance using consistent hashing of IP or other variables, consider the nginx.ingress.kubernetes.io/upstream-hash-by annotation. To load balance using session cookies, consider the nginx.ingress.kubernetes.io/affinity annotation. References: https://nginx.org/en/docs/http/load_balancing.html variables-hash-bucket-size \u00b6 Sets the bucket size for the variables hash table. References: https://nginx.org/en/docs/http/ngx_http_map_module.html#variables_hash_bucket_size variables-hash-max-size \u00b6 Sets the maximum size of the variables hash table. References: https://nginx.org/en/docs/http/ngx_http_map_module.html#variables_hash_max_size upstream-keepalive-connections \u00b6 Activates the cache for connections to upstream servers. The connections parameter sets the maximum number of idle keepalive connections to upstream servers that are preserved in the cache of each worker process. When this number is exceeded, the least recently used connections are closed. default: 320 References: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive upstream-keepalive-time \u00b6 Sets the maximum time during which requests can be processed through one keepalive connection. default: \"1h\" References: http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_time upstream-keepalive-timeout \u00b6 Sets a timeout during which an idle keepalive connection to an upstream server will stay open. default: 60 References: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_timeout upstream-keepalive-requests \u00b6 Sets the maximum number of requests that can be served through one keepalive connection. After the maximum number of requests is made, the connection is closed. default: 10000 References: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_requests limit-conn-zone-variable \u00b6 Sets parameters for a shared memory zone that will keep states for various keys of limit_conn_zone . The default of \"$binary_remote_addr\" variable\u2019s size is always 4 bytes for IPv4 addresses or 16 bytes for IPv6 addresses. proxy-stream-timeout \u00b6 Sets the timeout between two successive read or write operations on client or proxied server connections. If no data is transmitted within this time, the connection is closed. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout proxy-stream-next-upstream \u00b6 When a connection to the proxied server cannot be established, determines whether a client connection will be passed to the next server. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream proxy-stream-next-upstream-timeout \u00b6 Limits the time allowed to pass a connection to the next server. The 0 value turns off this limitation. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_timeout proxy-stream-next-upstream-tries \u00b6 Limits the number of possible tries a request should be passed to the next server. The 0 value turns off this limitation. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_tries proxy-stream-responses \u00b6 Sets the number of datagrams expected from the proxied server in response to the client request if the UDP protocol is used. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_responses bind-address \u00b6 Sets the addresses on which the server will accept requests instead of *. It should be noted that these addresses must exist in the runtime environment or the controller will crash loop. use-forwarded-headers \u00b6 If true, NGINX passes the incoming X-Forwarded-* headers to upstreams. Use this option when NGINX is behind another L7 proxy / load balancer that is setting these headers. If false, NGINX ignores incoming X-Forwarded-* headers, filling them with the request information it sees. Use this option if NGINX is exposed directly to the internet, or it's behind a L3/packet-based load balancer that doesn't alter the source IP in the packets. enable-real-ip \u00b6 enable-real-ip enables the configuration of https://nginx.org/en/docs/http/ngx_http_realip_module.html . Specific attributes of the module can be configured further by using forwarded-for-header and proxy-real-ip-cidr settings. forwarded-for-header \u00b6 Sets the header field for identifying the originating IP address of a client. default: X-Forwarded-For compute-full-forwarded-for \u00b6 Append the remote address to the X-Forwarded-For header instead of replacing it. When this option is enabled, the upstream application is responsible for extracting the client IP based on its own list of trusted proxies. proxy-add-original-uri-header \u00b6 Adds an X-Original-Uri header with the original request URI to the backend request generate-request-id \u00b6 Ensures that X-Request-ID is defaulted to a random value, if no X-Request-ID is present in the request enable-opentracing \u00b6 Enables the nginx Opentracing extension. default: is disabled References: https://github.com/opentracing-contrib/nginx-opentracing opentracing-operation-name \u00b6 Specifies a custom name for the server span. default: is empty For example, set to \"HTTP $request_method $uri\". opentracing-location-operation-name \u00b6 Specifies a custom name for the location span. default: is empty For example, set to \"HTTP $request_method $uri\". zipkin-collector-host \u00b6 Specifies the host to use when uploading traces. It must be a valid URL. zipkin-collector-port \u00b6 Specifies the port to use when uploading traces. default: 9411 zipkin-service-name \u00b6 Specifies the service name to use for any traces created. default: nginx zipkin-sample-rate \u00b6 Specifies sample rate for any traces created. default: 1.0 jaeger-collector-host \u00b6 Specifies the host to use when uploading traces. It must be a valid URL. jaeger-collector-port \u00b6 Specifies the port to use when uploading traces. default: 6831 jaeger-endpoint \u00b6 Specifies the endpoint to use when uploading traces to a collector. This takes priority over jaeger-collector-host if both are specified. jaeger-service-name \u00b6 Specifies the service name to use for any traces created. default: nginx jaeger-propagation-format \u00b6 Specifies the traceparent/tracestate propagation format. default: jaeger jaeger-sampler-type \u00b6 Specifies the sampler to be used when sampling traces. The available samplers are: const, probabilistic, ratelimiting, remote. default: const jaeger-sampler-param \u00b6 Specifies the argument to be passed to the sampler constructor. Must be a number. For const this should be 0 to never sample and 1 to always sample. default: 1 jaeger-sampler-host \u00b6 Specifies the custom remote sampler host to be passed to the sampler constructor. Must be a valid URL. Leave blank to use default value (localhost). default: http://127.0.0.1 jaeger-sampler-port \u00b6 Specifies the custom remote sampler port to be passed to the sampler constructor. Must be a number. default: 5778 jaeger-trace-context-header-name \u00b6 Specifies the header name used for passing trace context. default: uber-trace-id jaeger-debug-header \u00b6 Specifies the header name used for force sampling. default: jaeger-debug-id jaeger-baggage-header \u00b6 Specifies the header name used to submit baggage if there is no root span. default: jaeger-baggage jaeger-tracer-baggage-header-prefix \u00b6 Specifies the header prefix used to propagate baggage. default: uberctx- datadog-collector-host \u00b6 Specifies the datadog agent host to use when uploading traces. It must be a valid URL. datadog-collector-port \u00b6 Specifies the port to use when uploading traces. default: 8126 datadog-service-name \u00b6 Specifies the service name to use for any traces created. default: nginx datadog-environment \u00b6 Specifies the environment this trace belongs to. default: prod datadog-operation-name-override \u00b6 Overrides the operation name to use for any traces crated. default: nginx.handle datadog-priority-sampling \u00b6 Specifies to use client-side sampling. If true disables client-side sampling (thus ignoring sample_rate ) and enables distributed priority sampling, where traces are sampled based on a combination of user-assigned priorities and configuration from the agent. default: true datadog-sample-rate \u00b6 Specifies sample rate for any traces created. This is effective only when datadog-priority-sampling is false default: 1.0 main-snippet \u00b6 Adds custom configuration to the main section of the nginx configuration. http-snippet \u00b6 Adds custom configuration to the http section of the nginx configuration. server-snippet \u00b6 Adds custom configuration to all the servers in the nginx configuration. stream-snippet \u00b6 Adds custom configuration to the stream section of the nginx configuration. location-snippet \u00b6 Adds custom configuration to all the locations in the nginx configuration. You can not use this to add new locations that proxy to the Kubernetes pods, as the snippet does not have access to the Go template functions. If you want to add custom locations you will have to provide your own nginx.tmpl . custom-http-errors \u00b6 Enables which HTTP codes should be passed for processing with the error_page directive Setting at least one code also enables proxy_intercept_errors which are required to process error_page. Example usage: custom-http-errors: 404,415 proxy-body-size \u00b6 Sets the maximum allowed size of the client request body. See NGINX client_max_body_size . proxy-connect-timeout \u00b6 Sets the timeout for establishing a connection with a proxied server . It should be noted that this timeout cannot usually exceed 75 seconds. proxy-read-timeout \u00b6 Sets the timeout in seconds for reading a response from the proxied server . The timeout is set only between two successive read operations, not for the transmission of the whole response. proxy-send-timeout \u00b6 Sets the timeout in seconds for transmitting a request to the proxied server . The timeout is set only between two successive write operations, not for the transmission of the whole request. proxy-buffers-number \u00b6 Sets the number of the buffer used for reading the first part of the response received from the proxied server. This part usually contains a small response header. proxy-buffer-size \u00b6 Sets the size of the buffer used for reading the first part of the response received from the proxied server. This part usually contains a small response header. proxy-cookie-path \u00b6 Sets a text that should be changed in the path attribute of the \u201cSet-Cookie\u201d header fields of a proxied server response. proxy-cookie-domain \u00b6 Sets a text that should be changed in the domain attribute of the \u201cSet-Cookie\u201d header fields of a proxied server response. proxy-next-upstream \u00b6 Specifies in which cases a request should be passed to the next server. proxy-next-upstream-timeout \u00b6 Limits the time in seconds during which a request can be passed to the next server. proxy-next-upstream-tries \u00b6 Limit the number of possible tries a request should be passed to the next server. proxy-redirect-from \u00b6 Sets the original text that should be changed in the \"Location\" and \"Refresh\" header fields of a proxied server response. default: off References: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_redirect proxy-request-buffering \u00b6 Enables or disables buffering of a client request body . ssl-redirect \u00b6 Sets the global value of redirects (301) to HTTPS if the server has a TLS certificate (defined in an Ingress rule). default: \"true\" force-ssl-redirect \u00b6 Sets the global value of redirects (308) to HTTPS if the server has a default TLS certificate (defined in extra-args). default: \"false\" whitelist-source-range \u00b6 Sets the default whitelisted IPs for each server block. This can be overwritten by an annotation on an Ingress rule. See ngx_http_access_module . skip-access-log-urls \u00b6 Sets a list of URLs that should not appear in the NGINX access log. This is useful with urls like /health or health-check that make \"complex\" reading the logs. default: is empty limit-rate \u00b6 Limits the rate of response transmission to a client. The rate is specified in bytes per second. The zero value disables rate limiting. The limit is set per a request, and so if a client simultaneously opens two connections, the overall rate will be twice as much as the specified limit. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate limit-rate-after \u00b6 Sets the initial amount after which the further transmission of a response to a client will be rate limited. lua-shared-dicts \u00b6 Customize default Lua shared dictionaries or define more. You can use the following syntax to do so: lua-shared-dicts: \"<my dict name>: <my dict size>, [<my dict name>: <my dict size>], ...\" For example following will set default certificate_data dictionary to 100M and will introduce a new dictionary called my_custom_plugin : lua-shared-dicts: \"certificate_data: 100, my_custom_plugin: 5\" You can optionally set a size unit to allow for kilobyte-granularity. Allowed units are 'm' or 'k' (case-insensitive), and it defaults to MB if no unit is provided. Here is a similar example, but the my_custom_plugin dict is only 512KB. lua-shared-dicts: \"certificate_data: 100, my_custom_plugin: 512k\" References: https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate_after http-redirect-code \u00b6 Sets the HTTP status code to be used in redirects. Supported codes are 301 , 302 , 307 and 308 default: 308 Why the default code is 308? RFC 7238 was created to define the 308 (Permanent Redirect) status code that is similar to 301 (Moved Permanently) but it keeps the payload in the redirect. This is important if we send a redirect in methods like POST. proxy-buffering \u00b6 Enables or disables buffering of responses from the proxied server . limit-req-status-code \u00b6 Sets the status code to return in response to rejected requests . default: 503 limit-conn-status-code \u00b6 Sets the status code to return in response to rejected connections . default: 503 enable-syslog \u00b6 Enable syslog feature for access log and error log. default: false syslog-host \u00b6 Sets the address of syslog server. The address can be specified as a domain name or IP address. syslog-port \u00b6 Sets the port of syslog server. default: 514 no-tls-redirect-locations \u00b6 A comma-separated list of locations on which http requests will never get redirected to their https counterpart. default: \"/.well-known/acme-challenge\" global-auth-url \u00b6 A url to an existing service that provides authentication for all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-url . Locations that should not get authenticated can be listed using no-auth-locations See no-auth-locations . In addition, each service can be excluded from authentication via annotation enable-global-auth set to \"false\". default: \"\" References: https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/annotations.md#external-authentication global-auth-method \u00b6 A HTTP method to use for an existing service that provides authentication for all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-method . default: \"\" global-auth-signin \u00b6 Sets the location of the error page for an existing service that provides authentication for all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-signin . default: \"\" global-auth-signin-redirect-param \u00b6 Sets the query parameter in the error page signin URL which contains the original URL of the request that failed authentication. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-signin-redirect-param . default: \"rd\" global-auth-response-headers \u00b6 Sets the headers to pass to backend once authentication request completes. Applied to all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-response-headers . default: \"\" global-auth-request-redirect \u00b6 Sets the X-Auth-Request-Redirect header value. Applied to all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-request-redirect . default: \"\" global-auth-snippet \u00b6 Sets a custom snippet to use with external authentication. Applied to all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-request-redirect . default: \"\" global-auth-cache-key \u00b6 Enables caching for global auth requests. Specify a lookup key for auth responses, e.g. $remote_user$http_authorization . global-auth-cache-duration \u00b6 Set a caching time for auth responses based on their response codes, e.g. 200 202 30m . See proxy_cache_valid for details. You may specify multiple, comma-separated values: 200 202 10m, 401 5m . defaults to 200 202 401 5m . global-auth-always-set-cookie \u00b6 Always set a cookie returned by auth request. By default, the cookie will be set only if an upstream reports with the code 200, 201, 204, 206, 301, 302, 303, 304, 307, or 308. default: false no-auth-locations \u00b6 A comma-separated list of locations that should not get authenticated. default: \"/.well-known/acme-challenge\" block-cidrs \u00b6 A comma-separated list of IP addresses (or subnets), request from which have to be blocked globally. References: https://nginx.org/en/docs/http/ngx_http_access_module.html#deny block-user-agents \u00b6 A comma-separated list of User-Agent, request from which have to be blocked globally. It's possible to use here full strings and regular expressions. More details about valid patterns can be found at map Nginx directive documentation. References: https://nginx.org/en/docs/http/ngx_http_map_module.html#map block-referers \u00b6 A comma-separated list of Referers, request from which have to be blocked globally. It's possible to use here full strings and regular expressions. More details about valid patterns can be found at map Nginx directive documentation. References: https://nginx.org/en/docs/http/ngx_http_map_module.html#map proxy-ssl-location-only \u00b6 Set if proxy-ssl parameters should be applied only on locations and not on servers. default: is disabled default-type \u00b6 Sets the default MIME type of a response. default: text/html References: https://nginx.org/en/docs/http/ngx_http_core_module.html#default_type global-rate-limit \u00b6 global-rate-limit-status-code : configure HTTP status code to return when rejecting requests. Defaults to 429. Configure memcached client for Global Rate Limiting . global-rate-limit-memcached-host : IP/FQDN of memcached server to use. Required to enable Global Rate Limiting. global-rate-limit-memcached-port : port of memcached server to use. Defaults default memcached port of 11211 . global-rate-limit-memcached-connect-timeout : configure timeout for connect, send and receive operations. Unit is millisecond. Defaults to 50ms. global-rate-limit-memcached-max-idle-timeout : configure timeout for cleaning idle connections. Unit is millisecond. Defaults to 50ms. global-rate-limit-memcached-pool-size : configure number of max connections to keep alive. Make sure your memcached server can handle global-rate-limit-memcached-pool-size * worker-processes * <number of ingress-nginx replicas> simultaneous connections. These settings get used by lua-resty-global-throttle that ingress-nginx includes. Refer to the link to learn more about lua-resty-global-throttle . service-upstream \u00b6 Set if the service's Cluster IP and port should be used instead of a list of all endpoints. This can be overwritten by an annotation on an Ingress rule. default: \"false\" ssl-reject-handshake \u00b6 Set to reject SSL handshake to an unknown virtualhost. This parameter helps to mitigate the fingerprinting using default certificate of ingress. default: \"false\" References: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_reject_handshake debug-connections \u00b6 Enables debugging log for selected client connections. default: \"\" References: http://nginx.org/en/docs/ngx_core_module.html#debug_connection","title":"ConfigMap"},{"location":"user-guide/nginx-configuration/configmap/#configmaps","text":"ConfigMaps allow you to decouple configuration artifacts from image content to keep containerized applications portable. The ConfigMap API resource stores configuration data as key-value pairs. The data provides the configurations for system components for the nginx-controller. In order to overwrite nginx-controller configuration values as seen in config.go , you can add key-value pairs to the data section of the config-map. For Example: data : map-hash-bucket-size : \"128\" ssl-protocols : SSLv2 Important The key and values in a ConfigMap can only be strings. This means that we want a value with boolean values we need to quote the values, like \"true\" or \"false\". Same for numbers, like \"100\". \"Slice\" types (defined below as []string or []int ) can be provided as a comma-delimited string.","title":"ConfigMaps"},{"location":"user-guide/nginx-configuration/configmap/#configuration-options","text":"The following table shows a configuration option's name, type, and the default value: name type default add-headers string \"\" allow-backend-server-header bool \"false\" allow-snippet-annotations bool true annotation-value-word-blocklist string array \"\" hide-headers string array empty access-log-params string \"\" access-log-path string \"/var/log/nginx/access.log\" http-access-log-path string \"\" stream-access-log-path string \"\" enable-access-log-for-default-backend bool \"false\" error-log-path string \"/var/log/nginx/error.log\" enable-modsecurity bool \"false\" modsecurity-snippet string \"\" enable-owasp-modsecurity-crs bool \"false\" client-header-buffer-size string \"1k\" client-header-timeout int 60 client-body-buffer-size string \"8k\" client-body-timeout int 60 disable-access-log bool false disable-ipv6 bool false disable-ipv6-dns bool false enable-underscores-in-headers bool false enable-ocsp bool false ignore-invalid-headers bool true retry-non-idempotent bool \"false\" error-log-level string \"notice\" http2-max-field-size string \"4k\" http2-max-header-size string \"16k\" http2-max-requests int 1000 http2-max-concurrent-streams int 128 hsts bool \"true\" hsts-include-subdomains bool \"true\" hsts-max-age string \"15724800\" hsts-preload bool \"false\" keep-alive int 75 keep-alive-requests int 100 large-client-header-buffers string \"4 8k\" log-format-escape-none bool \"false\" log-format-escape-json bool \"false\" log-format-upstream string $remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id log-format-stream string [$remote_addr] [$time_local] $protocol $status $bytes_sent $bytes_received $session_time enable-multi-accept bool \"true\" max-worker-connections int 16384 max-worker-open-files int 0 map-hash-bucket-size int 64 nginx-status-ipv4-whitelist []string \"127.0.0.1\" nginx-status-ipv6-whitelist []string \"::1\" proxy-real-ip-cidr []string \"0.0.0.0/0\" proxy-set-headers string \"\" server-name-hash-max-size int 1024 server-name-hash-bucket-size int <size of the processor\u2019s cache line> proxy-headers-hash-max-size int 512 proxy-headers-hash-bucket-size int 64 plugins []string reuse-port bool \"true\" server-tokens bool \"false\" ssl-ciphers string \"ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384\" ssl-ecdh-curve string \"auto\" ssl-dh-param string \"\" ssl-protocols string \"TLSv1.2 TLSv1.3\" ssl-session-cache bool \"true\" ssl-session-cache-size string \"10m\" ssl-session-tickets bool \"false\" ssl-session-ticket-key string <Randomly Generated> ssl-session-timeout string \"10m\" ssl-buffer-size string \"4k\" use-proxy-protocol bool \"false\" proxy-protocol-header-timeout string \"5s\" use-gzip bool \"false\" use-geoip bool \"true\" use-geoip2 bool \"false\" enable-brotli bool \"false\" brotli-level int 4 brotli-min-length int 20 brotli-types string \"application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/javascript text/plain text/x-component\" use-http2 bool \"true\" gzip-level int 1 gzip-types string \"application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/javascript text/plain text/x-component\" worker-processes string <Number of CPUs> worker-cpu-affinity string \"\" worker-shutdown-timeout string \"240s\" load-balance string \"round_robin\" variables-hash-bucket-size int 128 variables-hash-max-size int 2048 upstream-keepalive-connections int 320 upstream-keepalive-time string \"1h\" upstream-keepalive-timeout int 60 upstream-keepalive-requests int 10000 limit-conn-zone-variable string \"$binary_remote_addr\" proxy-stream-timeout string \"600s\" proxy-stream-next-upstream bool \"true\" proxy-stream-next-upstream-timeout string \"600s\" proxy-stream-next-upstream-tries int 3 proxy-stream-responses int 1 bind-address []string \"\" use-forwarded-headers bool \"false\" enable-real-ip bool \"false\" forwarded-for-header string \"X-Forwarded-For\" compute-full-forwarded-for bool \"false\" proxy-add-original-uri-header bool \"false\" generate-request-id bool \"true\" enable-opentracing bool \"false\" opentracing-operation-name string \"\" opentracing-location-operation-name string \"\" zipkin-collector-host string \"\" zipkin-collector-port int 9411 zipkin-service-name string \"nginx\" zipkin-sample-rate float 1.0 jaeger-collector-host string \"\" jaeger-collector-port int 6831 jaeger-endpoint string \"\" jaeger-service-name string \"nginx\" jaeger-propagation-format string \"jaeger\" jaeger-sampler-type string \"const\" jaeger-sampler-param string \"1\" jaeger-sampler-host string \"http://127.0.0.1\" jaeger-sampler-port int 5778 jaeger-trace-context-header-name string uber-trace-id jaeger-debug-header string uber-debug-id jaeger-baggage-header string jaeger-baggage jaeger-trace-baggage-header-prefix string uberctx- datadog-collector-host string \"\" datadog-collector-port int 8126 datadog-service-name string \"nginx\" datadog-environment string \"prod\" datadog-operation-name-override string \"nginx.handle\" datadog-priority-sampling bool \"true\" datadog-sample-rate float 1.0 main-snippet string \"\" http-snippet string \"\" server-snippet string \"\" stream-snippet string \"\" location-snippet string \"\" custom-http-errors []int []int{} proxy-body-size string \"1m\" proxy-connect-timeout int 5 proxy-read-timeout int 60 proxy-send-timeout int 60 proxy-buffers-number int 4 proxy-buffer-size string \"4k\" proxy-cookie-path string \"off\" proxy-cookie-domain string \"off\" proxy-next-upstream string \"error timeout\" proxy-next-upstream-timeout int 0 proxy-next-upstream-tries int 3 proxy-redirect-from string \"off\" proxy-request-buffering string \"on\" ssl-redirect bool \"true\" force-ssl-redirect bool \"false\" whitelist-source-range []string []string{} skip-access-log-urls []string []string{} limit-rate int 0 limit-rate-after int 0 lua-shared-dicts string \"\" http-redirect-code int 308 proxy-buffering string \"off\" limit-req-status-code int 503 limit-conn-status-code int 503 enable-syslog bool false syslog-host string \"\" syslog-port int 514 no-tls-redirect-locations string \"/.well-known/acme-challenge\" global-auth-url string \"\" global-auth-method string \"\" global-auth-signin string \"\" global-auth-signin-redirect-param string \"rd\" global-auth-response-headers string \"\" global-auth-request-redirect string \"\" global-auth-snippet string \"\" global-auth-cache-key string \"\" global-auth-cache-duration string \"200 202 401 5m\" no-auth-locations string \"/.well-known/acme-challenge\" block-cidrs []string \"\" block-user-agents []string \"\" block-referers []string \"\" proxy-ssl-location-only bool \"false\" default-type string \"text/html\" global-rate-limit-memcached-host string \"\" global-rate-limit-memcached-port int 11211 global-rate-limit-memcached-connect-timeout int 50 global-rate-limit-memcached-max-idle-timeout int 10000 global-rate-limit-memcached-pool-size int 50 global-rate-limit-status-code int 429 service-upstream bool \"false\" ssl-reject-handshake bool \"false\" debug-connections []string \"127.0.0.1,1.1.1.1/24\"","title":"Configuration options"},{"location":"user-guide/nginx-configuration/configmap/#add-headers","text":"Sets custom headers from named configmap before sending traffic to the client. See proxy-set-headers . example","title":"add-headers"},{"location":"user-guide/nginx-configuration/configmap/#allow-backend-server-header","text":"Enables the return of the header Server from the backend instead of the generic nginx string. default: is disabled","title":"allow-backend-server-header"},{"location":"user-guide/nginx-configuration/configmap/#allow-snippet-annotations","text":"Enables Ingress to parse and add -snippet annotations/directives created by the user. _**default:* _ true Warning: We recommend enabling this option only if you TRUST users with permission to create Ingress objects, as this may allow a user to add restricted configurations to the final nginx.conf file","title":"allow-snippet-annotations"},{"location":"user-guide/nginx-configuration/configmap/#annotation-value-word-blocklist","text":"Contains a comma-separated value of chars/words that are well known of being used to abuse Ingress configuration and must be blocked. Related to CVE-2021-25742 When an annotation is detected with a value that matches one of the blocked bad words, the whole Ingress won't be configured. default: \"\" When doing this, the default blocklist is override, which means that the Ingress admin should add all the words that should be blocked, here is a suggested block list. suggested: \"load_module,lua_package,_by_lua,location,root,proxy_pass,serviceaccount,{,},',\\\"\"","title":"annotation-value-word-blocklist"},{"location":"user-guide/nginx-configuration/configmap/#hide-headers","text":"Sets additional header that will not be passed from the upstream server to the client response. default: empty References: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header","title":"hide-headers"},{"location":"user-guide/nginx-configuration/configmap/#access-log-params","text":"Additional params for access_log. For example, buffer=16k, gzip, flush=1m References: https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log","title":"access-log-params"},{"location":"user-guide/nginx-configuration/configmap/#access-log-path","text":"Access log path for both http and stream context. Goes to /var/log/nginx/access.log by default. Note: the file /var/log/nginx/access.log is a symlink to /dev/stdout","title":"access-log-path"},{"location":"user-guide/nginx-configuration/configmap/#http-access-log-path","text":"Access log path for http context globally. default: \"\" Note: If not specified, the access-log-path will be used.","title":"http-access-log-path"},{"location":"user-guide/nginx-configuration/configmap/#stream-access-log-path","text":"Access log path for stream context globally. default: \"\" Note: If not specified, the access-log-path will be used.","title":"stream-access-log-path"},{"location":"user-guide/nginx-configuration/configmap/#enable-access-log-for-default-backend","text":"Enables logging access to default backend. default: is disabled.","title":"enable-access-log-for-default-backend"},{"location":"user-guide/nginx-configuration/configmap/#error-log-path","text":"Error log path. Goes to /var/log/nginx/error.log by default. Note: the file /var/log/nginx/error.log is a symlink to /dev/stderr References: https://nginx.org/en/docs/ngx_core_module.html#error_log","title":"error-log-path"},{"location":"user-guide/nginx-configuration/configmap/#enable-modsecurity","text":"Enables the modsecurity module for NGINX. default: is disabled","title":"enable-modsecurity"},{"location":"user-guide/nginx-configuration/configmap/#enable-owasp-modsecurity-crs","text":"Enables the OWASP ModSecurity Core Rule Set (CRS). default: is disabled","title":"enable-owasp-modsecurity-crs"},{"location":"user-guide/nginx-configuration/configmap/#modsecurity-snippet","text":"Adds custom rules to modsecurity section of nginx configuration","title":"modsecurity-snippet"},{"location":"user-guide/nginx-configuration/configmap/#client-header-buffer-size","text":"Allows to configure a custom buffer size for reading client request header. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_buffer_size","title":"client-header-buffer-size"},{"location":"user-guide/nginx-configuration/configmap/#client-header-timeout","text":"Defines a timeout for reading client request header, in seconds. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_timeout","title":"client-header-timeout"},{"location":"user-guide/nginx-configuration/configmap/#client-body-buffer-size","text":"Sets buffer size for reading client request body. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size","title":"client-body-buffer-size"},{"location":"user-guide/nginx-configuration/configmap/#client-body-timeout","text":"Defines a timeout for reading client request body, in seconds. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_timeout","title":"client-body-timeout"},{"location":"user-guide/nginx-configuration/configmap/#disable-access-log","text":"Disables the Access Log from the entire Ingress Controller. default: false References: https://nginx.org/en/docs/http/ngx_http_log_module.html#access_log","title":"disable-access-log"},{"location":"user-guide/nginx-configuration/configmap/#disable-ipv6","text":"Disable listening on IPV6. default: false ; IPv6 listening is enabled","title":"disable-ipv6"},{"location":"user-guide/nginx-configuration/configmap/#disable-ipv6-dns","text":"Disable IPV6 for nginx DNS resolver. default: false ; IPv6 resolving enabled.","title":"disable-ipv6-dns"},{"location":"user-guide/nginx-configuration/configmap/#enable-underscores-in-headers","text":"Enables underscores in header names. default: is disabled","title":"enable-underscores-in-headers"},{"location":"user-guide/nginx-configuration/configmap/#enable-ocsp","text":"Enables Online Certificate Status Protocol stapling (OCSP) support. default: is disabled","title":"enable-ocsp"},{"location":"user-guide/nginx-configuration/configmap/#ignore-invalid-headers","text":"Set if header fields with invalid names should be ignored. default: is enabled","title":"ignore-invalid-headers"},{"location":"user-guide/nginx-configuration/configmap/#retry-non-idempotent","text":"Since 1.9.13 NGINX will not retry non-idempotent requests (POST, LOCK, PATCH) in case of an error in the upstream server. The previous behavior can be restored using the value \"true\".","title":"retry-non-idempotent"},{"location":"user-guide/nginx-configuration/configmap/#error-log-level","text":"Configures the logging level of errors. Log levels above are listed in the order of increasing severity. References: https://nginx.org/en/docs/ngx_core_module.html#error_log","title":"error-log-level"},{"location":"user-guide/nginx-configuration/configmap/#http2-max-field-size","text":"Warning This feature was deprecated in 1.1.3 and will be removed in 1.3.0. Use large-client-header-buffers instead. Limits the maximum size of an HPACK-compressed request header field. References: https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_field_size","title":"http2-max-field-size"},{"location":"user-guide/nginx-configuration/configmap/#http2-max-header-size","text":"Warning This feature was deprecated in 1.1.3 and will be removed in 1.3.0. Use large-client-header-buffers instead. Limits the maximum size of the entire request header list after HPACK decompression. References: https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_header_size","title":"http2-max-header-size"},{"location":"user-guide/nginx-configuration/configmap/#http2-max-requests","text":"Warning This feature was deprecated in 1.1.3 and will be removed in 1.3.0. Use upstream-keepalive-requests instead. Sets the maximum number of requests (including push requests) that can be served through one HTTP/2 connection, after which the next client request will lead to connection closing and the need of establishing a new connection. References: https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_requests","title":"http2-max-requests"},{"location":"user-guide/nginx-configuration/configmap/#http2-max-concurrent-streams","text":"Sets the maximum number of concurrent HTTP/2 streams in a connection. References: https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_concurrent_streams","title":"http2-max-concurrent-streams"},{"location":"user-guide/nginx-configuration/configmap/#hsts","text":"Enables or disables the header HSTS in servers running SSL. HTTP Strict Transport Security (often abbreviated as HSTS) is a security feature (HTTP header) that tell browsers that it should only be communicated with using HTTPS, instead of using HTTP. It provides protection against protocol downgrade attacks and cookie theft. References: https://developer.mozilla.org/en-US/docs/Web/Security/HTTP_strict_transport_security https://blog.qualys.com/securitylabs/2016/03/28/the-importance-of-a-proper-http-strict-transport-security-implementation-on-your-web-server","title":"hsts"},{"location":"user-guide/nginx-configuration/configmap/#hsts-include-subdomains","text":"Enables or disables the use of HSTS in all the subdomains of the server-name.","title":"hsts-include-subdomains"},{"location":"user-guide/nginx-configuration/configmap/#hsts-max-age","text":"Sets the time, in seconds, that the browser should remember that this site is only to be accessed using HTTPS.","title":"hsts-max-age"},{"location":"user-guide/nginx-configuration/configmap/#hsts-preload","text":"Enables or disables the preload attribute in the HSTS feature (when it is enabled).","title":"hsts-preload"},{"location":"user-guide/nginx-configuration/configmap/#keep-alive","text":"Sets the time, in seconds, during which a keep-alive client connection will stay open on the server side. The zero value disables keep-alive client connections. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout Important Setting keep-alive: '0' will most likely break concurrent http/2 requests due to changes introduced with nginx 1.19.7 Changes with nginx 1.19.7 16 Feb 2021 *) Change: connections handling in HTTP/2 has been changed to better match HTTP/1.x; the \"http2_recv_timeout\", \"http2_idle_timeout\", and \"http2_max_requests\" directives have been removed, the \"keepalive_timeout\" and \"keepalive_requests\" directives should be used instead. References: nginx change log nginx issue tracker nginx mailing list","title":"keep-alive"},{"location":"user-guide/nginx-configuration/configmap/#keep-alive-requests","text":"Sets the maximum number of requests that can be served through one keep-alive connection. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests","title":"keep-alive-requests"},{"location":"user-guide/nginx-configuration/configmap/#large-client-header-buffers","text":"Sets the maximum number and size of buffers used for reading large client request header. default: 4 8k References: https://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers","title":"large-client-header-buffers"},{"location":"user-guide/nginx-configuration/configmap/#log-format-escape-none","text":"Sets if the escape parameter is disabled entirely for character escaping in variables (\"true\") or controlled by log-format-escape-json (\"false\") Sets the nginx log format .","title":"log-format-escape-none"},{"location":"user-guide/nginx-configuration/configmap/#log-format-escape-json","text":"Sets if the escape parameter allows JSON (\"true\") or default characters escaping in variables (\"false\") Sets the nginx log format .","title":"log-format-escape-json"},{"location":"user-guide/nginx-configuration/configmap/#log-format-upstream","text":"Sets the nginx log format . Example for json output: log - f orma t - ups trea m : ' { \"time\" : \"$time_iso8601\" , \"remote_addr\" : \"$proxy_protocol_addr\" , \"x_forwarded_for\" : \"$proxy_add_x_forwarded_for\" , \"request_id\" : \"$req_id\" , \"remote_user\" : \"$remote_user\" , \"bytes_sent\" : $by tes _se nt , \"request_time\" : $reques t _ t ime , \"status\" : $s tatus , \"vhost\" : \"$host\" , \"request_proto\" : \"$server_protocol\" , \"path\" : \"$uri\" , \"request_query\" : \"$args\" , \"request_length\" : $reques t _le n g t h , \"duration\" : $reques t _ t ime , \"method\" : \"$request_method\" , \"http_referrer\" : \"$http_referer\" , \"http_user_agent\" : \"$http_user_agent\" } ' Please check the log-format for definition of each field.","title":"log-format-upstream"},{"location":"user-guide/nginx-configuration/configmap/#log-format-stream","text":"Sets the nginx stream format .","title":"log-format-stream"},{"location":"user-guide/nginx-configuration/configmap/#enable-multi-accept","text":"If disabled, a worker process will accept one new connection at a time. Otherwise, a worker process will accept all new connections at a time. default: true References: https://nginx.org/en/docs/ngx_core_module.html#multi_accept","title":"enable-multi-accept"},{"location":"user-guide/nginx-configuration/configmap/#max-worker-connections","text":"Sets the maximum number of simultaneous connections that can be opened by each worker process. 0 will use the value of max-worker-open-files . default: 16384 Tip Using 0 in scenarios of high load improves performance at the cost of increasing RAM utilization (even on idle).","title":"max-worker-connections"},{"location":"user-guide/nginx-configuration/configmap/#max-worker-open-files","text":"Sets the maximum number of files that can be opened by each worker process. The default of 0 means \"max open files (system's limit) - 1024\". default: 0","title":"max-worker-open-files"},{"location":"user-guide/nginx-configuration/configmap/#map-hash-bucket-size","text":"Sets the bucket size for the map variables hash tables . The details of setting up hash tables are provided in a separate document .","title":"map-hash-bucket-size"},{"location":"user-guide/nginx-configuration/configmap/#proxy-real-ip-cidr","text":"If use-forwarded-headers or use-proxy-protocol is enabled, proxy-real-ip-cidr defines the default IP/network address of your external load balancer. Can be a comma-separated list of CIDR blocks. default: \"0.0.0.0/0\"","title":"proxy-real-ip-cidr"},{"location":"user-guide/nginx-configuration/configmap/#proxy-set-headers","text":"Sets custom headers from named configmap before sending traffic to backends. The value format is namespace/name. See example","title":"proxy-set-headers"},{"location":"user-guide/nginx-configuration/configmap/#server-name-hash-max-size","text":"Sets the maximum size of the server names hash tables used in server names,map directive\u2019s values, MIME types, names of request header strings, etc. References: https://nginx.org/en/docs/hash.html","title":"server-name-hash-max-size"},{"location":"user-guide/nginx-configuration/configmap/#server-name-hash-bucket-size","text":"Sets the size of the bucket for the server names hash tables. References: https://nginx.org/en/docs/hash.html https://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_bucket_size","title":"server-name-hash-bucket-size"},{"location":"user-guide/nginx-configuration/configmap/#proxy-headers-hash-max-size","text":"Sets the maximum size of the proxy headers hash tables. References: https://nginx.org/en/docs/hash.html https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_headers_hash_max_size","title":"proxy-headers-hash-max-size"},{"location":"user-guide/nginx-configuration/configmap/#reuse-port","text":"Instructs NGINX to create an individual listening socket for each worker process (using the SO_REUSEPORT socket option), allowing a kernel to distribute incoming connections between worker processes default: true","title":"reuse-port"},{"location":"user-guide/nginx-configuration/configmap/#proxy-headers-hash-bucket-size","text":"Sets the size of the bucket for the proxy headers hash tables. References: https://nginx.org/en/docs/hash.html https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_headers_hash_bucket_size","title":"proxy-headers-hash-bucket-size"},{"location":"user-guide/nginx-configuration/configmap/#plugins","text":"Activates plugins installed in /etc/nginx/lua/plugins . Refer to ingress-nginx plugins README for more information on how to write and install a plugin.","title":"plugins"},{"location":"user-guide/nginx-configuration/configmap/#server-tokens","text":"Send NGINX Server header in responses and display NGINX version in error pages. default: is disabled","title":"server-tokens"},{"location":"user-guide/nginx-configuration/configmap/#ssl-ciphers","text":"Sets the ciphers list to enable. The ciphers are specified in the format understood by the OpenSSL library. The default cipher list is: ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 . The ordering of a ciphersuite is very important because it decides which algorithms are going to be selected in priority. The recommendation above prioritizes algorithms that provide perfect forward secrecy . DHE-based cyphers will not be available until DH parameter is configured Custom DH parameters for perfect forward secrecy Please check the Mozilla SSL Configuration Generator . Note: ssl_prefer_server_ciphers directive will be enabled by default for http context.","title":"ssl-ciphers"},{"location":"user-guide/nginx-configuration/configmap/#ssl-ecdh-curve","text":"Specifies a curve for ECDHE ciphers. References: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ecdh_curve","title":"ssl-ecdh-curve"},{"location":"user-guide/nginx-configuration/configmap/#ssl-dh-param","text":"Sets the name of the secret that contains Diffie-Hellman key to help with \"Perfect Forward Secrecy\". References: https://wiki.openssl.org/index.php/Diffie-Hellman_parameters https://wiki.mozilla.org/Security/Server_Side_TLS#DHE_handshake_and_dhparam https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_dhparam","title":"ssl-dh-param"},{"location":"user-guide/nginx-configuration/configmap/#ssl-protocols","text":"Sets the SSL protocols to use. The default is: TLSv1.2 TLSv1.3 . Please check the result of the configuration using https://ssllabs.com/ssltest/analyze.html or https://testssl.sh .","title":"ssl-protocols"},{"location":"user-guide/nginx-configuration/configmap/#ssl-early-data","text":"Enables or disables TLS 1.3 early data , also known as Zero Round Trip Time Resumption (0-RTT). This requires ssl-protocols to have TLSv1.3 enabled. Enable this with caution, because requests sent within early data are subject to replay attacks . ssl_early_data . The default is: false .","title":"ssl-early-data"},{"location":"user-guide/nginx-configuration/configmap/#ssl-session-cache","text":"Enables or disables the use of shared SSL cache among worker processes.","title":"ssl-session-cache"},{"location":"user-guide/nginx-configuration/configmap/#ssl-session-cache-size","text":"Sets the size of the SSL shared session cache between all worker processes.","title":"ssl-session-cache-size"},{"location":"user-guide/nginx-configuration/configmap/#ssl-session-tickets","text":"Enables or disables session resumption through TLS session tickets .","title":"ssl-session-tickets"},{"location":"user-guide/nginx-configuration/configmap/#ssl-session-ticket-key","text":"Sets the secret key used to encrypt and decrypt TLS session tickets. The value must be a valid base64 string. To create a ticket: openssl rand 80 | openssl enc -A -base64 TLS session ticket-key , by default, a randomly generated key is used.","title":"ssl-session-ticket-key"},{"location":"user-guide/nginx-configuration/configmap/#ssl-session-timeout","text":"Sets the time during which a client may reuse the session parameters stored in a cache.","title":"ssl-session-timeout"},{"location":"user-guide/nginx-configuration/configmap/#ssl-buffer-size","text":"Sets the size of the SSL buffer used for sending data. The default of 4k helps NGINX to improve TLS Time To First Byte (TTTFB). References: https://www.igvita.com/2013/12/16/optimizing-nginx-tls-time-to-first-byte/","title":"ssl-buffer-size"},{"location":"user-guide/nginx-configuration/configmap/#use-proxy-protocol","text":"Enables or disables the PROXY protocol to receive client connection (real IP address) information passed through proxy servers and load balancers such as HAProxy and Amazon Elastic Load Balancer (ELB).","title":"use-proxy-protocol"},{"location":"user-guide/nginx-configuration/configmap/#proxy-protocol-header-timeout","text":"Sets the timeout value for receiving the proxy-protocol headers. The default of 5 seconds prevents the TLS passthrough handler from waiting indefinitely on a dropped connection. default: 5s","title":"proxy-protocol-header-timeout"},{"location":"user-guide/nginx-configuration/configmap/#use-gzip","text":"Enables or disables compression of HTTP responses using the \"gzip\" module . MIME types to compress are controlled by gzip-types . default: false","title":"use-gzip"},{"location":"user-guide/nginx-configuration/configmap/#use-geoip","text":"Enables or disables \"geoip\" module that creates variables with values depending on the client IP address, using the precompiled MaxMind databases. default: true Note: MaxMind legacy databases are discontinued and will not receive updates after 2019-01-02, cf. discontinuation notice . Consider use-geoip2 below.","title":"use-geoip"},{"location":"user-guide/nginx-configuration/configmap/#use-geoip2","text":"Enables the geoip2 module for NGINX. Since 0.27.0 and due to a change in the MaxMind databases a license is required to have access to the databases. For this reason, it is required to define a new flag --maxmind-license-key in the ingress controller deployment to download the databases needed during the initialization of the ingress controller. Alternatively, it is possible to use a volume to mount the files /etc/nginx/geoip/GeoLite2-City.mmdb and /etc/nginx/geoip/GeoLite2-ASN.mmdb , avoiding the overhead of the download. Important If the feature is enabled but the files are missing, GeoIP2 will not be enabled. default: false","title":"use-geoip2"},{"location":"user-guide/nginx-configuration/configmap/#enable-brotli","text":"Enables or disables compression of HTTP responses using the \"brotli\" module . The default mime type list to compress is: application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component . default: is disabled Note: Brotli does not works in Safari < 11. For more information see https://caniuse.com/#feat=brotli","title":"enable-brotli"},{"location":"user-guide/nginx-configuration/configmap/#brotli-level","text":"Sets the Brotli Compression Level that will be used. default: 4","title":"brotli-level"},{"location":"user-guide/nginx-configuration/configmap/#brotli-min-length","text":"Minimum length of responses, in bytes, that will be eligible for brotli compression. default: 20","title":"brotli-min-length"},{"location":"user-guide/nginx-configuration/configmap/#brotli-types","text":"Sets the MIME Types that will be compressed on-the-fly by brotli. default: application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component","title":"brotli-types"},{"location":"user-guide/nginx-configuration/configmap/#use-http2","text":"Enables or disables HTTP/2 support in secure connections.","title":"use-http2"},{"location":"user-guide/nginx-configuration/configmap/#gzip-level","text":"Sets the gzip Compression Level that will be used. default: 1","title":"gzip-level"},{"location":"user-guide/nginx-configuration/configmap/#gzip-min-length","text":"Minimum length of responses to be returned to the client before it is eligible for gzip compression, in bytes. default: 256","title":"gzip-min-length"},{"location":"user-guide/nginx-configuration/configmap/#gzip-types","text":"Sets the MIME types in addition to \"text/html\" to compress. The special value \"*\" matches any MIME type. Responses with the \"text/html\" type are always compressed if use-gzip is enabled. default: application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component .","title":"gzip-types"},{"location":"user-guide/nginx-configuration/configmap/#worker-processes","text":"Sets the number of worker processes . The default of \"auto\" means number of available CPU cores.","title":"worker-processes"},{"location":"user-guide/nginx-configuration/configmap/#worker-cpu-affinity","text":"Binds worker processes to the sets of CPUs. worker_cpu_affinity . By default worker processes are not bound to any specific CPUs. The value can be: \"\": empty string indicate no affinity is applied. cpumask: e.g. 0001 0010 0100 1000 to bind processes to specific cpus. auto: binding worker processes automatically to available CPUs.","title":"worker-cpu-affinity"},{"location":"user-guide/nginx-configuration/configmap/#worker-shutdown-timeout","text":"Sets a timeout for Nginx to wait for worker to gracefully shutdown . default: \"240s\"","title":"worker-shutdown-timeout"},{"location":"user-guide/nginx-configuration/configmap/#load-balance","text":"Sets the algorithm to use for load balancing. The value can either be: round_robin: to use the default round robin loadbalancer ewma: to use the Peak EWMA method for routing ( implementation ) The default is round_robin . To load balance using consistent hashing of IP or other variables, consider the nginx.ingress.kubernetes.io/upstream-hash-by annotation. To load balance using session cookies, consider the nginx.ingress.kubernetes.io/affinity annotation. References: https://nginx.org/en/docs/http/load_balancing.html","title":"load-balance"},{"location":"user-guide/nginx-configuration/configmap/#variables-hash-bucket-size","text":"Sets the bucket size for the variables hash table. References: https://nginx.org/en/docs/http/ngx_http_map_module.html#variables_hash_bucket_size","title":"variables-hash-bucket-size"},{"location":"user-guide/nginx-configuration/configmap/#variables-hash-max-size","text":"Sets the maximum size of the variables hash table. References: https://nginx.org/en/docs/http/ngx_http_map_module.html#variables_hash_max_size","title":"variables-hash-max-size"},{"location":"user-guide/nginx-configuration/configmap/#upstream-keepalive-connections","text":"Activates the cache for connections to upstream servers. The connections parameter sets the maximum number of idle keepalive connections to upstream servers that are preserved in the cache of each worker process. When this number is exceeded, the least recently used connections are closed. default: 320 References: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive","title":"upstream-keepalive-connections"},{"location":"user-guide/nginx-configuration/configmap/#upstream-keepalive-time","text":"Sets the maximum time during which requests can be processed through one keepalive connection. default: \"1h\" References: http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_time","title":"upstream-keepalive-time"},{"location":"user-guide/nginx-configuration/configmap/#upstream-keepalive-timeout","text":"Sets a timeout during which an idle keepalive connection to an upstream server will stay open. default: 60 References: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_timeout","title":"upstream-keepalive-timeout"},{"location":"user-guide/nginx-configuration/configmap/#upstream-keepalive-requests","text":"Sets the maximum number of requests that can be served through one keepalive connection. After the maximum number of requests is made, the connection is closed. default: 10000 References: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive_requests","title":"upstream-keepalive-requests"},{"location":"user-guide/nginx-configuration/configmap/#limit-conn-zone-variable","text":"Sets parameters for a shared memory zone that will keep states for various keys of limit_conn_zone . The default of \"$binary_remote_addr\" variable\u2019s size is always 4 bytes for IPv4 addresses or 16 bytes for IPv6 addresses.","title":"limit-conn-zone-variable"},{"location":"user-guide/nginx-configuration/configmap/#proxy-stream-timeout","text":"Sets the timeout between two successive read or write operations on client or proxied server connections. If no data is transmitted within this time, the connection is closed. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout","title":"proxy-stream-timeout"},{"location":"user-guide/nginx-configuration/configmap/#proxy-stream-next-upstream","text":"When a connection to the proxied server cannot be established, determines whether a client connection will be passed to the next server. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream","title":"proxy-stream-next-upstream"},{"location":"user-guide/nginx-configuration/configmap/#proxy-stream-next-upstream-timeout","text":"Limits the time allowed to pass a connection to the next server. The 0 value turns off this limitation. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_timeout","title":"proxy-stream-next-upstream-timeout"},{"location":"user-guide/nginx-configuration/configmap/#proxy-stream-next-upstream-tries","text":"Limits the number of possible tries a request should be passed to the next server. The 0 value turns off this limitation. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_next_upstream_tries","title":"proxy-stream-next-upstream-tries"},{"location":"user-guide/nginx-configuration/configmap/#proxy-stream-responses","text":"Sets the number of datagrams expected from the proxied server in response to the client request if the UDP protocol is used. References: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_responses","title":"proxy-stream-responses"},{"location":"user-guide/nginx-configuration/configmap/#bind-address","text":"Sets the addresses on which the server will accept requests instead of *. It should be noted that these addresses must exist in the runtime environment or the controller will crash loop.","title":"bind-address"},{"location":"user-guide/nginx-configuration/configmap/#use-forwarded-headers","text":"If true, NGINX passes the incoming X-Forwarded-* headers to upstreams. Use this option when NGINX is behind another L7 proxy / load balancer that is setting these headers. If false, NGINX ignores incoming X-Forwarded-* headers, filling them with the request information it sees. Use this option if NGINX is exposed directly to the internet, or it's behind a L3/packet-based load balancer that doesn't alter the source IP in the packets.","title":"use-forwarded-headers"},{"location":"user-guide/nginx-configuration/configmap/#enable-real-ip","text":"enable-real-ip enables the configuration of https://nginx.org/en/docs/http/ngx_http_realip_module.html . Specific attributes of the module can be configured further by using forwarded-for-header and proxy-real-ip-cidr settings.","title":"enable-real-ip"},{"location":"user-guide/nginx-configuration/configmap/#forwarded-for-header","text":"Sets the header field for identifying the originating IP address of a client. default: X-Forwarded-For","title":"forwarded-for-header"},{"location":"user-guide/nginx-configuration/configmap/#compute-full-forwarded-for","text":"Append the remote address to the X-Forwarded-For header instead of replacing it. When this option is enabled, the upstream application is responsible for extracting the client IP based on its own list of trusted proxies.","title":"compute-full-forwarded-for"},{"location":"user-guide/nginx-configuration/configmap/#proxy-add-original-uri-header","text":"Adds an X-Original-Uri header with the original request URI to the backend request","title":"proxy-add-original-uri-header"},{"location":"user-guide/nginx-configuration/configmap/#generate-request-id","text":"Ensures that X-Request-ID is defaulted to a random value, if no X-Request-ID is present in the request","title":"generate-request-id"},{"location":"user-guide/nginx-configuration/configmap/#enable-opentracing","text":"Enables the nginx Opentracing extension. default: is disabled References: https://github.com/opentracing-contrib/nginx-opentracing","title":"enable-opentracing"},{"location":"user-guide/nginx-configuration/configmap/#opentracing-operation-name","text":"Specifies a custom name for the server span. default: is empty For example, set to \"HTTP $request_method $uri\".","title":"opentracing-operation-name"},{"location":"user-guide/nginx-configuration/configmap/#opentracing-location-operation-name","text":"Specifies a custom name for the location span. default: is empty For example, set to \"HTTP $request_method $uri\".","title":"opentracing-location-operation-name"},{"location":"user-guide/nginx-configuration/configmap/#zipkin-collector-host","text":"Specifies the host to use when uploading traces. It must be a valid URL.","title":"zipkin-collector-host"},{"location":"user-guide/nginx-configuration/configmap/#zipkin-collector-port","text":"Specifies the port to use when uploading traces. default: 9411","title":"zipkin-collector-port"},{"location":"user-guide/nginx-configuration/configmap/#zipkin-service-name","text":"Specifies the service name to use for any traces created. default: nginx","title":"zipkin-service-name"},{"location":"user-guide/nginx-configuration/configmap/#zipkin-sample-rate","text":"Specifies sample rate for any traces created. default: 1.0","title":"zipkin-sample-rate"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-collector-host","text":"Specifies the host to use when uploading traces. It must be a valid URL.","title":"jaeger-collector-host"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-collector-port","text":"Specifies the port to use when uploading traces. default: 6831","title":"jaeger-collector-port"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-endpoint","text":"Specifies the endpoint to use when uploading traces to a collector. This takes priority over jaeger-collector-host if both are specified.","title":"jaeger-endpoint"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-service-name","text":"Specifies the service name to use for any traces created. default: nginx","title":"jaeger-service-name"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-propagation-format","text":"Specifies the traceparent/tracestate propagation format. default: jaeger","title":"jaeger-propagation-format"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-sampler-type","text":"Specifies the sampler to be used when sampling traces. The available samplers are: const, probabilistic, ratelimiting, remote. default: const","title":"jaeger-sampler-type"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-sampler-param","text":"Specifies the argument to be passed to the sampler constructor. Must be a number. For const this should be 0 to never sample and 1 to always sample. default: 1","title":"jaeger-sampler-param"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-sampler-host","text":"Specifies the custom remote sampler host to be passed to the sampler constructor. Must be a valid URL. Leave blank to use default value (localhost). default: http://127.0.0.1","title":"jaeger-sampler-host"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-sampler-port","text":"Specifies the custom remote sampler port to be passed to the sampler constructor. Must be a number. default: 5778","title":"jaeger-sampler-port"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-trace-context-header-name","text":"Specifies the header name used for passing trace context. default: uber-trace-id","title":"jaeger-trace-context-header-name"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-debug-header","text":"Specifies the header name used for force sampling. default: jaeger-debug-id","title":"jaeger-debug-header"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-baggage-header","text":"Specifies the header name used to submit baggage if there is no root span. default: jaeger-baggage","title":"jaeger-baggage-header"},{"location":"user-guide/nginx-configuration/configmap/#jaeger-tracer-baggage-header-prefix","text":"Specifies the header prefix used to propagate baggage. default: uberctx-","title":"jaeger-tracer-baggage-header-prefix"},{"location":"user-guide/nginx-configuration/configmap/#datadog-collector-host","text":"Specifies the datadog agent host to use when uploading traces. It must be a valid URL.","title":"datadog-collector-host"},{"location":"user-guide/nginx-configuration/configmap/#datadog-collector-port","text":"Specifies the port to use when uploading traces. default: 8126","title":"datadog-collector-port"},{"location":"user-guide/nginx-configuration/configmap/#datadog-service-name","text":"Specifies the service name to use for any traces created. default: nginx","title":"datadog-service-name"},{"location":"user-guide/nginx-configuration/configmap/#datadog-environment","text":"Specifies the environment this trace belongs to. default: prod","title":"datadog-environment"},{"location":"user-guide/nginx-configuration/configmap/#datadog-operation-name-override","text":"Overrides the operation name to use for any traces crated. default: nginx.handle","title":"datadog-operation-name-override"},{"location":"user-guide/nginx-configuration/configmap/#datadog-priority-sampling","text":"Specifies to use client-side sampling. If true disables client-side sampling (thus ignoring sample_rate ) and enables distributed priority sampling, where traces are sampled based on a combination of user-assigned priorities and configuration from the agent. default: true","title":"datadog-priority-sampling"},{"location":"user-guide/nginx-configuration/configmap/#datadog-sample-rate","text":"Specifies sample rate for any traces created. This is effective only when datadog-priority-sampling is false default: 1.0","title":"datadog-sample-rate"},{"location":"user-guide/nginx-configuration/configmap/#main-snippet","text":"Adds custom configuration to the main section of the nginx configuration.","title":"main-snippet"},{"location":"user-guide/nginx-configuration/configmap/#http-snippet","text":"Adds custom configuration to the http section of the nginx configuration.","title":"http-snippet"},{"location":"user-guide/nginx-configuration/configmap/#server-snippet","text":"Adds custom configuration to all the servers in the nginx configuration.","title":"server-snippet"},{"location":"user-guide/nginx-configuration/configmap/#stream-snippet","text":"Adds custom configuration to the stream section of the nginx configuration.","title":"stream-snippet"},{"location":"user-guide/nginx-configuration/configmap/#location-snippet","text":"Adds custom configuration to all the locations in the nginx configuration. You can not use this to add new locations that proxy to the Kubernetes pods, as the snippet does not have access to the Go template functions. If you want to add custom locations you will have to provide your own nginx.tmpl .","title":"location-snippet"},{"location":"user-guide/nginx-configuration/configmap/#custom-http-errors","text":"Enables which HTTP codes should be passed for processing with the error_page directive Setting at least one code also enables proxy_intercept_errors which are required to process error_page. Example usage: custom-http-errors: 404,415","title":"custom-http-errors"},{"location":"user-guide/nginx-configuration/configmap/#proxy-body-size","text":"Sets the maximum allowed size of the client request body. See NGINX client_max_body_size .","title":"proxy-body-size"},{"location":"user-guide/nginx-configuration/configmap/#proxy-connect-timeout","text":"Sets the timeout for establishing a connection with a proxied server . It should be noted that this timeout cannot usually exceed 75 seconds.","title":"proxy-connect-timeout"},{"location":"user-guide/nginx-configuration/configmap/#proxy-read-timeout","text":"Sets the timeout in seconds for reading a response from the proxied server . The timeout is set only between two successive read operations, not for the transmission of the whole response.","title":"proxy-read-timeout"},{"location":"user-guide/nginx-configuration/configmap/#proxy-send-timeout","text":"Sets the timeout in seconds for transmitting a request to the proxied server . The timeout is set only between two successive write operations, not for the transmission of the whole request.","title":"proxy-send-timeout"},{"location":"user-guide/nginx-configuration/configmap/#proxy-buffers-number","text":"Sets the number of the buffer used for reading the first part of the response received from the proxied server. This part usually contains a small response header.","title":"proxy-buffers-number"},{"location":"user-guide/nginx-configuration/configmap/#proxy-buffer-size","text":"Sets the size of the buffer used for reading the first part of the response received from the proxied server. This part usually contains a small response header.","title":"proxy-buffer-size"},{"location":"user-guide/nginx-configuration/configmap/#proxy-cookie-path","text":"Sets a text that should be changed in the path attribute of the \u201cSet-Cookie\u201d header fields of a proxied server response.","title":"proxy-cookie-path"},{"location":"user-guide/nginx-configuration/configmap/#proxy-cookie-domain","text":"Sets a text that should be changed in the domain attribute of the \u201cSet-Cookie\u201d header fields of a proxied server response.","title":"proxy-cookie-domain"},{"location":"user-guide/nginx-configuration/configmap/#proxy-next-upstream","text":"Specifies in which cases a request should be passed to the next server.","title":"proxy-next-upstream"},{"location":"user-guide/nginx-configuration/configmap/#proxy-next-upstream-timeout","text":"Limits the time in seconds during which a request can be passed to the next server.","title":"proxy-next-upstream-timeout"},{"location":"user-guide/nginx-configuration/configmap/#proxy-next-upstream-tries","text":"Limit the number of possible tries a request should be passed to the next server.","title":"proxy-next-upstream-tries"},{"location":"user-guide/nginx-configuration/configmap/#proxy-redirect-from","text":"Sets the original text that should be changed in the \"Location\" and \"Refresh\" header fields of a proxied server response. default: off References: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_redirect","title":"proxy-redirect-from"},{"location":"user-guide/nginx-configuration/configmap/#proxy-request-buffering","text":"Enables or disables buffering of a client request body .","title":"proxy-request-buffering"},{"location":"user-guide/nginx-configuration/configmap/#ssl-redirect","text":"Sets the global value of redirects (301) to HTTPS if the server has a TLS certificate (defined in an Ingress rule). default: \"true\"","title":"ssl-redirect"},{"location":"user-guide/nginx-configuration/configmap/#force-ssl-redirect","text":"Sets the global value of redirects (308) to HTTPS if the server has a default TLS certificate (defined in extra-args). default: \"false\"","title":"force-ssl-redirect"},{"location":"user-guide/nginx-configuration/configmap/#whitelist-source-range","text":"Sets the default whitelisted IPs for each server block. This can be overwritten by an annotation on an Ingress rule. See ngx_http_access_module .","title":"whitelist-source-range"},{"location":"user-guide/nginx-configuration/configmap/#skip-access-log-urls","text":"Sets a list of URLs that should not appear in the NGINX access log. This is useful with urls like /health or health-check that make \"complex\" reading the logs. default: is empty","title":"skip-access-log-urls"},{"location":"user-guide/nginx-configuration/configmap/#limit-rate","text":"Limits the rate of response transmission to a client. The rate is specified in bytes per second. The zero value disables rate limiting. The limit is set per a request, and so if a client simultaneously opens two connections, the overall rate will be twice as much as the specified limit. References: https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate","title":"limit-rate"},{"location":"user-guide/nginx-configuration/configmap/#limit-rate-after","text":"Sets the initial amount after which the further transmission of a response to a client will be rate limited.","title":"limit-rate-after"},{"location":"user-guide/nginx-configuration/configmap/#lua-shared-dicts","text":"Customize default Lua shared dictionaries or define more. You can use the following syntax to do so: lua-shared-dicts: \"<my dict name>: <my dict size>, [<my dict name>: <my dict size>], ...\" For example following will set default certificate_data dictionary to 100M and will introduce a new dictionary called my_custom_plugin : lua-shared-dicts: \"certificate_data: 100, my_custom_plugin: 5\" You can optionally set a size unit to allow for kilobyte-granularity. Allowed units are 'm' or 'k' (case-insensitive), and it defaults to MB if no unit is provided. Here is a similar example, but the my_custom_plugin dict is only 512KB. lua-shared-dicts: \"certificate_data: 100, my_custom_plugin: 512k\" References: https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate_after","title":"lua-shared-dicts"},{"location":"user-guide/nginx-configuration/configmap/#http-redirect-code","text":"Sets the HTTP status code to be used in redirects. Supported codes are 301 , 302 , 307 and 308 default: 308 Why the default code is 308? RFC 7238 was created to define the 308 (Permanent Redirect) status code that is similar to 301 (Moved Permanently) but it keeps the payload in the redirect. This is important if we send a redirect in methods like POST.","title":"http-redirect-code"},{"location":"user-guide/nginx-configuration/configmap/#proxy-buffering","text":"Enables or disables buffering of responses from the proxied server .","title":"proxy-buffering"},{"location":"user-guide/nginx-configuration/configmap/#limit-req-status-code","text":"Sets the status code to return in response to rejected requests . default: 503","title":"limit-req-status-code"},{"location":"user-guide/nginx-configuration/configmap/#limit-conn-status-code","text":"Sets the status code to return in response to rejected connections . default: 503","title":"limit-conn-status-code"},{"location":"user-guide/nginx-configuration/configmap/#enable-syslog","text":"Enable syslog feature for access log and error log. default: false","title":"enable-syslog"},{"location":"user-guide/nginx-configuration/configmap/#syslog-host","text":"Sets the address of syslog server. The address can be specified as a domain name or IP address.","title":"syslog-host"},{"location":"user-guide/nginx-configuration/configmap/#syslog-port","text":"Sets the port of syslog server. default: 514","title":"syslog-port"},{"location":"user-guide/nginx-configuration/configmap/#no-tls-redirect-locations","text":"A comma-separated list of locations on which http requests will never get redirected to their https counterpart. default: \"/.well-known/acme-challenge\"","title":"no-tls-redirect-locations"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-url","text":"A url to an existing service that provides authentication for all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-url . Locations that should not get authenticated can be listed using no-auth-locations See no-auth-locations . In addition, each service can be excluded from authentication via annotation enable-global-auth set to \"false\". default: \"\" References: https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/annotations.md#external-authentication","title":"global-auth-url"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-method","text":"A HTTP method to use for an existing service that provides authentication for all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-method . default: \"\"","title":"global-auth-method"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-signin","text":"Sets the location of the error page for an existing service that provides authentication for all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-signin . default: \"\"","title":"global-auth-signin"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-signin-redirect-param","text":"Sets the query parameter in the error page signin URL which contains the original URL of the request that failed authentication. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-signin-redirect-param . default: \"rd\"","title":"global-auth-signin-redirect-param"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-response-headers","text":"Sets the headers to pass to backend once authentication request completes. Applied to all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-response-headers . default: \"\"","title":"global-auth-response-headers"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-request-redirect","text":"Sets the X-Auth-Request-Redirect header value. Applied to all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-request-redirect . default: \"\"","title":"global-auth-request-redirect"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-snippet","text":"Sets a custom snippet to use with external authentication. Applied to all the locations. Similar to the Ingress rule annotation nginx.ingress.kubernetes.io/auth-request-redirect . default: \"\"","title":"global-auth-snippet"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-cache-key","text":"Enables caching for global auth requests. Specify a lookup key for auth responses, e.g. $remote_user$http_authorization .","title":"global-auth-cache-key"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-cache-duration","text":"Set a caching time for auth responses based on their response codes, e.g. 200 202 30m . See proxy_cache_valid for details. You may specify multiple, comma-separated values: 200 202 10m, 401 5m . defaults to 200 202 401 5m .","title":"global-auth-cache-duration"},{"location":"user-guide/nginx-configuration/configmap/#global-auth-always-set-cookie","text":"Always set a cookie returned by auth request. By default, the cookie will be set only if an upstream reports with the code 200, 201, 204, 206, 301, 302, 303, 304, 307, or 308. default: false","title":"global-auth-always-set-cookie"},{"location":"user-guide/nginx-configuration/configmap/#no-auth-locations","text":"A comma-separated list of locations that should not get authenticated. default: \"/.well-known/acme-challenge\"","title":"no-auth-locations"},{"location":"user-guide/nginx-configuration/configmap/#block-cidrs","text":"A comma-separated list of IP addresses (or subnets), request from which have to be blocked globally. References: https://nginx.org/en/docs/http/ngx_http_access_module.html#deny","title":"block-cidrs"},{"location":"user-guide/nginx-configuration/configmap/#block-user-agents","text":"A comma-separated list of User-Agent, request from which have to be blocked globally. It's possible to use here full strings and regular expressions. More details about valid patterns can be found at map Nginx directive documentation. References: https://nginx.org/en/docs/http/ngx_http_map_module.html#map","title":"block-user-agents"},{"location":"user-guide/nginx-configuration/configmap/#block-referers","text":"A comma-separated list of Referers, request from which have to be blocked globally. It's possible to use here full strings and regular expressions. More details about valid patterns can be found at map Nginx directive documentation. References: https://nginx.org/en/docs/http/ngx_http_map_module.html#map","title":"block-referers"},{"location":"user-guide/nginx-configuration/configmap/#proxy-ssl-location-only","text":"Set if proxy-ssl parameters should be applied only on locations and not on servers. default: is disabled","title":"proxy-ssl-location-only"},{"location":"user-guide/nginx-configuration/configmap/#default-type","text":"Sets the default MIME type of a response. default: text/html References: https://nginx.org/en/docs/http/ngx_http_core_module.html#default_type","title":"default-type"},{"location":"user-guide/nginx-configuration/configmap/#global-rate-limit","text":"global-rate-limit-status-code : configure HTTP status code to return when rejecting requests. Defaults to 429. Configure memcached client for Global Rate Limiting . global-rate-limit-memcached-host : IP/FQDN of memcached server to use. Required to enable Global Rate Limiting. global-rate-limit-memcached-port : port of memcached server to use. Defaults default memcached port of 11211 . global-rate-limit-memcached-connect-timeout : configure timeout for connect, send and receive operations. Unit is millisecond. Defaults to 50ms. global-rate-limit-memcached-max-idle-timeout : configure timeout for cleaning idle connections. Unit is millisecond. Defaults to 50ms. global-rate-limit-memcached-pool-size : configure number of max connections to keep alive. Make sure your memcached server can handle global-rate-limit-memcached-pool-size * worker-processes * <number of ingress-nginx replicas> simultaneous connections. These settings get used by lua-resty-global-throttle that ingress-nginx includes. Refer to the link to learn more about lua-resty-global-throttle .","title":"global-rate-limit"},{"location":"user-guide/nginx-configuration/configmap/#service-upstream","text":"Set if the service's Cluster IP and port should be used instead of a list of all endpoints. This can be overwritten by an annotation on an Ingress rule. default: \"false\"","title":"service-upstream"},{"location":"user-guide/nginx-configuration/configmap/#ssl-reject-handshake","text":"Set to reject SSL handshake to an unknown virtualhost. This parameter helps to mitigate the fingerprinting using default certificate of ingress. default: \"false\" References: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_reject_handshake","title":"ssl-reject-handshake"},{"location":"user-guide/nginx-configuration/configmap/#debug-connections","text":"Enables debugging log for selected client connections. default: \"\" References: http://nginx.org/en/docs/ngx_core_module.html#debug_connection","title":"debug-connections"},{"location":"user-guide/nginx-configuration/custom-template/","text":"Custom NGINX template \u00b6 The NGINX template is located in the file /etc/nginx/template/nginx.tmpl . Using a Volume it is possible to use a custom template. This includes using a Configmap as source of the template volumeMounts : - mountPath : /etc/nginx/template name : nginx-template-volume readOnly : true volumes : - name : nginx-template-volume configMap : name : nginx-template items : - key : nginx.tmpl path : nginx.tmpl Please note the template is tied to the Go code. Do not change names in the variable $cfg . For more information about the template syntax please check the Go template package . In addition to the built-in functions provided by the Go package the following functions are also available: empty: returns true if the specified parameter (string) is empty contains: strings.Contains hasPrefix: strings.HasPrefix hasSuffix: strings.HasSuffix toUpper: strings.ToUpper toLower: strings.ToLower split: strings.Split quote: wraps a string in double quotes buildLocation: helps to build the NGINX Location section in each server buildProxyPass: builds the reverse proxy configuration buildRateLimit: helps to build a limit zone inside a location if contains a rate limit annotation TODO: buildAuthLocation: buildAuthResponseHeaders: buildResolvers: buildDenyVariable: buildUpstreamName: buildForwardedFor: buildAuthSignURL: buildNextUpstream: filterRateLimits: formatIP: getenv: getIngressInformation: serverConfig: isLocationAllowed: isValidClientBodyBufferSize:","title":"Custom NGINX template"},{"location":"user-guide/nginx-configuration/custom-template/#custom-nginx-template","text":"The NGINX template is located in the file /etc/nginx/template/nginx.tmpl . Using a Volume it is possible to use a custom template. This includes using a Configmap as source of the template volumeMounts : - mountPath : /etc/nginx/template name : nginx-template-volume readOnly : true volumes : - name : nginx-template-volume configMap : name : nginx-template items : - key : nginx.tmpl path : nginx.tmpl Please note the template is tied to the Go code. Do not change names in the variable $cfg . For more information about the template syntax please check the Go template package . In addition to the built-in functions provided by the Go package the following functions are also available: empty: returns true if the specified parameter (string) is empty contains: strings.Contains hasPrefix: strings.HasPrefix hasSuffix: strings.HasSuffix toUpper: strings.ToUpper toLower: strings.ToLower split: strings.Split quote: wraps a string in double quotes buildLocation: helps to build the NGINX Location section in each server buildProxyPass: builds the reverse proxy configuration buildRateLimit: helps to build a limit zone inside a location if contains a rate limit annotation TODO: buildAuthLocation: buildAuthResponseHeaders: buildResolvers: buildDenyVariable: buildUpstreamName: buildForwardedFor: buildAuthSignURL: buildNextUpstream: filterRateLimits: formatIP: getenv: getIngressInformation: serverConfig: isLocationAllowed: isValidClientBodyBufferSize:","title":"Custom NGINX template"},{"location":"user-guide/nginx-configuration/log-format/","text":"Log format \u00b6 The default configuration uses a custom logging format to add additional information about upstreams, response time and status. log_format upstreaminfo '$remote_addr - $remote_user [$time_local] \"$request\" ' '$status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" ' '$request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr ' '$upstream_response_length $upstream_response_time $upstream_status $req_id'; Placeholder Description $proxy_protocol_addr remote address if proxy protocol is enabled $remote_addr the source IP address of the client $remote_user user name supplied with the Basic authentication $time_local local time in the Common Log Format $request full original request line $status response status $body_bytes_sent number of bytes sent to a client, not counting the response header $http_referer value of the Referer header $http_user_agent value of User-Agent header $request_length request length (including request line, header, and request body) $request_time time elapsed since the first bytes were read from the client $proxy_upstream_name name of the upstream. The format is upstream-<namespace>-<service name>-<service port> $proxy_alternative_upstream_name name of the alternative upstream. The format is upstream-<namespace>-<service name>-<service port> $upstream_addr the IP address and port (or the path to the domain socket) of the upstream server. If several servers were contacted during request processing, their addresses are separated by commas. $upstream_response_length the length of the response obtained from the upstream server $upstream_response_time time spent on receiving the response from the upstream server as seconds with millisecond resolution $upstream_status status code of the response obtained from the upstream server $req_id value of the X-Request-ID HTTP header. If the header is not set, a randomly generated ID. Additional available variables: Placeholder Description $namespace namespace of the ingress $ingress_name name of the ingress $service_name name of the service $service_port port of the service Sources: Upstream variables Embedded variables","title":"Log format"},{"location":"user-guide/nginx-configuration/log-format/#log-format","text":"The default configuration uses a custom logging format to add additional information about upstreams, response time and status. log_format upstreaminfo '$remote_addr - $remote_user [$time_local] \"$request\" ' '$status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" ' '$request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr ' '$upstream_response_length $upstream_response_time $upstream_status $req_id'; Placeholder Description $proxy_protocol_addr remote address if proxy protocol is enabled $remote_addr the source IP address of the client $remote_user user name supplied with the Basic authentication $time_local local time in the Common Log Format $request full original request line $status response status $body_bytes_sent number of bytes sent to a client, not counting the response header $http_referer value of the Referer header $http_user_agent value of User-Agent header $request_length request length (including request line, header, and request body) $request_time time elapsed since the first bytes were read from the client $proxy_upstream_name name of the upstream. The format is upstream-<namespace>-<service name>-<service port> $proxy_alternative_upstream_name name of the alternative upstream. The format is upstream-<namespace>-<service name>-<service port> $upstream_addr the IP address and port (or the path to the domain socket) of the upstream server. If several servers were contacted during request processing, their addresses are separated by commas. $upstream_response_length the length of the response obtained from the upstream server $upstream_response_time time spent on receiving the response from the upstream server as seconds with millisecond resolution $upstream_status status code of the response obtained from the upstream server $req_id value of the X-Request-ID HTTP header. If the header is not set, a randomly generated ID. Additional available variables: Placeholder Description $namespace namespace of the ingress $ingress_name name of the ingress $service_name name of the service $service_port port of the service Sources: Upstream variables Embedded variables","title":"Log format"},{"location":"user-guide/third-party-addons/modsecurity/","text":"ModSecurity Web Application Firewall \u00b6 ModSecurity is an open source, cross platform web application firewall (WAF) engine for Apache, IIS and Nginx that is developed by Trustwave's SpiderLabs. It has a robust event-based programming language which provides protection from a range of attacks against web applications and allows for HTTP traffic monitoring, logging and real-time analysis - https://www.modsecurity.org The ModSecurity-nginx connector is the connection point between NGINX and libmodsecurity (ModSecurity v3). The default ModSecurity configuration file is located in /etc/nginx/modsecurity/modsecurity.conf . This is the only file located in this directory and contains the default recommended configuration. Using a volume we can replace this file with the desired configuration. To enable the ModSecurity feature we need to specify enable-modsecurity: \"true\" in the configuration configmap. Note: the default configuration use detection only, because that minimizes the chances of post-installation disruption. Due to the value of the setting SecAuditLogType=Concurrent the ModSecurity log is stored in multiple files inside the directory /var/log/audit . The default Serial value in SecAuditLogType can impact performance. The OWASP ModSecurity Core Rule Set (CRS) is a set of generic attack detection rules for use with ModSecurity or compatible web application firewalls. The CRS aims to protect web applications from a wide range of attacks, including the OWASP Top Ten, with a minimum of false alerts. The directory /etc/nginx/owasp-modsecurity-crs contains the OWASP ModSecurity Core Rule Set repository . Using enable-owasp-modsecurity-crs: \"true\" we enable the use of the rules.","title":"ModSecurity Web Application Firewall"},{"location":"user-guide/third-party-addons/modsecurity/#modsecurity-web-application-firewall","text":"ModSecurity is an open source, cross platform web application firewall (WAF) engine for Apache, IIS and Nginx that is developed by Trustwave's SpiderLabs. It has a robust event-based programming language which provides protection from a range of attacks against web applications and allows for HTTP traffic monitoring, logging and real-time analysis - https://www.modsecurity.org The ModSecurity-nginx connector is the connection point between NGINX and libmodsecurity (ModSecurity v3). The default ModSecurity configuration file is located in /etc/nginx/modsecurity/modsecurity.conf . This is the only file located in this directory and contains the default recommended configuration. Using a volume we can replace this file with the desired configuration. To enable the ModSecurity feature we need to specify enable-modsecurity: \"true\" in the configuration configmap. Note: the default configuration use detection only, because that minimizes the chances of post-installation disruption. Due to the value of the setting SecAuditLogType=Concurrent the ModSecurity log is stored in multiple files inside the directory /var/log/audit . The default Serial value in SecAuditLogType can impact performance. The OWASP ModSecurity Core Rule Set (CRS) is a set of generic attack detection rules for use with ModSecurity or compatible web application firewalls. The CRS aims to protect web applications from a wide range of attacks, including the OWASP Top Ten, with a minimum of false alerts. The directory /etc/nginx/owasp-modsecurity-crs contains the OWASP ModSecurity Core Rule Set repository . Using enable-owasp-modsecurity-crs: \"true\" we enable the use of the rules.","title":"ModSecurity Web Application Firewall"},{"location":"user-guide/third-party-addons/opentracing/","text":"OpenTracing \u00b6 Enables requests served by NGINX for distributed tracing via The OpenTracing Project. Using the third party module opentracing-contrib/nginx-opentracing the NGINX ingress controller can configure NGINX to enable OpenTracing instrumentation. By default this feature is disabled. Usage \u00b6 To enable the instrumentation we must enable OpenTracing in the configuration ConfigMap: data: enable-opentracing: \"true\" To enable or disable instrumentation for a single Ingress, use the enable-opentracing annotation: kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/enable-opentracing: \"true\" We must also set the host to use when uploading traces: zipkin-collector-host: zipkin.default.svc.cluster.local jaeger-collector-host: jaeger-agent.default.svc.cluster.local datadog-collector-host: datadog-agent.default.svc.cluster.local NOTE: While the option is called jaeger-collector-host , you will need to point this to a jaeger-agent , and not the jaeger-collector component. Alternatively, you can set jaeger-endpoint and specify the full endpoint for uploading traces. This will use TCP and should be used for a collector rather than an agent. Next you will need to deploy a distributed tracing system which uses OpenTracing. Zipkin and Jaeger and Datadog have been tested. Other optional configuration options: # specifies the name to use for the server span opentracing-operation-name # specifies specifies the name to use for the location span opentracing-location-operation-name # sets whether or not to trust incoming tracing spans opentracing-trust-incoming-span # specifies the port to use when uploading traces, Default: 9411 zipkin-collector-port # specifies the service name to use for any traces created, Default: nginx zipkin-service-name # specifies sample rate for any traces created, Default: 1.0 zipkin-sample-rate # specifies the port to use when uploading traces, Default: 6831 jaeger-collector-port # specifies the endpoint to use when uploading traces to a collector instead of an agent jaeger-endpoint # specifies the service name to use for any traces created, Default: nginx jaeger-service-name # specifies the traceparent/tracestate propagation format jaeger-propagation-format # specifies the sampler to be used when sampling traces. # The available samplers are: const, probabilistic, ratelimiting, remote, Default: const jaeger-sampler-type # specifies the argument to be passed to the sampler constructor, Default: 1 jaeger-sampler-param # Specifies the custom remote sampler host to be passed to the sampler constructor. Must be a valid URL. # Default: http://127.0.0.1 jaeger-sampler-host # Specifies the custom remote sampler port to be passed to the sampler constructor. Must be a number. Default: 5778 jaeger-sampler-port # Specifies the header name used for passing trace context. Must be a string. Default: uber-trace-id jaeger-trace-context-header-name # Specifies the header name used for force sampling. Must be a string. Default: jaeger-debug-id jaeger-debug-header # Specifies the header name used to submit baggage if there is no root span. Must be a string. Default: jaeger-baggage jaeger-baggage-header # Specifies the header prefix used to propagate baggage. Must be a string. Default: uberctx- jaeger-tracer-baggage-header-prefix # specifies the port to use when uploading traces, Default 8126 datadog-collector-port # specifies the service name to use for any traces created, Default: nginx datadog-service-name # specifies the environment this trace belongs to, Default: prod datadog-environment # specifies the operation name to use for any traces collected, Default: nginx.handle datadog-operation-name-override # Specifies to use client-side sampling for distributed priority sampling and ignore sample rate, Default: true datadog-priority-sampling # specifies sample rate for any traces created, Default: 1.0 datadog-sample-rate All these options (including host) allow environment variables, such as $HOSTNAME or $HOST_IP . In the case of Jaeger, if you have a Jaeger agent running on each machine in your cluster, you can use something like $HOST_IP (which can be 'mounted' with the status.hostIP fieldpath, as described here ) to make sure traces will be sent to the local agent. Note that you can also set whether to trust incoming spans (global default is true) per-location using annotations like the following: kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/opentracing-trust-incoming-span: \"true\" Examples \u00b6 The following examples show how to deploy and test different distributed tracing systems. These example can be performed using Minikube. Zipkin \u00b6 In the rnburn/zipkin-date-server GitHub repository is an example of a dockerized date service. To install the example and Zipkin collector run: kubectl create -f https://raw.githubusercontent.com/rnburn/zipkin-date-server/master/kubernetes/zipkin.yaml kubectl create -f https://raw.githubusercontent.com/rnburn/zipkin-date-server/master/kubernetes/deployment.yaml Also we need to configure the Ingress-NGINX controller ConfigMap with the required values: $ echo ' apiVersion: v1 kind: ConfigMap data: enable-opentracing: \"true\" zipkin-collector-host: zipkin.default.svc.cluster.local metadata: name: ingress-nginx-controller namespace: kube-system ' | kubectl replace -f - In the Zipkin interface we can see the details: Jaeger \u00b6 Enable Ingress addon in Minikube: $ minikube addons enable ingress Add Minikube IP to /etc/hosts: $ echo \"$(minikube ip) example.com\" | sudo tee -a /etc/hosts Apply a basic Service and Ingress Resource: # Create Echoheaders Deployment $ kubectl run echoheaders --image=registry.k8s.io/echoserver:1.4 --replicas=1 --port=8080 # Expose as a Cluster-IP $ kubectl expose deployment echoheaders --port=80 --target-port=8080 --name=echoheaders-x # Apply the Ingress Resource $ echo ' apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: echo-ingress spec: ingressClassName: nginx rules: - host: example.com http: paths: - path: /echo pathType: Prefix backend: service: name: echoheaders-x port: number: 80 ' | kubectl apply -f - Enable OpenTracing and set the jaeger-collector-host: $ echo ' apiVersion: v1 kind: ConfigMap data: enable-opentracing: \"true\" jaeger-collector-host: jaeger-agent.default.svc.cluster.local metadata: name: ingress-nginx-controller namespace: kube-system ' | kubectl replace -f - Apply the Jaeger All-In-One Template: $ kubectl apply -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml Make a few requests to the Service: $ curl example.com/echo -d \"meow\" CLIENT VALUES: client_address=172.17.0.5 command=POST real path=/echo query=nil request_version=1.1 request_uri=http://example.com:8080/echo SERVER VALUES: server_version=nginx: 1.10.0 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=close content-length=4 content-type=application/x-www-form-urlencoded host=example.com user-agent=curl/7.54.0 x-forwarded-for=192.168.99.1 x-forwarded-host=example.com x-forwarded-port=80 x-forwarded-proto=http x-original-uri=/echo x-real-ip=192.168.99.1 x-scheme=http BODY: meow View the Jaeger UI: $ minikube service jaeger-query --url http://192.168.99.100:30183 In the Jaeger interface we can see the details:","title":"OpenTracing"},{"location":"user-guide/third-party-addons/opentracing/#opentracing","text":"Enables requests served by NGINX for distributed tracing via The OpenTracing Project. Using the third party module opentracing-contrib/nginx-opentracing the NGINX ingress controller can configure NGINX to enable OpenTracing instrumentation. By default this feature is disabled.","title":"OpenTracing"},{"location":"user-guide/third-party-addons/opentracing/#usage","text":"To enable the instrumentation we must enable OpenTracing in the configuration ConfigMap: data: enable-opentracing: \"true\" To enable or disable instrumentation for a single Ingress, use the enable-opentracing annotation: kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/enable-opentracing: \"true\" We must also set the host to use when uploading traces: zipkin-collector-host: zipkin.default.svc.cluster.local jaeger-collector-host: jaeger-agent.default.svc.cluster.local datadog-collector-host: datadog-agent.default.svc.cluster.local NOTE: While the option is called jaeger-collector-host , you will need to point this to a jaeger-agent , and not the jaeger-collector component. Alternatively, you can set jaeger-endpoint and specify the full endpoint for uploading traces. This will use TCP and should be used for a collector rather than an agent. Next you will need to deploy a distributed tracing system which uses OpenTracing. Zipkin and Jaeger and Datadog have been tested. Other optional configuration options: # specifies the name to use for the server span opentracing-operation-name # specifies specifies the name to use for the location span opentracing-location-operation-name # sets whether or not to trust incoming tracing spans opentracing-trust-incoming-span # specifies the port to use when uploading traces, Default: 9411 zipkin-collector-port # specifies the service name to use for any traces created, Default: nginx zipkin-service-name # specifies sample rate for any traces created, Default: 1.0 zipkin-sample-rate # specifies the port to use when uploading traces, Default: 6831 jaeger-collector-port # specifies the endpoint to use when uploading traces to a collector instead of an agent jaeger-endpoint # specifies the service name to use for any traces created, Default: nginx jaeger-service-name # specifies the traceparent/tracestate propagation format jaeger-propagation-format # specifies the sampler to be used when sampling traces. # The available samplers are: const, probabilistic, ratelimiting, remote, Default: const jaeger-sampler-type # specifies the argument to be passed to the sampler constructor, Default: 1 jaeger-sampler-param # Specifies the custom remote sampler host to be passed to the sampler constructor. Must be a valid URL. # Default: http://127.0.0.1 jaeger-sampler-host # Specifies the custom remote sampler port to be passed to the sampler constructor. Must be a number. Default: 5778 jaeger-sampler-port # Specifies the header name used for passing trace context. Must be a string. Default: uber-trace-id jaeger-trace-context-header-name # Specifies the header name used for force sampling. Must be a string. Default: jaeger-debug-id jaeger-debug-header # Specifies the header name used to submit baggage if there is no root span. Must be a string. Default: jaeger-baggage jaeger-baggage-header # Specifies the header prefix used to propagate baggage. Must be a string. Default: uberctx- jaeger-tracer-baggage-header-prefix # specifies the port to use when uploading traces, Default 8126 datadog-collector-port # specifies the service name to use for any traces created, Default: nginx datadog-service-name # specifies the environment this trace belongs to, Default: prod datadog-environment # specifies the operation name to use for any traces collected, Default: nginx.handle datadog-operation-name-override # Specifies to use client-side sampling for distributed priority sampling and ignore sample rate, Default: true datadog-priority-sampling # specifies sample rate for any traces created, Default: 1.0 datadog-sample-rate All these options (including host) allow environment variables, such as $HOSTNAME or $HOST_IP . In the case of Jaeger, if you have a Jaeger agent running on each machine in your cluster, you can use something like $HOST_IP (which can be 'mounted' with the status.hostIP fieldpath, as described here ) to make sure traces will be sent to the local agent. Note that you can also set whether to trust incoming spans (global default is true) per-location using annotations like the following: kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/opentracing-trust-incoming-span: \"true\"","title":"Usage"},{"location":"user-guide/third-party-addons/opentracing/#examples","text":"The following examples show how to deploy and test different distributed tracing systems. These example can be performed using Minikube.","title":"Examples"},{"location":"user-guide/third-party-addons/opentracing/#zipkin","text":"In the rnburn/zipkin-date-server GitHub repository is an example of a dockerized date service. To install the example and Zipkin collector run: kubectl create -f https://raw.githubusercontent.com/rnburn/zipkin-date-server/master/kubernetes/zipkin.yaml kubectl create -f https://raw.githubusercontent.com/rnburn/zipkin-date-server/master/kubernetes/deployment.yaml Also we need to configure the Ingress-NGINX controller ConfigMap with the required values: $ echo ' apiVersion: v1 kind: ConfigMap data: enable-opentracing: \"true\" zipkin-collector-host: zipkin.default.svc.cluster.local metadata: name: ingress-nginx-controller namespace: kube-system ' | kubectl replace -f - In the Zipkin interface we can see the details:","title":"Zipkin"},{"location":"user-guide/third-party-addons/opentracing/#jaeger","text":"Enable Ingress addon in Minikube: $ minikube addons enable ingress Add Minikube IP to /etc/hosts: $ echo \"$(minikube ip) example.com\" | sudo tee -a /etc/hosts Apply a basic Service and Ingress Resource: # Create Echoheaders Deployment $ kubectl run echoheaders --image=registry.k8s.io/echoserver:1.4 --replicas=1 --port=8080 # Expose as a Cluster-IP $ kubectl expose deployment echoheaders --port=80 --target-port=8080 --name=echoheaders-x # Apply the Ingress Resource $ echo ' apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: echo-ingress spec: ingressClassName: nginx rules: - host: example.com http: paths: - path: /echo pathType: Prefix backend: service: name: echoheaders-x port: number: 80 ' | kubectl apply -f - Enable OpenTracing and set the jaeger-collector-host: $ echo ' apiVersion: v1 kind: ConfigMap data: enable-opentracing: \"true\" jaeger-collector-host: jaeger-agent.default.svc.cluster.local metadata: name: ingress-nginx-controller namespace: kube-system ' | kubectl replace -f - Apply the Jaeger All-In-One Template: $ kubectl apply -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml Make a few requests to the Service: $ curl example.com/echo -d \"meow\" CLIENT VALUES: client_address=172.17.0.5 command=POST real path=/echo query=nil request_version=1.1 request_uri=http://example.com:8080/echo SERVER VALUES: server_version=nginx: 1.10.0 - lua: 10001 HEADERS RECEIVED: accept=*/* connection=close content-length=4 content-type=application/x-www-form-urlencoded host=example.com user-agent=curl/7.54.0 x-forwarded-for=192.168.99.1 x-forwarded-host=example.com x-forwarded-port=80 x-forwarded-proto=http x-original-uri=/echo x-real-ip=192.168.99.1 x-scheme=http BODY: meow View the Jaeger UI: $ minikube service jaeger-query --url http://192.168.99.100:30183 In the Jaeger interface we can see the details:","title":"Jaeger"}]}
\ No newline at end of file
diff --git a/sitemap.xml b/sitemap.xml
index da864f9d1..d208198c3 100644
--- a/sitemap.xml
+++ b/sitemap.xml
@@ -1,207 +1,207 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><url>
      <loc>https://kubernetes.github.io/ingress-nginx/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/how-it-works/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/troubleshooting/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/kubectl-plugin/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/deploy/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/deploy/baremetal/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/deploy/rbac/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/deploy/upgrade/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/deploy/hardening-guide/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/basic-usage/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/custom-template/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/log-format/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/cli-arguments/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/custom-errors/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/default-backend/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/fcgi-services/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/ingress-path-matching/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/external-articles/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/tls/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/modsecurity/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/opentracing/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/PREREQUISITES/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/auth/basic/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/auth/client-certs/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/auth/external-auth/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/auth/oauth-external-auth/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/customization/configuration-snippets/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/customization/custom-configuration/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/customization/custom-errors/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/customization/custom-headers/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/customization/external-auth-headers/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/customization/ssl-dh-param/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/customization/sysctl/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/docker-registry/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/grpc/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/multi-tls/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/rewrite/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/static-ip/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/tls-termination/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/examples/psp/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/developer-guide/getting-started/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url><url>
      <loc>https://kubernetes.github.io/ingress-nginx/developer-guide/code-overview/</loc>
-     <lastmod>2022-11-10</lastmod>
+     <lastmod>2022-11-15</lastmod>
      <changefreq>daily</changefreq>
     </url>
 </urlset>
\ No newline at end of file
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index 92393866e..b3eb5d746 100644
Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ