This section demonstrates how to use the Layer 2 configuration mode of MetalLB together with the NGINX Ingress controller in a Kubernetes cluster that has publicly accessible nodes. In this mode, one node attracts all -the traffic for the ingress-nginx Service IP. See Traffic policies for more details.
ingress-nginx
Note
MetalLB can be deployed either with a simple Kubernetes manifest or with Helm. The rest of this example assumes MetalLB was deployed following the Installation instructions.
MetalLB requires a pool of IP addresses in order to be able to take ownership of the ingress-nginx Service. This pool -can be defined in a ConfigMap named config located in the same namespace as the MetalLB controller. This pool of IPs must be dedicated to MetalLB's use, you can't reuse the Kubernetes node IPs or IPs handed out by a DHCP server.
config
MetalLB requires a pool of IP addresses in order to be able to take ownership of the ingress-nginx Service. This pool +can be defined in a ConfigMap named config located in the same namespace as the MetalLB controller. This pool of IPs must be dedicated to MetalLB's use, you can't reuse the Kubernetes node IPs or IPs handed out by a DHCP server.
Example
Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is <None>)
$ kubectl get node +$ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 - +
$ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 -
After creating the following ConfigMap, MetalLB takes ownership of one of the IP addresses in the pool and updates -the loadBalancer IP field of the ingress-nginx Service accordingly.
apiVersion: v1 +the loadBalancer IP field of the ingress-nginx Service accordingly. +apiVersion: v1 kind: ConfigMap metadata: namespace: metallb-system @@ -1303,26 +1303,26 @@ the loadBalancer IP field of the ingress-nginx protocol: layer2 addresses: - 203.0.113.10-203.0.113.15 - +
apiVersion: v1 kind: ConfigMap metadata: namespace: metallb-system @@ -1303,26 +1303,26 @@ the loadBalancer IP field of the ingress-nginx protocol: layer2 addresses: - 203.0.113.10-203.0.113.15 -
ingress-nginx protocol: layer2 addresses: - 203.0.113.10-203.0.113.15 -
$ kubectl -n ingress-nginx get svc +$ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 <none> 80/TCP ingress-nginx LoadBalancer 10.0.220.217 203.0.113.10 80:30100/TCP,443:30101/TCP - +
$ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 <none> 80/TCP ingress-nginx LoadBalancer 10.0.220.217 203.0.113.10 80:30100/TCP,443:30101/TCP -
As soon as MetalLB sets the external IP address of the ingress-nginx LoadBalancer Service, the corresponding entries +
As soon as MetalLB sets the external IP address of the ingress-nginx LoadBalancer Service, the corresponding entries are created in the iptables NAT table and the node with the selected IP address starts responding to HTTP requests on the ports configured in the LoadBalancer Service:
$ curl -D- http://203.0.113.3 -H 'Host: myapp.example.com' +$ curl -D- http://203.0.113.3 -H 'Host: myapp.example.com' HTTP/1.1 200 OK Server: nginx/1.15.2 - +
$ curl -D- http://203.0.113.3 -H 'Host: myapp.example.com' HTTP/1.1 200 OK Server: nginx/1.15.2 -
Tip
In order to preserve the source IP address in HTTP requests sent to NGINX, it is necessary to use the Local +
Local
In order to preserve the source IP address in HTTP requests sent to NGINX, it is necessary to use the Local traffic policy. Traffic policies are described in more details in Traffic policies as well as in the next section.
Info
A Service of type NodePort exposes, via the kube-proxy component, the same unprivileged port (default: +
NodePort
kube-proxy
A Service of type NodePort exposes, via the kube-proxy component, the same unprivileged port (default: 30000-32767) on every Kubernetes node, masters included. For more information, see Services.
In this configuration, the NGINX container remains isolated from the host network. As a result, it can safely bind to any port, including the standard HTTP ports 80 and 443. However, due to the container namespace isolation, a client located outside the cluster network (e.g. on the public internet) is not able to access Ingress hosts directly on ports -80 and 443. Instead, the external client must append the NodePort allocated to the ingress-nginx Service to HTTP +80 and 443. Instead, the external client must append the NodePort allocated to the ingress-nginx Service to HTTP requests.
Given the NodePort 30100 allocated to the ingress-nginx Service
30100
$ kubectl -n ingress-nginx get svc +Given the NodePort 30100 allocated to the ingress-nginx Service +$ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 80/TCP ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP - +
$ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 80/TCP ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP -
and a Kubernetes node with the public IP address 203.0.113.2 (the external IP is added as an example, in most +
203.0.113.2
and a Kubernetes node with the public IP address 203.0.113.2 (the external IP is added as an example, in most bare-metal environments this value is <None>)
a client would reach an Ingress with host: myapp.example.com at http://myapp.example.com:30100, where the +
host: myapp.example.com
http://myapp.example.com:30100
a client would reach an Ingress with host: myapp.example.com at http://myapp.example.com:30100, where the myapp.example.com subdomain resolves to the 203.0.113.2 IP address.
Impact on the host system
While it may sound tempting to reconfigure the NodePort range using the --service-node-port-range API server flag +
--service-node-port-range
While it may sound tempting to reconfigure the NodePort range using the --service-node-port-range API server flag to include unprivileged ports and be able to expose ports 80 and 443, doing so may result in unexpected issues including (but not limited to) the use of ports otherwise reserved to system daemons and the necessity to grant -kube-proxy privileges it may otherwise not require.
This practice is therefore discouraged. See the other approaches proposed in this page for alternatives.
This approach has a few other limitations one ought to be aware of:
Services of type NodePort perform source address translation by default. This means the source IP of a HTTP request is always the IP address of the Kubernetes node that received the request from the perspective of NGINX.
The recommended way to preserve the source IP in a NodePort setup is to set the value of the externalTrafficPolicy -field of the ingress-nginx Service spec to Local (example).
externalTrafficPolicy
The recommended way to preserve the source IP in a NodePort setup is to set the value of the externalTrafficPolicy +field of the ingress-nginx Service spec to Local (example).
Warning
This setting effectively drops packets sent to Kubernetes nodes which are not running any instance of the NGINX @@ -1388,40 +1388,40 @@ the NGINX Ingress controller should be scheduled or not scheduled.
In a Kubernetes cluster composed of 3 nodes (the external IP is added as an example, in most bare-metal environments this value is <None>)
with a nginx-ingress-controller Deployment composed of 2 replicas
nginx-ingress-controller
$ kubectl -n ingress-nginx get pod -o wide +with a nginx-ingress-controller Deployment composed of 2 replicas +$ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 nginx-ingress-controller-cf9ff8c96-8vvf8 1/1 Running 172.17.0.3 host-3 nginx-ingress-controller-cf9ff8c96-pxsds 1/1 Running 172.17.1.4 host-2 - +
$ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 nginx-ingress-controller-cf9ff8c96-8vvf8 1/1 Running 172.17.0.3 host-3 nginx-ingress-controller-cf9ff8c96-pxsds 1/1 Running 172.17.1.4 host-2 -
Requests sent to host-2 and host-3 would be forwarded to NGINX and original client's IP would be preserved, -while requests to host-1 would get dropped because there is no NGINX replica running on that node.
host-2
host-3
host-1
Requests sent to host-2 and host-3 would be forwarded to NGINX and original client's IP would be preserved, +while requests to host-1 would get dropped because there is no NGINX replica running on that node.
Because NodePort Services do not get a LoadBalancerIP assigned by definition, the NGINX Ingress controller does not update the status of Ingress objects it manages.
$ kubectl get ingress +$ kubectl get ingress NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 80 - +
$ kubectl get ingress NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 80 -
Despite the fact there is no load balancer providing a public IP address to the NGINX Ingress controller, it is possible -to force the status update of all managed Ingress objects by setting the externalIPs field of the ingress-nginx +to force the status update of all managed Ingress objects by setting the externalIPs field of the ingress-nginx Service.
externalIPs
There is more to setting externalIPs than just enabling the NGINX Ingress controller to update the status of +
There is more to setting externalIPs than just enabling the NGINX Ingress controller to update the status of Ingress objects. Please read about this option in the Services page of official Kubernetes documentation as well as the section about External IPs in this document for more information.
one could edit the ingress-nginx Service and add the following field to the object spec
spec: +one could edit the ingress-nginx Service and add the following field to the object spec +spec: externalIPs: - 203.0.113.1 - 203.0.113.2 - 203.0.113.3 - +
spec: externalIPs: - 203.0.113.1 - 203.0.113.2 - 203.0.113.3 -
which would in turn be reflected on Ingress objects as follows:
$ kubectl get ingress -o wide +$ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.1,203.0.113.2,203.0.113.3 80 - +
$ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.1,203.0.113.2,203.0.113.3 80 -
Redirects generated by NGINX, for instance HTTP to HTTPS or domain to www.domain, are generated without +
domain
www.domain
Redirects generated by NGINX, for instance HTTP to HTTPS or domain to www.domain, are generated without NodePort:
$ curl -D- http://myapp.example.com:30100` +$ curl -D- http://myapp.example.com:30100` HTTP/1.1 308 Permanent Redirect Server: nginx/1.15.2 Location: https://myapp.example.com/ #-> missing NodePort in HTTPS redirect - +
$ curl -D- http://myapp.example.com:30100` HTTP/1.1 308 Permanent Redirect Server: nginx/1.15.2 Location: https://myapp.example.com/ #-> missing NodePort in HTTPS redirect -
In a setup where there is no external load balancer available but using NodePorts is not an option, one can configure -ingress-nginx Pods to use the network of the host they run on instead of a dedicated network namespace. The benefit of +ingress-nginx Pods to use the network of the host they run on instead of a dedicated network namespace. The benefit of this approach is that the NGINX Ingress controller can bind ports 80 and 443 directly to Kubernetes nodes' network interfaces, without the extra network translation imposed by NodePort Services.
This approach does not leverage any Service object to expose the NGINX Ingress controller. If the ingress-nginx +
This approach does not leverage any Service object to expose the NGINX Ingress controller. If the ingress-nginx Service exists in the target cluster, it is recommended to delete it.
This can be achieved by enabling the hostNetwork option in the Pods' spec.
hostNetwork
template: +This can be achieved by enabling the hostNetwork option in the Pods' spec. +template: spec: hostNetwork: true - +
template: spec: hostNetwork: true -
Security considerations
Consider this nginx-ingress-controller Deployment composed of 2 replicas, NGINX Pods inherit from the IP address +
Consider this nginx-ingress-controller Deployment composed of 2 replicas, NGINX Pods inherit from the IP address of their host instead of an internal Pod IP.
$ kubectl -n ingress-nginx get pod -o wide +$ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 nginx-ingress-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3 nginx-ingress-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2 - +
$ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 nginx-ingress-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3 nginx-ingress-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2 -
One major limitation of this deployment approach is that only a single NGINX Ingress controller Pod may be scheduled on each cluster node, because binding the same port multiple times on the same network interface is technically impossible. Pods that are unschedulable due to such situation fail with the following event:
$ kubectl -n ingress-nginx describe pod <unschedulable-nginx-ingress-controller-pod> +$ kubectl -n ingress-nginx describe pod <unschedulable-nginx-ingress-controller-pod> ... Events: Type Reason From Message ---- ------ ---- ------- Warning FailedScheduling default-scheduler 0/3 nodes are available: 3 node(s) didn't have free ports for the requested pod ports. - +
$ kubectl -n ingress-nginx describe pod <unschedulable-nginx-ingress-controller-pod> ... Events: Type Reason From Message ---- ------ ---- ------- Warning FailedScheduling default-scheduler 0/3 nodes are available: 3 node(s) didn't have free ports for the requested pod ports. -
One way to ensure only schedulable Pods are created is to deploy the NGINX Ingress controller as a DaemonSet instead of a traditional Deployment.
Pods configured with hostNetwork: true do not use the internal DNS resolver (i.e. kube-dns or CoreDNS), unless -their dnsPolicy spec field is set to ClusterFirstWithHostNet. Consider using this setting if NGINX is +their dnsPolicy spec field is set to ClusterFirstWithHostNet. Consider using this setting if NGINX is expected to resolve internal names for any reason.
hostNetwork: true
dnsPolicy
ClusterFirstWithHostNet
Because there is no Service exposing the NGINX Ingress controller in a configuration using the host network, the default ---publish-service flag used in standard cloud setups does not apply and the status of all Ingress objects remains +--publish-service flag used in standard cloud setups does not apply and the status of all Ingress objects remains blank.
--publish-service
Instead, and because bare-metal nodes usually don't have an ExternalIP, one has to enable the ---report-node-internal-ip-address flag, which sets the status of all Ingress objects to the internal IP +--report-node-internal-ip-address flag, which sets the status of all Ingress objects to the internal IP address of all nodes running the NGINX Ingress controller.
--report-node-internal-ip-address
Given a nginx-ingress-controller DaemonSet composed of 2 replicas
$ kubectl -n ingress-nginx get pod -o wide +Given a nginx-ingress-controller DaemonSet composed of 2 replicas +$ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 nginx-ingress-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3 nginx-ingress-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2 - +
the controller sets the status of all Ingress objects it manages to the following value:
$ kubectl get ingress -o wide +$ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.2,203.0.113.3 80 - +
$ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.2,203.0.113.3 80 -
Alternatively, it is possible to override the address written to Ingress objects using the ---publish-status-address flag. See Command line arguments.
--publish-status-address
Similarly to cloud environments, this deployment approach requires an edge network component providing a public @@ -1581,49 +1581,50 @@ on the target nodes as shown in the diagram below:
This method does not allow preserving the source IP of HTTP requests in any manner, it is therefore not recommended to use it despite its apparent simplicity.
The externalIPs Service option was previously mentioned in the NodePort section.
As per the Services page of the official Kubernetes documentation, the externalIPs option causes -kube-proxy to route traffic sent to arbitrary IP addresses and on the Service ports to the endpoints of that +
As per the Services page of the official Kubernetes documentation, the externalIPs option causes +kube-proxy to route traffic sent to arbitrary IP addresses and on the Service ports to the endpoints of that Service. These IP addresses must belong to the target node.
and the following ingress-nginx NodePort Service
$ kubectl -n ingress-nginx get svc +and the following ingress-nginx NodePort Service +$ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP - +
$ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP -
One could set the following external IPs in the Service spec, and NGINX would become available on both the NodePort and the Service port:
spec: +spec: externalIPs: - 203.0.113.2 - 203.0.113.3 - +
spec: externalIPs: - 203.0.113.2 - 203.0.113.3 -
$ curl -D- http://myapp.example.com:30100 +$ curl -D- http://myapp.example.com:30100 HTTP/1.1 200 OK Server: nginx/1.15.2 $ curl -D- http://myapp.example.com HTTP/1.1 200 OK Server: nginx/1.15.2 - +
$ curl -D- http://myapp.example.com:30100 HTTP/1.1 200 OK Server: nginx/1.15.2 $ curl -D- http://myapp.example.com HTTP/1.1 200 OK Server: nginx/1.15.2 -
We assume the myapp.example.com subdomain above resolves to both 203.0.113.2 and 203.0.113.3 IP addresses.
Attention
The default configuration watches Ingress object from all the namespaces. -To change this behavior use the flag --watch-namespace to limit the scope to a particular namespace.
--watch-namespace
--watch-namespace< Attention If you're using GKE you need to initialize your user as a cluster-admin with the following command: -kubectl create clusterrolebinding cluster-admin-binding \ +kubectl create clusterrolebinding cluster-admin-binding \ --clusterrole cluster-admin \ --user $(gcloud config get-value account) - + The following Mandatory Command is required for all deployments. -kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/mandatory.yaml - +kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/mandatory.yaml + Tip -If you are using a Kubernetes version previous to 1.14, you need to change kubernetes.io/os to beta.kubernetes.io/os at line 217 of mandatory.yaml, see Labels details. +If you are using a Kubernetes version previous to 1.14, you need to change kubernetes.io/os to beta.kubernetes.io/os at line 217 of mandatory.yaml, see Labels details. Provider Specific Steps ¶ There are cloud provider specific yaml files. Docker for Mac ¶ Kubernetes is available in Docker for Mac (from version 18.06.0-ce) Create a service -kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml - +kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml + minikube ¶ For standard usage: -minikube addons enable ingress - +minikube addons enable ingress + For development: Disable the ingress addon: -minikube addons disable ingress - +minikube addons disable ingress + -Execute make dev-env -Confirm the nginx-ingress-controller deployment exists: +Execute make dev-env +Confirm the nginx-ingress-controller deployment exists: -$ kubectl get pods -n ingress-nginx +$ kubectl get pods -n ingress-nginx NAME READY STATUS RESTARTS AGE default-http-backend-66b447d9cf-rrlf9 1/1 Running 0 12s nginx-ingress-controller-fdcdcd6dd-vvpgs 1/1 Running 0 11s - + AWS ¶ -In AWS we use an Elastic Load Balancer (ELB) to expose the NGINX Ingress controller behind a Service of Type=LoadBalancer. +In AWS we use an Elastic Load Balancer (ELB) to expose the NGINX Ingress controller behind a Service of Type=LoadBalancer. Since Kubernetes v1.9.0 it is possible to use a classic load balancer (ELB) or network load balancer (NLB) Please check the elastic load balancing AWS details page Elastic Load Balancer - ELB ¶ @@ -1509,45 +1509,45 @@ Please check the Layer 7: use HTTP as the listener protocol for port 80 and terminate TLS in the ELB For L4: -Check that no change is necessary with regards to the ELB idle timeout. In some scenarios, users may want to modify the ELB idle timeout, so please check the ELB Idle Timeouts section for additional information. If a change is required, users will need to update the value of service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout in provider/aws/service-l4.yaml +Check that no change is necessary with regards to the ELB idle timeout. In some scenarios, users may want to modify the ELB idle timeout, so please check the ELB Idle Timeouts section for additional information. If a change is required, users will need to update the value of service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout in provider/aws/service-l4.yaml Then execute: -kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-l4.yaml +kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-l4.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/patch-configmap-l4.yaml - + For L7: -Change line of the file provider/aws/service-l7.yaml replacing the dummy id with a valid one "arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX" -Check that no change is necessary with regards to the ELB idle timeout. In some scenarios, users may want to modify the ELB idle timeout, so please check the ELB Idle Timeouts section for additional information. If a change is required, users will need to update the value of service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout in provider/aws/service-l7.yaml +Change line of the file provider/aws/service-l7.yaml replacing the dummy id with a valid one "arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX" +Check that no change is necessary with regards to the ELB idle timeout. In some scenarios, users may want to modify the ELB idle timeout, so please check the ELB Idle Timeouts section for additional information. If a change is required, users will need to update the value of service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout in provider/aws/service-l7.yaml Then execute: -kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-l7.yaml +kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-l7.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/patch-configmap-l7.yaml - + This example creates an ELB with just two listeners, one in port 80 and another in port 443 ELB Idle Timeouts ¶ -In some scenarios users will need to modify the value of the ELB idle timeout. Users need to ensure the idle timeout is less than the keepalive_timeout that is configured for NGINX. By default NGINX keepalive_timeout is set to 75s. -The default ELB idle timeout will work for most scenarios, unless the NGINX keepalive_timeout has been modified, in which case service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout will need to be modified to ensure it is less than the keepalive_timeout the user has configured. -Please Note: An idle timeout of 3600s is recommended when using WebSockets. +In some scenarios users will need to modify the value of the ELB idle timeout. Users need to ensure the idle timeout is less than the keepalive_timeout that is configured for NGINX. By default NGINX keepalive_timeout is set to 75s. +The default ELB idle timeout will work for most scenarios, unless the NGINX keepalive_timeout has been modified, in which case service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout will need to be modified to ensure it is less than the keepalive_timeout the user has configured. +Please Note: An idle timeout of 3600s is recommended when using WebSockets. More information with regards to idle timeouts for your Load Balancer can be found in the official AWS documentation. Network Load Balancer (NLB) ¶ This type of load balancer is supported since v1.10.0 as an ALPHA feature. -kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-nlb.yaml - +kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-nlb.yaml + GCE-GKE ¶ -kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml - +kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml + Important Note: proxy protocol is not supported in GCE/GKE Azure ¶ -kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml - +kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml + Bare-metal ¶ Using NodePort: -kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/baremetal/service-nodeport.yaml - +kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/baremetal/service-nodeport.yaml + Tip @@ -1555,42 +1555,43 @@ Please check the Verify installation ¶ To check if the ingress controller pods have started, run the following command: -kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx --watch - +kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx --watch + -Once the operator pods are running, you can cancel the above command by typing Ctrl+C. +Once the operator pods are running, you can cancel the above command by typing Ctrl+C. Now, you are ready to create your first ingress. Detect installed version ¶ -To detect which version of the ingress controller is running, exec into the pod and run nginx-ingress-controller version command. -POD_NAMESPACE=ingress-nginx +To detect which version of the ingress controller is running, exec into the pod and run nginx-ingress-controller version command. +POD_NAMESPACE=ingress-nginx POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}') kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version - + Using Helm ¶ NGINX Ingress controller can be installed via Helm using the chart stable/nginx-ingress from the official charts repository. -To install the chart with the release name my-nginx: -helm install my-nginx stable/nginx-ingress - +To install the chart with the release name my-nginx: +helm install my-nginx stable/nginx-ingress + If the kubernetes cluster has RBAC enabled, then run: -helm install my-nginx stable/nginx-ingress --set rbac.create=true - +helm install my-nginx stable/nginx-ingress --set rbac.create=true + -If you are using Helm 2 then specify release name using --name flag -helm install stable/nginx-ingress --name my-nginx - +If you are using Helm 2 then specify release name using --name flag +helm install stable/nginx-ingress --name my-nginx + or -helm install stable/nginx-ingress --name my-nginx --set rbac.create=true - +helm install stable/nginx-ingress --name my-nginx --set rbac.create=true + Detect installed version: -POD_NAME=$(kubectl get pods -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}') +POD_NAME=$(kubectl get pods -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}') kubectl exec -it $POD_NAME -- /nginx-ingress-controller --version - + + @@ -1645,9 +1646,9 @@ or @@ -1657,7 +1658,7 @@ or
If you're using GKE you need to initialize your user as a cluster-admin with the following command: -
kubectl create clusterrolebinding cluster-admin-binding \ +kubectl create clusterrolebinding cluster-admin-binding \ --clusterrole cluster-admin \ --user $(gcloud config get-value account) - +
kubectl create clusterrolebinding cluster-admin-binding \ --clusterrole cluster-admin \ --user $(gcloud config get-value account) -
The following Mandatory Command is required for all deployments.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/mandatory.yaml -
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/mandatory.yaml +
If you are using a Kubernetes version previous to 1.14, you need to change kubernetes.io/os to beta.kubernetes.io/os at line 217 of mandatory.yaml, see Labels details.
kubernetes.io/os
beta.kubernetes.io/os
There are cloud provider specific yaml files.
Kubernetes is available in Docker for Mac (from version 18.06.0-ce)
Create a service
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml -
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml +
For standard usage:
minikube addons enable ingress -
minikube addons enable ingress +
For development:
minikube addons disable ingress -
minikube addons disable ingress +
make dev-env
$ kubectl get pods -n ingress-nginx +$ kubectl get pods -n ingress-nginx NAME READY STATUS RESTARTS AGE default-http-backend-66b447d9cf-rrlf9 1/1 Running 0 12s nginx-ingress-controller-fdcdcd6dd-vvpgs 1/1 Running 0 11s - +
$ kubectl get pods -n ingress-nginx NAME READY STATUS RESTARTS AGE default-http-backend-66b447d9cf-rrlf9 1/1 Running 0 12s nginx-ingress-controller-fdcdcd6dd-vvpgs 1/1 Running 0 11s -
In AWS we use an Elastic Load Balancer (ELB) to expose the NGINX Ingress controller behind a Service of Type=LoadBalancer. +
Type=LoadBalancer
In AWS we use an Elastic Load Balancer (ELB) to expose the NGINX Ingress controller behind a Service of Type=LoadBalancer. Since Kubernetes v1.9.0 it is possible to use a classic load balancer (ELB) or network load balancer (NLB) Please check the elastic load balancing AWS details page
For L4:
Check that no change is necessary with regards to the ELB idle timeout. In some scenarios, users may want to modify the ELB idle timeout, so please check the ELB Idle Timeouts section for additional information. If a change is required, users will need to update the value of service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout in provider/aws/service-l4.yaml
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout
provider/aws/service-l4.yaml
Then execute:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-l4.yaml +kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-l4.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/patch-configmap-l4.yaml - +
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-l4.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/patch-configmap-l4.yaml -
For L7:
Change line of the file provider/aws/service-l7.yaml replacing the dummy id with a valid one "arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX"
provider/aws/service-l7.yaml
"arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX"
Check that no change is necessary with regards to the ELB idle timeout. In some scenarios, users may want to modify the ELB idle timeout, so please check the ELB Idle Timeouts section for additional information. If a change is required, users will need to update the value of service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout in provider/aws/service-l7.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-l7.yaml +kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-l7.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/patch-configmap-l7.yaml - +
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-l7.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/patch-configmap-l7.yaml -
This example creates an ELB with just two listeners, one in port 80 and another in port 443
In some scenarios users will need to modify the value of the ELB idle timeout. Users need to ensure the idle timeout is less than the keepalive_timeout that is configured for NGINX. By default NGINX keepalive_timeout is set to 75s.
keepalive_timeout
75s
The default ELB idle timeout will work for most scenarios, unless the NGINX keepalive_timeout has been modified, in which case service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout will need to be modified to ensure it is less than the keepalive_timeout the user has configured.
Please Note: An idle timeout of 3600s is recommended when using WebSockets.
3600s
More information with regards to idle timeouts for your Load Balancer can be found in the official AWS documentation.
This type of load balancer is supported since v1.10.0 as an ALPHA feature.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-nlb.yaml -
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-nlb.yaml +
Important Note: proxy protocol is not supported in GCE/GKE
Using NodePort:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/baremetal/service-nodeport.yaml -
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/baremetal/service-nodeport.yaml +
To check if the ingress controller pods have started, run the following command:
kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx --watch -
kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx --watch +
Once the operator pods are running, you can cancel the above command by typing Ctrl+C. +
Ctrl+C
Once the operator pods are running, you can cancel the above command by typing Ctrl+C. Now, you are ready to create your first ingress.
To detect which version of the ingress controller is running, exec into the pod and run nginx-ingress-controller version command.
nginx-ingress-controller version
POD_NAMESPACE=ingress-nginx +To detect which version of the ingress controller is running, exec into the pod and run nginx-ingress-controller version command. +POD_NAMESPACE=ingress-nginx POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}') kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version - +
POD_NAMESPACE=ingress-nginx POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}') kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version -
NGINX Ingress controller can be installed via Helm using the chart stable/nginx-ingress from the official charts repository. -To install the chart with the release name my-nginx:
my-nginx
helm install my-nginx stable/nginx-ingress -
helm install my-nginx stable/nginx-ingress +
If the kubernetes cluster has RBAC enabled, then run:
helm install my-nginx stable/nginx-ingress --set rbac.create=true -
helm install my-nginx stable/nginx-ingress --set rbac.create=true +
If you are using Helm 2 then specify release name using --name flag
--name
helm install stable/nginx-ingress --name my-nginx -
helm install stable/nginx-ingress --name my-nginx +
helm install stable/nginx-ingress --name my-nginx --set rbac.create=true -
helm install stable/nginx-ingress --name my-nginx --set rbac.create=true +
Detect installed version:
POD_NAME=$(kubectl get pods -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}') +POD_NAME=$(kubectl get pods -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}') kubectl exec -it $POD_NAME -- /nginx-ingress-controller --version - +
POD_NAME=$(kubectl get pods -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}') kubectl exec -it $POD_NAME -- /nginx-ingress-controller --version -
This example applies to nginx-ingress-controllers being deployed in an environment with RBAC enabled.
Role Based Access Control is comprised of four layers:
ClusterRole
ClusterRoleBinding
Role
RoleBinding
In order for RBAC to be applied to an nginx-ingress-controller, that controller -should be assigned to a ServiceAccount. That ServiceAccount should be -bound to the Roles and ClusterRoles defined for the nginx-ingress-controller.
ServiceAccount
One ServiceAccount is created in this example, nginx-ingress-serviceaccount.
nginx-ingress-serviceaccount
There are two sets of permissions defined in this example. Cluster-wide -permissions defined by the ClusterRole named nginx-ingress-clusterrole, and -namespace specific permissions defined by the Role named nginx-ingress-role.
nginx-ingress-clusterrole
nginx-ingress-role
These permissions are granted in order for the nginx-ingress-controller to be able to function as an ingress across the cluster. These permissions are -granted to the ClusterRole named nginx-ingress-clusterrole
configmaps
endpoints
nodes
pods
secrets
services
ingresses
events
ingresses/status
These permissions are granted specific to the nginx-ingress namespace. These -permissions are granted to the Role named nginx-ingress-role
Furthermore to support leader-election, the nginx-ingress-controller needs to -have access to a configmap using the resourceName ingress-controller-leader-nginx
configmap
ingress-controller-leader-nginx
Note that resourceNames can NOT be used to limit requests using the “create” verb because authorizers only have access to information that can be obtained @@ -1321,27 +1321,28 @@ from the request URL, method, and headers (resource names in a “create” requ are part of the request body).
This resourceName is the concatenation of the election-id and the -ingress-class as defined by the ingress-controller, which defaults to:
election-id
ingress-class
This resourceName is the concatenation of the election-id and the +ingress-class as defined by the ingress-controller, which defaults to:
ingress-controller-leader
nginx
resourceName
<election-id>-<ingress-class>
Please adapt accordingly if you overwrite either parameter when launching the nginx-ingress-controller.
The ServiceAccount nginx-ingress-serviceaccount is bound to the Role -nginx-ingress-role and the ClusterRole nginx-ingress-clusterrole.
The ServiceAccount nginx-ingress-serviceaccount is bound to the Role +nginx-ingress-role and the ClusterRole nginx-ingress-clusterrole.
The serviceAccountName associated with the containers in the deployment must match the serviceAccount. The namespace references in the Deployment metadata, container arguments, and POD_NAMESPACE should be in the nginx-ingress namespace.
I.e. if your deployment resource looks like (partial example):
kind: Deployment +kind: Deployment metadata: name: nginx-ingress-controller namespace: ingress-nginx @@ -1231,23 +1231,24 @@ in the controller Deployment. - name: nginx-ingress-controller image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0 args: ... - +
kind: Deployment metadata: name: nginx-ingress-controller namespace: ingress-nginx @@ -1231,23 +1231,24 @@ in the controller Deployment. - name: nginx-ingress-controller image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0 args: ... -
simply change the 0.9.0 tag to the version you wish to upgrade to. +
0.9.0
simply change the 0.9.0 tag to the version you wish to upgrade to. The easiest way to do this is e.g. (do note you may need to change the name parameter according to your installation):
kubectl set image deployment/nginx-ingress-controller \ - nginx-ingress-controller=quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0 -
kubectl set image deployment/nginx-ingress-controller \ + nginx-ingress-controller=quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0 +
For interactive editing, use kubectl edit deployment nginx-ingress-controller.
kubectl edit deployment nginx-ingress-controller
If you installed ingress-nginx using the Helm command in the deployment docs so its name is ngx-ingress, +
ngx-ingress
If you installed ingress-nginx using the Helm command in the deployment docs so its name is ngx-ingress, you should be able to upgrade using
helm upgrade --reuse-values ngx-ingress stable/nginx-ingress -
helm upgrade --reuse-values ngx-ingress stable/nginx-ingress +
Validating webhook must be served using TLS, you need to generate a certificate. Note that kube API server is checking the hostname of the certificate, the common name of your certificate will need to match the service name.
To run the validating webhook with a service named ingress-validation-webhook in the namespace ingress-nginx, run
ingress-validation-webhook
openssl req -x509 -newkey rsa:2048 -keyout certificate.pem -out key.pem -days 365 -nodes -subj "/CN=ingress-validation-webhook.ingress-nginx.svc" -
openssl req -x509 -newkey rsa:2048 -keyout certificate.pem -out key.pem -days 365 -nodes -subj "/CN=ingress-validation-webhook.ingress-nginx.svc" +
Kubernetes also provides primitives to sign a certificate request. Here is an example on how to use it
#!/bin/bash +#!/bin/bash SERVICE_NAME=ingress-nginx NAMESPACE=ingress-nginx @@ -1423,17 +1423,17 @@ kubectl create secret generic ingress-nginx.svc \ --from-file=key.pem=${TEMP_DIRECTORY}/server-key.pem \ --from-file=cert.pem=${TEMP_DIRECTORY}/server-cert.pem \ -n ${NAMESPACE} - +
#!/bin/bash SERVICE_NAME=ingress-nginx NAMESPACE=ingress-nginx @@ -1423,17 +1423,17 @@ kubectl create secret generic ingress-nginx.svc \ --from-file=key.pem=${TEMP_DIRECTORY}/server-key.pem \ --from-file=cert.pem=${TEMP_DIRECTORY}/server-cert.pem \ -n ${NAMESPACE} -
To generate the certificate using helm, you can use the following snippet
{{- $cn := printf "%s.%s.svc" ( include "nginx-ingress.validatingWebhook.fullname" . ) .Release.Namespace }} +{{- $cn := printf "%s.%s.svc" ( include "nginx-ingress.validatingWebhook.fullname" . ) .Release.Namespace }} {{- $ca := genCA (printf "%s-ca" ( include "nginx-ingress.validatingWebhook.fullname" . )) .Values.validatingWebhook.certificateValidity -}} {{- $cert := genSignedCert $cn nil nil .Values.validatingWebhook.certificateValidity $ca -}} - +
{{- $cn := printf "%s.%s.svc" ( include "nginx-ingress.validatingWebhook.fullname" . ) .Release.Namespace }} {{- $ca := genCA (printf "%s-ca" ( include "nginx-ingress.validatingWebhook.fullname" . )) .Values.validatingWebhook.certificateValidity -}} {{- $cert := genSignedCert $cn nil nil .Values.validatingWebhook.certificateValidity $ca -}} -
--validating-webhook
:8080
--validating-webhook-certificate
/usr/local/certificates/validating-webhook.pem
--validating-webhook-key
/usr/local/certificates/validating-webhook-key.pem
Once both the ingress controller and the kube API server are configured to serve the webhook, add the you can configure the webhook with the following objects:
apiVersion: v1 +apiVersion: v1 kind: Service metadata: name: ingress-validation-webhook @@ -1507,10 +1507,11 @@ To check that your kube API server runs with the required flags, please refer to name: ingress-validation-webhook path: /networking.k8s.io/v1beta1/ingress caBundle: <pem encoded ca cert that signs the server cert used by the webhook> - +
apiVersion: v1 kind: Service metadata: name: ingress-validation-webhook @@ -1507,10 +1507,11 @@ To check that your kube API server runs with the required flags, please refer to name: ingress-validation-webhook path: /networking.k8s.io/v1beta1/ingress caBundle: <pem encoded ca cert that signs the server cert used by the webhook> -
The code must be checked out as a subdirectory of k8s.io, and not github.com.
mkdir -p $GOPATH/src/k8s.io -cd $GOPATH/src/k8s.io -# Replace "$YOUR_GITHUB_USERNAME" below with your github username -git clone https://github.com/$YOUR_GITHUB_USERNAME/ingress-nginx.git -cd ingress-nginx -
mkdir -p $GOPATH/src/k8s.io +cd $GOPATH/src/k8s.io +# Replace "$YOUR_GITHUB_USERNAME" below with your github username +git clone https://github.com/$YOUR_GITHUB_USERNAME/ingress-nginx.git +cd ingress-nginx +
Prequisites: Minikube must be installed. See releases for installation instructions.
If you are using MacOS and deploying to minikube, the following command will build the local nginx controller container image and deploy the ingress controller onto a minikube cluster with RBAC enabled in the namespace ingress-nginx:
$ make dev-env -
$ make dev-env +
The nginx controller container image can be rebuilt using: -
$ ARCH=amd64 TAG=dev REGISTRY=$USER/ingress-controller make build container -
$ ARCH=amd64 TAG=dev REGISTRY=$USER/ingress-controller make build container +
The image will only be used by pods created after the rebuild. To delete old pods which will cause new ones to spin up: -
$ kubectl get pods -n ingress-nginx +$ kubectl get pods -n ingress-nginx $ kubectl delete pod -n ingress-nginx nginx-ingress-controller-<unique-pod-id> - +
$ kubectl get pods -n ingress-nginx $ kubectl delete pod -n ingress-nginx nginx-ingress-controller-<unique-pod-id> -
The build uses dependencies in the vendor directory, which +
vendor
The build uses dependencies in the vendor directory, which must be installed before building a binary/image. Occasionally, you might need to update the dependencies.
This guide requires you to install go 1.13 or newer.
This will automatically save the dependencies to the vendor/ directory.
vendor/
$ go get +This will automatically save the dependencies to the vendor/ directory. +$ go get $ make dep-ensure - +
$ go get $ make dep-ensure -
All ingress controllers are built through a Makefile. Depending on your requirements you can build a raw server binary, a local container image, or push an image to a remote repository.
In order to use your local Docker, you may need to set the following environment variables:
# "gcloud docker" (default) or "docker" +# "gcloud docker" (default) or "docker" $ export DOCKER=<docker> # "quay.io/kubernetes-ingress-controller" (default), "index.docker.io", or your own registry $ export REGISTRY=<your-docker-registry> - +
# "gcloud docker" (default) or "docker" $ export DOCKER=<docker> # "quay.io/kubernetes-ingress-controller" (default), "index.docker.io", or your own registry $ export REGISTRY=<your-docker-registry> -
To find the registry simply run: docker system info | grep Registry
docker system info | grep Registry
The e2e test image can also be built through the Makefile.
$ make e2e-test-image -
$ make e2e-test-image +
You can then make this image available on your minikube host by exporting the image and loading it with the minikube docker context:
$ docker save nginx-ingress-controller:e2e | (eval $(minikube docker-env) && docker load) -
$ docker save nginx-ingress-controller:e2e | (eval $(minikube docker-env) && docker load) +
Build a raw server binary -
$ make build -
$ make build +
TODO: add more specific instructions needed for raw server binary.
Build a local container image
$ TAG=<tag> REGISTRY=$USER/ingress-controller make container -
$ TAG=<tag> REGISTRY=$USER/ingress-controller make container +
Push the container image to a remote repository
$ TAG=<tag> REGISTRY=$USER/ingress-controller make push -
$ TAG=<tag> REGISTRY=$USER/ingress-controller make push +
There are several ways to deploy the ingress controller onto a cluster. Please check the deployment guide
To run unit-tests, just run
$ cd $GOPATH/src/k8s.io/ingress-nginx +$ cd $GOPATH/src/k8s.io/ingress-nginx $ make test - +
$ cd $GOPATH/src/k8s.io/ingress-nginx $ make test -
If you have access to a Kubernetes cluster, you can also run e2e tests using ginkgo.
$ cd $GOPATH/src/k8s.io/ingress-nginx +$ cd $GOPATH/src/k8s.io/ingress-nginx $ make e2e-test - +
$ cd $GOPATH/src/k8s.io/ingress-nginx $ make e2e-test -
NOTE: if your e2e pod keeps hanging in an ImagePullBackoff, make sure you've made your e2e nginx-ingress-controller image available to minikube as explained in the Building the e2e test image section
To run unit-tests for lua code locally, run:
$ cd $GOPATH/src/k8s.io/ingress-nginx +$ cd $GOPATH/src/k8s.io/ingress-nginx $ ./rootfs/etc/nginx/lua/test/up.sh $ make lua-test - +
$ cd $GOPATH/src/k8s.io/ingress-nginx $ ./rootfs/etc/nginx/lua/test/up.sh $ make lua-test -
Lua tests are located in $GOPATH/src/k8s.io/ingress-nginx/rootfs/etc/nginx/lua/test. When creating a new test file it must follow the naming convention <mytest>_test.lua or it will be ignored.
$GOPATH/src/k8s.io/ingress-nginx/rootfs/etc/nginx/lua/test
<mytest>_test.lua
All Makefiles will produce a release binary, as shown above. To publish this to a wider Kubernetes user base, push the image to a container registry, like -gcr.io. All release images are hosted under gcr.io/google_containers and +gcr.io. All release images are hosted under gcr.io/google_containers and tagged according to a semver scheme.
gcr.io/google_containers
An example release might look like: -
$ make release -
$ make release +
Please follow these guidelines to cut a release:
controller-release-version
The static configuration implies reloads, something that affects the majority of the users.
--enable-dynamic-certificates
ssl_certificate
ssl_certificate_key
http
failure-domain.beta.kubernetes.io/zone
lua_ingress.lua
init_by_lua
How do we extract zones for endpoints? We can have the controller watch create and update events on nodes in the entire cluster and based on that keep the map of nodes to zones in the memory. -And when we generate endpoints list, we can access node name using .subsets.addresses[i].nodeName +And when we generate endpoints list, we can access node name using .subsets.addresses[i].nodeName and based on that fetch zone from the map in memory and store it as a field on the endpoint. -This solution assumes failure-domain.beta.kubernetes.io/zone annotation does not change until the end of node's life. Otherwise we have to +This solution assumes failure-domain.beta.kubernetes.io/zone annotation does not change until the end of node's life. Otherwise we have to watch update events as well on the nodes and that'll add even more overhead.
.subsets.addresses[i].nodeName
Alternatively, we can get the list of nodes only when there's no node in the memory for given node name. This is probably a better solution because then we would avoid watching for API changes on node resources. We can eagrly fetch all the nodes and build node name to zone mapping on start. @@ -1279,6 +1279,7 @@ see no endpoints for the backend and therefore we will use general balancer.
This is the title of the KEP. Keep it simple and descriptive. A good title can help communicate what the KEP is and should be considered as part of any review.
The title should be lowercased and spaces/punctuation should be replaced with -.
-
To get started with this template:
YYYYMMDD-my-title.md
YYYYMMDD
provisional
The canonical place for the latest set of instructions (and the likely source of this file) is here.
The Metadata section above is intended to support the creation of tooling around the KEP process. +
Metadata
The Metadata section above is intended to support the creation of tooling around the KEP process. This will be a YAML section that is fenced as a code block. See the KEP process for details on each of these items.
A table of contents is helpful for quickly jumping to sections of a KEP and for highlighting any additional information provided beyond the standard KEP template.
Ensure the TOC is wrapped with <!-- toc --&rt;<!-- /toc --&rt; tags, and then generate with hack/update-toc.sh.
<!-- toc --&rt;<!-- /toc --&rt;
hack/update-toc.sh
The Summary section is incredibly important for producing high quality user-focused documentation such as release notes or a development roadmap. +
Summary
The Summary section is incredibly important for producing high quality user-focused documentation such as release notes or a development roadmap. It should be possible to collect this information before implementation begins in order to avoid requiring implementors to split their attention between writing release notes and implementing the feature itself.
A good summary is probably at least a paragraph in length.
Major milestones in the life cycle of a KEP should be tracked in Implementation History. +
Implementation History
Major milestones in the life cycle of a KEP should be tracked in Implementation History. Major milestones might include
Motivation
Proposal
Why should this KEP not be implemented.
Similar to the Drawbacks section the Alternatives section is used to highlight and record other possible approaches to delivering the value proposed by a KEP.
Drawbacks
Alternatives
A Kubernetes Enhancement Proposal (KEP) is a way to propose, communicate and coordinate on new efforts for the Kubernetes project. For this reason, the ingress-nginx project is adopting it.
Follow the process outlined in the KEP template
Unless otherwise mentioned, the TLS secret used in examples is a 2048 bit RSA key/cert pair with an arbitrarily chosen hostname, created as follows
$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginxsvc/O=nginxsvc" +$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginxsvc/O=nginxsvc" Generating a 2048 bit RSA private key ................+++ ................+++ @@ -1235,7 +1235,7 @@ key/cert pair with an arbitrarily chosen hostname, created as follows $ kubectl create secret tls tls-secret --key tls.key --cert tls.crt secret "tls-secret" created - +
$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginxsvc/O=nginxsvc" Generating a 2048 bit RSA private key ................+++ ................+++ @@ -1235,7 +1235,7 @@ key/cert pair with an arbitrarily chosen hostname, created as follows $ kubectl create secret tls tls-secret --key tls.key --cert tls.crt secret "tls-secret" created -
Note: If using CA Authentication, described below, you will need to sign the server certificate with the CA.
These instructions are based on the following blog
Generate the CA Key and Certificate:
openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=My Cert Authority' -
openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=My Cert Authority' +
Generate the Server Key, and Certificate and Sign with the CA Certificate:
openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=mydomain.com' +openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=mydomain.com' openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt - +
openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=mydomain.com' openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt -
Generate the Client Key, and Certificate and Sign with the CA Certificate:
openssl req -new -newkey rsa:4096 -keyout client.key -out client.csr -nodes -subj '/CN=My Client' +openssl req -new -newkey rsa:4096 -keyout client.key -out client.csr -nodes -subj '/CN=My Client' openssl x509 -req -sha256 -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 02 -out client.crt - +
openssl req -new -newkey rsa:4096 -keyout client.key -out client.csr -nodes -subj '/CN=My Client' openssl x509 -req -sha256 -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 02 -out client.crt -
Once this is complete you can continue to follow the instructions here
All examples that require a test HTTP Service use the standard http-svc pod, which you can deploy as follows
$ kubectl create -f http-svc.yaml +$ kubectl create -f http-svc.yaml service "http-svc" created replicationcontroller "http-svc" created @@ -1274,10 +1274,10 @@ which you can deploy as follows $ kubectl get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE http-svc 10.0.122.116 <pending> 80:30301/TCP 1d - +
$ kubectl create -f http-svc.yaml service "http-svc" created replicationcontroller "http-svc" created @@ -1274,10 +1274,10 @@ which you can deploy as follows $ kubectl get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE http-svc 10.0.122.116 <pending> 80:30301/TCP 1d -
You can test that the HTTP Service works by exposing it temporarily
$ kubectl patch svc http-svc -p '{"spec":{"type": "LoadBalancer"}}' +$ kubectl patch svc http-svc -p '{"spec":{"type": "LoadBalancer"}}' "http-svc" patched $ kubectl get svc http-svc @@ -1324,10 +1324,11 @@ which you can deploy as follows $ kubectl patch svc http-svc -p '{"spec":{"type": "NodePort"}}' "http-svc" patched - +
$ kubectl patch svc http-svc -p '{"spec":{"type": "LoadBalancer"}}' "http-svc" patched $ kubectl get svc http-svc @@ -1324,10 +1324,11 @@ which you can deploy as follows $ kubectl patch svc http-svc -p '{"spec":{"type": "NodePort"}}' "http-svc" patched -
cookie
balanced
persistent
INGRESSCOOKIE
None
Lax
Strict
SameSite=None
"true"
"false"
Max-Age
Expires
false
true
You can create the example Ingress to test this:
kubectl create -f ingress.yaml -
kubectl create -f ingress.yaml +
You can confirm that the Ingress works:
$ kubectl describe ing nginx-test +$ kubectl describe ing nginx-test Name: nginx-test Namespace: default Address: @@ -1305,10 +1305,10 @@ Last-Modified: Tue, 24 Jan 2017 14:02:19 GMT ETag: "58875e6b-264" Accept-Ranges: bytes - +
$ kubectl describe ing nginx-test Name: nginx-test Namespace: default Address: @@ -1305,10 +1305,10 @@ Last-Modified: Tue, 24 Jan 2017 14:02:19 GMT ETag: "58875e6b-264" Accept-Ranges: bytes -
In the example above, you can see that the response contains a Set-Cookie header with the settings we have defined. -This cookie is created by NGINX, it contains a randomly generated key corresponding to the upstream used for that request (selected using consistent hashing) and has an Expires directive. +
Set-Cookie
In the example above, you can see that the response contains a Set-Cookie header with the settings we have defined. +This cookie is created by NGINX, it contains a randomly generated key corresponding to the upstream used for that request (selected using consistent hashing) and has an Expires directive. If the user changes this cookie, NGINX creates a new one and redirects the user to another upstream.
If the backend pool grows NGINX will keep sending the requests through the same server of the first request, even if it's overloaded.
When the backend server is removed, the requests are re-routed to another upstream server. This does not require the cookie to be updated because the key's consistent hash will change.
This example shows how to add authentication in a Ingress rule using a secret that contains a file generated with htpasswd. -It's important the file generated is named auth (actually - that the secret has a key data.auth), otherwise the ingress-controller returns a 503.
htpasswd
auth
data.auth
$ htpasswd -c auth foo +This example shows how to add authentication in a Ingress rule using a secret that contains a file generated with htpasswd. +It's important the file generated is named auth (actually - that the secret has a key data.auth), otherwise the ingress-controller returns a 503. +$ htpasswd -c auth foo New password: <bar> New password: Re-type new password: Adding password for user foo - +
This example shows how to add authentication in a Ingress rule using a secret that contains a file generated with htpasswd. +It's important the file generated is named auth (actually - that the secret has a key data.auth), otherwise the ingress-controller returns a 503.
$ htpasswd -c auth foo New password: <bar> New password: Re-type new password: Adding password for user foo -
$ kubectl create secret generic basic-auth --from-file=auth +$ kubectl create secret generic basic-auth --from-file=auth secret "basic-auth" created - +
$ kubectl create secret generic basic-auth --from-file=auth secret "basic-auth" created -
$ kubectl get secret basic-auth -o yaml +$ kubectl get secret basic-auth -o yaml apiVersion: v1 data: auth: Zm9vOiRhcHIxJE9GRzNYeWJwJGNrTDBGSERBa29YWUlsSDkuY3lzVDAK @@ -1173,9 +1173,9 @@ It's important the file generated is named auth name: basic-auth namespace: default type: Opaque - +
$ kubectl get secret basic-auth -o yaml apiVersion: v1 data: auth: Zm9vOiRhcHIxJE9GRzNYeWJwJGNrTDBGSERBa29YWUlsSDkuY3lzVDAK @@ -1173,9 +1173,9 @@ It's important the file generated is named auth name: basic-auth namespace: default type: Opaque -
echo " +echo " apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: @@ -1197,9 +1197,9 @@ It's important the file generated is named auth serviceName: http-svc servicePort: 80 " | kubectl create -f - - +
echo " apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: @@ -1197,9 +1197,9 @@ It's important the file generated is named auth serviceName: http-svc servicePort: 80 " | kubectl create -f - -
$ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' +$ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' * Trying 10.2.29.4... * Connected to 10.2.29.4 (10.2.29.4) port 80 (#0) > GET / HTTP/1.1 @@ -1223,9 +1223,9 @@ It's important the file generated is named auth </body> </html> * Connection #0 to host 10.2.29.4 left intact - +
$ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' * Trying 10.2.29.4... * Connected to 10.2.29.4 (10.2.29.4) port 80 (#0) > GET / HTTP/1.1 @@ -1223,9 +1223,9 @@ It's important the file generated is named auth </body> </html> * Connection #0 to host 10.2.29.4 left intact -
$ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' -u 'foo:bar' +$ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' -u 'foo:bar' * Trying 10.2.29.4... * Connected to 10.2.29.4 (10.2.29.4) port 80 (#0) * Server auth using Basic with user 'foo' @@ -1268,10 +1268,11 @@ x-real-ip=10.2.29.1 BODY: * Connection #0 to host 10.2.29.4 left intact -no body in request- - +
$ curl -v http://10.2.29.4/ -H 'Host: foo.bar.com' -u 'foo:bar' * Trying 10.2.29.4... * Connected to 10.2.29.4 (10.2.29.4) port 80 (#0) * Server auth using Basic with user 'foo' @@ -1268,10 +1268,11 @@ x-real-ip=10.2.29.1 BODY: * Connection #0 to host 10.2.29.4 left intact -no body in request- -
For more details on the generation process, checkout the Prerequisite docs.
You can have as many certificates as you want. If they're in the binary DER format, you can convert them as the following:
openssl x509 -in certificate.der -inform der -out certificate.crt -outform pem -
openssl x509 -in certificate.der -inform der -out certificate.crt -outform pem +
Then, you can concatenate them all in only one file, named 'ca.crt' as the following:
cat certificate1.crt certificate2.crt certificate3.crt >> ca.crt -
cat certificate1.crt certificate2.crt certificate3.crt >> ca.crt +
Note: Make sure that the Key Size is greater than 1024 and Hashing Algorithm(Digest) is something better than md5 for each certificate generated. Otherwise you will receive an error.
You can create a secret containing just the CA certificate and another Secret containing the Server Certificate which is Signed by the CA.
kubectl create secret generic ca-secret --from-file=ca.crt=ca.crt +kubectl create secret generic ca-secret --from-file=ca.crt=ca.crt kubectl create secret generic tls-secret --from-file=tls.crt=server.crt --from-file=tls.key=server.key - +
kubectl create secret generic ca-secret --from-file=ca.crt=ca.crt kubectl create secret generic tls-secret --from-file=tls.crt=server.crt --from-file=tls.key=server.key -
You can create a secret containing CA certificate along with the Server Certificate, that can be used for both TLS and Client Auth.
kubectl create secret generic ca-secret --from-file=tls.crt=server.crt --from-file=tls.key=server.key --from-file=ca.crt=ca.crt -
kubectl create secret generic ca-secret --from-file=tls.crt=server.crt --from-file=tls.key=server.key --from-file=ca.crt=ca.crt +
If you want to also enable Certificate Revocation List verification you can create the secret also containing the CRL file in PEM format: -
kubectl create secret generic ca-secret --from-file=ca.crt=ca.crt --from-file=ca.crl=ca.crl -
kubectl create secret generic ca-secret --from-file=ca.crt=ca.crt --from-file=ca.crl=ca.crl +
Note: The CA Certificate must contain the trusted certificate authority chain to verify client certificates.
Use an external service (Basic Auth) located in https://httpbin.org
https://httpbin.org
$ kubectl create -f ingress.yaml +Use an external service (Basic Auth) located in https://httpbin.org +$ kubectl create -f ingress.yaml ingress "external-auth" created $ kubectl get ing external-auth @@ -1232,10 +1232,10 @@ status: ingress: - ip: 172.17.4.99 $ - +
$ kubectl create -f ingress.yaml ingress "external-auth" created $ kubectl get ing external-auth @@ -1232,10 +1232,10 @@ status: ingress: - ip: 172.17.4.99 $ -
Test 1: no username/password (expect code 401)
$ curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' +$ curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) @@ -1260,10 +1260,10 @@ $ </body> </html> * Connection #0 to host 172.17.4.99 left intact - +
$ curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) @@ -1260,10 +1260,10 @@ $ </body> </html> * Connection #0 to host 172.17.4.99 left intact -
Test 2: valid username/password (expect code 200) -
$ curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' -u 'user:passwd' +$ curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' -u 'user:passwd' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) @@ -1306,9 +1306,9 @@ x-real-ip=10.2.60.1 BODY: * Connection #0 to host 172.17.4.99 left intact -no body in request- - +
$ curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' -u 'user:passwd' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) @@ -1306,9 +1306,9 @@ x-real-ip=10.2.60.1 BODY: * Connection #0 to host 172.17.4.99 left intact -no body in request- -
Test 3: invalid username/password (expect code 401) -
curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' -u 'user:user' +curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' -u 'user:user' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) @@ -1336,10 +1336,11 @@ BODY: </body> </html> * Connection #0 to host 172.17.4.99 left intact - +
curl -k http://172.17.4.99 -v -H 'Host: external-auth-01.sample.com' -u 'user:user' * Rebuilt URL to: http://172.17.4.99/ * Trying 172.17.4.99... * Connected to 172.17.4.99 (172.17.4.99) port 80 (#0) @@ -1336,10 +1336,11 @@ BODY: </body> </html> * Connection #0 to host 172.17.4.99 left intact -
The auth-url and auth-signin annotations allow you to use an external +
auth-url
auth-signin
The auth-url and auth-signin annotations allow you to use an external authentication provider to protect your Ingress resources.
Important
This annotation requires nginx-ingress-controller v0.9.0 or greater.)
nginx-ingress-controller v0.9.0
This functionality is enabled by deploying multiple Ingress objects for a single host. One Ingress object has no special annotations and handles authentication.
Other Ingress objects can then be annotated in such a way that require the user to -authenticate against the first Ingress's endpoint, and can redirect 401s to the +authenticate against the first Ingress's endpoint, and can redirect 401s to the same endpoint.
401
Sample:
... +... metadata: name: application annotations: nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth" nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri" ... - +
... metadata: name: application annotations: nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth" nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri" ... -
This example will show you how to deploy oauth2_proxy +
oauth2_proxy
This example will show you how to deploy oauth2_proxy into a Kubernetes cluster and use it to protect the Kubernetes Dashboard using github as oAuth2 provider
kubectl create -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/kubernetes-dashboard/v1.10.1.yaml -
kubectl create -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/kubernetes-dashboard/v1.10.1.yaml +
https://foo.bar.com
/oauth2
https://foo.bar.com/oauth2
Configure oauth2_proxy values in the file oauth2-proxy.yaml with the values:
OAUTH2_PROXY_CLIENT_ID with the github <Client ID>
<Client ID>
<Client Secret>
OAUTH2_PROXY_COOKIE_SECRET with value of python -c 'import os,base64; print(base64.b64encode(os.urandom(16)).decode("ascii"))'
python -c 'import os,base64; print(base64.b64encode(os.urandom(16)).decode("ascii"))'
Customize the contents of the file dashboard-ingress.yaml:
Replace __INGRESS_HOST__ with a valid FQDN and __INGRESS_SECRET__ with a Secret with a valid SSL certificate.
__INGRESS_HOST__
__INGRESS_SECRET__
$ kubectl create -f oauth2-proxy.yaml,dashboard-ingress.yaml -
$ kubectl create -f oauth2-proxy.yaml,dashboard-ingress.yaml +
Test the oauth integration accessing the configured URL, like https://foo.bar.com
The Ingress in this example adds a custom header to Nginx configuration that only applies to that specific Ingress. If you want to add headers that apply globally to all Ingresses, please have a look at this example.
$ kubectl apply -f ingress.yaml -
$ kubectl apply -f ingress.yaml +
Check if the contents of the annotation are present in the nginx.conf file using: -kubectl exec nginx-ingress-controller-873061567-4n3k2 -n kube-system cat /etc/nginx/nginx.conf
kubectl exec nginx-ingress-controller-873061567-4n3k2 -n kube-system cat /etc/nginx/nginx.conf
Using a ConfigMap is possible to customize the NGINX configuration
For example, if we want to change the timeouts we need to create a ConfigMap:
$ cat configmap.yaml +$ cat configmap.yaml apiVersion: v1 data: proxy-connect-timeout: "10" @@ -1162,16 +1162,17 @@ data: kind: ConfigMap metadata: name: nginx-configuration - +
$ cat configmap.yaml apiVersion: v1 data: proxy-connect-timeout: "10" @@ -1162,16 +1162,17 @@ data: kind: ConfigMap metadata: name: nginx-configuration -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/customization/custom-configuration/configmap.yaml \ - | kubectl apply -f - -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/customization/custom-configuration/configmap.yaml \ + | kubectl apply -f - +
If the Configmap it is updated, NGINX will be reloaded with the new configuration.
This example demonstrates how to use a custom backend to render custom error pages.
First, create the custom default-backend. It will be used by the Ingress controller later on.
default-backend
$ kubectl create -f custom-default-backend.yaml +First, create the custom default-backend. It will be used by the Ingress controller later on. +$ kubectl create -f custom-default-backend.yaml service "nginx-errors" created deployment.apps "nginx-errors" created - +
$ kubectl create -f custom-default-backend.yaml service "nginx-errors" created deployment.apps "nginx-errors" created -
This should have created a Deployment and a Service with the name nginx-errors.
nginx-errors
$ kubectl get deploy,svc +This should have created a Deployment and a Service with the name nginx-errors. +$ kubectl get deploy,svc NAME DESIRED CURRENT READY AGE deployment.apps/nginx-errors 1 1 1 10s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nginx-errors ClusterIP 10.0.0.12 <none> 80/TCP 10s - +
$ kubectl get deploy,svc NAME DESIRED CURRENT READY AGE deployment.apps/nginx-errors 1 1 1 10s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nginx-errors ClusterIP 10.0.0.12 <none> 80/TCP 10s -
If you do not already have an instance of the NGINX Ingress controller running, deploy it according to the deployment guide, then follow these steps:
Edit the nginx-ingress-controller Deployment and set the value of the --default-backend flag to the name of the +
--default-backend
Edit the nginx-ingress-controller Deployment and set the value of the --default-backend flag to the name of the newly created error backend.
Edit the nginx-configuration ConfigMap and create the key custom-http-errors with a value of 404,503.
nginx-configuration
custom-http-errors
404,503
Take note of the IP address assigned to the NGINX Ingress controller Service. -
$ kubectl get svc ingress-nginx + $ kubectl get svc ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx ClusterIP 10.0.0.13 <none> 80/TCP,443/TCP 10m - +
$ kubectl get svc ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx ClusterIP 10.0.0.13 <none> 80/TCP,443/TCP 10m -
The ingress-nginx Service is of type ClusterIP in this example. This may vary depending on your environment. +
ClusterIP
The ingress-nginx Service is of type ClusterIP in this example. This may vary depending on your environment. Make sure you can use the Service to reach NGINX before proceeding with the rest of this example.
Let us send a couple of HTTP requests using cURL and validate everything is working as expected.
A request to the default backend returns a 404 error with a custom message:
$ curl -D- http://10.0.0.13/ +$ curl -D- http://10.0.0.13/ HTTP/1.1 404 Not Found Server: nginx/1.13.12 Date: Tue, 12 Jun 2018 19:11:24 GMT @@ -1277,10 +1277,10 @@ Transfer-Encoding: chunked Connection: keep-alive <span>The page you're looking for could not be found.</span> - +
$ curl -D- http://10.0.0.13/ HTTP/1.1 404 Not Found Server: nginx/1.13.12 Date: Tue, 12 Jun 2018 19:11:24 GMT @@ -1277,10 +1277,10 @@ Transfer-Encoding: chunked Connection: keep-alive <span>The page you're looking for could not be found.</span> -
A request with a custom Accept header returns the corresponding document type (JSON):
Accept
$ curl -D- -H 'Accept: application/json' http://10.0.0.13/ +A request with a custom Accept header returns the corresponding document type (JSON): +$ curl -D- -H 'Accept: application/json' http://10.0.0.13/ HTTP/1.1 404 Not Found Server: nginx/1.13.12 Date: Tue, 12 Jun 2018 19:12:36 GMT @@ -1290,13 +1290,14 @@ Connection: keep-alive Vary: Accept-Encoding { "message": "The page you're looking for could not be found" } - +
$ curl -D- -H 'Accept: application/json' http://10.0.0.13/ HTTP/1.1 404 Not Found Server: nginx/1.13.12 Date: Tue, 12 Jun 2018 19:12:36 GMT @@ -1290,13 +1290,14 @@ Connection: keep-alive Vary: Accept-Encoding { "message": "The page you're looking for could not be found" } -
To go further with this example, feel free to deploy your own applications and Ingress objects, and validate that the responses are still in the correct format when a backend returns 503 (eg. if you scale a Deployment down to 0 replica).
This example demonstrates configuration of the nginx ingress controller via a ConfigMap to pass a custom list of headers to the upstream server.
custom-headers.yaml defines a ConfigMap in the ingress-nginx namespace named custom-headers, holding several custom X-prefixed HTTP headers.
custom-headers
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/customization/custom-headers/custom-headers.yaml -
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/customization/custom-headers/custom-headers.yaml +
configmap.yaml defines a ConfigMap in the ingress-nginx namespace named nginx-configuration. This controls the global configuration of the ingress controller, and already exists in a standard installation. The key proxy-set-headers is set to cite the previously-created ingress-nginx/custom-headers ConfigMap.
proxy-set-headers
ingress-nginx/custom-headers
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/customization/custom-headers/configmap.yaml -
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/customization/custom-headers/configmap.yaml +
The nginx ingress controller will read the ingress-nginx/nginx-configuration ConfigMap, find the proxy-set-headers key, read HTTP headers from the ingress-nginx/custom-headers ConfigMap, and include those HTTP headers in all requests flowing from nginx to the backends.
ingress-nginx/nginx-configuration
Check the contents of the ConfigMaps are present in the nginx.conf file using: -kubectl exec nginx-ingress-controller-873061567-4n3k2 -n ingress-nginx cat /etc/nginx/nginx.conf
kubectl exec nginx-ingress-controller-873061567-4n3k2 -n ingress-nginx cat /etc/nginx/nginx.conf
Sample configuration includes:
User
internal
UserID
UserRole
You can deploy the controller as follows:
$ kubectl create -f deploy/ +$ kubectl create -f deploy/ deployment "demo-auth-service" created service "demo-auth-service" created ingress "demo-auth-service" created @@ -1183,10 +1183,10 @@ follows: NAME HOSTS ADDRESS PORTS AGE public-demo-echo-service public-demo-echo-service.kube.local 80 1m secure-demo-echo-service secure-demo-echo-service.kube.local 80 1m - +
$ kubectl create -f deploy/ deployment "demo-auth-service" created service "demo-auth-service" created ingress "demo-auth-service" created @@ -1183,10 +1183,10 @@ follows: NAME HOSTS ADDRESS PORTS AGE public-demo-echo-service public-demo-echo-service.kube.local 80 1m secure-demo-echo-service secure-demo-echo-service.kube.local 80 1m -
Test 1: public service with no auth header
$ curl -H 'Host: public-demo-echo-service.kube.local' -v 192.168.99.100 +$ curl -H 'Host: public-demo-echo-service.kube.local' -v 192.168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) @@ -1204,10 +1204,10 @@ follows: < * Connection #0 to host 192.168.99.100 left intact UserID: , UserRole: - +
$ curl -H 'Host: public-demo-echo-service.kube.local' -v 192.168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) @@ -1204,10 +1204,10 @@ follows: < * Connection #0 to host 192.168.99.100 left intact UserID: , UserRole: -
Test 2: secure service with no auth header
$ curl -H 'Host: secure-demo-echo-service.kube.local' -v 192.168.99.100 +$ curl -H 'Host: secure-demo-echo-service.kube.local' -v 192.168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) @@ -1231,10 +1231,10 @@ follows: </body> </html> * Connection #0 to host 192.168.99.100 left intact - +
$ curl -H 'Host: secure-demo-echo-service.kube.local' -v 192.168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) @@ -1231,10 +1231,10 @@ follows: </body> </html> * Connection #0 to host 192.168.99.100 left intact -
Test 3: public service with valid auth header
$ curl -H 'Host: public-demo-echo-service.kube.local' -H 'User:internal' -v 192.168.99.100 +$ curl -H 'Host: public-demo-echo-service.kube.local' -H 'User:internal' -v 192.168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) @@ -1253,10 +1253,10 @@ follows: < * Connection #0 to host 192.168.99.100 left intact UserID: 1443635317331776148, UserRole: admin - +
$ curl -H 'Host: public-demo-echo-service.kube.local' -H 'User:internal' -v 192.168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) @@ -1253,10 +1253,10 @@ follows: < * Connection #0 to host 192.168.99.100 left intact UserID: 1443635317331776148, UserRole: admin -
Test 4: secure service with valid auth header
$ curl -H 'Host: secure-demo-echo-service.kube.local' -H 'User:internal' -v 192.168.99.100 +$ curl -H 'Host: secure-demo-echo-service.kube.local' -H 'User:internal' -v 192.168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) @@ -1275,10 +1275,11 @@ follows: < * Connection #0 to host 192.168.99.100 left intact UserID: 605394647632969758, UserRole: admin - +
$ curl -H 'Host: secure-demo-echo-service.kube.local' -H 'User:internal' -v 192.168.99.100 * Rebuilt URL to: 192.168.99.100/ * Trying 192.168.99.100... * Connected to 192.168.99.100 (192.168.99.100) port 80 (#0) @@ -1275,10 +1275,11 @@ follows: < * Connection #0 to host 192.168.99.100 left intact UserID: 605394647632969758, UserRole: admin -
$ cat configmap.yaml +$ cat configmap.yaml apiVersion: v1 data: ssl-dh-param: "ingress-nginx/lb-dhparam" @@ -1239,17 +1239,17 @@ use a ConfigMap to configure custom Diffie-Hellman parameters file to help with labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx - +
$ cat configmap.yaml apiVersion: v1 data: ssl-dh-param: "ingress-nginx/lb-dhparam" @@ -1239,17 +1239,17 @@ use a ConfigMap to configure custom Diffie-Hellman parameters file to help with labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx -
$ kubectl create -f configmap.yaml -
$ kubectl create -f configmap.yaml +
$> openssl dhparam 1024 2> /dev/null | base64 +$> openssl dhparam 1024 2> /dev/null | base64 LS0tLS1CRUdJTiBESCBQQVJBTUVURVJ... - +
$> openssl dhparam 1024 2> /dev/null | base64 LS0tLS1CRUdJTiBESCBQQVJBTUVURVJ... -
$ cat ssl-dh-param.yaml +$ cat ssl-dh-param.yaml apiVersion: v1 data: dhparam.pem: "LS0tLS1CRUdJTiBESCBQQVJBTUVURVJ..." @@ -1260,17 +1260,18 @@ use a ConfigMap to configure custom Diffie-Hellman parameters file to help with labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx - +
$ cat ssl-dh-param.yaml apiVersion: v1 data: dhparam.pem: "LS0tLS1CRUdJTiBESCBQQVJBTUVURVJ..." @@ -1260,17 +1260,18 @@ use a ConfigMap to configure custom Diffie-Hellman parameters file to help with labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx -
$ kubectl create -f ssl-dh-param.yaml -
$ kubectl create -f ssl-dh-param.yaml +
Check the contents of the configmap is present in the nginx.conf file using: -kubectl exec nginx-ingress-controller-873061567-4n3k2 -n kube-system cat /etc/nginx/nginx.conf
This example aims to demonstrate the use of an Init Container to adjust sysctl default values using kubectl patch
kubectl patch
kubectl patch deployment -n ingress-nginx nginx-ingress-controller \ +This example aims to demonstrate the use of an Init Container to adjust sysctl default values using kubectl patch +kubectl patch deployment -n ingress-nginx nginx-ingress-controller \ --patch="$(curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/customization/sysctl/patch.json)" - +
kubectl patch deployment -n ingress-nginx nginx-ingress-controller \ --patch="$(curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/customization/sysctl/patch.json)" -
Changes:
net.core.somaxconn
128
32768
net.ipv4.ip_local_port_range
32768 60999
1024 65000
In a post from the NGINX blog, it is possible to see an explanation for the changes.
This example demonstrates how to deploy a docker registry in the cluster and configure Ingress enable access from Internet
First we deploy the docker registry in the cluster:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/docker-registry/deployment.yaml -
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/docker-registry/deployment.yaml +
DO NOT RUN THIS IN PRODUCTION
This deployment uses emptyDir in the volumeMount which means the contents of the registry will be deleted when the pod dies.
emptyDir
volumeMount
The next required step is creation of the ingress rules. To do this we have two options: with and without TLS
Download and edit the yaml deployment replacing registry.<your domain> with a valid DNS name pointing to the ingress controller:
registry.<your domain>
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/docker-registry/ingress-without-tls.yaml -
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/docker-registry/ingress-without-tls.yaml +
Running a docker registry without TLS requires we configure our local docker daemon with the insecure registry flag.
Please check deploy a plain http registry
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/docker-registry/ingress-with-tls.yaml -
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/docker-registry/ingress-with-tls.yaml +
Deploy kube lego use Let's Encrypt certificates or edit the ingress rule to use a secret with an existing SSL certificate.
To test the registry is working correctly we download a known image from docker hub, create a tag pointing to the new registry and upload the image:
docker pull ubuntu:16.04 +docker pull ubuntu:16.04 docker tag ubuntu:16.04 `registry.<your domain>/ubuntu:16.04` docker push `registry.<your domain>/ubuntu:16.04` - +
docker pull ubuntu:16.04 docker tag ubuntu:16.04 `registry.<your domain>/ubuntu:16.04` docker push `registry.<your domain>/ubuntu:16.04` -
Please replace registry.<your domain> with your domain.
example.com
fortune-teller.stack.build
Deployment
$ kubectl create -f app.yaml -
$ kubectl create -f app.yaml +
This is a standard kubernetes deployment object. It is running a grpc service -listening on port 50051.
50051
The sample application fortune-teller-app is a grpc server implemented in go. Here's the stripped-down implementation:
func main() { +func main() { grpcServer := grpc.NewServer() fortune.RegisterFortuneTellerServer(grpcServer, &FortuneTeller{}) lis, _ := net.Listen("tcp", ":50051") grpcServer.Serve(lis) } - +
func main() { grpcServer := grpc.NewServer() fortune.RegisterFortuneTellerServer(grpcServer, &FortuneTeller{}) lis, _ := net.Listen("tcp", ":50051") grpcServer.Serve(lis) } -
The takeaway is that we are not doing any TLS configuration on the server (as we are terminating TLS at the ingress level, grpc traffic will travel unencrypted inside the cluster and arrive "insecure").
For your own application you may or may not want to do this. If you prefer to forward encrypted traffic to your POD and terminate TLS at the gRPC server -itself, add the ingress annotation nginx.ingress.kubernetes.io/backend-protocol: "GRPCS".
nginx.ingress.kubernetes.io/backend-protocol: "GRPCS"
Service
$ kubectl create -f svc.yaml -
$ kubectl create -f svc.yaml +
Here we have a typical service. Nothing special, just routing traffic to the -backend application on port 50051.
Ingress
$ kubectl create -f ingress.yaml -
$ kubectl create -f ingress.yaml +
A few things to note:
nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
https://fortune-teller.stack.build:443
Once we've applied our configuration to kubernetes, it's time to test that we can actually talk to the backend. To do this, we'll use the grpcurl utility:
$ grpcurl fortune-teller.stack.build:443 build.stack.fortune.FortuneTeller/Predict +$ grpcurl fortune-teller.stack.build:443 build.stack.fortune.FortuneTeller/Predict { "message": "Let us endeavor so to live that when we come to die even the undertaker will be sorry.\n\t\t-- Mark Twain, \"Pudd'nhead Wilson's Calendar\"" } - +
$ grpcurl fortune-teller.stack.build:443 build.stack.fortune.FortuneTeller/Predict { "message": "Let us endeavor so to live that when we come to die even the undertaker will be sorry.\n\t\t-- Mark Twain, \"Pudd'nhead Wilson's Calendar\"" } -
GODEBUG=http2debug=2
grpc_read_timeout
grpc_send_timeout
client_body_timeout
Values for the timeouts must be specified as e.g. "1200s".
"1200s"
-On the most recent versions of nginx-ingress, changing these timeouts requires using the nginx.ingress.kubernetes.io/server-snippet annotation. There are plans for future releases to allow using the Kubernetes annotations to define each timeout seperately. +On the most recent versions of nginx-ingress, changing these timeouts requires using the nginx.ingress.kubernetes.io/server-snippet annotation. There are plans for future releases to allow using the Kubernetes annotations to define each timeout seperately.
On the most recent versions of nginx-ingress, changing these timeouts requires using the nginx.ingress.kubernetes.io/server-snippet annotation. There are plans for future releases to allow using the Kubernetes annotations to define each timeout seperately.
nginx.ingress.kubernetes.io/server-snippet
This should generate a segment like: -
$ kubectl exec -it nginx-ingress-controller-6vwd1 -- cat /etc/nginx/nginx.conf | grep "foo.bar.com" -B 7 -A 35 +$ kubectl exec -it nginx-ingress-controller-6vwd1 -- cat /etc/nginx/nginx.conf | grep "foo.bar.com" -B 7 -A 35 server { listen 80; listen 443 ssl http2; @@ -1198,9 +1198,9 @@ proxy_pass http://default-http-svc-80; } - +
$ kubectl exec -it nginx-ingress-controller-6vwd1 -- cat /etc/nginx/nginx.conf | grep "foo.bar.com" -B 7 -A 35 server { listen 80; listen 443 ssl http2; @@ -1198,9 +1198,9 @@ proxy_pass http://default-http-svc-80; } -
And you should be able to reach your nginx service or http-svc service using a hostname switch: -
$ kubectl get ing +$ kubectl get ing NAME RULE BACKEND ADDRESS AGE foo-tls - 104.154.30.67 13m foo.bar.com @@ -1237,10 +1237,11 @@ $ curl 104.154.30.67 default backend - 404 - +
$ kubectl get ing NAME RULE BACKEND ADDRESS AGE foo-tls - 104.154.30.67 13m foo.bar.com @@ -1237,10 +1237,11 @@ $ curl 104.154.30.67 default backend - 404 -
Before applying any objects, first apply the PSP permissions by running: -
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/psp/psp.yaml -
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/psp/psp.yaml +
Now that the pod security policy is applied, we can continue as usual by applying the mandatory.yaml according to the Installation Guide.
Starting in Version 0.22.0, ingress definitions using the annotation nginx.ingress.kubernetes.io/rewrite-target are not backwards compatible with previous versions. In Version 0.22.0 and beyond, any substrings within the request URI that need to be passed to the rewritten path must explicitly be defined in a capture group.
nginx.ingress.kubernetes.io/rewrite-target
Captured groups are saved in numbered placeholders, chronologically, in the form $1, $2 ... $n. These placeholders can be used as parameters in the rewrite-target annotation.
$1
$2
$n
rewrite-target
Create an Ingress rule with a rewrite annotation:
$ echo ' +$ echo ' apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: @@ -1334,18 +1334,18 @@ and that you have an ingress controller running in y servicePort: 80 path: /something(/|$)(.*) ' | kubectl create -f - - +
$ echo ' apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: @@ -1334,18 +1334,18 @@ and that you have an ingress controller running in y servicePort: 80 path: /something(/|$)(.*) ' | kubectl create -f - -
In this ingress definition, any characters captured by (.*) will be assigned to the placeholder $2, which is then used as a parameter in the rewrite-target annotation.
(.*)
For example, the ingress definition above will result in the following rewrites:
rewrite.bar.com/something
rewrite.bar.com/
rewrite.bar.com/something/
rewrite.bar.com/something/new
rewrite.bar.com/new
Create an Ingress rule with a app-root annotation: -
$ echo " +$ echo " apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: @@ -1363,9 +1363,9 @@ and that you have an ingress controller running in y servicePort: 80 path: / " | kubectl create -f - - +
$ echo " apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: @@ -1363,9 +1363,9 @@ and that you have an ingress controller running in y servicePort: 80 path: / " | kubectl create -f - -
Check the rewrite is working
$ curl -I -k http://approot.bar.com/ +$ curl -I -k http://approot.bar.com/ HTTP/1.1 302 Moved Temporarily Server: nginx/1.11.10 Date: Mon, 13 Mar 2017 14:57:15 GMT @@ -1373,10 +1373,11 @@ Content-Type: text/html Content-Length: 162 Location: http://stickyingress.example.com/app1 Connection: keep-alive - +
$ curl -I -k http://approot.bar.com/ HTTP/1.1 302 Moved Temporarily Server: nginx/1.11.10 Date: Mon, 13 Mar 2017 14:57:15 GMT @@ -1373,10 +1373,11 @@ Content-Type: text/html Content-Length: 162 Location: http://stickyingress.example.com/app1 Connection: keep-alive -
To acquire a static IP for the nginx ingress controller, simply put it -behind a Service of Type=LoadBalancer.
First, create a loadbalancer Service and wait for it to acquire an IP
$ kubectl create -f static-ip-svc.yaml +$ kubectl create -f static-ip-svc.yaml service "nginx-ingress-lb" created $ kubectl get svc nginx-ingress-lb NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-ingress-lb 10.0.138.113 104.154.109.191 80:31457/TCP,443:32240/TCP 15m - +
$ kubectl create -f static-ip-svc.yaml service "nginx-ingress-lb" created $ kubectl get svc nginx-ingress-lb NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-ingress-lb 10.0.138.113 104.154.109.191 80:31457/TCP,443:32240/TCP 15m -
then, update the ingress controller so it adopts the static IP of the Service -by passing the --publish-service flag (the example yaml used in the next step +by passing the --publish-service flag (the example yaml used in the next step already has it set to "nginx-ingress-lb").
$ kubectl create -f nginx-ingress-controller.yaml +$ kubectl create -f nginx-ingress-controller.yaml deployment "nginx-ingress-controller" created - +
$ kubectl create -f nginx-ingress-controller.yaml deployment "nginx-ingress-controller" created -
From here on every Ingress created with the ingress.class annotation set to -nginx will get the IP allocated in the previous step
ingress.class
$ kubectl create -f nginx-ingress.yaml +From here on every Ingress created with the ingress.class annotation set to +nginx will get the IP allocated in the previous step +$ kubectl create -f nginx-ingress.yaml ingress "nginx-ingress" created $ kubectl get ing ingress-nginx @@ -1298,11 +1298,11 @@ already has it set to "nginx-ingress-lb"). request_version=1.1 request_uri=http://104.154.109.191:8080/ ... - +
From here on every Ingress created with the ingress.class annotation set to +nginx will get the IP allocated in the previous step
$ kubectl create -f nginx-ingress.yaml ingress "nginx-ingress" created $ kubectl get ing ingress-nginx @@ -1298,11 +1298,11 @@ already has it set to "nginx-ingress-lb"). request_version=1.1 request_uri=http://104.154.109.191:8080/ ... -
You can test retention by deleting the Ingress
$ kubectl delete ing nginx-ingress +$ kubectl delete ing nginx-ingress ingress "nginx-ingress" deleted $ kubectl create -f nginx-ingress.yaml @@ -1311,7 +1311,7 @@ already has it set to "nginx-ingress-lb"). $ kubectl get ing nginx-ingress NAME HOSTS ADDRESS PORTS AGE nginx-ingress * 104.154.109.191 80, 443 13m - +
$ kubectl delete ing nginx-ingress ingress "nginx-ingress" deleted $ kubectl create -f nginx-ingress.yaml @@ -1311,7 +1311,7 @@ already has it set to "nginx-ingress-lb"). $ kubectl get ing nginx-ingress NAME HOSTS ADDRESS PORTS AGE nginx-ingress * 104.154.109.191 80, 443 13m -
Note that unlike the GCE Ingress, the same loadbalancer IP is shared amongst all @@ -1320,14 +1320,14 @@ controllers.
To promote the allocated IP to static, you can update the Service manifest
$ kubectl patch svc nginx-ingress-lb -p '{"spec": {"loadBalancerIP": "104.154.109.191"}}' +$ kubectl patch svc nginx-ingress-lb -p '{"spec": {"loadBalancerIP": "104.154.109.191"}}' "nginx-ingress-lb" patched - +
$ kubectl patch svc nginx-ingress-lb -p '{"spec": {"loadBalancerIP": "104.154.109.191"}}' "nginx-ingress-lb" patched -
and promote the IP to static (promotion works differently for cloudproviders, provided example is for GKE/GCE) ` -
$ gcloud compute addresses create nginx-ingress-lb --addresses 104.154.109.191 --region us-central1 +$ gcloud compute addresses create nginx-ingress-lb --addresses 104.154.109.191 --region us-central1 Created [https://www.googleapis.com/compute/v1/projects/kubernetesdev/regions/us-central1/addresses/nginx-ingress-lb]. --- address: 104.154.109.191 @@ -1341,12 +1341,13 @@ provided example is for GKE/GCE) status: IN_USE users: - us-central1/forwardingRules/a09f6913ae80e11e6a8c542010af0000 - +
$ gcloud compute addresses create nginx-ingress-lb --addresses 104.154.109.191 --region us-central1 Created [https://www.googleapis.com/compute/v1/projects/kubernetesdev/regions/us-central1/addresses/nginx-ingress-lb]. --- address: 104.154.109.191 @@ -1341,12 +1341,13 @@ provided example is for GKE/GCE) status: IN_USE users: - us-central1/forwardingRules/a09f6913ae80e11e6a8c542010af0000 -
Now even if the Service is deleted, the IP will persist, so you can recreate the -Service with spec.loadBalancerIP set to 104.154.109.191.
spec.loadBalancerIP
104.154.109.191
powered by - MkDocs + MkDocs and - + Material for MkDocs
- + diff --git a/examples/tls-termination/index.html b/examples/tls-termination/index.html index 7710b5fd5..546263674 100644 --- a/examples/tls-termination/index.html +++ b/examples/tls-termination/index.html @@ -34,7 +34,7 @@ - + @@ -42,7 +42,7 @@ - + @@ -53,12 +53,12 @@ - + - + @@ -114,7 +114,7 @@ - + Skip to content @@ -123,7 +123,7 @@ - + public @@ -154,7 +154,7 @@ - + @@ -1226,8 +1226,8 @@ Prerequisites ¶ You need a TLS cert and a test HTTP service for this example. Deployment ¶ -Create a values.yaml file. -apiVersion: networking.k8s.io/v1beta1 +Create a values.yaml file. +apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: nginx-test @@ -1247,16 +1247,16 @@ # This assumes http-svc exists and routes to healthy endpoints serviceName: http-svc servicePort: 80 - + The following command instructs the controller to terminate traffic using the provided TLS cert, and forward un-encrypted HTTP traffic to the test HTTP service. -kubectl apply -f ingress.yaml - +kubectl apply -f ingress.yaml + Validation ¶ You can confirm that the Ingress works. -$ kubectl describe ing nginx-test +$ kubectl describe ing nginx-test Name: nginx-test Namespace: default Address: 104.198.183.6 @@ -1303,10 +1303,11 @@ TLS cert, and forward un-encrypted HTTP traffic to the test HTTP service. x-forwarded-for=104.132.0.80, 35.186.221.137 x-forwarded-proto=https BODY: - + + @@ -1361,9 +1362,9 @@ TLS cert, and forward un-encrypted HTTP traffic to the test HTTP service. @@ -1373,7 +1374,7 @@ TLS cert, and forward un-encrypted HTTP traffic to the test HTTP service. - + diff --git a/how-it-works/index.html b/how-it-works/index.html index a19c7855e..d00e28ac4 100644 --- a/how-it-works/index.html +++ b/how-it-works/index.html @@ -34,7 +34,7 @@ - + @@ -42,7 +42,7 @@ - + @@ -53,12 +53,12 @@ - + - + @@ -114,7 +114,7 @@ - + Skip to content @@ -123,7 +123,7 @@ - + public @@ -154,7 +154,7 @@ - + @@ -1292,10 +1292,10 @@ How it works ¶ The objective of this document is to explain how the NGINX Ingress controller works, in particular how the NGINX model is built and why we need one. NGINX configuration ¶ -The goal of this Ingress controller is the assembly of a configuration file (nginx.conf). The main implication of this requirement is the need to reload NGINX after any change in the configuration file. Though it is important to note that we don't reload Nginx on changes that impact only an upstream configuration (i.e Endpoints change when you deploy your app). We use lua-nginx-module to achieve this. Check below to learn more about how it's done. +The goal of this Ingress controller is the assembly of a configuration file (nginx.conf). The main implication of this requirement is the need to reload NGINX after any change in the configuration file. Though it is important to note that we don't reload Nginx on changes that impact only an upstream configuration (i.e Endpoints change when you deploy your app). We use lua-nginx-module to achieve this. Check below to learn more about how it's done. NGINX model ¶ Usually, a Kubernetes Controller utilizes the synchronization loop pattern to check if the desired state in the controller is updated or a change is required. To this purpose, we need to build a model using different objects from the cluster, in particular (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster. -To get this object from the cluster, we use Kubernetes Informers, in particular, FilteredSharedInformer. This informers allows reacting to changes in using callbacks to individual changes when a new object is added, modified or removed. Unfortunately, there is no way to know if a particular change is going to affect the final configuration file. Therefore on every change, we have to rebuild a new model from scratch based on the state of cluster and compare it to the current model. If the new model equals to the current one, then we avoid generating a new NGINX configuration and triggering a reload. Otherwise, we check if the difference is only about Endpoints. If so we then send the new list of Endpoints to a Lua handler running inside Nginx using HTTP POST request and again avoid generating a new NGINX configuration and triggering a reload. If the difference between running and new model is about more than just Endpoints we create a new NGINX configuration based on the new model, replace the current model and trigger a reload. +To get this object from the cluster, we use Kubernetes Informers, in particular, FilteredSharedInformer. This informers allows reacting to changes in using callbacks to individual changes when a new object is added, modified or removed. Unfortunately, there is no way to know if a particular change is going to affect the final configuration file. Therefore on every change, we have to rebuild a new model from scratch based on the state of cluster and compare it to the current model. If the new model equals to the current one, then we avoid generating a new NGINX configuration and triggering a reload. Otherwise, we check if the difference is only about Endpoints. If so we then send the new list of Endpoints to a Lua handler running inside Nginx using HTTP POST request and again avoid generating a new NGINX configuration and triggering a reload. If the difference between running and new model is about more than just Endpoints we create a new NGINX configuration based on the new model, replace the current model and trigger a reload. One of the uses of the model is to avoid unnecessary reloads when there's no change in the state and to detect conflicts in definitions. The final representation of the NGINX configuration is generated from a Go template using the new model as input for the variables required by the template. Building the NGINX model ¶ @@ -1303,7 +1303,7 @@ Operations to build the model: -Order Ingress rules by CreationTimestamp field, i.e., old rules first. +Order Ingress rules by CreationTimestamp field, i.e., old rules first. If the same path for the same host is defined in more than one Ingress, the oldest rule wins. @@ -1325,7 +1325,7 @@ New Ingress Resource Created. TLS section is added to existing Ingress. -Change in Ingress annotations that impacts more than just upstream configuration. For instance load-balance annotation does not require a reload. +Change in Ingress annotations that impacts more than just upstream configuration. For instance load-balance annotation does not require a reload. A path is added/removed from an Ingress. An Ingress, Service, Secret is removed. Some missing referenced object from the Ingress is available, like a Service or Secret. @@ -1334,15 +1334,16 @@ Avoiding reloads ¶ In some cases, it is possible to avoid reloads, in particular when there is a change in the endpoints, i.e., a pod is started or replaced. It is out of the scope of this Ingress controller to remove reloads completely. This would require an incredible amount of work and at some point makes no sense. This can change only if NGINX changes the way new configurations are read, basically, new changes do not replace worker processes. Avoiding reloads on Endpoints changes ¶ -On every endpoint change the controller fetches endpoints from all the services it sees and generates corresponding Backend objects. It then sends these objects to a Lua handler running inside Nginx. The Lua code in turn stores those backends in a shared memory zone. Then for every request Lua code running in balancer_by_lua context detects what endpoints it should choose upstream peer from and applies the configured load balancing algorithm to choose the peer. Then Nginx takes care of the rest. This way we avoid reloading Nginx on endpoint changes. Note that this includes annotation changes that affects only upstream configuration in Nginx as well. +On every endpoint change the controller fetches endpoints from all the services it sees and generates corresponding Backend objects. It then sends these objects to a Lua handler running inside Nginx. The Lua code in turn stores those backends in a shared memory zone. Then for every request Lua code running in balancer_by_lua context detects what endpoints it should choose upstream peer from and applies the configured load balancing algorithm to choose the peer. Then Nginx takes care of the rest. This way we avoid reloading Nginx on endpoint changes. Note that this includes annotation changes that affects only upstream configuration in Nginx as well. In a relatively big clusters with frequently deploying apps this feature saves significant number of Nginx reloads which can otherwise affect response latency, load balancing quality (after every reload Nginx resets the state of load balancing) and so on. Avoiding outage from wrong configuration ¶ -Because the ingress controller works using the synchronization loop pattern, it is applying the configuration for all matching objects. In case some Ingress objects have a broken configuration, for example a syntax error in the nginx.ingress.kubernetes.io/configuration-snippet annotation, the generated configuration becomes invalid, does not reload and hence no more ingresses will be taken into account. +Because the ingress controller works using the synchronization loop pattern, it is applying the configuration for all matching objects. In case some Ingress objects have a broken configuration, for example a syntax error in the nginx.ingress.kubernetes.io/configuration-snippet annotation, the generated configuration becomes invalid, does not reload and hence no more ingresses will be taken into account. To prevent this situation to happen, the nginx ingress controller optionally exposes a validating admission webhook server to ensure the validity of incoming ingress objects. This webhook appends the incoming ingress objects to the list of ingresses, generates the configuration and calls nginx to ensure the configuration has no syntax errors. + @@ -1397,9 +1398,9 @@ This webhook appends the incoming ingress objects to the list of ingresses, gene @@ -1409,7 +1410,7 @@ This webhook appends the incoming ingress objects to the list of ingresses, gene - + diff --git a/index.html b/index.html index f800d9138..c2ac90a9f 100644 --- a/index.html +++ b/index.html @@ -34,7 +34,7 @@ - + @@ -42,7 +42,7 @@ - + @@ -53,12 +53,12 @@ - + - + @@ -114,7 +114,7 @@ - + Skip to content @@ -123,7 +123,7 @@ - + public @@ -154,7 +154,7 @@ - + @@ -1202,6 +1202,7 @@ + @@ -1242,9 +1243,9 @@ @@ -1254,7 +1255,7 @@ - + diff --git a/kubectl-plugin/index.html b/kubectl-plugin/index.html index 85c81899f..0010e8b0b 100644 --- a/kubectl-plugin/index.html +++ b/kubectl-plugin/index.html @@ -34,7 +34,7 @@ - + @@ -42,7 +42,7 @@ - + @@ -53,12 +53,12 @@ - + - + @@ -114,7 +114,7 @@ - + Skip to content @@ -123,7 +123,7 @@ - + public @@ -154,7 +154,7 @@ - + @@ -1384,15 +1384,15 @@ Do not move it without providing redirects. The ingress-nginx kubectl plugin ¶ Installation ¶ Install krew, then run -kubectl krew install ingress-nginx - +kubectl krew install ingress-nginx + to install the plugin. Then run -kubectl ingress-nginx --help - +kubectl ingress-nginx --help + to make sure the plugin is properly installed and to get a list of commands: -kubectl ingress-nginx --help +kubectl ingress-nginx --help A kubectl plugin for inspecting your ingress-nginx deployments Usage: @@ -1430,29 +1430,29 @@ Do not move it without providing redirects. --user string The name of the kubeconfig user to use Use "ingress-nginx [command] --help" for more information about a command. - + -If a new ingress-nginx version has just been released, the plugin may not yet have been updated inside the repository. In that case, you can install the latest version of the plugin by running: -( +If a new ingress-nginx version has just been released, the plugin may not yet have been updated inside the repository. In that case, you can install the latest version of the plugin by running: +( set -x; cd "$(mktemp -d)" && curl -fsSLO "https://github.com/kubernetes/ingress-nginx/releases/download/nginx-0.24.0/{ingress-nginx.yaml,kubectl-ingress_nginx-$(uname | tr '[:upper:]' '[:lower:]')-amd64.tar.gz}" && kubectl krew install \ --manifest=ingress-nginx.yaml --archive=kubectl-ingress_nginx-$(uname | tr '[:upper:]' '[:lower:]')-amd64.tar.gz ) - + -Replacing 0.24.0 with the recently released version. +Replacing 0.24.0 with the recently released version. Common Flags ¶ -Every subcommand supports the basic kubectl configuration flags like --namespace, --context, --client-key and so on. -Subcommands that act on a particular ingress-nginx pod (backends, certs, conf, exec, general, logs, ssh), support the --deployment <deployment> and --pod <pod> flags to select either a pod from a deployment with the given name, or a pod with the given name. The --deployment flag defaults to nginx-ingress-controller. -Subcommands that inspect resources (ingresses, lint) support the --all-namespaces flag, which causes them to inspect resources in every namespace. +Every subcommand supports the basic kubectl configuration flags like --namespace, --context, --client-key and so on. +Subcommands that act on a particular ingress-nginx pod (backends, certs, conf, exec, general, logs, ssh), support the --deployment <deployment> and --pod <pod> flags to select either a pod from a deployment with the given name, or a pod with the given name. The --deployment flag defaults to nginx-ingress-controller. +Subcommands that inspect resources (ingresses, lint) support the --all-namespaces flag, which causes them to inspect resources in every namespace. Subcommands ¶ -Note that backends, general, certs, and conf require ingress-nginx version 0.23.0 or higher. +Note that backends, general, certs, and conf require ingress-nginx version 0.23.0 or higher. backends ¶ -Run kubectl ingress-nginx backends to get a JSON array of the backends that an ingress-nginx controller currently knows about: -$ kubectl ingress-nginx backends -n ingress-nginx +Run kubectl ingress-nginx backends to get a JSON array of the backends that an ingress-nginx controller currently knows about: +$ kubectl ingress-nginx backends -n ingress-nginx [ { "name": "default-apple-service-5678", @@ -1513,13 +1513,13 @@ Do not move it without providing redirects. ... } ] - + -Add the --list option to show only the backend names. Add the --backend <backend> option to show only the backend with the given name. +Add the --list option to show only the backend names. Add the --backend <backend> option to show only the backend with the given name. certs ¶ -Use kubectl ingress-nginx certs --host <hostname> to dump the SSL cert/key information for a given host. Requires that --enable-dynamic-certificates is true (this is the default as of version 0.24.0). +Use kubectl ingress-nginx certs --host <hostname> to dump the SSL cert/key information for a given host. Requires that --enable-dynamic-certificates is true (this is the default as of version 0.24.0). WARNING: This command will dump sensitive private key information. Don't blindly share the output, and certainly don't log it anywhere. -$ kubectl ingress-nginx certs -n ingress-nginx --host testaddr.local +$ kubectl ingress-nginx certs -n ingress-nginx --host testaddr.local -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- @@ -1530,11 +1530,11 @@ Do not move it without providing redirects. -----BEGIN RSA PRIVATE KEY----- <REDACTED! DO NOT SHARE THIS!> -----END RSA PRIVATE KEY----- - + conf ¶ -Use kubectl ingress-nginx conf to dump the generated nginx.conf file. Add the --host <hostname> option to view only the server block for that host: -kubectl ingress-nginx conf -n ingress-nginx --host testaddr.local +Use kubectl ingress-nginx conf to dump the generated nginx.conf file. Add the --host <hostname> option to view only the server block for that host: +kubectl ingress-nginx conf -n ingress-nginx --host testaddr.local server { server_name testaddr.local ; @@ -1556,11 +1556,11 @@ Do not move it without providing redirects. set $location_path "/"; ... - + exec ¶ -kubectl ingress-nginx exec is exactly the same as kubectl exec, with the same command flags. It will automatically choose an ingress-nginx pod to run the command in. -$ kubectl ingress-nginx exec -i -n ingress-nginx -- ls /etc/nginx +kubectl ingress-nginx exec is exactly the same as kubectl exec, with the same command flags. It will automatically choose an ingress-nginx pod to run the command in. +$ kubectl ingress-nginx exec -i -n ingress-nginx -- ls /etc/nginx fastcgi_params geoip lua @@ -1571,44 +1571,44 @@ Do not move it without providing redirects. opentracing.json owasp-modsecurity-crs template - + general ¶ -kubectl ingress-nginx general dumps miscellaneous controller state as a JSON object. Currently it just shows the number of controller pods known to a particular controller pod. -$ kubectl ingress-nginx general -n ingress-nginx +kubectl ingress-nginx general dumps miscellaneous controller state as a JSON object. Currently it just shows the number of controller pods known to a particular controller pod. +$ kubectl ingress-nginx general -n ingress-nginx { "controllerPodsCount": 1 } - + info ¶ -Shows the internal and external IP/CNAMES for an ingress-nginx service. -$ kubectl ingress-nginx info -n ingress-nginx +Shows the internal and external IP/CNAMES for an ingress-nginx service. +$ kubectl ingress-nginx info -n ingress-nginx Service cluster IP address: 10.187.253.31 LoadBalancer IP|CNAME: 35.123.123.123 - + -Use the --service <service> flag if your ingress-nginx LoadBalancer service is not named ingress-nginx. +Use the --service <service> flag if your ingress-nginx LoadBalancer service is not named ingress-nginx. ingresses ¶ -kubectl ingress-nginx ingresses, alternately kubectl ingress-nginx ing, shows a more detailed view of the ingress definitions in a namespace. Compare: -$ kubectl get ingresses --all-namespaces +kubectl ingress-nginx ingresses, alternately kubectl ingress-nginx ing, shows a more detailed view of the ingress definitions in a namespace. Compare: +$ kubectl get ingresses --all-namespaces NAMESPACE NAME HOSTS ADDRESS PORTS AGE default example-ingress1 testaddr.local,testaddr2.local localhost 80 5d default test-ingress-2 * localhost 80 5d - + vs -$ kubectl ingress-nginx ingresses --all-namespaces +$ kubectl ingress-nginx ingresses --all-namespaces NAMESPACE INGRESS NAME HOST+PATH ADDRESSES TLS SERVICE SERVICE PORT ENDPOINTS default example-ingress1 testaddr.local/etameta localhost NO pear-service 5678 5 default example-ingress1 testaddr2.local/otherpath localhost NO apple-service 5678 1 default example-ingress1 testaddr2.local/otherotherpath localhost NO pear-service 5678 5 default test-ingress-2 * localhost NO echo-service 8080 2 - + lint ¶ -kubectl ingress-nginx lint can check a namespace or entire cluster for potential configuration issues. This command is especially useful when upgrading between ingress-nginx versions. -$ kubectl ingress-nginx lint --all-namespaces --verbose +kubectl ingress-nginx lint can check a namespace or entire cluster for potential configuration issues. This command is especially useful when upgrading between ingress-nginx versions. +$ kubectl ingress-nginx lint --all-namespaces --verbose Checking ingresses... ✗ anamespace/this-nginx - Contains the removed session-cookie-hash annotation. @@ -1627,10 +1627,10 @@ Do not move it without providing redirects. - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808 - + -to show the lints added only for a particular ingress-nginx release, use the --from-version and --to-version flags: -$ kubectl ingress-nginx lint --all-namespaces --verbose --from-version 0.24.0 --to-version 0.24.0 +to show the lints added only for a particular ingress-nginx release, use the --from-version and --to-version flags: +$ kubectl ingress-nginx lint --all-namespaces --verbose --from-version 0.24.0 --to-version 0.24.0 Checking ingresses... ✗ anamespace/this-nginx - Contains the removed session-cookie-hash annotation. @@ -1642,11 +1642,11 @@ Do not move it without providing redirects. - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808 - + logs ¶ -kubectl ingress-nginx logs is almost the same as kubectl logs, with fewer flags. It will automatically choose an ingress-nginx pod to read logs from. -$ kubectl ingress-nginx logs -n ingress-nginx +kubectl ingress-nginx logs is almost the same as kubectl logs, with fewer flags. It will automatically choose an ingress-nginx pod to read logs from. +$ kubectl ingress-nginx logs -n ingress-nginx ------------------------------------------------------------------------------- NGINX Ingress controller Release: dev @@ -1662,16 +1662,17 @@ Do not move it without providing redirects. I0405 16:53:46.183359 7 nginx.go:265] Starting NGINX Ingress controller I0405 16:53:46.193913 7 event.go:209] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"82258915-563e-11e9-9c52-025000000001", APIVersion:"v1", ResourceVersion:"494", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services ... - + ssh ¶ -kubectl ingress-nginx ssh is exactly the same as kubectl ingress-nginx exec -it -- /bin/bash. Use it when you want to quickly be dropped into a shell inside a running ingress-nginx container. -$ kubectl ingress-nginx ssh -n ingress-nginx +kubectl ingress-nginx ssh is exactly the same as kubectl ingress-nginx exec -it -- /bin/bash. Use it when you want to quickly be dropped into a shell inside a running ingress-nginx container. +$ kubectl ingress-nginx ssh -n ingress-nginx www-data@nginx-ingress-controller-7cbf77c976-wx5pn:/etc/nginx$ - + + @@ -1726,9 +1727,9 @@ Do not move it without providing redirects. @@ -1738,7 +1739,7 @@ Do not move it without providing redirects. - + diff --git a/search/search_index.json b/search/search_index.json index 573e83c48..1340c42d6 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome \u00b6 This is the documentation for the NGINX Ingress Controller. It is built around the Kubernetes Ingress resource , using a ConfigMap to store the NGINX configuration. Learn more about using Ingress on k8s.io . Getting Started \u00b6 See Deployment for a whirlwind tour that will get you started.","title":"Welcome"},{"location":"#welcome","text":"This is the documentation for the NGINX Ingress Controller. It is built around the Kubernetes Ingress resource , using a ConfigMap to store the NGINX configuration. Learn more about using Ingress on k8s.io .","title":"Welcome"},{"location":"#getting-started","text":"See Deployment for a whirlwind tour that will get you started.","title":"Getting Started"},{"location":"development/","text":"Developing for NGINX Ingress Controller \u00b6 This document explains how to get started with developing for NGINX Ingress controller. It includes how to build, test, and release ingress controllers. Quick Start \u00b6 Getting the code \u00b6 The code must be checked out as a subdirectory of k8s.io, and not github.com. mkdir -p $GOPATH/src/k8s.io cd $GOPATH/src/k8s.io # Replace \"$YOUR_GITHUB_USERNAME\" below with your github username git clone https://github.com/$YOUR_GITHUB_USERNAME/ingress-nginx.git cd ingress-nginx Initial developer environment build \u00b6 Prequisites : Minikube must be installed. See releases for installation instructions. If you are using MacOS and deploying to minikube , the following command will build the local nginx controller container image and deploy the ingress controller onto a minikube cluster with RBAC enabled in the namespace ingress-nginx : $ make dev-env Updating the deployment \u00b6 The nginx controller container image can be rebuilt using: $ ARCH = amd64 TAG = dev REGISTRY = $USER /ingress-controller make build container The image will only be used by pods created after the rebuild. To delete old pods which will cause new ones to spin up: $ kubectl get pods -n ingress-nginx $ kubectl delete pod -n ingress-nginx nginx-ingress-controller- Dependencies \u00b6 The build uses dependencies in the vendor directory, which must be installed before building a binary/image. Occasionally, you might need to update the dependencies. This guide requires you to install go 1.13 or newer. This will automatically save the dependencies to the vendor/ directory. $ go get $ make dep-ensure Building \u00b6 All ingress controllers are built through a Makefile. Depending on your requirements you can build a raw server binary, a local container image, or push an image to a remote repository. In order to use your local Docker, you may need to set the following environment variables: # \"gcloud docker\" ( default ) or \"docker\" $ export DOCKER = # \"quay.io/kubernetes-ingress-controller\" ( default ) , \"index.docker.io\" , or your own registry $ export REGISTRY = To find the registry simply run: docker system info | grep Registry Building the e2e test image \u00b6 The e2e test image can also be built through the Makefile. $ make e2e-test-image You can then make this image available on your minikube host by exporting the image and loading it with the minikube docker context: $ docker save nginx-ingress-controller:e2e | ( eval $( minikube docker-env ) && docker load ) Nginx Controller \u00b6 Build a raw server binary $ make build TODO : add more specific instructions needed for raw server binary. Build a local container image $ TAG = REGISTRY = $USER /ingress-controller make container Push the container image to a remote repository $ TAG = REGISTRY = $USER /ingress-controller make push Deploying \u00b6 There are several ways to deploy the ingress controller onto a cluster. Please check the deployment guide Testing \u00b6 To run unit-tests, just run $ cd $GOPATH /src/k8s.io/ingress-nginx $ make test If you have access to a Kubernetes cluster, you can also run e2e tests using ginkgo. $ cd $GOPATH /src/k8s.io/ingress-nginx $ make e2e-test NOTE: if your e2e pod keeps hanging in an ImagePullBackoff, make sure you've made your e2e nginx-ingress-controller image available to minikube as explained in the Building the e2e test image section To run unit-tests for lua code locally, run: $ cd $GOPATH /src/k8s.io/ingress-nginx $ ./rootfs/etc/nginx/lua/test/up.sh $ make lua-test Lua tests are located in $GOPATH/src/k8s.io/ingress-nginx/rootfs/etc/nginx/lua/test . When creating a new test file it must follow the naming convention _test.lua or it will be ignored. Releasing \u00b6 All Makefiles will produce a release binary, as shown above. To publish this to a wider Kubernetes user base, push the image to a container registry, like gcr.io . All release images are hosted under gcr.io/google_containers and tagged according to a semver scheme. An example release might look like: $ make release Please follow these guidelines to cut a release: Update the release page with a short description of the major changes that correspond to a given image tag. Cut a release branch, if appropriate. Release branches follow the format of controller-release-version . Typically, pre-releases are cut from HEAD. All major feature work is done in HEAD. Specific bug fixes are cherry-picked into a release branch. If you're not confident about the stability of the code, tag it as alpha or beta. Typically, a release branch should have stable code.","title":"Development"},{"location":"development/#developing-for-nginx-ingress-controller","text":"This document explains how to get started with developing for NGINX Ingress controller. It includes how to build, test, and release ingress controllers.","title":"Developing for NGINX Ingress Controller"},{"location":"development/#quick-start","text":"","title":"Quick Start"},{"location":"development/#getting-the-code","text":"The code must be checked out as a subdirectory of k8s.io, and not github.com. mkdir -p $GOPATH/src/k8s.io cd $GOPATH/src/k8s.io # Replace \"$YOUR_GITHUB_USERNAME\" below with your github username git clone https://github.com/$YOUR_GITHUB_USERNAME/ingress-nginx.git cd ingress-nginx","title":"Getting the code"},{"location":"development/#initial-developer-environment-build","text":"Prequisites : Minikube must be installed. See releases for installation instructions. If you are using MacOS and deploying to minikube , the following command will build the local nginx controller container image and deploy the ingress controller onto a minikube cluster with RBAC enabled in the namespace ingress-nginx : $ make dev-env","title":"Initial developer environment build"},{"location":"development/#updating-the-deployment","text":"The nginx controller container image can be rebuilt using: $ ARCH = amd64 TAG = dev REGISTRY = $USER /ingress-controller make build container The image will only be used by pods created after the rebuild. To delete old pods which will cause new ones to spin up: $ kubectl get pods -n ingress-nginx $ kubectl delete pod -n ingress-nginx nginx-ingress-controller-","title":"Updating the deployment"},{"location":"development/#dependencies","text":"The build uses dependencies in the vendor directory, which must be installed before building a binary/image. Occasionally, you might need to update the dependencies. This guide requires you to install go 1.13 or newer. This will automatically save the dependencies to the vendor/ directory. $ go get $ make dep-ensure","title":"Dependencies"},{"location":"development/#building","text":"All ingress controllers are built through a Makefile. Depending on your requirements you can build a raw server binary, a local container image, or push an image to a remote repository. In order to use your local Docker, you may need to set the following environment variables: # \"gcloud docker\" ( default ) or \"docker\" $ export DOCKER = # \"quay.io/kubernetes-ingress-controller\" ( default ) , \"index.docker.io\" , or your own registry $ export REGISTRY = To find the registry simply run: docker system info | grep Registry","title":"Building"},{"location":"development/#building-the-e2e-test-image","text":"The e2e test image can also be built through the Makefile. $ make e2e-test-image You can then make this image available on your minikube host by exporting the image and loading it with the minikube docker context: $ docker save nginx-ingress-controller:e2e | ( eval $( minikube docker-env ) && docker load )","title":"Building the e2e test image"},{"location":"development/#nginx-controller","text":"Build a raw server binary $ make build TODO : add more specific instructions needed for raw server binary. Build a local container image $ TAG = REGISTRY = $USER /ingress-controller make container Push the container image to a remote repository $ TAG = REGISTRY = $USER /ingress-controller make push","title":"Nginx Controller"},{"location":"development/#deploying","text":"There are several ways to deploy the ingress controller onto a cluster. Please check the deployment guide","title":"Deploying"},{"location":"development/#testing","text":"To run unit-tests, just run $ cd $GOPATH /src/k8s.io/ingress-nginx $ make test If you have access to a Kubernetes cluster, you can also run e2e tests using ginkgo. $ cd $GOPATH /src/k8s.io/ingress-nginx $ make e2e-test NOTE: if your e2e pod keeps hanging in an ImagePullBackoff, make sure you've made your e2e nginx-ingress-controller image available to minikube as explained in the Building the e2e test image section To run unit-tests for lua code locally, run: $ cd $GOPATH /src/k8s.io/ingress-nginx $ ./rootfs/etc/nginx/lua/test/up.sh $ make lua-test Lua tests are located in $GOPATH/src/k8s.io/ingress-nginx/rootfs/etc/nginx/lua/test . When creating a new test file it must follow the naming convention _test.lua or it will be ignored.","title":"Testing"},{"location":"development/#releasing","text":"All Makefiles will produce a release binary, as shown above. To publish this to a wider Kubernetes user base, push the image to a container registry, like gcr.io . All release images are hosted under gcr.io/google_containers and tagged according to a semver scheme. An example release might look like: $ make release Please follow these guidelines to cut a release: Update the release page with a short description of the major changes that correspond to a given image tag. Cut a release branch, if appropriate. Release branches follow the format of controller-release-version . Typically, pre-releases are cut from HEAD. All major feature work is done in HEAD. Specific bug fixes are cherry-picked into a release branch. If you're not confident about the stability of the code, tag it as alpha or beta. Typically, a release branch should have stable code.","title":"Releasing"},{"location":"how-it-works/","text":"How it works \u00b6 The objective of this document is to explain how the NGINX Ingress controller works, in particular how the NGINX model is built and why we need one. NGINX configuration \u00b6 The goal of this Ingress controller is the assembly of a configuration file (nginx.conf). The main implication of this requirement is the need to reload NGINX after any change in the configuration file. Though it is important to note that we don't reload Nginx on changes that impact only an upstream configuration (i.e Endpoints change when you deploy your app) . We use lua-nginx-module to achieve this. Check below to learn more about how it's done. NGINX model \u00b6 Usually, a Kubernetes Controller utilizes the synchronization loop pattern to check if the desired state in the controller is updated or a change is required. To this purpose, we need to build a model using different objects from the cluster, in particular (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster. To get this object from the cluster, we use Kubernetes Informers , in particular, FilteredSharedInformer . This informers allows reacting to changes in using callbacks to individual changes when a new object is added, modified or removed. Unfortunately, there is no way to know if a particular change is going to affect the final configuration file. Therefore on every change, we have to rebuild a new model from scratch based on the state of cluster and compare it to the current model. If the new model equals to the current one, then we avoid generating a new NGINX configuration and triggering a reload. Otherwise, we check if the difference is only about Endpoints. If so we then send the new list of Endpoints to a Lua handler running inside Nginx using HTTP POST request and again avoid generating a new NGINX configuration and triggering a reload. If the difference between running and new model is about more than just Endpoints we create a new NGINX configuration based on the new model, replace the current model and trigger a reload. One of the uses of the model is to avoid unnecessary reloads when there's no change in the state and to detect conflicts in definitions. The final representation of the NGINX configuration is generated from a Go template using the new model as input for the variables required by the template. Building the NGINX model \u00b6 Building a model is an expensive operation, for this reason, the use of the synchronization loop is a must. By using a work queue it is possible to not lose changes and remove the use of sync.Mutex to force a single execution of the sync loop and additionally it is possible to create a time window between the start and end of the sync loop that allows us to discard unnecessary updates. It is important to understand that any change in the cluster could generate events that the informer will send to the controller and one of the reasons for the work queue . Operations to build the model: Order Ingress rules by CreationTimestamp field, i.e., old rules first. If the same path for the same host is defined in more than one Ingress, the oldest rule wins. If more than one Ingress contains a TLS section for the same host, the oldest rule wins. If multiple Ingresses define an annotation that affects the configuration of the Server block, the oldest rule wins. Create a list of NGINX Servers (per hostname) Create a list of NGINX Upstreams If multiple Ingresses define different paths for the same host, the ingress controller will merge the definitions. Annotations are applied to all the paths in the Ingress. Multiple Ingresses can define different annotations. These definitions are not shared between Ingresses. When a reload is required \u00b6 The next list describes the scenarios when a reload is required: New Ingress Resource Created. TLS section is added to existing Ingress. Change in Ingress annotations that impacts more than just upstream configuration. For instance load-balance annotation does not require a reload. A path is added/removed from an Ingress. An Ingress, Service, Secret is removed. Some missing referenced object from the Ingress is available, like a Service or Secret. A Secret is updated. Avoiding reloads \u00b6 In some cases, it is possible to avoid reloads, in particular when there is a change in the endpoints, i.e., a pod is started or replaced. It is out of the scope of this Ingress controller to remove reloads completely. This would require an incredible amount of work and at some point makes no sense. This can change only if NGINX changes the way new configurations are read, basically, new changes do not replace worker processes. Avoiding reloads on Endpoints changes \u00b6 On every endpoint change the controller fetches endpoints from all the services it sees and generates corresponding Backend objects. It then sends these objects to a Lua handler running inside Nginx. The Lua code in turn stores those backends in a shared memory zone. Then for every request Lua code running in balancer_by_lua context detects what endpoints it should choose upstream peer from and applies the configured load balancing algorithm to choose the peer. Then Nginx takes care of the rest. This way we avoid reloading Nginx on endpoint changes. Note that this includes annotation changes that affects only upstream configuration in Nginx as well. In a relatively big clusters with frequently deploying apps this feature saves significant number of Nginx reloads which can otherwise affect response latency, load balancing quality (after every reload Nginx resets the state of load balancing) and so on. Avoiding outage from wrong configuration \u00b6 Because the ingress controller works using the synchronization loop pattern , it is applying the configuration for all matching objects. In case some Ingress objects have a broken configuration, for example a syntax error in the nginx.ingress.kubernetes.io/configuration-snippet annotation, the generated configuration becomes invalid, does not reload and hence no more ingresses will be taken into account. To prevent this situation to happen, the nginx ingress controller optionally exposes a validating admission webhook server to ensure the validity of incoming ingress objects. This webhook appends the incoming ingress objects to the list of ingresses, generates the configuration and calls nginx to ensure the configuration has no syntax errors.","title":"How it works"},{"location":"how-it-works/#how-it-works","text":"The objective of this document is to explain how the NGINX Ingress controller works, in particular how the NGINX model is built and why we need one.","title":"How it works"},{"location":"how-it-works/#nginx-configuration","text":"The goal of this Ingress controller is the assembly of a configuration file (nginx.conf). The main implication of this requirement is the need to reload NGINX after any change in the configuration file. Though it is important to note that we don't reload Nginx on changes that impact only an upstream configuration (i.e Endpoints change when you deploy your app) . We use lua-nginx-module to achieve this. Check below to learn more about how it's done.","title":"NGINX configuration"},{"location":"how-it-works/#nginx-model","text":"Usually, a Kubernetes Controller utilizes the synchronization loop pattern to check if the desired state in the controller is updated or a change is required. To this purpose, we need to build a model using different objects from the cluster, in particular (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster. To get this object from the cluster, we use Kubernetes Informers , in particular, FilteredSharedInformer . This informers allows reacting to changes in using callbacks to individual changes when a new object is added, modified or removed. Unfortunately, there is no way to know if a particular change is going to affect the final configuration file. Therefore on every change, we have to rebuild a new model from scratch based on the state of cluster and compare it to the current model. If the new model equals to the current one, then we avoid generating a new NGINX configuration and triggering a reload. Otherwise, we check if the difference is only about Endpoints. If so we then send the new list of Endpoints to a Lua handler running inside Nginx using HTTP POST request and again avoid generating a new NGINX configuration and triggering a reload. If the difference between running and new model is about more than just Endpoints we create a new NGINX configuration based on the new model, replace the current model and trigger a reload. One of the uses of the model is to avoid unnecessary reloads when there's no change in the state and to detect conflicts in definitions. The final representation of the NGINX configuration is generated from a Go template using the new model as input for the variables required by the template.","title":"NGINX model"},{"location":"how-it-works/#building-the-nginx-model","text":"Building a model is an expensive operation, for this reason, the use of the synchronization loop is a must. By using a work queue it is possible to not lose changes and remove the use of sync.Mutex to force a single execution of the sync loop and additionally it is possible to create a time window between the start and end of the sync loop that allows us to discard unnecessary updates. It is important to understand that any change in the cluster could generate events that the informer will send to the controller and one of the reasons for the work queue . Operations to build the model: Order Ingress rules by CreationTimestamp field, i.e., old rules first. If the same path for the same host is defined in more than one Ingress, the oldest rule wins. If more than one Ingress contains a TLS section for the same host, the oldest rule wins. If multiple Ingresses define an annotation that affects the configuration of the Server block, the oldest rule wins. Create a list of NGINX Servers (per hostname) Create a list of NGINX Upstreams If multiple Ingresses define different paths for the same host, the ingress controller will merge the definitions. Annotations are applied to all the paths in the Ingress. Multiple Ingresses can define different annotations. These definitions are not shared between Ingresses.","title":"Building the NGINX model"},{"location":"how-it-works/#when-a-reload-is-required","text":"The next list describes the scenarios when a reload is required: New Ingress Resource Created. TLS section is added to existing Ingress. Change in Ingress annotations that impacts more than just upstream configuration. For instance load-balance annotation does not require a reload. A path is added/removed from an Ingress. An Ingress, Service, Secret is removed. Some missing referenced object from the Ingress is available, like a Service or Secret. A Secret is updated.","title":"When a reload is required"},{"location":"how-it-works/#avoiding-reloads","text":"In some cases, it is possible to avoid reloads, in particular when there is a change in the endpoints, i.e., a pod is started or replaced. It is out of the scope of this Ingress controller to remove reloads completely. This would require an incredible amount of work and at some point makes no sense. This can change only if NGINX changes the way new configurations are read, basically, new changes do not replace worker processes.","title":"Avoiding reloads"},{"location":"how-it-works/#avoiding-reloads-on-endpoints-changes","text":"On every endpoint change the controller fetches endpoints from all the services it sees and generates corresponding Backend objects. It then sends these objects to a Lua handler running inside Nginx. The Lua code in turn stores those backends in a shared memory zone. Then for every request Lua code running in balancer_by_lua context detects what endpoints it should choose upstream peer from and applies the configured load balancing algorithm to choose the peer. Then Nginx takes care of the rest. This way we avoid reloading Nginx on endpoint changes. Note that this includes annotation changes that affects only upstream configuration in Nginx as well. In a relatively big clusters with frequently deploying apps this feature saves significant number of Nginx reloads which can otherwise affect response latency, load balancing quality (after every reload Nginx resets the state of load balancing) and so on.","title":"Avoiding reloads on Endpoints changes"},{"location":"how-it-works/#avoiding-outage-from-wrong-configuration","text":"Because the ingress controller works using the synchronization loop pattern , it is applying the configuration for all matching objects. In case some Ingress objects have a broken configuration, for example a syntax error in the nginx.ingress.kubernetes.io/configuration-snippet annotation, the generated configuration becomes invalid, does not reload and hence no more ingresses will be taken into account. To prevent this situation to happen, the nginx ingress controller optionally exposes a validating admission webhook server to ensure the validity of incoming ingress objects. This webhook appends the incoming ingress objects to the list of ingresses, generates the configuration and calls nginx to ensure the configuration has no syntax errors.","title":"Avoiding outage from wrong configuration"},{"location":"kubectl-plugin/","text":"The ingress-nginx kubectl plugin \u00b6 Installation \u00b6 Install krew , then run kubectl krew install ingress-nginx to install the plugin. Then run kubectl ingress-nginx --help to make sure the plugin is properly installed and to get a list of commands: kubectl ingress-nginx --help A kubectl plugin for inspecting your ingress-nginx deployments Usage: ingress-nginx [command] Available Commands: backends Inspect the dynamic backend information of an ingress-nginx instance certs Output the certificate data stored in an ingress-nginx pod conf Inspect the generated nginx.conf exec Execute a command inside an ingress-nginx pod general Inspect the other dynamic ingress-nginx information help Help about any command info Show information about the ingress-nginx service ingresses Provide a short summary of all of the ingress definitions lint Inspect kubernetes resources for possible issues logs Get the kubernetes logs for an ingress-nginx pod ssh ssh into a running ingress-nginx pod Flags: --as string Username to impersonate for the operation --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. --cache-dir string Default HTTP cache directory (default \"/Users/alexkursell/.kube/http-cache\") --certificate-authority string Path to a cert file for the certificate authority --client-certificate string Path to a client certificate file for TLS --client-key string Path to a client key file for TLS --cluster string The name of the kubeconfig cluster to use --context string The name of the kubeconfig context to use -h, --help help for ingress-nginx --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure --kubeconfig string Path to the kubeconfig file to use for CLI requests. -n, --namespace string If present, the namespace scope for this CLI request --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\") -s, --server string The address and port of the Kubernetes API server --token string Bearer token for authentication to the API server --user string The name of the kubeconfig user to use Use \"ingress-nginx [command] --help\" for more information about a command. If a new ingress-nginx version has just been released, the plugin may not yet have been updated inside the repository. In that case, you can install the latest version of the plugin by running: ( set -x; cd \"$(mktemp -d)\" && curl -fsSLO \"https://github.com/kubernetes/ingress-nginx/releases/download/nginx-0.24.0/{ingress-nginx.yaml,kubectl-ingress_nginx-$(uname | tr '[:upper:]' '[:lower:]')-amd64.tar.gz}\" && kubectl krew install \\ --manifest=ingress-nginx.yaml --archive=kubectl-ingress_nginx-$(uname | tr '[:upper:]' '[:lower:]')-amd64.tar.gz ) Replacing 0.24.0 with the recently released version. Common Flags \u00b6 Every subcommand supports the basic kubectl configuration flags like --namespace , --context , --client-key and so on. Subcommands that act on a particular ingress-nginx pod ( backends , certs , conf , exec , general , logs , ssh ), support the --deployment and --pod flags to select either a pod from a deployment with the given name, or a pod with the given name. The --deployment flag defaults to nginx-ingress-controller . Subcommands that inspect resources ( ingresses , lint ) support the --all-namespaces flag, which causes them to inspect resources in every namespace. Subcommands \u00b6 Note that backends , general , certs , and conf require ingress-nginx version 0.23.0 or higher. backends \u00b6 Run kubectl ingress-nginx backends to get a JSON array of the backends that an ingress-nginx controller currently knows about: $ kubectl ingress-nginx backends -n ingress-nginx [ { \"name\": \"default-apple-service-5678\", \"service\": { \"metadata\": { \"creationTimestamp\": null }, \"spec\": { \"ports\": [ { \"protocol\": \"TCP\", \"port\": 5678, \"targetPort\": 5678 } ], \"selector\": { \"app\": \"apple\" }, \"clusterIP\": \"10.97.230.121\", \"type\": \"ClusterIP\", \"sessionAffinity\": \"None\" }, \"status\": { \"loadBalancer\": {} } }, \"port\": 0, \"sslPassthrough\": false, \"endpoints\": [ { \"address\": \"10.1.3.86\", \"port\": \"5678\" } ], \"sessionAffinityConfig\": { \"name\": \"\", \"cookieSessionAffinity\": { \"name\": \"\" } }, \"upstreamHashByConfig\": { \"upstream-hash-by-subset-size\": 3 }, \"noServer\": false, \"trafficShapingPolicy\": { \"weight\": 0, \"header\": \"\", \"headerValue\": \"\", \"cookie\": \"\" } }, { \"name\": \"default-echo-service-8080\", ... }, { \"name\": \"upstream-default-backend\", ... } ] Add the --list option to show only the backend names. Add the --backend option to show only the backend with the given name. certs \u00b6 Use kubectl ingress-nginx certs --host to dump the SSL cert/key information for a given host. Requires that --enable-dynamic-certificates is true (this is the default as of version 0.24.0 ). WARNING: This command will dump sensitive private key information. Don't blindly share the output, and certainly don't log it anywhere. $ kubectl ingress-nginx certs -n ingress-nginx --host testaddr.local -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN RSA PRIVATE KEY----- -----END RSA PRIVATE KEY----- conf \u00b6 Use kubectl ingress-nginx conf to dump the generated nginx.conf file. Add the --host option to view only the server block for that host: kubectl ingress-nginx conf -n ingress-nginx --host testaddr.local server { server_name testaddr.local ; listen 80; set $proxy_upstream_name \"-\"; set $pass_access_scheme $scheme; set $pass_server_port $server_port; set $best_http_host $http_host; set $pass_port $pass_server_port; location / { set $namespace \"\"; set $ingress_name \"\"; set $service_name \"\"; set $service_port \"0\"; set $location_path \"/\"; ... exec \u00b6 kubectl ingress-nginx exec is exactly the same as kubectl exec , with the same command flags. It will automatically choose an ingress-nginx pod to run the command in. $ kubectl ingress-nginx exec -i -n ingress-nginx -- ls /etc/nginx fastcgi_params geoip lua mime.types modsecurity modules nginx.conf opentracing.json owasp-modsecurity-crs template general \u00b6 kubectl ingress-nginx general dumps miscellaneous controller state as a JSON object. Currently it just shows the number of controller pods known to a particular controller pod. $ kubectl ingress-nginx general -n ingress-nginx { \"controllerPodsCount\": 1 } info \u00b6 Shows the internal and external IP/CNAMES for an ingress-nginx service. $ kubectl ingress-nginx info -n ingress-nginx Service cluster IP address: 10.187.253.31 LoadBalancer IP|CNAME: 35.123.123.123 Use the --service flag if your ingress-nginx LoadBalancer service is not named ingress-nginx . ingresses \u00b6 kubectl ingress-nginx ingresses , alternately kubectl ingress-nginx ing , shows a more detailed view of the ingress definitions in a namespace. Compare: $ kubectl get ingresses --all-namespaces NAMESPACE NAME HOSTS ADDRESS PORTS AGE default example-ingress1 testaddr.local,testaddr2.local localhost 80 5d default test-ingress-2 * localhost 80 5d vs $ kubectl ingress-nginx ingresses --all-namespaces NAMESPACE INGRESS NAME HOST+PATH ADDRESSES TLS SERVICE SERVICE PORT ENDPOINTS default example-ingress1 testaddr.local/etameta localhost NO pear-service 5678 5 default example-ingress1 testaddr2.local/otherpath localhost NO apple-service 5678 1 default example-ingress1 testaddr2.local/otherotherpath localhost NO pear-service 5678 5 default test-ingress-2 * localhost NO echo-service 8080 2 lint \u00b6 kubectl ingress-nginx lint can check a namespace or entire cluster for potential configuration issues. This command is especially useful when upgrading between ingress-nginx versions. $ kubectl ingress-nginx lint --all-namespaces --verbose Checking ingresses... \u2717 anamespace/this-nginx - Contains the removed session-cookie-hash annotation. Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3743 \u2717 othernamespace/ingress-definition-blah - The rewrite-target annotation value does not reference a capture group Lint added for version 0.22.0 https://github.com/kubernetes/ingress-nginx/issues/3174 Checking deployments... \u2717 namespace2/nginx-ingress-controller - Uses removed config flag --sort-backends Lint added for version 0.22.0 https://github.com/kubernetes/ingress-nginx/issues/3655 - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808 to show the lints added only for a particular ingress-nginx release, use the --from-version and --to-version flags: $ kubectl ingress-nginx lint --all-namespaces --verbose --from-version 0 .24.0 --to-version 0 .24.0 Checking ingresses... \u2717 anamespace/this-nginx - Contains the removed session-cookie-hash annotation. Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3743 Checking deployments... \u2717 namespace2/nginx-ingress-controller - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808 logs \u00b6 kubectl ingress-nginx logs is almost the same as kubectl logs , with fewer flags. It will automatically choose an ingress-nginx pod to read logs from. $ kubectl ingress-nginx logs -n ingress-nginx ------------------------------------------------------------------------------- NGINX Ingress controller Release: dev Build: git-48dc3a867 Repository: git@github.com:kubernetes/ingress-nginx.git ------------------------------------------------------------------------------- W0405 16:53:46.061589 7 flags.go:214] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false) nginx version: nginx/1.15.9 W0405 16:53:46.070093 7 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0405 16:53:46.070499 7 main.go:205] Creating API client for https://10.96.0.1:443 I0405 16:53:46.077784 7 main.go:249] Running in Kubernetes cluster version v1.10 (v1.10.11) - git (clean) commit 637c7e288581ee40ab4ca210618a89a555b6e7e9 - platform linux/amd64 I0405 16:53:46.183359 7 nginx.go:265] Starting NGINX Ingress controller I0405 16:53:46.193913 7 event.go:209] Event(v1.ObjectReference{Kind:\"ConfigMap\", Namespace:\"ingress-nginx\", Name:\"udp-services\", UID:\"82258915-563e-11e9-9c52-025000000001\", APIVersion:\"v1\", ResourceVersion:\"494\", FieldPath:\"\"}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services ... ssh \u00b6 kubectl ingress-nginx ssh is exactly the same as kubectl ingress-nginx exec -it -- /bin/bash . Use it when you want to quickly be dropped into a shell inside a running ingress-nginx container. $ kubectl ingress-nginx ssh -n ingress-nginx www-data@nginx-ingress-controller-7cbf77c976-wx5pn:/etc/nginx$","title":"kubectl plugin"},{"location":"kubectl-plugin/#the-ingress-nginx-kubectl-plugin","text":"","title":"The ingress-nginx kubectl plugin"},{"location":"kubectl-plugin/#installation","text":"Install krew , then run kubectl krew install ingress-nginx to install the plugin. Then run kubectl ingress-nginx --help to make sure the plugin is properly installed and to get a list of commands: kubectl ingress-nginx --help A kubectl plugin for inspecting your ingress-nginx deployments Usage: ingress-nginx [command] Available Commands: backends Inspect the dynamic backend information of an ingress-nginx instance certs Output the certificate data stored in an ingress-nginx pod conf Inspect the generated nginx.conf exec Execute a command inside an ingress-nginx pod general Inspect the other dynamic ingress-nginx information help Help about any command info Show information about the ingress-nginx service ingresses Provide a short summary of all of the ingress definitions lint Inspect kubernetes resources for possible issues logs Get the kubernetes logs for an ingress-nginx pod ssh ssh into a running ingress-nginx pod Flags: --as string Username to impersonate for the operation --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. --cache-dir string Default HTTP cache directory (default \"/Users/alexkursell/.kube/http-cache\") --certificate-authority string Path to a cert file for the certificate authority --client-certificate string Path to a client certificate file for TLS --client-key string Path to a client key file for TLS --cluster string The name of the kubeconfig cluster to use --context string The name of the kubeconfig context to use -h, --help help for ingress-nginx --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure --kubeconfig string Path to the kubeconfig file to use for CLI requests. -n, --namespace string If present, the namespace scope for this CLI request --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\") -s, --server string The address and port of the Kubernetes API server --token string Bearer token for authentication to the API server --user string The name of the kubeconfig user to use Use \"ingress-nginx [command] --help\" for more information about a command. If a new ingress-nginx version has just been released, the plugin may not yet have been updated inside the repository. In that case, you can install the latest version of the plugin by running: ( set -x; cd \"$(mktemp -d)\" && curl -fsSLO \"https://github.com/kubernetes/ingress-nginx/releases/download/nginx-0.24.0/{ingress-nginx.yaml,kubectl-ingress_nginx-$(uname | tr '[:upper:]' '[:lower:]')-amd64.tar.gz}\" && kubectl krew install \\ --manifest=ingress-nginx.yaml --archive=kubectl-ingress_nginx-$(uname | tr '[:upper:]' '[:lower:]')-amd64.tar.gz ) Replacing 0.24.0 with the recently released version.","title":"Installation"},{"location":"kubectl-plugin/#common-flags","text":"Every subcommand supports the basic kubectl configuration flags like --namespace , --context , --client-key and so on. Subcommands that act on a particular ingress-nginx pod ( backends , certs , conf , exec , general , logs , ssh ), support the --deployment and --pod flags to select either a pod from a deployment with the given name, or a pod with the given name. The --deployment flag defaults to nginx-ingress-controller . Subcommands that inspect resources ( ingresses , lint ) support the --all-namespaces flag, which causes them to inspect resources in every namespace.","title":"Common Flags"},{"location":"kubectl-plugin/#subcommands","text":"Note that backends , general , certs , and conf require ingress-nginx version 0.23.0 or higher.","title":"Subcommands"},{"location":"kubectl-plugin/#backends","text":"Run kubectl ingress-nginx backends to get a JSON array of the backends that an ingress-nginx controller currently knows about: $ kubectl ingress-nginx backends -n ingress-nginx [ { \"name\": \"default-apple-service-5678\", \"service\": { \"metadata\": { \"creationTimestamp\": null }, \"spec\": { \"ports\": [ { \"protocol\": \"TCP\", \"port\": 5678, \"targetPort\": 5678 } ], \"selector\": { \"app\": \"apple\" }, \"clusterIP\": \"10.97.230.121\", \"type\": \"ClusterIP\", \"sessionAffinity\": \"None\" }, \"status\": { \"loadBalancer\": {} } }, \"port\": 0, \"sslPassthrough\": false, \"endpoints\": [ { \"address\": \"10.1.3.86\", \"port\": \"5678\" } ], \"sessionAffinityConfig\": { \"name\": \"\", \"cookieSessionAffinity\": { \"name\": \"\" } }, \"upstreamHashByConfig\": { \"upstream-hash-by-subset-size\": 3 }, \"noServer\": false, \"trafficShapingPolicy\": { \"weight\": 0, \"header\": \"\", \"headerValue\": \"\", \"cookie\": \"\" } }, { \"name\": \"default-echo-service-8080\", ... }, { \"name\": \"upstream-default-backend\", ... } ] Add the --list option to show only the backend names. Add the --backend option to show only the backend with the given name.","title":"backends"},{"location":"kubectl-plugin/#certs","text":"Use kubectl ingress-nginx certs --host to dump the SSL cert/key information for a given host. Requires that --enable-dynamic-certificates is true (this is the default as of version 0.24.0 ). WARNING: This command will dump sensitive private key information. Don't blindly share the output, and certainly don't log it anywhere. $ kubectl ingress-nginx certs -n ingress-nginx --host testaddr.local -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN RSA PRIVATE KEY----- -----END RSA PRIVATE KEY-----","title":"certs"},{"location":"kubectl-plugin/#conf","text":"Use kubectl ingress-nginx conf to dump the generated nginx.conf file. Add the --host option to view only the server block for that host: kubectl ingress-nginx conf -n ingress-nginx --host testaddr.local server { server_name testaddr.local ; listen 80; set $proxy_upstream_name \"-\"; set $pass_access_scheme $scheme; set $pass_server_port $server_port; set $best_http_host $http_host; set $pass_port $pass_server_port; location / { set $namespace \"\"; set $ingress_name \"\"; set $service_name \"\"; set $service_port \"0\"; set $location_path \"/\"; ...","title":"conf"},{"location":"kubectl-plugin/#exec","text":"kubectl ingress-nginx exec is exactly the same as kubectl exec , with the same command flags. It will automatically choose an ingress-nginx pod to run the command in. $ kubectl ingress-nginx exec -i -n ingress-nginx -- ls /etc/nginx fastcgi_params geoip lua mime.types modsecurity modules nginx.conf opentracing.json owasp-modsecurity-crs template","title":"exec"},{"location":"kubectl-plugin/#general","text":"kubectl ingress-nginx general dumps miscellaneous controller state as a JSON object. Currently it just shows the number of controller pods known to a particular controller pod. $ kubectl ingress-nginx general -n ingress-nginx { \"controllerPodsCount\": 1 }","title":"general"},{"location":"kubectl-plugin/#info","text":"Shows the internal and external IP/CNAMES for an ingress-nginx service. $ kubectl ingress-nginx info -n ingress-nginx Service cluster IP address: 10.187.253.31 LoadBalancer IP|CNAME: 35.123.123.123 Use the --service flag if your ingress-nginx LoadBalancer service is not named ingress-nginx .","title":"info"},{"location":"kubectl-plugin/#ingresses","text":"kubectl ingress-nginx ingresses , alternately kubectl ingress-nginx ing , shows a more detailed view of the ingress definitions in a namespace. Compare: $ kubectl get ingresses --all-namespaces NAMESPACE NAME HOSTS ADDRESS PORTS AGE default example-ingress1 testaddr.local,testaddr2.local localhost 80 5d default test-ingress-2 * localhost 80 5d vs $ kubectl ingress-nginx ingresses --all-namespaces NAMESPACE INGRESS NAME HOST+PATH ADDRESSES TLS SERVICE SERVICE PORT ENDPOINTS default example-ingress1 testaddr.local/etameta localhost NO pear-service 5678 5 default example-ingress1 testaddr2.local/otherpath localhost NO apple-service 5678 1 default example-ingress1 testaddr2.local/otherotherpath localhost NO pear-service 5678 5 default test-ingress-2 * localhost NO echo-service 8080 2","title":"ingresses"},{"location":"kubectl-plugin/#lint","text":"kubectl ingress-nginx lint can check a namespace or entire cluster for potential configuration issues. This command is especially useful when upgrading between ingress-nginx versions. $ kubectl ingress-nginx lint --all-namespaces --verbose Checking ingresses... \u2717 anamespace/this-nginx - Contains the removed session-cookie-hash annotation. Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3743 \u2717 othernamespace/ingress-definition-blah - The rewrite-target annotation value does not reference a capture group Lint added for version 0.22.0 https://github.com/kubernetes/ingress-nginx/issues/3174 Checking deployments... \u2717 namespace2/nginx-ingress-controller - Uses removed config flag --sort-backends Lint added for version 0.22.0 https://github.com/kubernetes/ingress-nginx/issues/3655 - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808 to show the lints added only for a particular ingress-nginx release, use the --from-version and --to-version flags: $ kubectl ingress-nginx lint --all-namespaces --verbose --from-version 0 .24.0 --to-version 0 .24.0 Checking ingresses... \u2717 anamespace/this-nginx - Contains the removed session-cookie-hash annotation. Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3743 Checking deployments... \u2717 namespace2/nginx-ingress-controller - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808","title":"lint"},{"location":"kubectl-plugin/#logs","text":"kubectl ingress-nginx logs is almost the same as kubectl logs , with fewer flags. It will automatically choose an ingress-nginx pod to read logs from. $ kubectl ingress-nginx logs -n ingress-nginx ------------------------------------------------------------------------------- NGINX Ingress controller Release: dev Build: git-48dc3a867 Repository: git@github.com:kubernetes/ingress-nginx.git ------------------------------------------------------------------------------- W0405 16:53:46.061589 7 flags.go:214] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false) nginx version: nginx/1.15.9 W0405 16:53:46.070093 7 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0405 16:53:46.070499 7 main.go:205] Creating API client for https://10.96.0.1:443 I0405 16:53:46.077784 7 main.go:249] Running in Kubernetes cluster version v1.10 (v1.10.11) - git (clean) commit 637c7e288581ee40ab4ca210618a89a555b6e7e9 - platform linux/amd64 I0405 16:53:46.183359 7 nginx.go:265] Starting NGINX Ingress controller I0405 16:53:46.193913 7 event.go:209] Event(v1.ObjectReference{Kind:\"ConfigMap\", Namespace:\"ingress-nginx\", Name:\"udp-services\", UID:\"82258915-563e-11e9-9c52-025000000001\", APIVersion:\"v1\", ResourceVersion:\"494\", FieldPath:\"\"}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services ...","title":"logs"},{"location":"kubectl-plugin/#ssh","text":"kubectl ingress-nginx ssh is exactly the same as kubectl ingress-nginx exec -it -- /bin/bash . Use it when you want to quickly be dropped into a shell inside a running ingress-nginx container. $ kubectl ingress-nginx ssh -n ingress-nginx www-data@nginx-ingress-controller-7cbf77c976-wx5pn:/etc/nginx$","title":"ssh"},{"location":"troubleshooting/","text":"Troubleshooting \u00b6 Ingress-Controller Logs and Events \u00b6 There are many ways to troubleshoot the ingress-controller. The following are basic troubleshooting methods to obtain more information. Check the Ingress Resource Events $ kubectl get ing -n NAME HOSTS ADDRESS PORTS AGE cafe-ingress cafe.com 10.0.2.15 80 25s $ kubectl describe ing -n Name: cafe-ingress Namespace: default Address: 10.0.2.15 Default backend: default-http-backend:80 (172.17.0.5:8080) Rules: Host Path Backends ---- ---- -------- cafe.com /tea tea-svc:80 () /coffee coffee-svc:80 () Annotations: kubectl.kubernetes.io/last-applied-configuration: {\"apiVersion\":\"networking.k8s.io/v1beta1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{},\"name\":\"cafe-ingress\",\"namespace\":\"default\",\"selfLink\":\"/apis/networking/v1beta1/namespaces/default/ingresses/cafe-ingress\"},\"spec\":{\"rules\":[{\"host\":\"cafe.com\",\"http\":{\"paths\":[{\"backend\":{\"serviceName\":\"tea-svc\",\"servicePort\":80},\"path\":\"/tea\"},{\"backend\":{\"serviceName\":\"coffee-svc\",\"servicePort\":80},\"path\":\"/coffee\"}]}}]},\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"169.48.142.110\"}]}}} Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal CREATE 1m nginx-ingress-controller Ingress default/cafe-ingress Normal UPDATE 58s nginx-ingress-controller Ingress default/cafe-ingress Check the Ingress Controller Logs $ kubectl get pods -n NAME READY STATUS RESTARTS AGE nginx-ingress-controller-67956bf89d-fv58j 1/1 Running 0 1m $ kubectl logs -n nginx-ingress-controller-67956bf89d-fv58j ------------------------------------------------------------------------------- NGINX Ingress controller Release: 0.14.0 Build: git-734361d Repository: https://github.com/kubernetes/ingress-nginx ------------------------------------------------------------------------------- .... Check the Nginx Configuration $ kubectl get pods -n NAME READY STATUS RESTARTS AGE nginx-ingress-controller-67956bf89d-fv58j 1/1 Running 0 1m $ kubectl exec -it -n nginx-ingress-controller-67956bf89d-fv58j cat /etc/nginx/nginx.conf daemon off; worker_processes 2; pid /run/nginx.pid; worker_rlimit_nofile 523264; worker_shutdown_timeout 240s; events { multi_accept on; worker_connections 16384; use epoll; } http { .... Check if used Services Exist $ kubectl get svc --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default coffee-svc ClusterIP 10.106.154.35 80/TCP 18m default kubernetes ClusterIP 10.96.0.1 443/TCP 30m default tea-svc ClusterIP 10.104.172.12 80/TCP 18m kube-system default-http-backend NodePort 10.108.189.236 80:30001/TCP 30m kube-system kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 30m kube-system kubernetes-dashboard NodePort 10.103.128.17 80:30000/TCP 30m Debug Logging \u00b6 Using the flag --v=XX it is possible to increase the level of logging. This is performed by editing the deployment. $ kubectl get deploy -n NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE default-http-backend 1 1 1 1 35m nginx-ingress-controller 1 1 1 1 35m $ kubectl edit deploy -n nginx-ingress-controller # Add --v = X to \"- args\" , where X is an integer --v=2 shows details using diff about the changes in the configuration in nginx --v=3 shows details about the service, Ingress rule, endpoint changes and it dumps the nginx configuration in JSON format --v=5 configures NGINX in debug mode Authentication to the Kubernetes API Server \u00b6 A number of components are involved in the authentication process and the first step is to narrow down the source of the problem, namely whether it is a problem with service authentication or with the kubeconfig file. Both authentications must work: +-------------+ service +------------+ | | authentication | | + apiserver +<-------------------+ ingress | | | | controller | +-------------+ +------------+ Service authentication The Ingress controller needs information from apiserver. Therefore, authentication is required, which can be achieved in two different ways: Service Account: This is recommended, because nothing has to be configured. The Ingress controller will use information provided by the system to communicate with the API server. See 'Service Account' section for details. Kubeconfig file: In some Kubernetes environments service accounts are not available. In this case a manual configuration is required. The Ingress controller binary can be started with the --kubeconfig flag. The value of the flag is a path to a file specifying how to connect to the API server. Using the --kubeconfig does not requires the flag --apiserver-host . The format of the file is identical to ~/.kube/config which is used by kubectl to connect to the API server. See 'kubeconfig' section for details. Using the flag --apiserver-host : Using this flag --apiserver-host=http://localhost:8080 it is possible to specify an unsecured API server or reach a remote kubernetes cluster using kubectl proxy . Please do not use this approach in production. In the diagram below you can see the full authentication flow with all options, starting with the browser on the lower left hand side. Kubernetes Workstation +---------------------------------------------------+ +------------------+ | | | | | +-----------+ apiserver +------------+ | | +------------+ | | | | proxy | | | | | | | | | apiserver | | ingress | | | | ingress | | | | | | controller | | | | controller | | | | | | | | | | | | | | | | | | | | | | | | | service account/ | | | | | | | | | | kubeconfig | | | | | | | | | +<-------------------+ | | | | | | | | | | | | | | | | | +------+----+ kubeconfig +------+-----+ | | +------+-----+ | | |<--------------------------------------------------------| | | | | | +---------------------------------------------------+ +------------------+ Service Account \u00b6 If using a service account to connect to the API server, Dashboard expects the file /var/run/secrets/kubernetes.io/serviceaccount/token to be present. It provides a secret token that is required to authenticate with the API server. Verify with the following commands: # start a container that contains curl $ kubectl run test --image = tutum/curl -- sleep 10000 # check that container is running $ kubectl get pods NAME READY STATUS RESTARTS AGE test-701078429-s5kca 1/1 Running 0 16s # check if secret exists $ kubectl exec test-701078429-s5kca ls /var/run/secrets/kubernetes.io/serviceaccount/ ca.crt namespace token # get service IP of master $ kubectl get services NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes 10.0.0.1 443/TCP 1d # check base connectivity from cluster inside $ kubectl exec test-701078429-s5kca -- curl -k https://10.0.0.1 Unauthorized # connect using tokens $ TOKEN_VALUE = $( kubectl exec test-701078429-s5kca -- cat /var/run/secrets/kubernetes.io/serviceaccount/token ) $ echo $TOKEN_VALUE eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3Mi....9A $ kubectl exec test-701078429-s5kca -- curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H \"Authorization: Bearer $TOKEN_VALUE \" https://10.0.0.1 { \"paths\": [ \"/api\", \"/api/v1\", \"/apis\", \"/apis/apps\", \"/apis/apps/v1alpha1\", \"/apis/authentication.k8s.io\", \"/apis/authentication.k8s.io/v1beta1\", \"/apis/authorization.k8s.io\", \"/apis/authorization.k8s.io/v1beta1\", \"/apis/autoscaling\", \"/apis/autoscaling/v1\", \"/apis/batch\", \"/apis/batch/v1\", \"/apis/batch/v2alpha1\", \"/apis/certificates.k8s.io\", \"/apis/certificates.k8s.io/v1alpha1\", \"/apis/networking\", \"/apis/networking/v1beta1\", \"/apis/policy\", \"/apis/policy/v1alpha1\", \"/apis/rbac.authorization.k8s.io\", \"/apis/rbac.authorization.k8s.io/v1alpha1\", \"/apis/storage.k8s.io\", \"/apis/storage.k8s.io/v1beta1\", \"/healthz\", \"/healthz/ping\", \"/logs\", \"/metrics\", \"/swaggerapi/\", \"/ui/\", \"/version\" ] } If it is not working, there are two possible reasons: The contents of the tokens are invalid. Find the secret name with kubectl get secrets | grep service-account and delete it with kubectl delete secret . It will automatically be recreated. You have a non-standard Kubernetes installation and the file containing the token may not be present. The API server will mount a volume containing this file, but only if the API server is configured to use the ServiceAccount admission controller. If you experience this error, verify that your API server is using the ServiceAccount admission controller. If you are configuring the API server by hand, you can set this with the --admission-control parameter. Note that you should use other admission controllers as well. Before configuring this option, you should read about admission controllers. More information: User Guide: Service Accounts Cluster Administrator Guide: Managing Service Accounts Kube-Config \u00b6 If you want to use a kubeconfig file for authentication, follow the deploy procedure and add the flag --kubeconfig=/etc/kubernetes/kubeconfig.yaml to the args section of the deployment. Using GDB with Nginx \u00b6 Gdb can be used to with nginx to perform a configuration dump. This allows us to see which configuration is being used, as well as older configurations. Note: The below is based on the nginx documentation . SSH into the worker $ ssh user@workerIP Obtain the Docker Container Running nginx $ docker ps | grep nginx-ingress-controller CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d9e1d243156a quay.io/kubernetes-ingress-controller/nginx-ingress-controller \"/usr/bin/dumb-init \u2026\" 19 minutes ago Up 19 minutes k8s_nginx-ingress-controller_nginx-ingress-controller-67956bf89d-mqxzt_kube-system_079f31ec-aa37-11e8-ad39-080027a227db_0 Exec into the container $ docker exec -it --user = 0 --privileged d9e1d243156a bash Make sure nginx is running in --with-debug $ nginx -V 2 > & 1 | grep -- '--with-debug' Get list of processes running on container $ ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 20:23 ? 00:00:00 /usr/bin/dumb-init /nginx-ingres root 5 1 0 20:23 ? 00:00:05 /nginx-ingress-controller --defa root 21 5 0 20:23 ? 00:00:00 nginx: master process /usr/sbin/ nobody 106 21 0 20:23 ? 00:00:00 nginx: worker process nobody 107 21 0 20:23 ? 00:00:00 nginx: worker process root 172 0 0 20:43 pts/0 00:00:00 bash Attach gdb to the nginx master process $ gdb -p 21 .... Attaching to process 21 Reading symbols from /usr/sbin/nginx...done. .... (gdb) Copy and paste the following: set $cd = ngx_cycle->config_dump set $nelts = $cd.nelts set $elts = (ngx_conf_dump_t*)($cd.elts) while ($nelts-- > 0) set $name = $elts[$nelts]->name.data printf \"Dumping %s to nginx_conf.txt\\n\", $name append memory nginx_conf.txt \\ $ elts [ $nelts ] ->buffer.start $elts [ $nelts ] ->buffer.end end Quit GDB by pressing CTRL+D Open nginx_conf.txt cat nginx_conf.txt","title":"Troubleshooting"},{"location":"troubleshooting/#troubleshooting","text":"","title":"Troubleshooting"},{"location":"troubleshooting/#ingress-controller-logs-and-events","text":"There are many ways to troubleshoot the ingress-controller. The following are basic troubleshooting methods to obtain more information. Check the Ingress Resource Events $ kubectl get ing -n NAME HOSTS ADDRESS PORTS AGE cafe-ingress cafe.com 10.0.2.15 80 25s $ kubectl describe ing -n Name: cafe-ingress Namespace: default Address: 10.0.2.15 Default backend: default-http-backend:80 (172.17.0.5:8080) Rules: Host Path Backends ---- ---- -------- cafe.com /tea tea-svc:80 () /coffee coffee-svc:80 () Annotations: kubectl.kubernetes.io/last-applied-configuration: {\"apiVersion\":\"networking.k8s.io/v1beta1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{},\"name\":\"cafe-ingress\",\"namespace\":\"default\",\"selfLink\":\"/apis/networking/v1beta1/namespaces/default/ingresses/cafe-ingress\"},\"spec\":{\"rules\":[{\"host\":\"cafe.com\",\"http\":{\"paths\":[{\"backend\":{\"serviceName\":\"tea-svc\",\"servicePort\":80},\"path\":\"/tea\"},{\"backend\":{\"serviceName\":\"coffee-svc\",\"servicePort\":80},\"path\":\"/coffee\"}]}}]},\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"169.48.142.110\"}]}}} Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal CREATE 1m nginx-ingress-controller Ingress default/cafe-ingress Normal UPDATE 58s nginx-ingress-controller Ingress default/cafe-ingress Check the Ingress Controller Logs $ kubectl get pods -n NAME READY STATUS RESTARTS AGE nginx-ingress-controller-67956bf89d-fv58j 1/1 Running 0 1m $ kubectl logs -n nginx-ingress-controller-67956bf89d-fv58j ------------------------------------------------------------------------------- NGINX Ingress controller Release: 0.14.0 Build: git-734361d Repository: https://github.com/kubernetes/ingress-nginx ------------------------------------------------------------------------------- .... Check the Nginx Configuration $ kubectl get pods -n NAME READY STATUS RESTARTS AGE nginx-ingress-controller-67956bf89d-fv58j 1/1 Running 0 1m $ kubectl exec -it -n nginx-ingress-controller-67956bf89d-fv58j cat /etc/nginx/nginx.conf daemon off; worker_processes 2; pid /run/nginx.pid; worker_rlimit_nofile 523264; worker_shutdown_timeout 240s; events { multi_accept on; worker_connections 16384; use epoll; } http { .... Check if used Services Exist $ kubectl get svc --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default coffee-svc ClusterIP 10.106.154.35 80/TCP 18m default kubernetes ClusterIP 10.96.0.1 443/TCP 30m default tea-svc ClusterIP 10.104.172.12 80/TCP 18m kube-system default-http-backend NodePort 10.108.189.236 80:30001/TCP 30m kube-system kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 30m kube-system kubernetes-dashboard NodePort 10.103.128.17 80:30000/TCP 30m","title":"Ingress-Controller Logs and Events"},{"location":"troubleshooting/#debug-logging","text":"Using the flag --v=XX it is possible to increase the level of logging. This is performed by editing the deployment. $ kubectl get deploy -n NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE default-http-backend 1 1 1 1 35m nginx-ingress-controller 1 1 1 1 35m $ kubectl edit deploy -n nginx-ingress-controller # Add --v = X to \"- args\" , where X is an integer --v=2 shows details using diff about the changes in the configuration in nginx --v=3 shows details about the service, Ingress rule, endpoint changes and it dumps the nginx configuration in JSON format --v=5 configures NGINX in debug mode","title":"Debug Logging"},{"location":"troubleshooting/#authentication-to-the-kubernetes-api-server","text":"A number of components are involved in the authentication process and the first step is to narrow down the source of the problem, namely whether it is a problem with service authentication or with the kubeconfig file. Both authentications must work: +-------------+ service +------------+ | | authentication | | + apiserver +<-------------------+ ingress | | | | controller | +-------------+ +------------+ Service authentication The Ingress controller needs information from apiserver. Therefore, authentication is required, which can be achieved in two different ways: Service Account: This is recommended, because nothing has to be configured. The Ingress controller will use information provided by the system to communicate with the API server. See 'Service Account' section for details. Kubeconfig file: In some Kubernetes environments service accounts are not available. In this case a manual configuration is required. The Ingress controller binary can be started with the --kubeconfig flag. The value of the flag is a path to a file specifying how to connect to the API server. Using the --kubeconfig does not requires the flag --apiserver-host . The format of the file is identical to ~/.kube/config which is used by kubectl to connect to the API server. See 'kubeconfig' section for details. Using the flag --apiserver-host : Using this flag --apiserver-host=http://localhost:8080 it is possible to specify an unsecured API server or reach a remote kubernetes cluster using kubectl proxy . Please do not use this approach in production. In the diagram below you can see the full authentication flow with all options, starting with the browser on the lower left hand side. Kubernetes Workstation +---------------------------------------------------+ +------------------+ | | | | | +-----------+ apiserver +------------+ | | +------------+ | | | | proxy | | | | | | | | | apiserver | | ingress | | | | ingress | | | | | | controller | | | | controller | | | | | | | | | | | | | | | | | | | | | | | | | service account/ | | | | | | | | | | kubeconfig | | | | | | | | | +<-------------------+ | | | | | | | | | | | | | | | | | +------+----+ kubeconfig +------+-----+ | | +------+-----+ | | |<--------------------------------------------------------| | | | | | +---------------------------------------------------+ +------------------+","title":"Authentication to the Kubernetes API Server"},{"location":"troubleshooting/#service-account","text":"If using a service account to connect to the API server, Dashboard expects the file /var/run/secrets/kubernetes.io/serviceaccount/token to be present. It provides a secret token that is required to authenticate with the API server. Verify with the following commands: # start a container that contains curl $ kubectl run test --image = tutum/curl -- sleep 10000 # check that container is running $ kubectl get pods NAME READY STATUS RESTARTS AGE test-701078429-s5kca 1/1 Running 0 16s # check if secret exists $ kubectl exec test-701078429-s5kca ls /var/run/secrets/kubernetes.io/serviceaccount/ ca.crt namespace token # get service IP of master $ kubectl get services NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes 10.0.0.1 443/TCP 1d # check base connectivity from cluster inside $ kubectl exec test-701078429-s5kca -- curl -k https://10.0.0.1 Unauthorized # connect using tokens $ TOKEN_VALUE = $( kubectl exec test-701078429-s5kca -- cat /var/run/secrets/kubernetes.io/serviceaccount/token ) $ echo $TOKEN_VALUE eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3Mi....9A $ kubectl exec test-701078429-s5kca -- curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H \"Authorization: Bearer $TOKEN_VALUE \" https://10.0.0.1 { \"paths\": [ \"/api\", \"/api/v1\", \"/apis\", \"/apis/apps\", \"/apis/apps/v1alpha1\", \"/apis/authentication.k8s.io\", \"/apis/authentication.k8s.io/v1beta1\", \"/apis/authorization.k8s.io\", \"/apis/authorization.k8s.io/v1beta1\", \"/apis/autoscaling\", \"/apis/autoscaling/v1\", \"/apis/batch\", \"/apis/batch/v1\", \"/apis/batch/v2alpha1\", \"/apis/certificates.k8s.io\", \"/apis/certificates.k8s.io/v1alpha1\", \"/apis/networking\", \"/apis/networking/v1beta1\", \"/apis/policy\", \"/apis/policy/v1alpha1\", \"/apis/rbac.authorization.k8s.io\", \"/apis/rbac.authorization.k8s.io/v1alpha1\", \"/apis/storage.k8s.io\", \"/apis/storage.k8s.io/v1beta1\", \"/healthz\", \"/healthz/ping\", \"/logs\", \"/metrics\", \"/swaggerapi/\", \"/ui/\", \"/version\" ] } If it is not working, there are two possible reasons: The contents of the tokens are invalid. Find the secret name with kubectl get secrets | grep service-account and delete it with kubectl delete secret . It will automatically be recreated. You have a non-standard Kubernetes installation and the file containing the token may not be present. The API server will mount a volume containing this file, but only if the API server is configured to use the ServiceAccount admission controller. If you experience this error, verify that your API server is using the ServiceAccount admission controller. If you are configuring the API server by hand, you can set this with the --admission-control parameter. Note that you should use other admission controllers as well. Before configuring this option, you should read about admission controllers. More information: User Guide: Service Accounts Cluster Administrator Guide: Managing Service Accounts","title":"Service Account"},{"location":"troubleshooting/#kube-config","text":"If you want to use a kubeconfig file for authentication, follow the deploy procedure and add the flag --kubeconfig=/etc/kubernetes/kubeconfig.yaml to the args section of the deployment.","title":"Kube-Config"},{"location":"troubleshooting/#using-gdb-with-nginx","text":"Gdb can be used to with nginx to perform a configuration dump. This allows us to see which configuration is being used, as well as older configurations. Note: The below is based on the nginx documentation . SSH into the worker $ ssh user@workerIP Obtain the Docker Container Running nginx $ docker ps | grep nginx-ingress-controller CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d9e1d243156a quay.io/kubernetes-ingress-controller/nginx-ingress-controller \"/usr/bin/dumb-init \u2026\" 19 minutes ago Up 19 minutes k8s_nginx-ingress-controller_nginx-ingress-controller-67956bf89d-mqxzt_kube-system_079f31ec-aa37-11e8-ad39-080027a227db_0 Exec into the container $ docker exec -it --user = 0 --privileged d9e1d243156a bash Make sure nginx is running in --with-debug $ nginx -V 2 > & 1 | grep -- '--with-debug' Get list of processes running on container $ ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 20:23 ? 00:00:00 /usr/bin/dumb-init /nginx-ingres root 5 1 0 20:23 ? 00:00:05 /nginx-ingress-controller --defa root 21 5 0 20:23 ? 00:00:00 nginx: master process /usr/sbin/ nobody 106 21 0 20:23 ? 00:00:00 nginx: worker process nobody 107 21 0 20:23 ? 00:00:00 nginx: worker process root 172 0 0 20:43 pts/0 00:00:00 bash Attach gdb to the nginx master process $ gdb -p 21 .... Attaching to process 21 Reading symbols from /usr/sbin/nginx...done. .... (gdb) Copy and paste the following: set $cd = ngx_cycle->config_dump set $nelts = $cd.nelts set $elts = (ngx_conf_dump_t*)($cd.elts) while ($nelts-- > 0) set $name = $elts[$nelts]->name.data printf \"Dumping %s to nginx_conf.txt\\n\", $name append memory nginx_conf.txt \\ $ elts [ $nelts ] ->buffer.start $elts [ $nelts ] ->buffer.end end Quit GDB by pressing CTRL+D Open nginx_conf.txt cat nginx_conf.txt","title":"Using GDB with Nginx"},{"location":"deploy/","text":"Installation Guide \u00b6 Contents \u00b6 Prerequisite Generic Deployment Command Provider Specific Steps Docker for Mac minikube AWS GCE - GKE Azure Bare-metal Verify installation Detect installed version Using Helm Prerequisite Generic Deployment Command \u00b6 Attention The default configuration watches Ingress object from all the namespaces . To change this behavior use the flag --watch-namespace to limit the scope to a particular namespace. Warning If multiple Ingresses define different paths for the same host, the ingress controller will merge the definitions. Attention If you're using GKE you need to initialize your user as a cluster-admin with the following command: kubectl create clusterrolebinding cluster-admin-binding \\ --clusterrole cluster-admin \\ --user $(gcloud config get-value account) The following Mandatory Command is required for all deployments. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/mandatory.yaml Tip If you are using a Kubernetes version previous to 1.14, you need to change kubernetes.io/os to beta.kubernetes.io/os at line 217 of mandatory.yaml , see Labels details . Provider Specific Steps \u00b6 There are cloud provider specific yaml files. Docker for Mac \u00b6 Kubernetes is available in Docker for Mac (from version 18.06.0-ce ) Create a service kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml minikube \u00b6 For standard usage: minikube addons enable ingress For development: Disable the ingress addon: minikube addons disable ingress Execute make dev-env Confirm the nginx-ingress-controller deployment exists: $ kubectl get pods -n ingress-nginx NAME READY STATUS RESTARTS AGE default-http-backend-66b447d9cf-rrlf9 1/1 Running 0 12s nginx-ingress-controller-fdcdcd6dd-vvpgs 1/1 Running 0 11s AWS \u00b6 In AWS we use an Elastic Load Balancer (ELB) to expose the NGINX Ingress controller behind a Service of Type=LoadBalancer . Since Kubernetes v1.9.0 it is possible to use a classic load balancer (ELB) or network load balancer (NLB) Please check the elastic load balancing AWS details page Elastic Load Balancer - ELB \u00b6 This setup requires to choose in which layer (L4 or L7) we want to configure the ELB: Layer 4 : use TCP as the listener protocol for ports 80 and 443. Layer 7 : use HTTP as the listener protocol for port 80 and terminate TLS in the ELB For L4: Check that no change is necessary with regards to the ELB idle timeout. In some scenarios, users may want to modify the ELB idle timeout, so please check the ELB Idle Timeouts section for additional information. If a change is required, users will need to update the value of service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout in provider/aws/service-l4.yaml Then execute: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-l4.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/patch-configmap-l4.yaml For L7: Change line of the file provider/aws/service-l7.yaml replacing the dummy id with a valid one \"arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX\" Check that no change is necessary with regards to the ELB idle timeout. In some scenarios, users may want to modify the ELB idle timeout, so please check the ELB Idle Timeouts section for additional information. If a change is required, users will need to update the value of service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout in provider/aws/service-l7.yaml Then execute: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-l7.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/patch-configmap-l7.yaml This example creates an ELB with just two listeners, one in port 80 and another in port 443 ELB Idle Timeouts \u00b6 In some scenarios users will need to modify the value of the ELB idle timeout. Users need to ensure the idle timeout is less than the keepalive_timeout that is configured for NGINX. By default NGINX keepalive_timeout is set to 75s . The default ELB idle timeout will work for most scenarios, unless the NGINX keepalive_timeout has been modified, in which case service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout will need to be modified to ensure it is less than the keepalive_timeout the user has configured. Please Note: An idle timeout of 3600s is recommended when using WebSockets. More information with regards to idle timeouts for your Load Balancer can be found in the official AWS documentation . Network Load Balancer (NLB) \u00b6 This type of load balancer is supported since v1.10.0 as an ALPHA feature. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-nlb.yaml GCE-GKE \u00b6 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml Important Note: proxy protocol is not supported in GCE/GKE Azure \u00b6 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml Bare-metal \u00b6 Using NodePort : kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/baremetal/service-nodeport.yaml Tip For extended notes regarding deployments on bare-metal, see Bare-metal considerations . Verify installation \u00b6 To check if the ingress controller pods have started, run the following command: kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx --watch Once the operator pods are running, you can cancel the above command by typing Ctrl+C . Now, you are ready to create your first ingress. Detect installed version \u00b6 To detect which version of the ingress controller is running, exec into the pod and run nginx-ingress-controller version command. POD_NAMESPACE=ingress-nginx POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}') kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version Using Helm \u00b6 NGINX Ingress controller can be installed via Helm using the chart stable/nginx-ingress from the official charts repository. To install the chart with the release name my-nginx : helm install my-nginx stable/nginx-ingress If the kubernetes cluster has RBAC enabled, then run: helm install my-nginx stable/nginx-ingress --set rbac.create=true If you are using Helm 2 then specify release name using --name flag helm install stable/nginx-ingress --name my-nginx or helm install stable/nginx-ingress --name my-nginx --set rbac.create=true Detect installed version: POD_NAME=$(kubectl get pods -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}') kubectl exec -it $POD_NAME -- /nginx-ingress-controller --version","title":"Installation Guide"},{"location":"deploy/#installation-guide","text":"","title":"Installation Guide"},{"location":"deploy/#contents","text":"Prerequisite Generic Deployment Command Provider Specific Steps Docker for Mac minikube AWS GCE - GKE Azure Bare-metal Verify installation Detect installed version Using Helm","title":"Contents"},{"location":"deploy/#prerequisite-generic-deployment-command","text":"Attention The default configuration watches Ingress object from all the namespaces . To change this behavior use the flag --watch-namespace to limit the scope to a particular namespace. Warning If multiple Ingresses define different paths for the same host, the ingress controller will merge the definitions. Attention If you're using GKE you need to initialize your user as a cluster-admin with the following command: kubectl create clusterrolebinding cluster-admin-binding \\ --clusterrole cluster-admin \\ --user $(gcloud config get-value account) The following Mandatory Command is required for all deployments. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/mandatory.yaml Tip If you are using a Kubernetes version previous to 1.14, you need to change kubernetes.io/os to beta.kubernetes.io/os at line 217 of mandatory.yaml , see Labels details .","title":"Prerequisite Generic Deployment Command"},{"location":"deploy/#provider-specific-steps","text":"There are cloud provider specific yaml files.","title":"Provider Specific Steps"},{"location":"deploy/#docker-for-mac","text":"Kubernetes is available in Docker for Mac (from version 18.06.0-ce ) Create a service kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml","title":"Docker for Mac"},{"location":"deploy/#minikube","text":"For standard usage: minikube addons enable ingress For development: Disable the ingress addon: minikube addons disable ingress Execute make dev-env Confirm the nginx-ingress-controller deployment exists: $ kubectl get pods -n ingress-nginx NAME READY STATUS RESTARTS AGE default-http-backend-66b447d9cf-rrlf9 1/1 Running 0 12s nginx-ingress-controller-fdcdcd6dd-vvpgs 1/1 Running 0 11s","title":"minikube"},{"location":"deploy/#aws","text":"In AWS we use an Elastic Load Balancer (ELB) to expose the NGINX Ingress controller behind a Service of Type=LoadBalancer . Since Kubernetes v1.9.0 it is possible to use a classic load balancer (ELB) or network load balancer (NLB) Please check the elastic load balancing AWS details page","title":"AWS"},{"location":"deploy/#elastic-load-balancer-elb","text":"This setup requires to choose in which layer (L4 or L7) we want to configure the ELB: Layer 4 : use TCP as the listener protocol for ports 80 and 443. Layer 7 : use HTTP as the listener protocol for port 80 and terminate TLS in the ELB For L4: Check that no change is necessary with regards to the ELB idle timeout. In some scenarios, users may want to modify the ELB idle timeout, so please check the ELB Idle Timeouts section for additional information. If a change is required, users will need to update the value of service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout in provider/aws/service-l4.yaml Then execute: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-l4.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/patch-configmap-l4.yaml For L7: Change line of the file provider/aws/service-l7.yaml replacing the dummy id with a valid one \"arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX\" Check that no change is necessary with regards to the ELB idle timeout. In some scenarios, users may want to modify the ELB idle timeout, so please check the ELB Idle Timeouts section for additional information. If a change is required, users will need to update the value of service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout in provider/aws/service-l7.yaml Then execute: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-l7.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/patch-configmap-l7.yaml This example creates an ELB with just two listeners, one in port 80 and another in port 443","title":"Elastic Load Balancer - ELB"},{"location":"deploy/#elb-idle-timeouts","text":"In some scenarios users will need to modify the value of the ELB idle timeout. Users need to ensure the idle timeout is less than the keepalive_timeout that is configured for NGINX. By default NGINX keepalive_timeout is set to 75s . The default ELB idle timeout will work for most scenarios, unless the NGINX keepalive_timeout has been modified, in which case service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout will need to be modified to ensure it is less than the keepalive_timeout the user has configured. Please Note: An idle timeout of 3600s is recommended when using WebSockets. More information with regards to idle timeouts for your Load Balancer can be found in the official AWS documentation .","title":"ELB Idle Timeouts"},{"location":"deploy/#network-load-balancer-nlb","text":"This type of load balancer is supported since v1.10.0 as an ALPHA feature. kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-nlb.yaml","title":"Network Load Balancer (NLB)"},{"location":"deploy/#gce-gke","text":"kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml Important Note: proxy protocol is not supported in GCE/GKE","title":"GCE-GKE"},{"location":"deploy/#azure","text":"kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml","title":"Azure"},{"location":"deploy/#bare-metal","text":"Using NodePort : kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/baremetal/service-nodeport.yaml Tip For extended notes regarding deployments on bare-metal, see Bare-metal considerations .","title":"Bare-metal"},{"location":"deploy/#verify-installation","text":"To check if the ingress controller pods have started, run the following command: kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx --watch Once the operator pods are running, you can cancel the above command by typing Ctrl+C . Now, you are ready to create your first ingress.","title":"Verify installation"},{"location":"deploy/#detect-installed-version","text":"To detect which version of the ingress controller is running, exec into the pod and run nginx-ingress-controller version command. POD_NAMESPACE=ingress-nginx POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}') kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version","title":"Detect installed version"},{"location":"deploy/#using-helm","text":"NGINX Ingress controller can be installed via Helm using the chart stable/nginx-ingress from the official charts repository. To install the chart with the release name my-nginx : helm install my-nginx stable/nginx-ingress If the kubernetes cluster has RBAC enabled, then run: helm install my-nginx stable/nginx-ingress --set rbac.create=true If you are using Helm 2 then specify release name using --name flag helm install stable/nginx-ingress --name my-nginx or helm install stable/nginx-ingress --name my-nginx --set rbac.create=true Detect installed version: POD_NAME=$(kubectl get pods -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}') kubectl exec -it $POD_NAME -- /nginx-ingress-controller --version","title":"Using Helm"},{"location":"deploy/baremetal/","text":"Bare-metal considerations \u00b6 In traditional cloud environments, where network load balancers are available on-demand, a single Kubernetes manifest suffices to provide a single point of contact to the NGINX Ingress controller to external clients and, indirectly, to any application running inside the cluster. Bare-metal environments lack this commodity, requiring a slightly different setup to offer the same kind of access to external consumers. The rest of this document describes a few recommended approaches to deploying the NGINX Ingress controller inside a Kubernetes cluster running on bare-metal. A pure software solution: MetalLB \u00b6 MetalLB provides a network load-balancer implementation for Kubernetes clusters that do not run on a supported cloud provider, effectively allowing the usage of LoadBalancer Services within any cluster. This section demonstrates how to use the Layer 2 configuration mode of MetalLB together with the NGINX Ingress controller in a Kubernetes cluster that has publicly accessible nodes . In this mode, one node attracts all the traffic for the ingress-nginx Service IP. See Traffic policies for more details. Note The description of other supported configuration modes is off-scope for this document. Warning MetalLB is currently in beta . Read about the Project maturity and make sure you inform yourself by reading the official documentation thoroughly. MetalLB can be deployed either with a simple Kubernetes manifest or with Helm. The rest of this example assumes MetalLB was deployed following the Installation instructions. MetalLB requires a pool of IP addresses in order to be able to take ownership of the ingress-nginx Service. This pool can be defined in a ConfigMap named config located in the same namespace as the MetalLB controller. This pool of IPs must be dedicated to MetalLB's use, you can't reuse the Kubernetes node IPs or IPs handed out by a DHCP server. Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is ) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 After creating the following ConfigMap, MetalLB takes ownership of one of the IP addresses in the pool and updates the loadBalancer IP field of the ingress-nginx Service accordingly. apiVersion : v1 kind : ConfigMap metadata : namespace : metallb-system name : config data : config : | address-pools: - name: default protocol: layer2 addresses: - 203.0.113.10-203.0.113.15 $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 80/TCP ingress-nginx LoadBalancer 10.0.220.217 203.0.113.10 80:30100/TCP,443:30101/TCP As soon as MetalLB sets the external IP address of the ingress-nginx LoadBalancer Service, the corresponding entries are created in the iptables NAT table and the node with the selected IP address starts responding to HTTP requests on the ports configured in the LoadBalancer Service: $ curl -D- http://203.0.113.3 -H 'Host: myapp.example.com' HTTP/1.1 200 OK Server: nginx/1.15.2 Tip In order to preserve the source IP address in HTTP requests sent to NGINX, it is necessary to use the Local traffic policy. Traffic policies are described in more details in Traffic policies as well as in the next section. Over a NodePort Service \u00b6 Due to its simplicity, this is the setup a user will deploy by default when following the steps described in the installation guide . Info A Service of type NodePort exposes, via the kube-proxy component, the same unprivileged port (default: 30000-32767) on every Kubernetes node, masters included. For more information, see Services . In this configuration, the NGINX container remains isolated from the host network. As a result, it can safely bind to any port, including the standard HTTP ports 80 and 443. However, due to the container namespace isolation, a client located outside the cluster network (e.g. on the public internet) is not able to access Ingress hosts directly on ports 80 and 443. Instead, the external client must append the NodePort allocated to the ingress-nginx Service to HTTP requests. Example Given the NodePort 30100 allocated to the ingress-nginx Service $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 80/TCP ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP and a Kubernetes node with the public IP address 203.0.113.2 (the external IP is added as an example, in most bare-metal environments this value is ) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 a client would reach an Ingress with host : myapp . example . com at http://myapp.example.com:30100 , where the myapp.example.com subdomain resolves to the 203.0.113.2 IP address. Impact on the host system While it may sound tempting to reconfigure the NodePort range using the --service-node-port-range API server flag to include unprivileged ports and be able to expose ports 80 and 443, doing so may result in unexpected issues including (but not limited to) the use of ports otherwise reserved to system daemons and the necessity to grant kube-proxy privileges it may otherwise not require. This practice is therefore discouraged . See the other approaches proposed in this page for alternatives. This approach has a few other limitations one ought to be aware of: Source IP address Services of type NodePort perform source address translation by default. This means the source IP of a HTTP request is always the IP address of the Kubernetes node that received the request from the perspective of NGINX. The recommended way to preserve the source IP in a NodePort setup is to set the value of the externalTrafficPolicy field of the ingress-nginx Service spec to Local ( example ). Warning This setting effectively drops packets sent to Kubernetes nodes which are not running any instance of the NGINX Ingress controller. Consider assigning NGINX Pods to specific nodes in order to control on what nodes the NGINX Ingress controller should be scheduled or not scheduled. Example In a Kubernetes cluster composed of 3 nodes (the external IP is added as an example, in most bare-metal environments this value is ) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 with a nginx-ingress-controller Deployment composed of 2 replicas $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 nginx-ingress-controller-cf9ff8c96-8vvf8 1/1 Running 172.17.0.3 host-3 nginx-ingress-controller-cf9ff8c96-pxsds 1/1 Running 172.17.1.4 host-2 Requests sent to host-2 and host-3 would be forwarded to NGINX and original client's IP would be preserved, while requests to host-1 would get dropped because there is no NGINX replica running on that node. Ingress status Because NodePort Services do not get a LoadBalancerIP assigned by definition, the NGINX Ingress controller does not update the status of Ingress objects it manages . $ kubectl get ingress NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 80 Despite the fact there is no load balancer providing a public IP address to the NGINX Ingress controller, it is possible to force the status update of all managed Ingress objects by setting the externalIPs field of the ingress-nginx Service. Warning There is more to setting externalIPs than just enabling the NGINX Ingress controller to update the status of Ingress objects. Please read about this option in the Services page of official Kubernetes documentation as well as the section about External IPs in this document for more information. Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is ) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 one could edit the ingress-nginx Service and add the following field to the object spec spec : externalIPs : - 203.0.113.1 - 203.0.113.2 - 203.0.113.3 which would in turn be reflected on Ingress objects as follows: $ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.1,203.0.113.2,203.0.113.3 80 Redirects As NGINX is not aware of the port translation operated by the NodePort Service , backend applications are responsible for generating redirect URLs that take into account the URL used by external clients, including the NodePort. Example Redirects generated by NGINX, for instance HTTP to HTTPS or domain to www.domain , are generated without NodePort: $ curl -D- http://myapp.example.com:30100 ` HTTP/1.1 308 Permanent Redirect Server: nginx/1.15.2 Location: https://myapp.example.com/ #-> missing NodePort in HTTPS redirect Via the host network \u00b6 In a setup where there is no external load balancer available but using NodePorts is not an option, one can configure ingress-nginx Pods to use the network of the host they run on instead of a dedicated network namespace. The benefit of this approach is that the NGINX Ingress controller can bind ports 80 and 443 directly to Kubernetes nodes' network interfaces, without the extra network translation imposed by NodePort Services. Note This approach does not leverage any Service object to expose the NGINX Ingress controller. If the ingress-nginx Service exists in the target cluster, it is recommended to delete it . This can be achieved by enabling the hostNetwork option in the Pods' spec. template : spec : hostNetwork : true Security considerations Enabling this option exposes every system daemon to the NGINX Ingress controller on any network interface, including the host's loopback. Please evaluate the impact this may have on the security of your system carefully. Example Consider this nginx-ingress-controller Deployment composed of 2 replicas, NGINX Pods inherit from the IP address of their host instead of an internal Pod IP. $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 nginx-ingress-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3 nginx-ingress-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2 One major limitation of this deployment approach is that only a single NGINX Ingress controller Pod may be scheduled on each cluster node, because binding the same port multiple times on the same network interface is technically impossible. Pods that are unschedulable due to such situation fail with the following event: $ kubectl -n ingress-nginx describe pod ... Events: Type Reason From Message ---- ------ ---- ------- Warning FailedScheduling default-scheduler 0/3 nodes are available: 3 node(s) didn't have free ports for the requested pod ports. One way to ensure only schedulable Pods are created is to deploy the NGINX Ingress controller as a DaemonSet instead of a traditional Deployment. Info A DaemonSet schedules exactly one type of Pod per cluster node, masters included, unless a node is configured to repel those Pods . For more information, see DaemonSet . Because most properties of DaemonSet objects are identical to Deployment objects, this documentation page leaves the configuration of the corresponding manifest at the user's discretion. Like with NodePorts, this approach has a few quirks it is important to be aware of. DNS resolution Pods configured with hostNetwork : true do not use the internal DNS resolver (i.e. kube-dns or CoreDNS ), unless their dnsPolicy spec field is set to ClusterFirstWithHostNet . Consider using this setting if NGINX is expected to resolve internal names for any reason. Ingress status Because there is no Service exposing the NGINX Ingress controller in a configuration using the host network, the default --publish-service flag used in standard cloud setups does not apply and the status of all Ingress objects remains blank. $ kubectl get ingress NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 80 Instead, and because bare-metal nodes usually don't have an ExternalIP, one has to enable the --report-node-internal-ip-address flag, which sets the status of all Ingress objects to the internal IP address of all nodes running the NGINX Ingress controller. Example Given a nginx-ingress-controller DaemonSet composed of 2 replicas $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 nginx-ingress-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3 nginx-ingress-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2 the controller sets the status of all Ingress objects it manages to the following value: $ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.2,203.0.113.3 80 Note Alternatively, it is possible to override the address written to Ingress objects using the --publish-status-address flag. See Command line arguments . Using a self-provisioned edge \u00b6 Similarly to cloud environments, this deployment approach requires an edge network component providing a public entrypoint to the Kubernetes cluster. This edge component can be either hardware (e.g. vendor appliance) or software (e.g. HAproxy ) and is usually managed outside of the Kubernetes landscape by operations teams. Such deployment builds upon the NodePort Service described above in Over a NodePort Service , with one significant difference: external clients do not access cluster nodes directly, only the edge component does. This is particularly suitable for private Kubernetes clusters where none of the nodes has a public IP address. On the edge side, the only prerequisite is to dedicate a public IP address that forwards all HTTP traffic to Kubernetes nodes and/or masters. Incoming traffic on TCP ports 80 and 443 is forwarded to the corresponding HTTP and HTTPS NodePort on the target nodes as shown in the diagram below: External IPs \u00b6 Source IP address This method does not allow preserving the source IP of HTTP requests in any manner, it is therefore not recommended to use it despite its apparent simplicity. The externalIPs Service option was previously mentioned in the NodePort section. As per the Services page of the official Kubernetes documentation, the externalIPs option causes kube-proxy to route traffic sent to arbitrary IP addresses and on the Service ports to the endpoints of that Service. These IP addresses must belong to the target node . Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is ) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 and the following ingress-nginx NodePort Service $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP One could set the following external IPs in the Service spec, and NGINX would become available on both the NodePort and the Service port: spec : externalIPs : - 203.0.113.2 - 203.0.113.3 $ curl -D- http://myapp.example.com:30100 HTTP/1.1 200 OK Server: nginx/1.15.2 $ curl -D- http://myapp.example.com HTTP/1.1 200 OK Server: nginx/1.15.2 We assume the myapp.example.com subdomain above resolves to both 203.0.113.2 and 203.0.113.3 IP addresses.","title":"Bare-metal considerations"},{"location":"deploy/baremetal/#bare-metal-considerations","text":"In traditional cloud environments, where network load balancers are available on-demand, a single Kubernetes manifest suffices to provide a single point of contact to the NGINX Ingress controller to external clients and, indirectly, to any application running inside the cluster. Bare-metal environments lack this commodity, requiring a slightly different setup to offer the same kind of access to external consumers. The rest of this document describes a few recommended approaches to deploying the NGINX Ingress controller inside a Kubernetes cluster running on bare-metal.","title":"Bare-metal considerations"},{"location":"deploy/baremetal/#a-pure-software-solution-metallb","text":"MetalLB provides a network load-balancer implementation for Kubernetes clusters that do not run on a supported cloud provider, effectively allowing the usage of LoadBalancer Services within any cluster. This section demonstrates how to use the Layer 2 configuration mode of MetalLB together with the NGINX Ingress controller in a Kubernetes cluster that has publicly accessible nodes . In this mode, one node attracts all the traffic for the ingress-nginx Service IP. See Traffic policies for more details. Note The description of other supported configuration modes is off-scope for this document. Warning MetalLB is currently in beta . Read about the Project maturity and make sure you inform yourself by reading the official documentation thoroughly. MetalLB can be deployed either with a simple Kubernetes manifest or with Helm. The rest of this example assumes MetalLB was deployed following the Installation instructions. MetalLB requires a pool of IP addresses in order to be able to take ownership of the ingress-nginx Service. This pool can be defined in a ConfigMap named config located in the same namespace as the MetalLB controller. This pool of IPs must be dedicated to MetalLB's use, you can't reuse the Kubernetes node IPs or IPs handed out by a DHCP server. Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is ) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 After creating the following ConfigMap, MetalLB takes ownership of one of the IP addresses in the pool and updates the loadBalancer IP field of the ingress-nginx Service accordingly. apiVersion : v1 kind : ConfigMap metadata : namespace : metallb-system name : config data : config : | address-pools: - name: default protocol: layer2 addresses: - 203.0.113.10-203.0.113.15 $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 80/TCP ingress-nginx LoadBalancer 10.0.220.217 203.0.113.10 80:30100/TCP,443:30101/TCP As soon as MetalLB sets the external IP address of the ingress-nginx LoadBalancer Service, the corresponding entries are created in the iptables NAT table and the node with the selected IP address starts responding to HTTP requests on the ports configured in the LoadBalancer Service: $ curl -D- http://203.0.113.3 -H 'Host: myapp.example.com' HTTP/1.1 200 OK Server: nginx/1.15.2 Tip In order to preserve the source IP address in HTTP requests sent to NGINX, it is necessary to use the Local traffic policy. Traffic policies are described in more details in Traffic policies as well as in the next section.","title":"A pure software solution: MetalLB"},{"location":"deploy/baremetal/#over-a-nodeport-service","text":"Due to its simplicity, this is the setup a user will deploy by default when following the steps described in the installation guide . Info A Service of type NodePort exposes, via the kube-proxy component, the same unprivileged port (default: 30000-32767) on every Kubernetes node, masters included. For more information, see Services . In this configuration, the NGINX container remains isolated from the host network. As a result, it can safely bind to any port, including the standard HTTP ports 80 and 443. However, due to the container namespace isolation, a client located outside the cluster network (e.g. on the public internet) is not able to access Ingress hosts directly on ports 80 and 443. Instead, the external client must append the NodePort allocated to the ingress-nginx Service to HTTP requests. Example Given the NodePort 30100 allocated to the ingress-nginx Service $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 80/TCP ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP and a Kubernetes node with the public IP address 203.0.113.2 (the external IP is added as an example, in most bare-metal environments this value is ) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 a client would reach an Ingress with host : myapp . example . com at http://myapp.example.com:30100 , where the myapp.example.com subdomain resolves to the 203.0.113.2 IP address. Impact on the host system While it may sound tempting to reconfigure the NodePort range using the --service-node-port-range API server flag to include unprivileged ports and be able to expose ports 80 and 443, doing so may result in unexpected issues including (but not limited to) the use of ports otherwise reserved to system daemons and the necessity to grant kube-proxy privileges it may otherwise not require. This practice is therefore discouraged . See the other approaches proposed in this page for alternatives. This approach has a few other limitations one ought to be aware of: Source IP address Services of type NodePort perform source address translation by default. This means the source IP of a HTTP request is always the IP address of the Kubernetes node that received the request from the perspective of NGINX. The recommended way to preserve the source IP in a NodePort setup is to set the value of the externalTrafficPolicy field of the ingress-nginx Service spec to Local ( example ). Warning This setting effectively drops packets sent to Kubernetes nodes which are not running any instance of the NGINX Ingress controller. Consider assigning NGINX Pods to specific nodes in order to control on what nodes the NGINX Ingress controller should be scheduled or not scheduled. Example In a Kubernetes cluster composed of 3 nodes (the external IP is added as an example, in most bare-metal environments this value is ) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 with a nginx-ingress-controller Deployment composed of 2 replicas $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 nginx-ingress-controller-cf9ff8c96-8vvf8 1/1 Running 172.17.0.3 host-3 nginx-ingress-controller-cf9ff8c96-pxsds 1/1 Running 172.17.1.4 host-2 Requests sent to host-2 and host-3 would be forwarded to NGINX and original client's IP would be preserved, while requests to host-1 would get dropped because there is no NGINX replica running on that node. Ingress status Because NodePort Services do not get a LoadBalancerIP assigned by definition, the NGINX Ingress controller does not update the status of Ingress objects it manages . $ kubectl get ingress NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 80 Despite the fact there is no load balancer providing a public IP address to the NGINX Ingress controller, it is possible to force the status update of all managed Ingress objects by setting the externalIPs field of the ingress-nginx Service. Warning There is more to setting externalIPs than just enabling the NGINX Ingress controller to update the status of Ingress objects. Please read about this option in the Services page of official Kubernetes documentation as well as the section about External IPs in this document for more information. Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is ) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 one could edit the ingress-nginx Service and add the following field to the object spec spec : externalIPs : - 203.0.113.1 - 203.0.113.2 - 203.0.113.3 which would in turn be reflected on Ingress objects as follows: $ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.1,203.0.113.2,203.0.113.3 80 Redirects As NGINX is not aware of the port translation operated by the NodePort Service , backend applications are responsible for generating redirect URLs that take into account the URL used by external clients, including the NodePort. Example Redirects generated by NGINX, for instance HTTP to HTTPS or domain to www.domain , are generated without NodePort: $ curl -D- http://myapp.example.com:30100 ` HTTP/1.1 308 Permanent Redirect Server: nginx/1.15.2 Location: https://myapp.example.com/ #-> missing NodePort in HTTPS redirect","title":"Over a NodePort Service"},{"location":"deploy/baremetal/#via-the-host-network","text":"In a setup where there is no external load balancer available but using NodePorts is not an option, one can configure ingress-nginx Pods to use the network of the host they run on instead of a dedicated network namespace. The benefit of this approach is that the NGINX Ingress controller can bind ports 80 and 443 directly to Kubernetes nodes' network interfaces, without the extra network translation imposed by NodePort Services. Note This approach does not leverage any Service object to expose the NGINX Ingress controller. If the ingress-nginx Service exists in the target cluster, it is recommended to delete it . This can be achieved by enabling the hostNetwork option in the Pods' spec. template : spec : hostNetwork : true Security considerations Enabling this option exposes every system daemon to the NGINX Ingress controller on any network interface, including the host's loopback. Please evaluate the impact this may have on the security of your system carefully. Example Consider this nginx-ingress-controller Deployment composed of 2 replicas, NGINX Pods inherit from the IP address of their host instead of an internal Pod IP. $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 nginx-ingress-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3 nginx-ingress-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2 One major limitation of this deployment approach is that only a single NGINX Ingress controller Pod may be scheduled on each cluster node, because binding the same port multiple times on the same network interface is technically impossible. Pods that are unschedulable due to such situation fail with the following event: $ kubectl -n ingress-nginx describe pod ... Events: Type Reason From Message ---- ------ ---- ------- Warning FailedScheduling default-scheduler 0/3 nodes are available: 3 node(s) didn't have free ports for the requested pod ports. One way to ensure only schedulable Pods are created is to deploy the NGINX Ingress controller as a DaemonSet instead of a traditional Deployment. Info A DaemonSet schedules exactly one type of Pod per cluster node, masters included, unless a node is configured to repel those Pods . For more information, see DaemonSet . Because most properties of DaemonSet objects are identical to Deployment objects, this documentation page leaves the configuration of the corresponding manifest at the user's discretion. Like with NodePorts, this approach has a few quirks it is important to be aware of. DNS resolution Pods configured with hostNetwork : true do not use the internal DNS resolver (i.e. kube-dns or CoreDNS ), unless their dnsPolicy spec field is set to ClusterFirstWithHostNet . Consider using this setting if NGINX is expected to resolve internal names for any reason. Ingress status Because there is no Service exposing the NGINX Ingress controller in a configuration using the host network, the default --publish-service flag used in standard cloud setups does not apply and the status of all Ingress objects remains blank. $ kubectl get ingress NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 80 Instead, and because bare-metal nodes usually don't have an ExternalIP, one has to enable the --report-node-internal-ip-address flag, which sets the status of all Ingress objects to the internal IP address of all nodes running the NGINX Ingress controller. Example Given a nginx-ingress-controller DaemonSet composed of 2 replicas $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 nginx-ingress-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3 nginx-ingress-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2 the controller sets the status of all Ingress objects it manages to the following value: $ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.2,203.0.113.3 80 Note Alternatively, it is possible to override the address written to Ingress objects using the --publish-status-address flag. See Command line arguments .","title":"Via the host network"},{"location":"deploy/baremetal/#using-a-self-provisioned-edge","text":"Similarly to cloud environments, this deployment approach requires an edge network component providing a public entrypoint to the Kubernetes cluster. This edge component can be either hardware (e.g. vendor appliance) or software (e.g. HAproxy ) and is usually managed outside of the Kubernetes landscape by operations teams. Such deployment builds upon the NodePort Service described above in Over a NodePort Service , with one significant difference: external clients do not access cluster nodes directly, only the edge component does. This is particularly suitable for private Kubernetes clusters where none of the nodes has a public IP address. On the edge side, the only prerequisite is to dedicate a public IP address that forwards all HTTP traffic to Kubernetes nodes and/or masters. Incoming traffic on TCP ports 80 and 443 is forwarded to the corresponding HTTP and HTTPS NodePort on the target nodes as shown in the diagram below:","title":"Using a self-provisioned edge"},{"location":"deploy/baremetal/#external-ips","text":"Source IP address This method does not allow preserving the source IP of HTTP requests in any manner, it is therefore not recommended to use it despite its apparent simplicity. The externalIPs Service option was previously mentioned in the NodePort section. As per the Services page of the official Kubernetes documentation, the externalIPs option causes kube-proxy to route traffic sent to arbitrary IP addresses and on the Service ports to the endpoints of that Service. These IP addresses must belong to the target node . Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is ) $ kubectl get node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 and the following ingress-nginx NodePort Service $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP One could set the following external IPs in the Service spec, and NGINX would become available on both the NodePort and the Service port: spec : externalIPs : - 203.0.113.2 - 203.0.113.3 $ curl -D- http://myapp.example.com:30100 HTTP/1.1 200 OK Server: nginx/1.15.2 $ curl -D- http://myapp.example.com HTTP/1.1 200 OK Server: nginx/1.15.2 We assume the myapp.example.com subdomain above resolves to both 203.0.113.2 and 203.0.113.3 IP addresses.","title":"External IPs"},{"location":"deploy/rbac/","text":"Role Based Access Control (RBAC) \u00b6 Overview \u00b6 This example applies to nginx-ingress-controllers being deployed in an environment with RBAC enabled. Role Based Access Control is comprised of four layers: ClusterRole - permissions assigned to a role that apply to an entire cluster ClusterRoleBinding - binding a ClusterRole to a specific account Role - permissions assigned to a role that apply to a specific namespace RoleBinding - binding a Role to a specific account In order for RBAC to be applied to an nginx-ingress-controller, that controller should be assigned to a ServiceAccount . That ServiceAccount should be bound to the Role s and ClusterRole s defined for the nginx-ingress-controller. Service Accounts created in this example \u00b6 One ServiceAccount is created in this example, nginx-ingress-serviceaccount . Permissions Granted in this example \u00b6 There are two sets of permissions defined in this example. Cluster-wide permissions defined by the ClusterRole named nginx-ingress-clusterrole , and namespace specific permissions defined by the Role named nginx-ingress-role . Cluster Permissions \u00b6 These permissions are granted in order for the nginx-ingress-controller to be able to function as an ingress across the cluster. These permissions are granted to the ClusterRole named nginx-ingress-clusterrole configmaps , endpoints , nodes , pods , secrets : list, watch nodes : get services , ingresses : get, list, watch events : create, patch ingresses/status : update Namespace Permissions \u00b6 These permissions are granted specific to the nginx-ingress namespace. These permissions are granted to the Role named nginx-ingress-role configmaps , pods , secrets : get endpoints : get Furthermore to support leader-election, the nginx-ingress-controller needs to have access to a configmap using the resourceName ingress-controller-leader-nginx Note that resourceNames can NOT be used to limit requests using the \u201ccreate\u201d verb because authorizers only have access to information that can be obtained from the request URL, method, and headers (resource names in a \u201ccreate\u201d request are part of the request body). configmaps : get, update (for resourceName ingress-controller-leader-nginx ) configmaps : create This resourceName is the concatenation of the election-id and the ingress-class as defined by the ingress-controller, which defaults to: election-id : ingress-controller-leader ingress-class : nginx resourceName : - Please adapt accordingly if you overwrite either parameter when launching the nginx-ingress-controller. Bindings \u00b6 The ServiceAccount nginx-ingress-serviceaccount is bound to the Role nginx-ingress-role and the ClusterRole nginx-ingress-clusterrole . The serviceAccountName associated with the containers in the deployment must match the serviceAccount. The namespace references in the Deployment metadata, container arguments, and POD_NAMESPACE should be in the nginx-ingress namespace.","title":"Role Based Access Control (RBAC)"},{"location":"deploy/rbac/#role-based-access-control-rbac","text":"","title":"Role Based Access Control (RBAC)"},{"location":"deploy/rbac/#overview","text":"This example applies to nginx-ingress-controllers being deployed in an environment with RBAC enabled. Role Based Access Control is comprised of four layers: ClusterRole - permissions assigned to a role that apply to an entire cluster ClusterRoleBinding - binding a ClusterRole to a specific account Role - permissions assigned to a role that apply to a specific namespace RoleBinding - binding a Role to a specific account In order for RBAC to be applied to an nginx-ingress-controller, that controller should be assigned to a ServiceAccount . That ServiceAccount should be bound to the Role s and ClusterRole s defined for the nginx-ingress-controller.","title":"Overview"},{"location":"deploy/rbac/#service-accounts-created-in-this-example","text":"One ServiceAccount is created in this example, nginx-ingress-serviceaccount .","title":"Service Accounts created in this example"},{"location":"deploy/rbac/#permissions-granted-in-this-example","text":"There are two sets of permissions defined in this example. Cluster-wide permissions defined by the ClusterRole named nginx-ingress-clusterrole , and namespace specific permissions defined by the Role named nginx-ingress-role .","title":"Permissions Granted in this example"},{"location":"deploy/rbac/#cluster-permissions","text":"These permissions are granted in order for the nginx-ingress-controller to be able to function as an ingress across the cluster. These permissions are granted to the ClusterRole named nginx-ingress-clusterrole configmaps , endpoints , nodes , pods , secrets : list, watch nodes : get services , ingresses : get, list, watch events : create, patch ingresses/status : update","title":"Cluster Permissions"},{"location":"deploy/rbac/#namespace-permissions","text":"These permissions are granted specific to the nginx-ingress namespace. These permissions are granted to the Role named nginx-ingress-role configmaps , pods , secrets : get endpoints : get Furthermore to support leader-election, the nginx-ingress-controller needs to have access to a configmap using the resourceName ingress-controller-leader-nginx Note that resourceNames can NOT be used to limit requests using the \u201ccreate\u201d verb because authorizers only have access to information that can be obtained from the request URL, method, and headers (resource names in a \u201ccreate\u201d request are part of the request body). configmaps : get, update (for resourceName ingress-controller-leader-nginx ) configmaps : create This resourceName is the concatenation of the election-id and the ingress-class as defined by the ingress-controller, which defaults to: election-id : ingress-controller-leader ingress-class : nginx resourceName : - Please adapt accordingly if you overwrite either parameter when launching the nginx-ingress-controller.","title":"Namespace Permissions"},{"location":"deploy/rbac/#bindings","text":"The ServiceAccount nginx-ingress-serviceaccount is bound to the Role nginx-ingress-role and the ClusterRole nginx-ingress-clusterrole . The serviceAccountName associated with the containers in the deployment must match the serviceAccount. The namespace references in the Deployment metadata, container arguments, and POD_NAMESPACE should be in the nginx-ingress namespace.","title":"Bindings"},{"location":"deploy/upgrade/","text":"Upgrading \u00b6 Important No matter the method you use for upgrading, if you use template overrides, make sure your templates are compatible with the new version of ingress-nginx . Without Helm \u00b6 To upgrade your ingress-nginx installation, it should be enough to change the version of the image in the controller Deployment. I.e. if your deployment resource looks like (partial example): kind : Deployment metadata : name : nginx-ingress-controller namespace : ingress-nginx spec : replicas : 1 selector : ... template : metadata : ... spec : containers : - name : nginx-ingress-controller image : quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0 args : ... simply change the 0.9.0 tag to the version you wish to upgrade to. The easiest way to do this is e.g. (do note you may need to change the name parameter according to your installation): kubectl set image deployment/nginx-ingress-controller \\ nginx-ingress-controller=quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0 For interactive editing, use kubectl edit deployment nginx-ingress-controller . With Helm \u00b6 If you installed ingress-nginx using the Helm command in the deployment docs so its name is ngx-ingress , you should be able to upgrade using helm upgrade --reuse-values ngx-ingress stable/nginx-ingress","title":"Upgrade"},{"location":"deploy/upgrade/#upgrading","text":"Important No matter the method you use for upgrading, if you use template overrides, make sure your templates are compatible with the new version of ingress-nginx .","title":"Upgrading"},{"location":"deploy/upgrade/#without-helm","text":"To upgrade your ingress-nginx installation, it should be enough to change the version of the image in the controller Deployment. I.e. if your deployment resource looks like (partial example): kind : Deployment metadata : name : nginx-ingress-controller namespace : ingress-nginx spec : replicas : 1 selector : ... template : metadata : ... spec : containers : - name : nginx-ingress-controller image : quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0 args : ... simply change the 0.9.0 tag to the version you wish to upgrade to. The easiest way to do this is e.g. (do note you may need to change the name parameter according to your installation): kubectl set image deployment/nginx-ingress-controller \\ nginx-ingress-controller=quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0 For interactive editing, use kubectl edit deployment nginx-ingress-controller .","title":"Without Helm"},{"location":"deploy/upgrade/#with-helm","text":"If you installed ingress-nginx using the Helm command in the deployment docs so its name is ngx-ingress , you should be able to upgrade using helm upgrade --reuse-values ngx-ingress stable/nginx-ingress","title":"With Helm"},{"location":"deploy/validating-webhook/","text":"Validating webhook (admission controller) \u00b6 Overview \u00b6 Nginx ingress controller offers the option to validate ingresses before they enter the cluster, ensuring controller will generate a valid configuration. This controller is called, when ValidatingAdmissionWebhook is enabled, by the Kubernetes API server each time a new ingress is to enter the cluster, and rejects objects for which the generated nginx configuration fails to be validated. This feature requires some further configuration of the cluster, hence it is an optional feature, this section explains how to enable it for your cluster. Configure the webhook \u00b6 Generate the webhook certificate \u00b6 Self signed certificate \u00b6 Validating webhook must be served using TLS, you need to generate a certificate. Note that kube API server is checking the hostname of the certificate, the common name of your certificate will need to match the service name. Example To run the validating webhook with a service named ingress-validation-webhook in the namespace ingress-nginx , run openssl req -x509 -newkey rsa:2048 -keyout certificate.pem -out key.pem -days 365 -nodes -subj \"/CN=ingress-validation-webhook.ingress-nginx.svc\" Using Kubernetes CA \u00b6 Kubernetes also provides primitives to sign a certificate request. Here is an example on how to use it Example #!/bin/bash SERVICE_NAME = ingress-nginx NAMESPACE = ingress-nginx TEMP_DIRECTORY = $( mktemp -d ) echo \"creating certs in directory ${ TEMP_DIRECTORY } \" cat <> ${TEMP_DIRECTORY}/csr.conf [req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = serverAuth subjectAltName = @alt_names [alt_names] DNS.1 = ${SERVICE_NAME} DNS.2 = ${SERVICE_NAME}.${NAMESPACE} DNS.3 = ${SERVICE_NAME}.${NAMESPACE}.svc EOF openssl genrsa -out ${ TEMP_DIRECTORY } /server-key.pem 2048 openssl req -new -key ${ TEMP_DIRECTORY } /server-key.pem \\ -subj \"/CN= ${ SERVICE_NAME } . ${ NAMESPACE } .svc\" \\ -out ${ TEMP_DIRECTORY } /server.csr \\ -config ${ TEMP_DIRECTORY } /csr.conf cat < & 2 exit 1 fi echo ${ SERVER_CERT } | openssl base64 -d -A -out ${ TEMP_DIRECTORY } /server-cert.pem kubectl create secret generic ingress-nginx.svc \\ --from-file = key.pem = ${ TEMP_DIRECTORY } /server-key.pem \\ --from-file = cert.pem = ${ TEMP_DIRECTORY } /server-cert.pem \\ -n ${ NAMESPACE } Using helm \u00b6 To generate the certificate using helm, you can use the following snippet Example {{ - $ cn := printf \"%s.%s.svc\" ( include \"nginx-ingress.validatingWebhook.fullname\" . ) .Release.Namespace }} {{ - $ ca := genCA ( printf \"%s-ca\" ( include \"nginx-ingress.validatingWebhook.fullname\" . )) .Values.validatingWebhook.certificateValidity - }} {{ - $ cert := genSignedCert $ cn nil nil .Values.validatingWebhook.certificateValidity $ ca - }} Ingress controller flags \u00b6 To enable the feature in the ingress controller, you need to provide 3 flags to the command line. flag description example usage --validating-webhook The address to start an admission controller on :8080 --validating-webhook-certificate The certificate the webhook is using for its TLS handling /usr/local/certificates/validating-webhook.pem --validating-webhook-key The key the webhook is using for its TLS handling /usr/local/certificates/validating-webhook-key.pem kube API server flags \u00b6 Validating webhook feature requires specific setup on the kube API server side. Depending on your kubernetes version, the flag can, or not, be enabled by default. To check that your kube API server runs with the required flags, please refer to the kubernetes documentation. Additional kubernetes objects \u00b6 Once both the ingress controller and the kube API server are configured to serve the webhook, add the you can configure the webhook with the following objects: apiVersion : v1 kind : Service metadata : name : ingress-validation-webhook namespace : ingress-nginx spec : ports : - name : admission port : 443 protocol : TCP targetPort : 8080 selector : app : nginx-ingress component : controller --- apiVersion : admissionregistration.k8s.io/v1beta1 kind : ValidatingWebhookConfiguration metadata : name : check-ingress webhooks : - name : validate.nginx.ingress.kubernetes.io rules : - apiGroups : - networking.k8s.io/v1beta1 apiVersions : - v1beta1 operations : - CREATE - UPDATE resources : - ingresses failurePolicy : Fail clientConfig : service : namespace : ingress-nginx name : ingress-validation-webhook path : /networking.k8s.io/v1beta1/ingress caBundle : ","title":"Validating Webhook (admission controller)"},{"location":"deploy/validating-webhook/#validating-webhook-admission-controller","text":"","title":"Validating webhook (admission controller)"},{"location":"deploy/validating-webhook/#overview","text":"Nginx ingress controller offers the option to validate ingresses before they enter the cluster, ensuring controller will generate a valid configuration. This controller is called, when ValidatingAdmissionWebhook is enabled, by the Kubernetes API server each time a new ingress is to enter the cluster, and rejects objects for which the generated nginx configuration fails to be validated. This feature requires some further configuration of the cluster, hence it is an optional feature, this section explains how to enable it for your cluster.","title":"Overview"},{"location":"deploy/validating-webhook/#configure-the-webhook","text":"","title":"Configure the webhook"},{"location":"deploy/validating-webhook/#generate-the-webhook-certificate","text":"","title":"Generate the webhook certificate"},{"location":"deploy/validating-webhook/#self-signed-certificate","text":"Validating webhook must be served using TLS, you need to generate a certificate. Note that kube API server is checking the hostname of the certificate, the common name of your certificate will need to match the service name. Example To run the validating webhook with a service named ingress-validation-webhook in the namespace ingress-nginx , run openssl req -x509 -newkey rsa:2048 -keyout certificate.pem -out key.pem -days 365 -nodes -subj \"/CN=ingress-validation-webhook.ingress-nginx.svc\"","title":"Self signed certificate"},{"location":"deploy/validating-webhook/#using-kubernetes-ca","text":"Kubernetes also provides primitives to sign a certificate request. Here is an example on how to use it Example #!/bin/bash SERVICE_NAME = ingress-nginx NAMESPACE = ingress-nginx TEMP_DIRECTORY = $( mktemp -d ) echo \"creating certs in directory ${ TEMP_DIRECTORY } \" cat <> ${TEMP_DIRECTORY}/csr.conf [req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = serverAuth subjectAltName = @alt_names [alt_names] DNS.1 = ${SERVICE_NAME} DNS.2 = ${SERVICE_NAME}.${NAMESPACE} DNS.3 = ${SERVICE_NAME}.${NAMESPACE}.svc EOF openssl genrsa -out ${ TEMP_DIRECTORY } /server-key.pem 2048 openssl req -new -key ${ TEMP_DIRECTORY } /server-key.pem \\ -subj \"/CN= ${ SERVICE_NAME } . ${ NAMESPACE } .svc\" \\ -out ${ TEMP_DIRECTORY } /server.csr \\ -config ${ TEMP_DIRECTORY } /csr.conf cat < & 2 exit 1 fi echo ${ SERVER_CERT } | openssl base64 -d -A -out ${ TEMP_DIRECTORY } /server-cert.pem kubectl create secret generic ingress-nginx.svc \\ --from-file = key.pem = ${ TEMP_DIRECTORY } /server-key.pem \\ --from-file = cert.pem = ${ TEMP_DIRECTORY } /server-cert.pem \\ -n ${ NAMESPACE }","title":"Using Kubernetes CA"},{"location":"deploy/validating-webhook/#using-helm","text":"To generate the certificate using helm, you can use the following snippet Example {{ - $ cn := printf \"%s.%s.svc\" ( include \"nginx-ingress.validatingWebhook.fullname\" . ) .Release.Namespace }} {{ - $ ca := genCA ( printf \"%s-ca\" ( include \"nginx-ingress.validatingWebhook.fullname\" . )) .Values.validatingWebhook.certificateValidity - }} {{ - $ cert := genSignedCert $ cn nil nil .Values.validatingWebhook.certificateValidity $ ca - }}","title":"Using helm"},{"location":"deploy/validating-webhook/#ingress-controller-flags","text":"To enable the feature in the ingress controller, you need to provide 3 flags to the command line. flag description example usage --validating-webhook The address to start an admission controller on :8080 --validating-webhook-certificate The certificate the webhook is using for its TLS handling /usr/local/certificates/validating-webhook.pem --validating-webhook-key The key the webhook is using for its TLS handling /usr/local/certificates/validating-webhook-key.pem","title":"Ingress controller flags"},{"location":"deploy/validating-webhook/#kube-api-server-flags","text":"Validating webhook feature requires specific setup on the kube API server side. Depending on your kubernetes version, the flag can, or not, be enabled by default. To check that your kube API server runs with the required flags, please refer to the kubernetes documentation.","title":"kube API server flags"},{"location":"deploy/validating-webhook/#additional-kubernetes-objects","text":"Once both the ingress controller and the kube API server are configured to serve the webhook, add the you can configure the webhook with the following objects: apiVersion : v1 kind : Service metadata : name : ingress-validation-webhook namespace : ingress-nginx spec : ports : - name : admission port : 443 protocol : TCP targetPort : 8080 selector : app : nginx-ingress component : controller --- apiVersion : admissionregistration.k8s.io/v1beta1 kind : ValidatingWebhookConfiguration metadata : name : check-ingress webhooks : - name : validate.nginx.ingress.kubernetes.io rules : - apiGroups : - networking.k8s.io/v1beta1 apiVersions : - v1beta1 operations : - CREATE - UPDATE resources : - ingresses failurePolicy : Fail clientConfig : service : namespace : ingress-nginx name : ingress-validation-webhook path : /networking.k8s.io/v1beta1/ingress caBundle : ","title":"Additional kubernetes objects"},{"location":"enhancements/","text":"Kubernetes Enhancement Proposals (KEPs) \u00b6 A Kubernetes Enhancement Proposal (KEP) is a way to propose, communicate and coordinate on new efforts for the Kubernetes project. For this reason, the ingress-nginx project is adopting it. Quick start for the KEP process \u00b6 Follow the process outlined in the KEP template Do I have to use the KEP process? \u00b6 No... but we hope that you will. Over time having a rich set of KEPs in one place will make it easier for people to track what is going on in the community and find a structured historic record. KEPs are only required when the changes are wide ranging and impact most of the project. Why would I want to use the KEP process? \u00b6 Our aim with KEPs is to clearly communicate new efforts to the Kubernetes contributor community. As such, we want to build a well curated set of clear proposals in a common format with useful metadata. Benefits to KEP users (in the limit): Exposure on a kubernetes blessed web site that is findable via web search engines. Cross indexing of KEPs so that users can find connections and the current status of any KEP. A clear process with approvers and reviewers for making decisions. This will lead to more structured decisions that stick as there is a discoverable record around the decisions. We are inspired by IETF RFCs, Python PEPs, and Rust RFCs.","title":"Kubernetes Enhancement Proposals (KEPs)"},{"location":"enhancements/#kubernetes-enhancement-proposals-keps","text":"A Kubernetes Enhancement Proposal (KEP) is a way to propose, communicate and coordinate on new efforts for the Kubernetes project. For this reason, the ingress-nginx project is adopting it.","title":"Kubernetes Enhancement Proposals (KEPs)"},{"location":"enhancements/#quick-start-for-the-kep-process","text":"Follow the process outlined in the KEP template","title":"Quick start for the KEP process"},{"location":"enhancements/#do-i-have-to-use-the-kep-process","text":"No... but we hope that you will. Over time having a rich set of KEPs in one place will make it easier for people to track what is going on in the community and find a structured historic record. KEPs are only required when the changes are wide ranging and impact most of the project.","title":"Do I have to use the KEP process?"},{"location":"enhancements/#why-would-i-want-to-use-the-kep-process","text":"Our aim with KEPs is to clearly communicate new efforts to the Kubernetes contributor community. As such, we want to build a well curated set of clear proposals in a common format with useful metadata. Benefits to KEP users (in the limit): Exposure on a kubernetes blessed web site that is findable via web search engines. Cross indexing of KEPs so that users can find connections and the current status of any KEP. A clear process with approvers and reviewers for making decisions. This will lead to more structured decisions that stick as there is a discoverable record around the decisions. We are inspired by IETF RFCs, Python PEPs, and Rust RFCs.","title":"Why would I want to use the KEP process?"},{"location":"enhancements/20190724-only-dynamic-ssl/","text":"Remove static SSL configuration mode \u00b6 Table of Contents \u00b6 Summary Motivation Goals Non-Goals Proposal Implementation Details/Notes/Constraints Drawbacks Alternatives Summary \u00b6 Since release 0.19.0 is possible to configure SSL certificates without the need of NGINX reloads (thanks to lua) and after release 0.24.0 the default enabled mode is dynamic. Motivation \u00b6 The static configuration implies reloads, something that affects the majority of the users. Goals \u00b6 Deprecation of the flag --enable-dynamic-certificates . Cleanup of the codebase. Non-Goals \u00b6 Features related to certificate authentication are not changed in any way. Proposal \u00b6 Remove static SSL configuration Implementation Details/Notes/Constraints \u00b6 Deprecate the flag Move the directives ssl_certificate and ssl_certificate_key from each server block to the http section. These settings are required to avoid NGINX errors in the logs. Remove any action of the flag --enable-dynamic-certificates Drawbacks \u00b6 Alternatives \u00b6 Keep both implementations","title":"Remove static SSL configuration mode"},{"location":"enhancements/20190724-only-dynamic-ssl/#remove-static-ssl-configuration-mode","text":"","title":"Remove static SSL configuration mode"},{"location":"enhancements/20190724-only-dynamic-ssl/#table-of-contents","text":"Summary Motivation Goals Non-Goals Proposal Implementation Details/Notes/Constraints Drawbacks Alternatives","title":"Table of Contents"},{"location":"enhancements/20190724-only-dynamic-ssl/#summary","text":"Since release 0.19.0 is possible to configure SSL certificates without the need of NGINX reloads (thanks to lua) and after release 0.24.0 the default enabled mode is dynamic.","title":"Summary"},{"location":"enhancements/20190724-only-dynamic-ssl/#motivation","text":"The static configuration implies reloads, something that affects the majority of the users.","title":"Motivation"},{"location":"enhancements/20190724-only-dynamic-ssl/#goals","text":"Deprecation of the flag --enable-dynamic-certificates . Cleanup of the codebase.","title":"Goals"},{"location":"enhancements/20190724-only-dynamic-ssl/#non-goals","text":"Features related to certificate authentication are not changed in any way.","title":"Non-Goals"},{"location":"enhancements/20190724-only-dynamic-ssl/#proposal","text":"Remove static SSL configuration","title":"Proposal"},{"location":"enhancements/20190724-only-dynamic-ssl/#implementation-detailsnotesconstraints","text":"Deprecate the flag Move the directives ssl_certificate and ssl_certificate_key from each server block to the http section. These settings are required to avoid NGINX errors in the logs. Remove any action of the flag --enable-dynamic-certificates","title":"Implementation Details/Notes/Constraints"},{"location":"enhancements/20190724-only-dynamic-ssl/#drawbacks","text":"","title":"Drawbacks"},{"location":"enhancements/20190724-only-dynamic-ssl/#alternatives","text":"Keep both implementations","title":"Alternatives"},{"location":"enhancements/20190815-zone-aware-routing/","text":"Availability zone aware routing \u00b6 Table of Contents \u00b6 Summary Motivation Goals Non-Goals Proposal Implementation History Drawbacks [optional] Summary \u00b6 Teach ingress-nginx about availability zones where endpoints are running in. This way ingress-nginx pod will do its best to proxy to zone-local endpoint. Motivation \u00b6 When users run their services across multiple availability zones they usually pay for egress traffic between zones. Providers such as GCP, Amazon EC charges money for that. ingress-nginx when picking an endpoint to route request to does not consider whether the endpoint is in different zone or the same one. That means it's at least equally likely that it will pick an endpoint from another zone and proxy the request to it. In this situation response from the endpoint to ingress-nginx pod is considered as inter zone traffic and costs money. At the time of this writing GCP charges $0.01 per GB of inter zone egress traffic according to https://cloud.google.com/compute/network-pricing. According to https://datapath.io/resources/blog/what-are-aws-data-transfer-costs-and-how-to-minimize-them/ Amazon also charges the same amount of money sa GCP for cross zone, egress traffic. This can be a lot of money depending on once's traffic. By teaching ingress-nginx about zones we can eliminate or at least decrease this cost. Arguably inter-zone network latency should also be better than cross zone. Goals \u00b6 Given a regional cluster running ingress-nginx, ingress-nginx should do best effort to pick zone-local endpoint when proxying This should not impact canary feature ingress-nginx should be able to operate successfully if there's no zonal endpoints Non-Goals \u00b6 This feature inherently assumes that endpoints are distributed across zones in a way that they can handle all the traffic from ingress-nginx pod(s) in that zone This feature will be relying on https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#failure-domainbetakubernetesiozone, it is not this KEP's goal to support other cases Proposal \u00b6 The idea here is to have controller part of ingress-nginx to (1) detect what zone its current pod is running in and (2) detect the zone for every endpoints it knows about. After that it will post that data as part of endpoints to Lua land. Then Lua balancer when picking an endpoint will try to pick zone-local endpoint first and if there is no zone-local endpoint then it will fallback to current behaviour. This feature at least in the beginning should be optional since it is going to make it harder to reason about the load balancing and not everyone might want that. How does controller know what zone it runs in? We can have the pod spec do pass node name using downward API as an environment variable. Then on start controller can get node details from the API based on node name. Once the node details is obtained we can extract the zone from failure-domain.beta.kubernetes.io/zone annotation. Then we can pass that value to Lua land through Nginx configuration when loading lua_ingress.lua module in init_by_lua phase. How do we extract zones for endpoints? We can have the controller watch create and update events on nodes in the entire cluster and based on that keep the map of nodes to zones in the memory. And when we generate endpoints list, we can access node name using .subsets.addresses [ i ]. nodeName and based on that fetch zone from the map in memory and store it as a field on the endpoint. This solution assumes failure-domain.beta.kubernetes.io/zone annotation does not change until the end of node's life. Otherwise we have to watch update events as well on the nodes and that'll add even more overhead. Alternatively, we can get the list of nodes only when there's no node in the memory for given node name. This is probably a better solution because then we would avoid watching for API changes on node resources. We can eagrly fetch all the nodes and build node name to zone mapping on start. And from thereon sync it during endpoints building in the main event loop iff there's no entry exist for the node of an endpoint. This means an extra API call in case cluster has expanded. How do we make sure we do our best to choose zone-local endpoint? This will be done on Lua side. For every backend we will initialize two balancer instances: (1) with all endpoints (2) with all endpoints corresponding to current zone for the backend. Then given the request once we choose what backend needs to serve the request, we will first try to use zonal balancer for that backend. If zonal balancer does not exist (i.e there's no zonal endpoint) then we will use general balancer. In case of zonal outages we assume that readiness probe will fail and controller will see no endpoints for the backend and therefore we will use general balancer. We can enable the feature using a configmap setting. Doing it this way makes it easier to rollback in case of a problem. Implementation History \u00b6 initial version of KEP is shipped proposal and implementation details is done Drawbacks [optional] \u00b6 More load on the Kubernetes API server.","title":"Availability zone aware routing"},{"location":"enhancements/20190815-zone-aware-routing/#availability-zone-aware-routing","text":"","title":"Availability zone aware routing"},{"location":"enhancements/20190815-zone-aware-routing/#table-of-contents","text":"Summary Motivation Goals Non-Goals Proposal Implementation History Drawbacks [optional]","title":"Table of Contents"},{"location":"enhancements/20190815-zone-aware-routing/#summary","text":"Teach ingress-nginx about availability zones where endpoints are running in. This way ingress-nginx pod will do its best to proxy to zone-local endpoint.","title":"Summary"},{"location":"enhancements/20190815-zone-aware-routing/#motivation","text":"When users run their services across multiple availability zones they usually pay for egress traffic between zones. Providers such as GCP, Amazon EC charges money for that. ingress-nginx when picking an endpoint to route request to does not consider whether the endpoint is in different zone or the same one. That means it's at least equally likely that it will pick an endpoint from another zone and proxy the request to it. In this situation response from the endpoint to ingress-nginx pod is considered as inter zone traffic and costs money. At the time of this writing GCP charges $0.01 per GB of inter zone egress traffic according to https://cloud.google.com/compute/network-pricing. According to https://datapath.io/resources/blog/what-are-aws-data-transfer-costs-and-how-to-minimize-them/ Amazon also charges the same amount of money sa GCP for cross zone, egress traffic. This can be a lot of money depending on once's traffic. By teaching ingress-nginx about zones we can eliminate or at least decrease this cost. Arguably inter-zone network latency should also be better than cross zone.","title":"Motivation"},{"location":"enhancements/20190815-zone-aware-routing/#goals","text":"Given a regional cluster running ingress-nginx, ingress-nginx should do best effort to pick zone-local endpoint when proxying This should not impact canary feature ingress-nginx should be able to operate successfully if there's no zonal endpoints","title":"Goals"},{"location":"enhancements/20190815-zone-aware-routing/#non-goals","text":"This feature inherently assumes that endpoints are distributed across zones in a way that they can handle all the traffic from ingress-nginx pod(s) in that zone This feature will be relying on https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#failure-domainbetakubernetesiozone, it is not this KEP's goal to support other cases","title":"Non-Goals"},{"location":"enhancements/20190815-zone-aware-routing/#proposal","text":"The idea here is to have controller part of ingress-nginx to (1) detect what zone its current pod is running in and (2) detect the zone for every endpoints it knows about. After that it will post that data as part of endpoints to Lua land. Then Lua balancer when picking an endpoint will try to pick zone-local endpoint first and if there is no zone-local endpoint then it will fallback to current behaviour. This feature at least in the beginning should be optional since it is going to make it harder to reason about the load balancing and not everyone might want that. How does controller know what zone it runs in? We can have the pod spec do pass node name using downward API as an environment variable. Then on start controller can get node details from the API based on node name. Once the node details is obtained we can extract the zone from failure-domain.beta.kubernetes.io/zone annotation. Then we can pass that value to Lua land through Nginx configuration when loading lua_ingress.lua module in init_by_lua phase. How do we extract zones for endpoints? We can have the controller watch create and update events on nodes in the entire cluster and based on that keep the map of nodes to zones in the memory. And when we generate endpoints list, we can access node name using .subsets.addresses [ i ]. nodeName and based on that fetch zone from the map in memory and store it as a field on the endpoint. This solution assumes failure-domain.beta.kubernetes.io/zone annotation does not change until the end of node's life. Otherwise we have to watch update events as well on the nodes and that'll add even more overhead. Alternatively, we can get the list of nodes only when there's no node in the memory for given node name. This is probably a better solution because then we would avoid watching for API changes on node resources. We can eagrly fetch all the nodes and build node name to zone mapping on start. And from thereon sync it during endpoints building in the main event loop iff there's no entry exist for the node of an endpoint. This means an extra API call in case cluster has expanded. How do we make sure we do our best to choose zone-local endpoint? This will be done on Lua side. For every backend we will initialize two balancer instances: (1) with all endpoints (2) with all endpoints corresponding to current zone for the backend. Then given the request once we choose what backend needs to serve the request, we will first try to use zonal balancer for that backend. If zonal balancer does not exist (i.e there's no zonal endpoint) then we will use general balancer. In case of zonal outages we assume that readiness probe will fail and controller will see no endpoints for the backend and therefore we will use general balancer. We can enable the feature using a configmap setting. Doing it this way makes it easier to rollback in case of a problem.","title":"Proposal"},{"location":"enhancements/20190815-zone-aware-routing/#implementation-history","text":"initial version of KEP is shipped proposal and implementation details is done","title":"Implementation History"},{"location":"enhancements/20190815-zone-aware-routing/#drawbacks-optional","text":"More load on the Kubernetes API server.","title":"Drawbacks [optional]"},{"location":"enhancements/YYYYMMDD-kep-template/","text":"Title \u00b6 This is the title of the KEP. Keep it simple and descriptive. A good title can help communicate what the KEP is and should be considered as part of any review. The title should be lowercased and spaces/punctuation should be replaced with - . To get started with this template: Make a copy of this template. Create a copy of this template and name it YYYYMMDD-my-title.md , where YYYYMMDD is the date the KEP was first drafted. Fill out the \"overview\" sections. This includes the Summary and Motivation sections. These should be easy if you've preflighted the idea of the KEP in an issue. Create a PR. Assign it to folks that are sponsoring this process. Create an issue When filing an enhancement tracking issue, please ensure to complete all fields in the template. Merge early. Avoid getting hung up on specific details and instead aim to get the goal of the KEP merged quickly. The best way to do this is to just start with the \"Overview\" sections and fill out details incrementally in follow on PRs. View anything marked as a provisional as a working document and subject to change. Aim for single topic PRs to keep discussions focused. If you disagree with what is already in a document, open a new PR with suggested changes. The canonical place for the latest set of instructions (and the likely source of this file) is here . The Metadata section above is intended to support the creation of tooling around the KEP process. This will be a YAML section that is fenced as a code block. See the KEP process for details on each of these items. Table of Contents \u00b6 A table of contents is helpful for quickly jumping to sections of a KEP and for highlighting any additional information provided beyond the standard KEP template. Ensure the TOC is wrapped with
You need a TLS cert and a test HTTP service for this example.
Create a values.yaml file.
values.yaml
apiVersion: networking.k8s.io/v1beta1 +Create a values.yaml file. +apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: nginx-test @@ -1247,16 +1247,16 @@ # This assumes http-svc exists and routes to healthy endpoints serviceName: http-svc servicePort: 80 - +
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: nginx-test @@ -1247,16 +1247,16 @@ # This assumes http-svc exists and routes to healthy endpoints serviceName: http-svc servicePort: 80 -
The following command instructs the controller to terminate traffic using the provided TLS cert, and forward un-encrypted HTTP traffic to the test HTTP service.
kubectl apply -f ingress.yaml -
kubectl apply -f ingress.yaml +
You can confirm that the Ingress works.
$ kubectl describe ing nginx-test +$ kubectl describe ing nginx-test Name: nginx-test Namespace: default Address: 104.198.183.6 @@ -1303,10 +1303,11 @@ TLS cert, and forward un-encrypted HTTP traffic to the test HTTP service. x-forwarded-for=104.132.0.80, 35.186.221.137 x-forwarded-proto=https BODY: - +
$ kubectl describe ing nginx-test Name: nginx-test Namespace: default Address: 104.198.183.6 @@ -1303,10 +1303,11 @@ TLS cert, and forward un-encrypted HTTP traffic to the test HTTP service. x-forwarded-for=104.132.0.80, 35.186.221.137 x-forwarded-proto=https BODY: -
The objective of this document is to explain how the NGINX Ingress controller works, in particular how the NGINX model is built and why we need one.
The goal of this Ingress controller is the assembly of a configuration file (nginx.conf). The main implication of this requirement is the need to reload NGINX after any change in the configuration file. Though it is important to note that we don't reload Nginx on changes that impact only an upstream configuration (i.e Endpoints change when you deploy your app). We use lua-nginx-module to achieve this. Check below to learn more about how it's done.
upstream
Usually, a Kubernetes Controller utilizes the synchronization loop pattern to check if the desired state in the controller is updated or a change is required. To this purpose, we need to build a model using different objects from the cluster, in particular (in no special order) Ingresses, Services, Endpoints, Secrets, and Configmaps to generate a point in time configuration file that reflects the state of the cluster.
To get this object from the cluster, we use Kubernetes Informers, in particular, FilteredSharedInformer. This informers allows reacting to changes in using callbacks to individual changes when a new object is added, modified or removed. Unfortunately, there is no way to know if a particular change is going to affect the final configuration file. Therefore on every change, we have to rebuild a new model from scratch based on the state of cluster and compare it to the current model. If the new model equals to the current one, then we avoid generating a new NGINX configuration and triggering a reload. Otherwise, we check if the difference is only about Endpoints. If so we then send the new list of Endpoints to a Lua handler running inside Nginx using HTTP POST request and again avoid generating a new NGINX configuration and triggering a reload. If the difference between running and new model is about more than just Endpoints we create a new NGINX configuration based on the new model, replace the current model and trigger a reload.
FilteredSharedInformer
One of the uses of the model is to avoid unnecessary reloads when there's no change in the state and to detect conflicts in definitions.
The final representation of the NGINX configuration is generated from a Go template using the new model as input for the variables required by the template.
Operations to build the model:
Order Ingress rules by CreationTimestamp field, i.e., old rules first.
CreationTimestamp
If the same path for the same host is defined in more than one Ingress, the oldest rule wins.
load-balance
In some cases, it is possible to avoid reloads, in particular when there is a change in the endpoints, i.e., a pod is started or replaced. It is out of the scope of this Ingress controller to remove reloads completely. This would require an incredible amount of work and at some point makes no sense. This can change only if NGINX changes the way new configurations are read, basically, new changes do not replace worker processes.
On every endpoint change the controller fetches endpoints from all the services it sees and generates corresponding Backend objects. It then sends these objects to a Lua handler running inside Nginx. The Lua code in turn stores those backends in a shared memory zone. Then for every request Lua code running in balancer_by_lua context detects what endpoints it should choose upstream peer from and applies the configured load balancing algorithm to choose the peer. Then Nginx takes care of the rest. This way we avoid reloading Nginx on endpoint changes. Note that this includes annotation changes that affects only upstream configuration in Nginx as well.
balancer_by_lua
In a relatively big clusters with frequently deploying apps this feature saves significant number of Nginx reloads which can otherwise affect response latency, load balancing quality (after every reload Nginx resets the state of load balancing) and so on.
Because the ingress controller works using the synchronization loop pattern, it is applying the configuration for all matching objects. In case some Ingress objects have a broken configuration, for example a syntax error in the nginx.ingress.kubernetes.io/configuration-snippet annotation, the generated configuration becomes invalid, does not reload and hence no more ingresses will be taken into account.
nginx.ingress.kubernetes.io/configuration-snippet
To prevent this situation to happen, the nginx ingress controller optionally exposes a validating admission webhook server to ensure the validity of incoming ingress objects. This webhook appends the incoming ingress objects to the list of ingresses, generates the configuration and calls nginx to ensure the configuration has no syntax errors.
Install krew, then run
kubectl krew install ingress-nginx -
kubectl krew install ingress-nginx +
to install the plugin. Then run
kubectl ingress-nginx --help -
kubectl ingress-nginx --help +
to make sure the plugin is properly installed and to get a list of commands:
kubectl ingress-nginx --help +kubectl ingress-nginx --help A kubectl plugin for inspecting your ingress-nginx deployments Usage: @@ -1430,29 +1430,29 @@ Do not move it without providing redirects. --user string The name of the kubeconfig user to use Use "ingress-nginx [command] --help" for more information about a command. - +
kubectl ingress-nginx --help A kubectl plugin for inspecting your ingress-nginx deployments Usage: @@ -1430,29 +1430,29 @@ Do not move it without providing redirects. --user string The name of the kubeconfig user to use Use "ingress-nginx [command] --help" for more information about a command. -
If a new ingress-nginx version has just been released, the plugin may not yet have been updated inside the repository. In that case, you can install the latest version of the plugin by running:
( +If a new ingress-nginx version has just been released, the plugin may not yet have been updated inside the repository. In that case, you can install the latest version of the plugin by running: +( set -x; cd "$(mktemp -d)" && curl -fsSLO "https://github.com/kubernetes/ingress-nginx/releases/download/nginx-0.24.0/{ingress-nginx.yaml,kubectl-ingress_nginx-$(uname | tr '[:upper:]' '[:lower:]')-amd64.tar.gz}" && kubectl krew install \ --manifest=ingress-nginx.yaml --archive=kubectl-ingress_nginx-$(uname | tr '[:upper:]' '[:lower:]')-amd64.tar.gz ) - +
( set -x; cd "$(mktemp -d)" && curl -fsSLO "https://github.com/kubernetes/ingress-nginx/releases/download/nginx-0.24.0/{ingress-nginx.yaml,kubectl-ingress_nginx-$(uname | tr '[:upper:]' '[:lower:]')-amd64.tar.gz}" && kubectl krew install \ --manifest=ingress-nginx.yaml --archive=kubectl-ingress_nginx-$(uname | tr '[:upper:]' '[:lower:]')-amd64.tar.gz ) -
Replacing 0.24.0 with the recently released version.
0.24.0
kubectl
--namespace
--context
--client-key
backends
certs
conf
exec
general
logs
ssh
--deployment <deployment>
--pod <pod>
--deployment
lint
--all-namespaces
Note that backends, general, certs, and conf require ingress-nginx version 0.23.0 or higher.
0.23.0
Run kubectl ingress-nginx backends to get a JSON array of the backends that an ingress-nginx controller currently knows about:
kubectl ingress-nginx backends
$ kubectl ingress-nginx backends -n ingress-nginx +Run kubectl ingress-nginx backends to get a JSON array of the backends that an ingress-nginx controller currently knows about: +$ kubectl ingress-nginx backends -n ingress-nginx [ { "name": "default-apple-service-5678", @@ -1513,13 +1513,13 @@ Do not move it without providing redirects. ... } ] - +
$ kubectl ingress-nginx backends -n ingress-nginx [ { "name": "default-apple-service-5678", @@ -1513,13 +1513,13 @@ Do not move it without providing redirects. ... } ] -
Add the --list option to show only the backend names. Add the --backend <backend> option to show only the backend with the given name.
--list
--backend <backend>
Use kubectl ingress-nginx certs --host <hostname> to dump the SSL cert/key information for a given host. Requires that --enable-dynamic-certificates is true (this is the default as of version 0.24.0).
kubectl ingress-nginx certs --host <hostname>
WARNING: This command will dump sensitive private key information. Don't blindly share the output, and certainly don't log it anywhere.
$ kubectl ingress-nginx certs -n ingress-nginx --host testaddr.local +$ kubectl ingress-nginx certs -n ingress-nginx --host testaddr.local -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- @@ -1530,11 +1530,11 @@ Do not move it without providing redirects. -----BEGIN RSA PRIVATE KEY----- <REDACTED! DO NOT SHARE THIS!> -----END RSA PRIVATE KEY----- - +
$ kubectl ingress-nginx certs -n ingress-nginx --host testaddr.local -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- @@ -1530,11 +1530,11 @@ Do not move it without providing redirects. -----BEGIN RSA PRIVATE KEY----- <REDACTED! DO NOT SHARE THIS!> -----END RSA PRIVATE KEY----- -
Use kubectl ingress-nginx conf to dump the generated nginx.conf file. Add the --host <hostname> option to view only the server block for that host:
kubectl ingress-nginx conf
nginx.conf
--host <hostname>
kubectl ingress-nginx conf -n ingress-nginx --host testaddr.local +Use kubectl ingress-nginx conf to dump the generated nginx.conf file. Add the --host <hostname> option to view only the server block for that host: +kubectl ingress-nginx conf -n ingress-nginx --host testaddr.local server { server_name testaddr.local ; @@ -1556,11 +1556,11 @@ Do not move it without providing redirects. set $location_path "/"; ... - +
kubectl ingress-nginx conf -n ingress-nginx --host testaddr.local server { server_name testaddr.local ; @@ -1556,11 +1556,11 @@ Do not move it without providing redirects. set $location_path "/"; ... -
kubectl ingress-nginx exec is exactly the same as kubectl exec, with the same command flags. It will automatically choose an ingress-nginx pod to run the command in.
kubectl ingress-nginx exec
kubectl exec
$ kubectl ingress-nginx exec -i -n ingress-nginx -- ls /etc/nginx +kubectl ingress-nginx exec is exactly the same as kubectl exec, with the same command flags. It will automatically choose an ingress-nginx pod to run the command in. +$ kubectl ingress-nginx exec -i -n ingress-nginx -- ls /etc/nginx fastcgi_params geoip lua @@ -1571,44 +1571,44 @@ Do not move it without providing redirects. opentracing.json owasp-modsecurity-crs template - +
$ kubectl ingress-nginx exec -i -n ingress-nginx -- ls /etc/nginx fastcgi_params geoip lua @@ -1571,44 +1571,44 @@ Do not move it without providing redirects. opentracing.json owasp-modsecurity-crs template -
kubectl ingress-nginx general dumps miscellaneous controller state as a JSON object. Currently it just shows the number of controller pods known to a particular controller pod.
kubectl ingress-nginx general
$ kubectl ingress-nginx general -n ingress-nginx +kubectl ingress-nginx general dumps miscellaneous controller state as a JSON object. Currently it just shows the number of controller pods known to a particular controller pod. +$ kubectl ingress-nginx general -n ingress-nginx { "controllerPodsCount": 1 } - +
$ kubectl ingress-nginx general -n ingress-nginx { "controllerPodsCount": 1 } -
Shows the internal and external IP/CNAMES for an ingress-nginx service.
$ kubectl ingress-nginx info -n ingress-nginx +Shows the internal and external IP/CNAMES for an ingress-nginx service. +$ kubectl ingress-nginx info -n ingress-nginx Service cluster IP address: 10.187.253.31 LoadBalancer IP|CNAME: 35.123.123.123 - +
$ kubectl ingress-nginx info -n ingress-nginx Service cluster IP address: 10.187.253.31 LoadBalancer IP|CNAME: 35.123.123.123 -
Use the --service <service> flag if your ingress-nginx LoadBalancer service is not named ingress-nginx.
--service <service>
LoadBalancer
kubectl ingress-nginx ingresses, alternately kubectl ingress-nginx ing, shows a more detailed view of the ingress definitions in a namespace. Compare:
kubectl ingress-nginx ingresses
kubectl ingress-nginx ing
$ kubectl get ingresses --all-namespaces +kubectl ingress-nginx ingresses, alternately kubectl ingress-nginx ing, shows a more detailed view of the ingress definitions in a namespace. Compare: +$ kubectl get ingresses --all-namespaces NAMESPACE NAME HOSTS ADDRESS PORTS AGE default example-ingress1 testaddr.local,testaddr2.local localhost 80 5d default test-ingress-2 * localhost 80 5d - +
$ kubectl get ingresses --all-namespaces NAMESPACE NAME HOSTS ADDRESS PORTS AGE default example-ingress1 testaddr.local,testaddr2.local localhost 80 5d default test-ingress-2 * localhost 80 5d -
vs
$ kubectl ingress-nginx ingresses --all-namespaces +$ kubectl ingress-nginx ingresses --all-namespaces NAMESPACE INGRESS NAME HOST+PATH ADDRESSES TLS SERVICE SERVICE PORT ENDPOINTS default example-ingress1 testaddr.local/etameta localhost NO pear-service 5678 5 default example-ingress1 testaddr2.local/otherpath localhost NO apple-service 5678 1 default example-ingress1 testaddr2.local/otherotherpath localhost NO pear-service 5678 5 default test-ingress-2 * localhost NO echo-service 8080 2 - +
$ kubectl ingress-nginx ingresses --all-namespaces NAMESPACE INGRESS NAME HOST+PATH ADDRESSES TLS SERVICE SERVICE PORT ENDPOINTS default example-ingress1 testaddr.local/etameta localhost NO pear-service 5678 5 default example-ingress1 testaddr2.local/otherpath localhost NO apple-service 5678 1 default example-ingress1 testaddr2.local/otherotherpath localhost NO pear-service 5678 5 default test-ingress-2 * localhost NO echo-service 8080 2 -
kubectl ingress-nginx lint can check a namespace or entire cluster for potential configuration issues. This command is especially useful when upgrading between ingress-nginx versions.
kubectl ingress-nginx lint
$ kubectl ingress-nginx lint --all-namespaces --verbose +kubectl ingress-nginx lint can check a namespace or entire cluster for potential configuration issues. This command is especially useful when upgrading between ingress-nginx versions. +$ kubectl ingress-nginx lint --all-namespaces --verbose Checking ingresses... ✗ anamespace/this-nginx - Contains the removed session-cookie-hash annotation. @@ -1627,10 +1627,10 @@ Do not move it without providing redirects. - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808 - +
$ kubectl ingress-nginx lint --all-namespaces --verbose Checking ingresses... ✗ anamespace/this-nginx - Contains the removed session-cookie-hash annotation. @@ -1627,10 +1627,10 @@ Do not move it without providing redirects. - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808 -
to show the lints added only for a particular ingress-nginx release, use the --from-version and --to-version flags:
--from-version
--to-version
$ kubectl ingress-nginx lint --all-namespaces --verbose --from-version 0.24.0 --to-version 0.24.0 +to show the lints added only for a particular ingress-nginx release, use the --from-version and --to-version flags: +$ kubectl ingress-nginx lint --all-namespaces --verbose --from-version 0.24.0 --to-version 0.24.0 Checking ingresses... ✗ anamespace/this-nginx - Contains the removed session-cookie-hash annotation. @@ -1642,11 +1642,11 @@ Do not move it without providing redirects. - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808 - +
$ kubectl ingress-nginx lint --all-namespaces --verbose --from-version 0.24.0 --to-version 0.24.0 Checking ingresses... ✗ anamespace/this-nginx - Contains the removed session-cookie-hash annotation. @@ -1642,11 +1642,11 @@ Do not move it without providing redirects. - Uses removed config flag --enable-dynamic-certificates Lint added for version 0.24.0 https://github.com/kubernetes/ingress-nginx/issues/3808 -
kubectl ingress-nginx logs is almost the same as kubectl logs, with fewer flags. It will automatically choose an ingress-nginx pod to read logs from.
kubectl ingress-nginx logs
kubectl logs
$ kubectl ingress-nginx logs -n ingress-nginx +kubectl ingress-nginx logs is almost the same as kubectl logs, with fewer flags. It will automatically choose an ingress-nginx pod to read logs from. +$ kubectl ingress-nginx logs -n ingress-nginx ------------------------------------------------------------------------------- NGINX Ingress controller Release: dev @@ -1662,16 +1662,17 @@ Do not move it without providing redirects. I0405 16:53:46.183359 7 nginx.go:265] Starting NGINX Ingress controller I0405 16:53:46.193913 7 event.go:209] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"82258915-563e-11e9-9c52-025000000001", APIVersion:"v1", ResourceVersion:"494", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services ... - +
$ kubectl ingress-nginx logs -n ingress-nginx ------------------------------------------------------------------------------- NGINX Ingress controller Release: dev @@ -1662,16 +1662,17 @@ Do not move it without providing redirects. I0405 16:53:46.183359 7 nginx.go:265] Starting NGINX Ingress controller I0405 16:53:46.193913 7 event.go:209] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"82258915-563e-11e9-9c52-025000000001", APIVersion:"v1", ResourceVersion:"494", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services ... -
kubectl ingress-nginx ssh is exactly the same as kubectl ingress-nginx exec -it -- /bin/bash. Use it when you want to quickly be dropped into a shell inside a running ingress-nginx container.
kubectl ingress-nginx ssh
kubectl ingress-nginx exec -it -- /bin/bash
$ kubectl ingress-nginx ssh -n ingress-nginx +kubectl ingress-nginx ssh is exactly the same as kubectl ingress-nginx exec -it -- /bin/bash. Use it when you want to quickly be dropped into a shell inside a running ingress-nginx container. +$ kubectl ingress-nginx ssh -n ingress-nginx www-data@nginx-ingress-controller-7cbf77c976-wx5pn:/etc/nginx$ - +
$ kubectl ingress-nginx ssh -n ingress-nginx www-data@nginx-ingress-controller-7cbf77c976-wx5pn:/etc/nginx$ -