From 4f06fb09d5e623c879cf158b0ad812e61928531d Mon Sep 17 00:00:00 2001
From: Travis Bot
+
+
The rest of this document describes a few recommended approaches to deploying the NGINX Ingress controller inside a Kubernetes cluster running on bare-metal.
+MetalLB provides a network load-balancer implementation for Kubernetes clusters that do not run on a +supported cloud provider, effectively allowing the usage of LoadBalancer Services within any cluster.
+This section demonstrates how to use the Layer 2 configuration mode of MetalLB together with the NGINX
+Ingress controller in a Kubernetes cluster that has publicly accessible nodes. In this mode, one node attracts all
+the traffic for the ingress-nginx
Service IP. See Traffic policies for more details.
Note
+The description of other supported configuration modes is off-scope for this document.
+Warning
+MetalLB is currently in beta. Read about the Project maturity and make sure you inform +yourself by reading the official documentation thoroughly.
+MetalLB can be deployed either with a simple Kubernetes manifest or with Helm. The rest of this example assumes MetalLB +was deployed following the Installation instructions.
+MetalLB requires a pool of IP addresses in order to be able to take ownership of the ingress-nginx
Service. This pool
+can be defined in a ConfigMap named config
located in the same namespace as the MetalLB controller. In the simplest
+possible scenario, the pool is composed of the IP addresses of Kubernetes nodes, but IP addresses can also be handed out
+by a DHCP server.
Example
+Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal +environments this value is <None>)
+$ kubectl describe node +NAME STATUS ROLES EXTERNAL-IP +host-1 Ready master 203.0.113.1 +host-2 Ready node 203.0.113.2 +host-3 Ready node 203.0.113.3 +
After creating the following ConfigMap, MetalLB takes ownership of one of the IP addresses in the pool and updates
+the loadBalancer IP field of the ingress-nginx
Service accordingly.
apiVersion: v1 +kind: ConfigMap +metadata: + namespace: metallb-system + name: config +data: + config: | + address-pools: + - name: default + protocol: layer2 + addresses: + - 203.0.113.2-203.0.113.3 +
$ kubectl -n ingress-nginx get svc +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) +default-http-backend ClusterIP 10.0.64.249 <none> 80/TCP +ingress-nginx LoadBalancer 10.0.220.217 203.0.113.3 80:30100/TCP,443:30101/TCP +
As soon as MetalLB sets the external IP address of the ingress-nginx
LoadBalancer Service, the corresponding entries
+are created in the iptables NAT table and the node with the selected IP address starts responding to HTTP requests on
+the ports configured in the LoadBalancer Service:
$ curl -D- http://203.0.113.3 -H 'Host: myapp.example.com' +HTTP/1.1 200 OK +Server: nginx/1.15.2 +
Tip
+In order to preserve the source IP address in HTTP requests sent to NGINX, it is necessary to use the Local
+traffic policy. Traffic policies are described in more details in Traffic policies as
+well as in the next section.
Due to its simplicity, this is the setup a user will deploy by default when following the steps described in the installation guide.
@@ -1166,7 +1264,7 @@ requests.and a Kubernetes node with the public IP address 203.0.113.2
(the external IP is added as an example, in most
bare-metal environments this value is <None>)
$ kubectl describe node +$ kubectl describe node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 @@ -1203,7 +1301,7 @@ the NGINX Ingress controller should be scheduled or not scheduled.Example
In a Kubernetes cluster composed of 3 nodes (the external IP is added as an example, in most bare-metal environments this value is <None>)
-$ kubectl describe node +$ kubectl describe node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 @@ -1234,11 +1332,17 @@ update the status of Ingress objects it manages.Despite the fact there is no load balancer providing a public IP address to the NGINX Ingress controller, it is possible to force the status update of all managed Ingress objects by setting the
+externalIPs
field of theingress-nginx
Service.+Warning
+There is more to setting
+externalIPs
than just enabling the NGINX Ingress controller to update the status of +Ingress objects. Please read about this option in the Services page of official Kubernetes +documentation as well as the section about External IPs in this document for more information.Example
Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal environments this value is <None>)
-$ kubectl describe node +$ kubectl describe node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 @@ -1269,7 +1373,7 @@ for generating redirect URLs that take into account the URL used by external cliExample
Redirects generated by NGINX, for instance HTTP to HTTPS or
-domain
towww.domain
, are generated without NodePort:$ curl http://myapp.example.com:30100` +$ curl -D- http://myapp.example.com:30100` HTTP/1.1 308 Permanent Redirect Server: nginx/1.15.2 Location: https://myapp.example.com/ #-> missing NodePort in HTTPS redirect @@ -1384,7 +1488,52 @@ This is particularly suitable for private Kubernetes clusters where none of the nodes and/or masters. Incoming traffic on TCP ports 80 and 443 is forwarded to the corresponding HTTP and HTTPS NodePort on the target nodes as shown in the diagram below:- +
External IPs¶
+++Source IP address
+This method does not allow preserving the source IP of HTTP requests in any manner, it is therefore not +recommended to use it despite its apparent simplicity.
+The
+externalIPs
Service option was previously mentioned in the NodePort section.As per the Services page of the official Kubernetes documentation, the
+externalIPs
option causes +kube-proxy
to route traffic sent to arbitrary IP addresses and on the Service ports to the endpoints of that +Service. These IP addresses must belong to the target node.+diff --git a/examples/PREREQUISITES/index.html b/examples/PREREQUISITES/index.html index d41c60b3b..cfd283079 100644 --- a/examples/PREREQUISITES/index.html +++ b/examples/PREREQUISITES/index.html @@ -1320,7 +1320,7 @@ which you can deploy as follows 1m 1m 1 {service-controller } Normal CreatingLoadBalancer Creating load balancer 16s 16s 1 {service-controller } Normal CreatedLoadBalancer Created load balancer -$ curl 108.59.87.126 +$ curl 108.59.87.136 CLIENT VALUES: client_address=10.240.0.3 command=GET diff --git a/images/baremetal/metallb.gliffy b/images/baremetal/metallb.gliffy new file mode 100644 index 000000000..2ad2de0ec --- /dev/null +++ b/images/baremetal/metallb.gliffy @@ -0,0 +1 @@ +{"contentType":"application/gliffy+json","version":"1.3","stage":{"background":"#FFFFFF","width":737,"height":427,"nodeIndex":402,"autoFit":true,"exportBorder":false,"gridOn":true,"snapToGrid":true,"drawingGuidesOn":true,"pageBreaksOn":false,"printGridOn":false,"printPaper":null,"printShrinkToFit":false,"printPortrait":false,"maxWidth":5000,"maxHeight":5000,"themeData":null,"imageCache":{},"viewportType":"default","fitBB":{"min":{"x":36.86442430045474,"y":20},"max":{"x":736.8644243004547,"y":427}},"printModel":{"pageSize":"Letter","portrait":true,"fitToOnePage":false,"displayPageBreaks":false},"objects":[{"x":50.0,"y":52.0,"rotation":0.0,"id":356,"width":146.0,"height":1.0,"uid":"com.gliffy.shape.basic.basic_v1.default.line","order":72,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Line","Line":{"strokeWidth":2.0,"strokeColor":"#999999","fillColor":"none","dashStyle":null,"startArrow":0,"endArrow":0,"startArrowRotation":"auto","endArrowRotation":"auto","interpolationType":"linear","cornerRadius":null,"controlPath":[[0.0,0.0],[125.0,0.0]],"lockSegments":{},"ortho":false}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":46.86442430045474,"y":29.0,"rotation":0.0,"id":354,"width":245.0,"height":22.0,"uid":"com.gliffy.shape.basic.basic_v1.default.text","order":71,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Text","Text":{"overflow":"none","paddingTop":2,"paddingRight":2,"paddingBottom":2,"paddingLeft":2,"outerPaddingTop":6,"outerPaddingRight":6,"outerPaddingBottom":2,"outerPaddingLeft":6,"type":"fixed","lineTValue":null,"linePerpValue":null,"cardinalityType":null,"html":"Example
+Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal +environments this value is <None>)
++ +$ kubectl describe node +NAME STATUS ROLES EXTERNAL-IP +host-1 Ready master 203.0.113.1 +host-2 Ready node 203.0.113.2 +host-3 Ready node 203.0.113.3 +and the following
+ingress-nginx
NodePort Service+ +$ kubectl -n ingress-nginx get svc +NAME TYPE CLUSTER-IP PORT(S) +ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP +One could set the following external IPs in the Service spec, and NGINX would become available on both the NodePort +and the Service port:
++ +spec: + externalIPs: + - 203.0.113.2 + - 203.0.113.3 ++ +$ curl -D- http://myapp.example.com:30100 +HTTP/1.1 200 OK +Server: nginx/1.15.2 + +$ curl -D- http://myapp.example.com +HTTP/1.1 200 OK +Server: nginx/1.15.2 +We assume the myapp.example.com subdomain above resolves to both 203.0.113.2 and 203.0.113.3 IP addresses.
+MetalLB: Layer 2
","tid":null,"valign":"middle","vposition":"none","hposition":"none"}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":326.86442430045474,"y":20.0,"rotation":0.0,"id":325,"width":120.0,"height":90.0,"uid":"com.gliffy.shape.basic.basic_v1.default.group","order":62,"lockAspectRatio":false,"lockShape":false,"children":[{"x":7.225609756097583,"y":20.0,"rotation":0.0,"id":317,"width":62.5,"height":50.0,"uid":"com.gliffy.shape.android.android_v1.icons.monitor","order":60,"lockAspectRatio":true,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.android.android_v1.icons.monitor","strokeWidth":1.0,"strokeColor":"#000000","fillColor":"#676767","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":74.72560975609758,"y":30.0,"rotation":0.0,"id":314,"width":38.048780487804876,"height":40.0,"uid":"com.gliffy.shape.android.android_v1.icons.person","order":56,"lockAspectRatio":true,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.android.android_v1.icons.person","strokeWidth":1.0,"strokeColor":"#000000","fillColor":"#676767","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":0.0,"y":0.0,"rotation":0.0,"id":315,"width":120.0,"height":90.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":13,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":2.0,"strokeColor":"#cccccc","fillColor":"#FFFFFF","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"}],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":547.0,"y":626.5,"rotation":0.0,"id":351,"width":46.0,"height":91.0,"uid":"com.gliffy.shape.basic.basic_v1.default.line","order":70,"lockAspectRatio":false,"lockShape":false,"constraints":{"constraints":[],"startConstraint":{"type":"StartPositionConstraint","StartPositionConstraint":{"nodeId":315,"py":1.0,"px":0.5}},"endConstraint":{"type":"EndPositionConstraint","EndPositionConstraint":{"nodeId":334,"py":0.0,"px":0.5}}},"graphic":{"type":"Line","Line":{"strokeWidth":2.0,"strokeColor":"#f50056","fillColor":"none","dashStyle":null,"startArrow":0,"endArrow":17,"startArrowRotation":"auto","endArrowRotation":"auto","interpolationType":"linear","cornerRadius":null,"controlPath":[[-160.13557569954526,-516.5],[-160.13557569954526,-482.25],[-66.802242366212,-482.25],[-66.802242366212,-398.0]],"lockSegments":{"1":true},"ortho":true}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":607.3644243004547,"y":228.5,"rotation":0.0,"id":338,"width":40.0,"height":16.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":5,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#ff0000","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":460.197757633788,"y":228.5,"rotation":0.0,"id":334,"width":40.0,"height":16.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":4,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#ff0000","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":314.0310909671214,"y":228.5,"rotation":0.0,"id":337,"width":40.0,"height":16.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":6,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#ff0000","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":36.86442430045474,"y":167.0,"rotation":0.0,"id":221,"width":700.0,"height":260.0,"uid":"com.gliffy.shape.basic.basic_v1.default.group","order":7,"lockAspectRatio":false,"lockShape":false,"children":[{"x":10.0,"y":8.5,"rotation":0.0,"id":153,"width":42.0,"height":41.0,"uid":"com.gliffy.shape.basic.basic_v1.default.image","order":9,"lockAspectRatio":true,"lockShape":false,"graphic":{"type":"Image","Image":{"url":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACoAAAApCAYAAABDV7v1AAAAAXNSR0IArs4c6QAAAAlwSFlzAAA9hAAAPYQB1ayvdAAAActpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IlhNUCBDb3JlIDUuNC4wIj4KICAgPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICAgICAgPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIKICAgICAgICAgICAgeG1sbnM6eG1wPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIgogICAgICAgICAgICB4bWxuczp0aWZmPSJodHRwOi8vbnMuYWRvYmUuY29tL3RpZmYvMS4wLyI+CiAgICAgICAgIDx4bXA6Q3JlYXRvclRvb2w+d3d3Lmlua3NjYXBlLm9yZzwveG1wOkNyZWF0b3JUb29sPgogICAgICAgICA8dGlmZjpPcmllbnRhdGlvbj4xPC90aWZmOk9yaWVudGF0aW9uPgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KGMtVWAAAD9hJREFUWAmdWAl0ltWZfr7l37JvSBIIq1AETTBhCUeBgAtKW9qisdYz6tixyAxLUceOMvYYempPrYwWgbFF0PbUnqqo48zRnsG2mlhFAiSAQDuCYRMSAglZ/iz/8n3fnee93/+ziY6de873/99y73uf+77Pu9xr4P/TlDJqVtVb9XU1LmAoEVGxpG2SZXn/ZCjzFgVlAGqz65nP7llfst+fgmPqOOYxjjH8MX/L1BT4tzRlVC1qsps2TEmmR1Utaa1UlrEUSn3bCuVleM6A/mTaGXDj3QME9bKpvLU71w7bdXbMop2Bpg1VTnqR6fdf9P/lgFKDVfddCPDqFe0z4LlLqZvbrFCu7SX7oTzHoUAtk2pWhmnbZiCTgHsc6vgVuN66XeuHf5gGVCWAf0nAX0LDXwy0Tpk1qDfr6+Zw9X6rXHpyLgxvGZ++YYXzDDfeS0hKNGwaUFbA9PslPb6GQWrAI5CAFcwRDcvHN6DMtc3rit/xewI1dcqul351Bkddul0aKAFWtTZZ55t48vK2+aZSy2GY86xgNtxElEi8BMXavAhSZgJ29fhzXZ1j6pfyxG/yR82ZwfPGbvE895nd68t+z2+6aQ2XVrmXAvwZoDV179pnNUiTVy5v+xbNuMy0gjWmFSLAPhFKDSqL/1p/IkSuvwwq/Pb2TH3/nZf6MTHii9fexu9sBKy1TA1nkTlxXol6w8AzzWtK3khT4AIM/jifT6l7gJqU1YgpejvabufAJaYdqqYiCLBX5hNTCriUgcW8QJBPnQmFLELf+mQxmQDMfrgdHXGFkrCBmMSGC1UiGualbKEEuQ064Tayen1O0ccvaUWlsLCfbmcnTIOsWt42p7ezrdkMRH5jBTOrvWRMEWRSI0qZOT1Y/hVReQRWGDTQQyZ39zroifJKKhSFDLj8xnB1/hC5l3ltAqPsaNJzYh4pUS1z9nZOaJ6yoq1Gm1/Appq+qa2lGanJqu+3jvCUetUOF1zFwY6b6E8K+eggAer+Ap3I1Bbf5BCgOFAXNXr4gIsBmn8gpnCQ990EG6SWcwKG7vsZuJRJa3C4MmQumdMO51/lut5rgkUw1da+IhQ7Z0J5UC5uskO5BU6sk0ZT4sWBOH22nU9iXkEq2pOWSRcapEnfa/WwtdPDyCwDDy0Io6jARlG+hYe+HkZZhoH3Ozy8d9JDgoaWMdJEhsgSmSe4qJjEBc4lczqxM3E7lFdAQ90kfQ/lj9HK1EM3T1qlsFleo0r/GmwKpsPeNuGOIc/e6VKozjZABerVCYDxBLf+GxHMKCeokiDycizYoma2nywp0jQ42pbEB7sH8eSf49hDml9bYKboAGyLKtxYaOBMnN7pkbAGTK7B0iRXagrFbBjTVeU18YYLYSgmLDF/S3HbNnrjFCfen8wOqMB7Jzz12r1ZxvxZ2XjutW4sf3UAVcMtNJ1myKsJ4c752RhTFhJcZ5s4krSLnAcHj8Tw/JtR/HRrAlVFJpqOufjFHRm455t5eKshioUbo5g5zEKfA8cKZEoC2dlcWDLd52qdadfeBpPKdFuGHB9Nfo8nT4TjlrglQjBOdroIh0wsu6MAoSBw36uDeOnuTNx6Yy6slPZa2xPYsW+QiQf41vU5MhL/8Yde2FTR1CsjKB4SwLhRYfxocQhXlHXj7pcHsOmuTHx3Yb7ue/w0vZBWk0XSkuJh8n5cVeep0dRmSw1mm/ah/CbhgMtZyxkrczwnLq5mDDgK02mmJVsG0XyoHSvvzMOiWwtw8zUJDBsagElKdNHN32zow6aGQTQ0Onj422EsvME3feP/JPDE5hjmTuvHousiuOnaLORm27jja3mYOz0Tw4uDWsuPv9iDX3/ioDrfxIAEP9pXeUkYVjCXgCv4piXamm2Y2aVV2lgkwFTDphkZUTTT2V/S4Iw8E5sOOhi7sgNHjsc1FwVktN/Fyl+cwV1PRhESvyTny4ZoB+UDUFbEe74TCtz+4yhWbTyDfnqfcFhAHjoWx/h/7cSvWxxcQ5BxASmRTBDAcEw7LBoWnkJ4yixk6DxuCHmVNrj0l84SIrWHTslmAJER57WMiInFC7IQYwj6FSeTVBAWT0u1kNzz9Z9Oefje14L4e/I5Qgqd38rz6ZxctMTaVEsL0KGXi9TOvXmz4eqR5YtPXsalX6FcyYx6RTJODxJt7WR4uf/GCEYNDyFJYMdPJmBxgoorMvD08kJs+GoE+NhDkPEy3YISTw572LggA6uXFaH8KxFNl+NtCSRJqzEjQrj/+gh2ciG2cE0PTSOml7CM4KuJGhtFaaBBOzmRShzmkRsSA2QyGSKBvJsxtDDPwLQrw/IaR07EMfGHHfhwV79+zsuxce8t+di6sQDXVhJwqs2qykDjswX4B37LkdzK9n5THyoe68BRypBWXR5BDkNeHxef8kv9nj8mfYVYjOEsxifISw3UNa1KKRJoele8TsMkUgH6EevgW8dYGEZeSduxP4Zom4d5/96DP25lBcXGqIsZkzMxati5UCXan1aeqb/Lz9sfMPys78UZWqfpLzH9vqw4gNsoezfnEK2mjMh/8WflSB1gmKiUL/5nT02VN2zMEQJUQmvKHEyN44fZyAibcGiyfYeTGFtmYjq1fMPjPXh1Sw+HMd+nUlbbqQTaTvsbAKZjkYnf/b4b837Sg7kFBsYOM7UMl8TMzLAwoZQcYXZiJNPhSWSlANNhGLKgpokMu2pRawbNXS4hwe/gC5ePurF7Ua5eBHO4h70nXLQc89BSbGIEAZ/qcsk5ap8zHTgcw91rurQDvnh/vo6dwsfuqIcrR5s6u4HW2JPvMgJ4mhJFjCq6+CPQixp3MMJTo0Iw2iriXg7PGqvICeI3xe/FJ6SYCAtxRE4qGIjWLh9iYs0dEUyZGMboYUEMLUolcI5rPeVg2w6iZrxu+zuHQJkPOf6+2nwsvM4hvxPY+dc4jrSzrEtZQCud08hcrq2YSg1Jp2JRzVOKHSsYbaWsCiuQFfKSfS4HmcIVqS1belKaPer55TtHZGVa+PHiQm0yPn6mTa/IwCs/dMVcmHpVhv4uMVfa0KKAvqZXZLLCOhchxBI44uFt6cQpx5BSUjI6JKHJstoIZIbc5MBk2/SMqaYVAC3vEqQVpakm0NSPXB9CKQN4yRAbV47zvVm0I7yS1seA39vv4VRnElkZJi4fGUaEPK69KVd/lx8JxAePxNE34GFooY1sen92pgmJwSn64vabc1E+LoSTHQ5OMJW+vj2OA91K5bCwpOaUadq0tDfNVoY3RfjpMRbksNDaQQ49Mi+Cuxb4ebizy0HLp3GMLA3qnH+I95v+qxcdveTpGQ9/2uvioZuDeGK57/ESAdJNwDzHvqvfjGN2uY2xTMmFTB7fW5Ct+TtIzgsdhD5XMc5Ks8wu3PXiAGrKLCXFN0t/vjUrSTBzJPctNK8ypb68jOR+f38CuVk9+Piog4074jj4iYf6x/Mwe2oWCnJZPR1zsOWQizkjqV2aat60iA5RMtG+A4NaWzKxmH1+NQP+tqSmz/N7HcwZbuJfcn1e79w3gFmPdqNwhIkHpwcxviyA+r0JlOQbxiAtK6FSYjvRjhS9fsq8Wuq6CYfFrTUqYqr/POIaz+3iJo5KmkotoNTA6w0DmF6ewZrTxhP35GLLz7pxfEDhm2MtVEzwtRHtc7FsQ7fwC2+tCjCY2ygfz0JlbB/2U/sibzXHFubbiHOyN/7MAFrMkMVN4MoGJoFYHMVMqyPCpkqw/KRxGOLDluslP+UO2HtUb30NUzKxm2SHUSy5ZnLl17BuFF+oKTTxzM4kGnb42ahiQgb+cG82Dn7oYsGUECt6X0PvNPahnrXq+7ze3e73FVBfZ5+PP3Dx1nezUTnJd7IPmvvx1PYkZlERMsc1nEPmHMG5BYNgMVhCCTYy9VFNqMplrSu4uXraSUQZHHRjP7+JQwQoSZxMBL72YAG+MtpPpw3b+3TJd/nIEE4w/9/y0070s584sk11vP5Ioa62pGg+xkr/uhnZWmgLK6fbVrOaYt88erhUTiL7XNMHEYrZ0vLi0RVN60rXaEDNa0t/7ib7XuBeRZ79gstPEZp7Sca8AgpklYcHftkNcShps6dl0dt9J9qxfxCNJzzQN/AJM+TONhfb9w7qfhIR0iCPtSbw8HNdaCZthnCXGmeGOgeSzqPnVa4VzrV4CvOCgJSXhmxBpIyqWsTNVaj1PZ4jVfOsiNMZktxlpF6rxOesANDIfU8l90pr7snRsTIdJ6U+7ex2EO3j5oeDchiGCvLsswWJBPjdfx3EP7/Qg3e5/5qZSyvRT86BPDtXwgrlBIlhW05h6UwpQwWjBqGPUnhCN3lF2yju6BrNQPgyVi8UowjtLFadGnMJfxfBynHE6Z8N4a4zoB0jJFvKS7QE00yQ1Y0sYtwPTqGbPjWLILupivMqptQkRpIHHgHXibUTWvXun5ccSWPT0butaYMnGt21PvtM6fQHP1LKvZM2l92gnGWeRSAZRwrdg8xaj04LYv7MLNTTgVb/rgcD3JXJ4UMXAbV3MvYeS2D7R4N49o0oItyBXTEmjEGWdw2HXYzMNLh9Zuw5F3NFYZzLZKbkrsj0FjavKW1Og5T1X0BhOcoRVV+95PgDdjj333znki6K21hxEn+CDmpj62NFuIzZZtYP2rGDhYreR1Nb8KtBne91bUa7TGYIkqMe0ercH3VAwksG1SncT4EVZsEOZptuPPpA87rSpy8+fzqrLfZDPVbJVOAZ5lM8DHs+EOZmxlCSGnQQlwOEPacUVs4JYwQz1f6DMexghX4Da8qaEoaXUhPVDDdyyf1svruRVdPu0ywPmQhkr/T9WWHs61QsQkQqQ7rf3EA4z5Q5BSRnM+rRoLGkvus1p+95SFbHIxR9SgcjXrzYiXc3WoFs0VFCVh6XspqOdLDVoVn78fI7jJUM1n3UsBzp0Pq68pHqR+7FJeUCT0xeYt/GPf04cIIftOkNRnSxqEqwQA44sZ5t8cIz/yhgamuJi1jkPt0uMH36ZZob5UuPjLaNYCMz1xBxLpo/EOBhxdGYZ7R3M7zQKcYQRIKWP0e3tBT/X/K97BSOsjhup7f7mUeCuvDOdx6eJZz2LGPa+c5zoZSLOHr+x0m1+4P7N09KVC5tnUcU/y1IuOOSMwZLwIYsZYjmJFh/Hsi0PAErm0QJDDFaJZ15mH/08Q3r4HlNa0vfFodu2mBIcv9Mu4Cj538VkEIDcmYLCfqA7KmkRiBc7kiUIablAdr/CVJkykKkL08jIWP5KCWHxzqYbFQPCkhx5M8DKTI+F6h83DxJB1EIwemNzwcyhwaoBEnsOtVekjcy8BJN+qb6eyJDZHnJ3k27nil9SrqnHfkSQ/WrLwSKOjmfPOdcyf72dcp1ZDcnJxLisXJ9+eaP4V7I7RZZiJWknEfOZy90nouF/i9FYjxzfww9WAAAAABJRU5ErkJggg==","urlHash":null,"fileName":null,"imageId":null,"strokeWidth":2.0,"strokeColor":"#000000","dropShadow":false,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":0.0,"y":0.0,"rotation":0.0,"id":3,"width":700.0,"height":260.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":11,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#f5f5f5","gradient":true,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"}],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":436.197757633788,"y":236.5,"rotation":0.0,"id":231,"width":88.0,"height":100.0,"uid":"com.gliffy.shape.basic.basic_v1.default.group","order":57,"lockAspectRatio":false,"lockShape":false,"children":[{"x":61.5,"y":56.0,"rotation":0.0,"id":190,"width":3.0,"height":3.0,"uid":"com.gliffy.shape.basic.basic_v1.default.circle","order":43,"lockAspectRatio":true,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.ellipse.basic_v1","strokeWidth":2.0,"strokeColor":"#333333","fillColor":"#000000","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":19.0,"y":52.5,"rotation":0.0,"id":191,"width":50.0,"height":10.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":41,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#FFFFFF","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":61.5,"y":43.0,"rotation":0.0,"id":193,"width":3.0,"height":3.0,"uid":"com.gliffy.shape.basic.basic_v1.default.circle","order":39,"lockAspectRatio":true,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.ellipse.basic_v1","strokeWidth":2.0,"strokeColor":"#333333","fillColor":"#000000","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":19.0,"y":39.5,"rotation":0.0,"id":194,"width":50.0,"height":10.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":37,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#FFFFFF","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":-6.0,"y":6.0,"rotation":90.0,"id":195,"width":100.0,"height":88.0,"uid":"com.gliffy.shape.basic.basic_v1.default.hexagon","order":35,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.hexagon.basic_v1","strokeWidth":0.0,"strokeColor":"#000000","fillColor":"#3a5de9","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"}],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":142.86442430045474,"y":236.5,"rotation":0.0,"id":235,"width":88.0,"height":100.0,"uid":"com.gliffy.shape.basic.basic_v1.default.group","order":61,"lockAspectRatio":false,"lockShape":false,"children":[{"x":61.5,"y":56.0,"rotation":0.0,"id":181,"width":3.0,"height":3.0,"uid":"com.gliffy.shape.basic.basic_v1.default.circle","order":23,"lockAspectRatio":true,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.ellipse.basic_v1","strokeWidth":2.0,"strokeColor":"#333333","fillColor":"#000000","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":19.0,"y":52.5,"rotation":0.0,"id":182,"width":50.0,"height":10.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":21,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#FFFFFF","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":61.5,"y":43.0,"rotation":0.0,"id":184,"width":3.0,"height":3.0,"uid":"com.gliffy.shape.basic.basic_v1.default.circle","order":19,"lockAspectRatio":true,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.ellipse.basic_v1","strokeWidth":2.0,"strokeColor":"#333333","fillColor":"#000000","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":19.0,"y":39.5,"rotation":0.0,"id":185,"width":50.0,"height":10.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":17,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#FFFFFF","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":-6.0,"y":6.0,"rotation":90.0,"id":186,"width":100.0,"height":88.0,"uid":"com.gliffy.shape.basic.basic_v1.default.hexagon","order":15,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.hexagon.basic_v1","strokeWidth":0.0,"strokeColor":"#000000","fillColor":"#3a5de9","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"}],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":392.0,"y":148.0,"rotation":0.0,"id":339,"width":46.0,"height":91.0,"uid":"com.gliffy.shape.basic.basic_v1.default.line","order":63,"lockAspectRatio":false,"lockShape":false,"constraints":{"constraints":[],"startConstraint":{"type":"StartPositionConstraint","StartPositionConstraint":{"nodeId":366,"py":0.0,"px":0.5}},"endConstraint":{"type":"EndPositionConstraint","EndPositionConstraint":{"nodeId":338,"py":0.0,"px":0.5}}},"graphic":{"type":"Line","Line":{"strokeWidth":2.0,"strokeColor":"#00aeff","fillColor":"none","dashStyle":null,"startArrow":0,"endArrow":0,"startArrowRotation":"auto","endArrowRotation":"auto","interpolationType":"linear","cornerRadius":null,"controlPath":[[-177.38557569954526,62.25],[-177.38557569954526,45.5],[235.36442430045474,45.5],[235.36442430045474,80.5]],"lockSegments":{"1":true},"ortho":true}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":537.0,"y":496.5,"rotation":0.0,"id":340,"width":46.0,"height":91.0,"uid":"com.gliffy.shape.basic.basic_v1.default.line","order":64,"lockAspectRatio":false,"lockShape":false,"constraints":{"constraints":[],"startConstraint":{"type":"StartPositionConstraint","StartPositionConstraint":{"nodeId":368,"py":0.0,"px":0.4999999999999999}},"endConstraint":{"type":"EndPositionConstraint","EndPositionConstraint":{"nodeId":337,"py":0.0,"px":0.5}}},"graphic":{"type":"Line","Line":{"strokeWidth":2.0,"strokeColor":"#00aeff","fillColor":"none","dashStyle":null,"startArrow":0,"endArrow":0,"startArrowRotation":"auto","endArrowRotation":"auto","interpolationType":"linear","cornerRadius":null,"controlPath":[[-322.38557569954526,-279.75],[-322.38557569954526,-302.75],[-202.96890903287863,-302.75],[-202.96890903287863,-268.0]],"lockSegments":{"1":true},"ortho":true}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":110.0,"y":180.0,"rotation":0.0,"id":341,"width":85.0,"height":38.0,"uid":"com.gliffy.shape.basic.basic_v1.default.text","order":65,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Text","Text":{"overflow":"none","paddingTop":2,"paddingRight":2,"paddingBottom":2,"paddingLeft":2,"outerPaddingTop":6,"outerPaddingRight":6,"outerPaddingBottom":2,"outerPaddingLeft":6,"type":"fixed","lineTValue":null,"linePerpValue":null,"cardinalityType":null,"html":"MetalLB controller
","tid":null,"valign":"middle","vposition":"none","hposition":"none"}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":144.36442430045474,"y":346.5,"rotation":0.0,"id":346,"width":85.0,"height":19.0,"uid":"com.gliffy.shape.basic.basic_v1.default.text","order":66,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Text","Text":{"overflow":"none","paddingTop":2,"paddingRight":2,"paddingBottom":2,"paddingLeft":2,"outerPaddingTop":6,"outerPaddingRight":6,"outerPaddingBottom":2,"outerPaddingLeft":6,"type":"fixed","lineTValue":null,"linePerpValue":null,"cardinalityType":null,"html":"master
","tid":null,"valign":"middle","vposition":"none","hposition":"none"}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":291.5310909671214,"y":346.5,"rotation":0.0,"id":348,"width":85.0,"height":19.0,"uid":"com.gliffy.shape.basic.basic_v1.default.text","order":67,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Text","Text":{"overflow":"none","paddingTop":2,"paddingRight":2,"paddingBottom":2,"paddingLeft":2,"outerPaddingTop":6,"outerPaddingRight":6,"outerPaddingBottom":2,"outerPaddingLeft":6,"type":"fixed","lineTValue":null,"linePerpValue":null,"cardinalityType":null,"html":"node
","tid":null,"valign":"middle","vposition":"none","hposition":"none"}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":439.197757633788,"y":346.5,"rotation":0.0,"id":349,"width":85.0,"height":19.0,"uid":"com.gliffy.shape.basic.basic_v1.default.text","order":68,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Text","Text":{"overflow":"none","paddingTop":2,"paddingRight":2,"paddingBottom":2,"paddingLeft":2,"outerPaddingTop":6,"outerPaddingRight":6,"outerPaddingBottom":2,"outerPaddingLeft":6,"type":"fixed","lineTValue":null,"linePerpValue":null,"cardinalityType":null,"html":"node
","tid":null,"valign":"middle","vposition":"none","hposition":"none"}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":584.8644243004547,"y":346.5,"rotation":0.0,"id":350,"width":85.0,"height":19.0,"uid":"com.gliffy.shape.basic.basic_v1.default.text","order":69,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Text","Text":{"overflow":"none","paddingTop":2,"paddingRight":2,"paddingBottom":2,"paddingLeft":2,"outerPaddingTop":6,"outerPaddingRight":6,"outerPaddingBottom":2,"outerPaddingLeft":6,"type":"fixed","lineTValue":null,"linePerpValue":null,"cardinalityType":null,"html":"node
","tid":null,"valign":"middle","vposition":"none","hposition":"none"}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":187.61442430045474,"y":210.25,"rotation":0.0,"id":377,"width":54.0,"height":54.0,"uid":"com.gliffy.shape.basic.basic_v1.default.group","order":81,"lockAspectRatio":false,"lockShape":false,"children":[{"x":37.63557569954526,"y":39.0,"rotation":0.0,"id":373,"width":23.0,"height":11.5,"uid":"com.gliffy.shape.basic.basic_v1.default.line","order":80,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Line","Line":{"strokeWidth":1.0,"strokeColor":"#3f51b5","fillColor":"none","dashStyle":null,"startArrow":0,"endArrow":0,"startArrowRotation":"auto","endArrowRotation":"auto","interpolationType":"linear","cornerRadius":null,"controlPath":[[6.0,2.0],[-26.0,-16.5]],"lockSegments":{},"ortho":false}},"linkMap":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":9.0,"y":31.0,"rotation":0.0,"id":371,"width":36.0,"height":10.0,"uid":"com.gliffy.shape.basic.basic_v1.default.triangle","order":78,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.triangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#3f51b5","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":9.0,"y":6.5,"rotation":0.0,"id":368,"width":36.0,"height":34.0,"uid":"com.gliffy.shape.basic.basic_v1.default.triangle","order":76,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.triangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#FFFFFF","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":0.0,"y":0.0,"rotation":0.0,"id":366,"width":54.0,"height":54.0,"uid":"com.gliffy.shape.basic.basic_v1.default.circle","order":74,"lockAspectRatio":true,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.ellipse.basic_v1","strokeWidth":4.0,"strokeColor":"#ffffff","fillColor":"#3f51b5","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"}],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":210.0,"y":170.0,"rotation":0.0,"id":378,"width":85.0,"height":19.0,"uid":"com.gliffy.shape.basic.basic_v1.default.text","order":82,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Text","Text":{"overflow":"none","paddingTop":2,"paddingRight":2,"paddingBottom":2,"paddingLeft":2,"outerPaddingTop":6,"outerPaddingRight":6,"outerPaddingBottom":2,"outerPaddingLeft":6,"type":"fixed","lineTValue":null,"linePerpValue":null,"cardinalityType":null,"html":"IP pool
","tid":null,"valign":"middle","vposition":"none","hposition":"none"}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":488.9951905283833,"y":233.75,"rotation":29.999999999999996,"id":380,"width":100.0,"height":19.0,"uid":"com.gliffy.shape.basic.basic_v1.default.text","order":83,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Text","Text":{"overflow":"none","paddingTop":2,"paddingRight":2,"paddingBottom":2,"paddingLeft":2,"outerPaddingTop":6,"outerPaddingRight":6,"outerPaddingBottom":2,"outerPaddingLeft":6,"type":"fixed","lineTValue":null,"linePerpValue":null,"cardinalityType":null,"html":"203.0.113.2
","tid":null,"valign":"middle","vposition":"none","hposition":"none"}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":340.0,"y":230.0,"rotation":29.999999999999996,"id":381,"width":85.0,"height":19.0,"uid":"com.gliffy.shape.basic.basic_v1.default.text","order":84,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Text","Text":{"overflow":"none","paddingTop":2,"paddingRight":2,"paddingBottom":2,"paddingLeft":2,"outerPaddingTop":6,"outerPaddingRight":6,"outerPaddingBottom":2,"outerPaddingLeft":6,"type":"fixed","lineTValue":null,"linePerpValue":null,"cardinalityType":null,"html":"203.0.113.1
","tid":null,"valign":"middle","vposition":"none","hposition":"none"}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":640.0,"y":230.0,"rotation":29.999999999999996,"id":382,"width":85.0,"height":19.0,"uid":"com.gliffy.shape.basic.basic_v1.default.text","order":85,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Text","Text":{"overflow":"none","paddingTop":2,"paddingRight":2,"paddingBottom":2,"paddingLeft":2,"outerPaddingTop":6,"outerPaddingRight":6,"outerPaddingBottom":2,"outerPaddingLeft":6,"type":"fixed","lineTValue":null,"linePerpValue":null,"cardinalityType":null,"html":"203.0.113.3
","tid":null,"valign":"middle","vposition":"none","hposition":"none"}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":582.8644243004547,"y":236.5,"rotation":0.0,"id":229,"width":88.0,"height":100.0,"uid":"com.gliffy.shape.basic.basic_v1.default.group","order":54,"lockAspectRatio":false,"lockShape":false,"children":[{"x":61.5,"y":56.0,"rotation":0.0,"id":199,"width":3.0,"height":3.0,"uid":"com.gliffy.shape.basic.basic_v1.default.circle","order":53,"lockAspectRatio":true,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.ellipse.basic_v1","strokeWidth":2.0,"strokeColor":"#333333","fillColor":"#000000","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":19.0,"y":52.5,"rotation":0.0,"id":200,"width":50.0,"height":10.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":51,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#FFFFFF","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":61.5,"y":43.0,"rotation":0.0,"id":202,"width":3.0,"height":3.0,"uid":"com.gliffy.shape.basic.basic_v1.default.circle","order":49,"lockAspectRatio":true,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.ellipse.basic_v1","strokeWidth":2.0,"strokeColor":"#333333","fillColor":"#000000","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":19.0,"y":39.5,"rotation":0.0,"id":203,"width":50.0,"height":10.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":47,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#FFFFFF","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":-6.0,"y":6.0,"rotation":90.0,"id":204,"width":100.0,"height":88.0,"uid":"com.gliffy.shape.basic.basic_v1.default.hexagon","order":45,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.hexagon.basic_v1","strokeWidth":0.0,"strokeColor":"#000000","fillColor":"#3a5de9","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"}],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":599.3644243004546,"y":256.5,"rotation":0.0,"id":383,"width":53.00000000000006,"height":60.0,"uid":"com.gliffy.shape.basic.basic_v1.default.group","order":86,"lockAspectRatio":false,"lockShape":false,"children":[{"x":0.0,"y":5.5,"rotation":0.0,"id":384,"width":53.0,"height":49.0,"uid":"com.gliffy.shape.basic.basic_v1.default.text","order":90,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Text","Text":{"overflow":"none","paddingTop":2,"paddingRight":2,"paddingBottom":2,"paddingLeft":2,"outerPaddingTop":6,"outerPaddingRight":6,"outerPaddingBottom":2,"outerPaddingLeft":6,"type":"fixed","lineTValue":null,"linePerpValue":null,"cardinalityType":null,"html":"N
","tid":null,"valign":"middle","vposition":"none","hposition":"none"}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":-3.599999999999966,"y":3.6000000000000227,"rotation":90.0,"id":385,"width":60.0,"height":52.8,"uid":"com.gliffy.shape.basic.basic_v1.default.hexagon","order":88,"lockAspectRatio":true,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.hexagon.basic_v1","strokeWidth":0.0,"strokeColor":"#000000","fillColor":"#00963a","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"}],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":289.5310909671214,"y":236.5,"rotation":0.0,"id":233,"width":88.0,"height":100.0,"uid":"com.gliffy.shape.basic.basic_v1.default.group","order":58,"lockAspectRatio":false,"lockShape":false,"children":[{"x":61.5,"y":56.0,"rotation":0.0,"id":161,"width":3.0,"height":3.0,"uid":"com.gliffy.shape.basic.basic_v1.default.circle","order":33,"lockAspectRatio":true,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.ellipse.basic_v1","strokeWidth":2.0,"strokeColor":"#333333","fillColor":"#000000","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":19.0,"y":52.5,"rotation":0.0,"id":162,"width":50.0,"height":10.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":31,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#FFFFFF","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":61.5,"y":43.0,"rotation":0.0,"id":164,"width":3.0,"height":3.0,"uid":"com.gliffy.shape.basic.basic_v1.default.circle","order":29,"lockAspectRatio":true,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.ellipse.basic_v1","strokeWidth":2.0,"strokeColor":"#333333","fillColor":"#000000","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":19.0,"y":39.5,"rotation":0.0,"id":165,"width":50.0,"height":10.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":27,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#FFFFFF","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":-6.0,"y":6.0,"rotation":90.0,"id":166,"width":100.0,"height":88.0,"uid":"com.gliffy.shape.basic.basic_v1.default.hexagon","order":25,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.hexagon.basic_v1","strokeWidth":0.0,"strokeColor":"#000000","fillColor":"#3a5de9","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"}],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":306.0310909671214,"y":256.5,"rotation":0.0,"id":387,"width":53.00000000000006,"height":60.0,"uid":"com.gliffy.shape.basic.basic_v1.default.group","order":91,"lockAspectRatio":false,"lockShape":false,"children":[{"x":0.0,"y":5.5,"rotation":0.0,"id":388,"width":53.0,"height":49.0,"uid":"com.gliffy.shape.basic.basic_v1.default.text","order":95,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Text","Text":{"overflow":"none","paddingTop":2,"paddingRight":2,"paddingBottom":2,"paddingLeft":2,"outerPaddingTop":6,"outerPaddingRight":6,"outerPaddingBottom":2,"outerPaddingLeft":6,"type":"fixed","lineTValue":null,"linePerpValue":null,"cardinalityType":null,"html":"N
","tid":null,"valign":"middle","vposition":"none","hposition":"none"}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":-3.599999999999966,"y":3.6000000000000227,"rotation":90.0,"id":389,"width":60.0,"height":52.8,"uid":"com.gliffy.shape.basic.basic_v1.default.hexagon","order":93,"lockAspectRatio":true,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.hexagon.basic_v1","strokeWidth":0.0,"strokeColor":"#000000","fillColor":"#00963a","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"}],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":587.75,"y":778.75,"rotation":0.0,"id":393,"width":46.0,"height":91.0,"uid":"com.gliffy.shape.basic.basic_v1.default.line","order":96,"lockAspectRatio":false,"lockShape":false,"constraints":{"constraints":[],"startConstraint":{"type":"StartPositionConstraint","StartPositionConstraint":{"nodeId":399,"py":0.0,"px":0.5}},"endConstraint":{"type":"EndPositionConstraint","EndPositionConstraint":{"nodeId":394,"py":1.0,"px":0.5}}},"graphic":{"type":"Line","Line":{"strokeWidth":2.0,"strokeColor":"#f50056","fillColor":"none","dashStyle":null,"startArrow":0,"endArrow":17,"startArrowRotation":"auto","endArrowRotation":"auto","interpolationType":"linear","cornerRadius":null,"controlPath":[[-57.75,-490.75],[-42.08333333333337,-490.75],[-26.41666666666663,-490.75],[-10.75,-490.75]],"lockSegments":{},"ortho":true}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":565.0,"y":280.0,"rotation":90.0,"id":394,"width":40.0,"height":16.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":0,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#ff0000","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":419.0,"y":280.0,"rotation":90.0,"id":396,"width":40.0,"height":16.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":3,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#ff0000","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":355.0,"y":280.0,"rotation":90.0,"id":398,"width":40.0,"height":16.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":2,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#ff0000","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":502.0,"y":280.0,"rotation":90.0,"id":399,"width":40.0,"height":16.0,"uid":"com.gliffy.shape.basic.basic_v1.default.rectangle","order":1,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Shape","Shape":{"tid":"com.gliffy.stencil.rectangle.basic_v1","strokeWidth":0.0,"strokeColor":"#333333","fillColor":"#ff0000","gradient":false,"dashStyle":null,"dropShadow":false,"state":0,"opacity":1.0,"shadowX":0.0,"shadowY":0.0}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":597.75,"y":788.75,"rotation":0.0,"id":400,"width":46.0,"height":91.0,"uid":"com.gliffy.shape.basic.basic_v1.default.line","order":97,"lockAspectRatio":false,"lockShape":false,"constraints":{"constraints":[],"startConstraint":{"type":"StartPositionConstraint","StartPositionConstraint":{"nodeId":396,"py":1.0,"px":0.5}},"endConstraint":{"type":"EndPositionConstraint","EndPositionConstraint":{"nodeId":398,"py":0.0,"px":0.5}}},"graphic":{"type":"Line","Line":{"strokeWidth":2.0,"strokeColor":"#f50056","fillColor":"none","dashStyle":null,"startArrow":0,"endArrow":17,"startArrowRotation":"auto","endArrowRotation":"auto","interpolationType":"linear","cornerRadius":null,"controlPath":[[-166.75,-500.75],[-182.75,-500.75],[-198.75,-500.75],[-214.75,-500.75]],"lockSegments":{},"ortho":true}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"},{"x":314.94392033916137,"y":118.0,"rotation":0.0,"id":401,"width":60.0,"height":38.0,"uid":"com.gliffy.shape.basic.basic_v1.default.text","order":98,"lockAspectRatio":false,"lockShape":false,"graphic":{"type":"Text","Text":{"overflow":"none","paddingTop":2,"paddingRight":2,"paddingBottom":2,"paddingLeft":2,"outerPaddingTop":6,"outerPaddingRight":6,"outerPaddingBottom":2,"outerPaddingLeft":6,"type":"fixed","lineTValue":null,"linePerpValue":null,"cardinalityType":null,"html":"80/tcp
443/tcp
","tid":null,"valign":"middle","vposition":"none","hposition":"none"}},"linkMap":[],"children":[],"hidden":false,"layerId":"mcqBzoFMGdMI"}],"layers":[{"guid":"mcqBzoFMGdMI","order":0,"name":"Layer 0","active":true,"locked":false,"visible":true,"nodeIndex":99}],"shapeStyles":{},"lineStyles":{"global":{"stroke":"#f50056","strokeWidth":1,"orthoMode":0,"endArrow":17,"startArrow":0}},"textStyles":{"global":{"bold":false,"face":"Roboto Slab","size":"13px","color":"#676767"}}},"metadata":{"title":"untitled","revision":0,"exportBorder":false,"loadPosition":"default","autosaveDisabled":false,"lastSerialized":1536676662123,"libraries":["com.gliffy.libraries.basic.basic_v1.default","com.gliffy.libraries.flowchart.flowchart_v1.default","com.gliffy.libraries.swimlanes.swimlanes_v1.default","com.gliffy.libraries.uml.uml_v2.class","com.gliffy.libraries.uml.uml_v2.sequence","com.gliffy.libraries.uml.uml_v2.activity","com.gliffy.libraries.erd.erd_v1.default","com.gliffy.libraries.ui.ui_v3.containers_content","com.gliffy.libraries.ui.ui_v3.forms_controls","com.gliffy.libraries.android.android_v1.icons","com.gliffy.libraries.images"]},"embeddedResources":{"index":0,"resources":[]}} \ No newline at end of file diff --git a/images/baremetal/metallb.jpg b/images/baremetal/metallb.jpg new file mode 100644 index 0000000000000000000000000000000000000000..28dad6dea3b71a5da1bdecef00115d6d467d11dc GIT binary patch literal 50191 zcmeFZ2RxkLwm1HW9xXz&V3Ht+7M&J=LB^_AaFg7@1m@d-%4A6_9NB?CYG2ak*I05u6-)1?D^JZj()H6A`S z-bEV#0mn&*_lE=c=L7E&J^>*SF$w8qGO$6#72p!s_ay>+LP7!pu(cofJwQNBc=e{B zJQ0o7Q{r1Lv_h|9vPszPmo#6~9z?MVTe`j`y-Y{Xz{tdLo0E(Cj) _xl2@DDj2@QMmHa0FkAu;KFa>|FC+`NzZ zp9(&gmX%jjR#n&3wzRgjcXa;f>K+;%866w{H8Ht>TwGdSSzTM-K=17z93CBGPELQD z7aoBBPtyV)|7l`>HZM>Jc$Wx3krMwlFT6`$;6FY!0pU$SqO0;+#7|vlZVA02p}ilI zUD8a-CajIRX6ZV3nT}m#fdl>9)c!EDzi(o%|6Mct Zp|2Kw2EQ!X)DL51#gK`}Vmt$f1 z&mFvX&;>wqxD8n(FvlBzG7K#(9=LAuo-M{&>179dO!C`Um230lu`(2Za7PpE;kG!^ zJO;Jf@@?Ij x?fw`MvwRDBp|V4bNhAF^GIjD302ppT?xKz`Kw% z+A7|NO2O{Zmd=);CXrJdI<(a%I J+~TIX3ETHjZyI3V^LG zlon2kJX;-<7Dw7HH^F9dbO>S+J6yX~xTqp%0yKc5EXu1EVTdc`0-?s^Vd^;_?W=0( z8e~&NswzVSMR;y0=4sPu7S)uxYA|lL13sK@JOpUg8VTyh_@-3-p0~8ks5VMc6bkXL zU;6X}b3<%8qq-M(IqBS%8J4Jt(J)5K NTI00DcDxI)7G}mw_LdA(~h0xFo@draNIwXa>KwKyTw- zOw<7$UuW9|fNx^&6A~#_OSqEYG;ZJ$ZkO8gMpw|U`EDyY6~STF oZpq}|Ak)W=4;5A`ZbCEle)RkIEFpzzk>{|$Zq=k53Rsx@98Tn;oQ#+&CD z#DV1ivC@C;?qwxg?4>b4E49hM*Kw@lw-WjIZ+^5$mz5`JtcdH=F12!EAd;KuSsV%w z-sXj?Iu^rh+H2>6D%L&JR1H&&MsU3!du~(&zk17{+)1eCR=$VtCU>Q4e)9*v$L?iV zN@r(ON=OCAkV4L6Ctt!(r;RS|nD~Vdp$Ta~YDtsW#_zl)2m1g$*}PzKwphHbNg>)B zlhS8*v*M+;{#h2?iDyPlCnF=U1Q!+v=a)8_cU00IthW#`mdoHNhTd0oa1euek}Md$ zyAm*OUHzQ_KSqx1vucqATmdP>REGpRnEYGDWjJg0R zRs<~@hb&Z0RTl67AF`@0=g=hDF%%GIi0ugxA3q{)T4x%v*W93?pw})ryEZ+x?BrNq zS zw#8>dlEG$AFMtlFN2uelTWSw%Ciu$JuX0w^sRbS<(O*|BdqPQkBZAx=pn)q5VdeAJ zWv`66GtAEEpowDWgPosL>oDi{Myf1p+>8$;$jIaaEYOHR=Xy@$IVG+|faZ3zUDoXw zma$h(Wu{8JPA121^}c l8bQ)#ZVuL8DfDffL|R2qEyeus z(FjL 3t*CIWu<>UDdOvw9gf$w ;kk_4_?_=`<}5CF%GwL!U{ zqmt2yf|fKkcP1T0>>l?5Hi&lg1yJC70kDGbMIicIbq;d~ zK+tf+hWQ21`RO-!vk1bQW1Nf}G1dZdn$(2e0Pzn7#6JxeK=f}SqwmD!?P7gEfYb;* z6hJJF0DtwQPt7nq$jA%e%{JV^*D<^Q(}bc`rH}@96!U8`Uhm24#9yJB6o3=B4u5k* zmpanHnM7|7AnY!Hr+!JW{rZNf%Hz%k!0TUR`EMBIKeNWn O8v9`${ykxfqWBqyo$Ny4S`yW8ve{KOp&y+5JE;4l0`D<%jJl-lO$b9U(74T_v z4j0G*_)-6k65vnIBoJN8SoI44VRiuw$GhVl`2qefiob( X69`1cqavX3H2*Bs;iTVE0U=4346SeD1j2L_{hCtEX0roAxty+5A+8<<0PBhhN z+*pV-)7PsSJaDg!=<>MvgtTS(b;MyO=VeI|yd#a~VuF6u=7LuUmeEsp#%}Bmf8{$} z^`$N@0uQ{I0GaPu6jc8sw6-0C&b9YBZl0rCJLL;Ht{V4wRwD0wi9S|IqU95FoE(H1 zD>mBXFc-|qEPD+V+UjU5h5z!{#AYBve|8?fYkTi4uyFw_yu?j69jNajP-*r6^grcY z!x<-x2U(QV=}ZmVMAE-LG~?Pd_Bhx#ot_Y(7X3M{`J=mrf$`)A>+1m7?kuV+m~HDb z39Mpu6-ucslX30TzYfp3JY68??M81{m%dek+)G(y32Cb|CSbW78ejntA>%5aBk&&< zQqHoX+m?Bko)ev=%iDMN9{dY&7A>Atvp)@n3RT?cvYB~BR&WBA(>eYyxM@saAk{5; zLG9-a2?@d}iG`O!T=;8iOHsCChPehxn%&*546H${$N ^m!awFk&$1Ia_w7xW53arrc<_ql78Vv?uoLFfoaEE>{@xA~8buWNLT>#PA zT8=p7sLI5MYT!gchxz9%@Fuah&dadeXlRG$EO`@Z1P94DSDEM+MfNfeAB8O#*gZ}2 zc}J`bit#hRg};KB5?<^pM6D{7q38p5Q-)2(Z`$oR<-AaKu!+riy76_3foS8B*xg4r z@aM(75+;>d(x(%jTg57GC}$BWuSAIl-}Z SSYU 8QK<^r$`omu| z(qW>tQWoXMa_LTC3E_Tkx1)a)WZZu=$3OV}Ie``LBjU-XE6^qV+6%xRa?VhWH#qR$ zgjFB}+%o@SWUS(7t0Oa%MTf=J;y3kX$jocJCHE_jjw+M%*PWdLKPlKUETzRNv&XFg z0J8=5E^%}JDI?tG0(h0!gpCH*>i_%+Rp1c`M!`YaOKg@0U9NK<`DjUXQ}5XmSCU(E zHv(q^WTQ$dWBa;i`%JoaW=UejsxzIF1%XSu<#%>u)B#3myrBTjEJ}fP1d%K~Y*GK- zZaDl}{zlR^Q=3gSxryb0c>|HgsvN@{suu#ns L3b&dlLd0X}I_Wa7~WJ2-lZM;J*1olnfGb`dD0Sh +jH*KZ9leCinB_{_#IKNdAv_?pKSczTw3brmYUnynBgZL2t+e+uo`JSwx5y zYY;pahc%(V^(zGtHjTh(0$kw&2!Ng^fbmB%Vx$l$fCX23>o=$!)Pu{jphnD7T>#dg zM&7nKyn_k9006|<9ng;mfk54$%>sje0ZjacyGfJIRT^(y0N)|F?f}8^;s{pA*kO@Q z>!ee7u^J*rC~u3$`lmvO98^4yI6u4#4P)kGc7#o)n^ZR*O;7x3Z}KbuSa(#I$4wi0 zXnGtt>LXA?|8$<@#Wb5qOe>LMsSXQ~9vlfdH}gtwDl*fg(X7j{H2e`)|NC|3?@+6M zxCwCxoEZeD`)D^@<2LlbO!#1y7kqER1qOu|fbjG>B>6v}BmaeS#ZxsN{c=FTdI6Y? z7$uIYS0?Vb=CCU<(OD9^1=!+u9Yjwm_H5BlW=6niKCRKAVJ|UXzn_1;p5vSEq^Gk- zz1-c){zS18Q}gsKu#TldEAF>3ea&C!XMiapNH>`rBbW-RGLCrOFO|kuUwy+if6aEn z?9u!p;j3OiAC=UpL4rxuLhZb9Nf*#T%VvrdK6P+$p usJ@#kLA6=Wo!j_S+< z;GG}f_~?qUz|U*77O!@sd2LXdgHD-xqsYcH&kwPk?Edmb?6x<%yV7ViW(9U|rU8_x z?f%S|)I~F2o$;^rRG%=8Se?9e++{((57!HBRC+I^oNiCjJ&O_ztG89PPL7{{b{W{N zx0x@phlZJ7AE_5y2uV_lZav1aI@BrOh&=e1khS-Gimg!SRtNUBq1&rI5@PCXn2sK5 z{FUW!y%8!$8HM$yMnz?RoN?H 4PtzstNpI X@Uq#= zu#lTAo8XF)raUc>R4Hbh@FnqHWE#ExFh%JD>8FF8**eW&JgiC~r7={*_HAg(ackSM zKZ&AVmDw& AcH7{KoGCV& z$vA_^pD+_{n2L6Kj?~`A?c$itIaCFX5scwavM+%77l_427^*|7dlx{KaVCcB0=RJj z1Q( qG`R1F!GBrDCdJdhZjJU|7MCqv5V0|Nj56MI-dPk9I#JOFgQp zPBXc`&rq}|q s9GK&oGbqKZ7QoKh8=2AY#+zeU?oXyHG-p?r z&4(nM*c^NxJy0a0;^$?QT~xh&FGG$S1JNE7fSWCjV>(@s4vf-f3*o+VS5hPIYz&Gl ztUjiwjbmL7zICOIc}bG$wT~vTK^E+lKO-gwX )#%LnYkG-?=3$+(JzPo@LG9{SqPk?L zL8_B{5hVZu{ZKFpgPtC+AkbGNGI4?SehYv86*2Z9B`3HECo^G)RtYS)5kEWyH|Sh6 zt^ov#)c7DLy(X4F3qkck!EFde7l>n!*)5hsw8 $-Wi9fFngcOKBeudxna7JDGQIdv@dG z%Pn!5HeGI|IHwiuEN+6B0Adv&+Zk20FdA9WkeE8(z@|}E*%Ye#2z1O76eC80Dm| n>R(fRvJu zQzHw^o}{9%>GE2w;;XL?`uh*WiXA>aSWS$J-Tl5V>-;`Mg8kV5kylG*kUtZ~AZv2H z)-NPUET5vfB2zz5HDn06!SNz1zC);uA}dzeQk+!{@Y4NjnAp{V9=eo>slt69&phS+ zq}0$dw 4c@Yrt&gEF>McK77`?pM~#(fcd)v z|NE1_Lt@~{vLF@=U{x!SJ)-*E3*d1|knc3#1;AF%c@7W7#YaR>jK;&sQF6S{B?{-c z^AyH4mt$-^X7}_&;OBwb9~J~2_CH+KQbqqtO9^z&2M@DIZ2yrQ4oMqIq}QoT(ORqK z-Q8*80v7uk_MBcvsdt~{dtyZWXO!=#{zJuO_;HvDFb&q!o_RzBCT0fJ?#t}ZfyD^3 zG`L0(p{Xb;ZGrhXc4#y7h^Pt#eYQ5)opXCSi%pwLO~M0fvJZdH;{5%|CNl10`ZX@R zkw031e?SuWKO67aG5Hr-Ehsbwr-{E6+0>~tc-+pL`b?)hiope2xCv9GF`&}9x?bnM zk~1M=`8A_pt3Pn@CkwL~@72%6xIq=EirhRqKBgH~y8?`+xkH9z2`V$NUI`jzIfl;N zs1z@vXb3OY{PCdfF>sRnD>5_`u!`_AhX@`S!mmAc^sx`m-)+&V{}`WJu0r)e*|$f} zYZdkWOa7GQK9i9r-ItO(#XB4Kodg42f-0OG3LU>1 #|Y4{e(X4tGt zYot6+)_oa{w>ABFBvQOD_b}GeAyI44XvV~h&nmtgQ-B72yHOR!k0S9E)f@FfKP#Di z-eT*esRzt;V>cF#OMm9r`Cm>u{C;Hr^2kUrs7zADDU_W(XQyD+HU7o2gONm)znN1b z0;&`)oThk(QRJzzWkcR0$+NypK8X%5E$x*$9hK=xk#ib%i*SD#bi#R6FkCW!jqcuZ zQTp~+zUW$t!~L;iqG{fnj3$(el2&^KxLcC2uh)BAyIngh^CY$6v$Ar!^Xxh^_G%pz z@)CDTB>3<434b2w`F1z&GCERO*KH+Ufb>l2ey@~@@l2D4mv4USwtr6tl(yXBy2<+8 zU(*i$WGN063fEI~>)#3|GFM-}OS)=HyPwALxQw$XsoO~_KTJh7DlKD1yiT>Ky|pHo zcQuUb`L~bRcYYa38c$Z}3yvOEd`TS}lljm{a{iC$FQo=@92 ?;?fhJ5rl;pVD z1g8YkI6a{{=VpYIbz %oeN?x8&qa%LZ6^$31^uHk+Aoh32#OtlA@`KF0NT4uhOVDny5 zaJ_f7eQnxlp@{;`uYNn}EAMlSIF%mI)V~orQTLLT634r#Nx>()n30YqHcMSd!m>48 zk;eao2~=&WK^>KSvK#v*x~0FYZMUsAX~i6qZ1pnf<;^GGgPt$|l#uxgpe-|8ma_pv z%384a$>{~$YM!UT%RI{QHjT{k%%`&JZJ%@>GS4o*{zx1TbSwstHcDg7(VVRpKt!S} zeLd!IAv1KAWmKBmyKqYN%UaCzD_>_9a^DB#gGu^E;j|3S@8 8H7sPUCU)< zE3Z!HwV@Y7@=R YqvpH9=v@*d; zEERDxGCVeLB{E-PQq^Wh<=)q%sVo?x*`Ebfv6J^{EwSoKe_`)Y_*m?bcTlO@zNONW z{4{vhOJI$NPF$q@IqmvTo5-lB?WCZ-S-El(7)H=BdMdx}{ybV79IpNOB+pyjz;Byd zq)~sU=UJ2&a`lzy9AsiG#VLtDrLvMK%_nkbrbpV5Gm%73DJaCE@#My95>wWSjhC~| zAzk^#Mb`Xw^6jpl%8Tpljb#!O9egu58J;mT-pmip4t=Q^tRU$o`SHYES;DAjwn|OP zC=twwYQSZf9^Mo2x!!rLVQQ{Z+AFSTpxPxgZW|9f^z$yI$va?6F#5DN`OM2~q|Qal zu0y)M(7MOTgg?b=r!VtRoa40o<>!R>XDcc3BPSVqtJ )s%x zs;F?{MT$kQKugcW%kZ_HqZp`O?x=nKLd$w>n8A&p6LRe=k+tKr6O(V&Y?* z8E-=6FOO<*_TtdI5O+r= {}i`Q$+k5{ThalUqy zq3YK|T KII+uxUtDPOgwE$Zp%?QyDHo}0x~-1FI^g8f6T^uLEGx-#w83_=(o zKU<*dFbkh^EPy=I82IaNiMm!8emz_;4J`lmgSdo^4Vn(Xd$VyJwE=N0_o5^HIZf fs=V8Sj0{Aeu2WX2( zI3v>0t0m>AtZ;q&UVp&QzH%akwtmXS?5ak}_Rk=d<;8^Ck#Y8vis$pJ(I2sdc-nl+ByODh4=i(tIH zrGpd*WfdKky@5d>HG+oC2e(8W?@8^?E6^LK%-)!ADVXM)PdQrVcH @0Fo~M9*uk>l?D>zbi`kLQR{V;Ne-^q!+Izuh&(ud^duGEUOPdvf$ zNF&*D;MwT_UfiY}tIU~>nhSmFBf8lYTk_h5FE0iqb;?73{3xzLoeCztn2z-f%4lNS zhhAHicqXVBXHY ~!iUwdff2|1x#efV4dCC0lDO0@$n9&6- z{F|JAedxbv}&F^5QSxe>?BC5&?sRa@*=P&`5^sx0W?!#f*CE& z3G~5E!SNIS>3E|tVTNE?(>~Y<*yC#m`m#{dDM=C7CD@z_V;xP57#Kw&u#^z28c3g% zTmbtknK*n95kQf&)`$aY6R=OP9|)?46*L^#g>4rAh8c=Ufg*2OT>z`SP%JTy1kuLT z4Kg&hs&K?$@(;R@D9v&KJTL$W9FznCErMvf-+2M_rywv?*ee%6m=-9wfj@{B%oOv& zq;Mg{G~m!*UI0k21}iK$`U1!VD^XDp%+3Gd5|*S2L*Pq8=Rcx;o_d909|#=fhw&av zv{iMU>jqaj-oYb&6=*)I+Wzays3n_zf@qhDn|`#OgZVjC3Q`@v`I$56y=xmycE;)A z-sj2YN2U5cPCiOl;r`dqH34KamPHLGy4SS%^(i(U6ql(+H7Gq7@MVAZ97aqY)Uf^e z=u HF*Z+41{uf&-#wM}@E;uI|yatDH_>e-;_qqca=`E9 G2tK`+ew>>{n!yNG@c@fBR@iz GQ_|dX{S`szl!?#KLxqVF)FMF zR0K|`cL8ujfghN!`#(N7J;C37{nxI;C5t(4KfMfi{E?@{qv)Rolj+63ms|hy)ZoKs z${1(4kw$KquO6lW)!!y7jq!kl3T!*N&+t5`8MCS!H@1DN%9mGLM{J9qt5doi@rDS~ z^ZGlh7MkZf{o~p%vLYB6G+P@}Z-0QhFC;iN+`Gc07hf-ml ;Oq-;~>Vn(4GDzL) LOL8&}%@B(6 |rs=dkHW9^=W2 z3xEYmWB6P*BUTOa5TnuFcolsVP9IiJgCyg#&`5eET3+|M%H!!MgO@%Kg=|#47F7KD z)4JuYMgIQNk)7a2Bc=mhirECr)V*$r9DmgEKkioI_i&F3KyNr|K14c6+vssVH5I=TF zF8~WL4#{JK9D#+?u{Y;4&+tLWu_husu(*4|jh;K)vq)ZZhur#G_QN|g7`za>5jKTu zu64!*j2@nZV&e><=n?E7O40zEC>eV0n|C5Sytzcx_*b(b{7QeGfDnhi^s~M6e=1H~ zy1zxMN!&eGNwxhH+8V>z0!FDxKN{H*p_pWQ5-%1}1{WsJ{rAkP7(7UuyhY7|6Qkj# zm0 AN{^}lR*D4Li~Efkwsm8(~>{j%*@Qfw8z`mjjz3J=UJkWY2A;dr82>#Sl^U$ zbwN!PLH$SQ>y2MdO^$fa(+~Q?Yb&de(M)#R(2#>EJ9o@G^s2cUqiMn6o9q6*!iq{N z7#v@}P2zppmph~E!w1T^Vhz=F471Re>HJb@VWe|_Xml`6Y79>LmHSLR&UKJfY+jA} zBqu>BCtDw4YB02nne>igVo32-_Ur$AB15TCKEqut8ypRa`dYmMDzFU=%Wu9{Xgtk- z6$arW9PwLB5Du{zVU;INN$ccmU5Pg%f9zB%TX6iCTR)xAO+K?AlTcO==De8s$`Nap z6+<{=$)ebt>7M8XSid_h*eHrp%Kia)yUM8dBJ=8+-l$Wi$GKB2L?N`8`JrUY)Oftp zU#P&}&`p^?(@pfs9Idq4LT+{WFb5{Nz)swA`=+CS{DkQFbS@R2C!rT`1$KkS=_HEK z&s;vBy}J8zi=vC?@y5)3p29j83#rPYkE< B~wCiz TusK}2^Yc?t6=&A9Mu8C7 zGII~1^>&^3ysaE=-(jt}I3@S=Ay<2_v@{d1qe-`AI44ynwTPs3N<~necDnK?rBLk4 z(uYjG*)G-s3nu@dzYxYB+$tUtQm)h)9f~6y5S_!vBqDL+IwJ=Ki{%% FN|B(ZewO*TOG$=^qF;b9JZRi*Y?p7)^0n>pV{aNpUEeJ`m1>8duD~ZJG=p{PLNuHNK3n@zg+R$6d zRnK5v;aKPM>_v&Wh0t~ugEpq24LVI0GSw)u^xJ(C)_QekFS=+*Tr(V;y!GX}v+ z`-e-@Qh!+&*}xdIa=OH~?Ijh Fl-f4-5Jd%z6Wm{O0)JJ zS{ahw=;AZhNa#tZVBD*Wnbm}U`s>Dv_&v-5r@9*1v~~R>XqG4}EZ!r!L<2#itI-R= zPe628y+@4KTExl7Hp-nLzvg|mOA*c4U~0^9uyTORWvp1g6+JDxLg@H_%H*1;N)U7I zDf`q}O#j>Vfo5%w5@F5PGhMk!y=$<>H;V_Tor>F4#8HR+w3z4DO+v>i$m%rt6eiqF zU|VlH>eAmQ2xfD?rgW7!n>K%puJ#B`WTNG%g4i%L`6l-Ej?|~e8t*G}n~C_Fyh2_} zL}u_abiYr#6OnGzExRf+6l6PjXV0;shi>tC#@Znj31+snz2Db?HK}0T;M660TkKx^ zCGJHd+H&XM=h|QJpSx5**<7`Q9}JFo&`BMSfuS`?7s2E52uka*vFz` zpSC`d?_{g1AFFK;K3HL6;~*O;CH$e7_o(Ghz|`N5qW)B`pI{Zx5fI8o7EE}j_~pp+ z==@#}Cdu!ydOFMbSkms|MYCjo>ctXWRL{0ZqnSk)<7mNF*6J@TZ0s;ywausZsOig$ zOfN3ajbmn{e@ywA< EQpq2S9dd1}Uj`sQIL-pXAycLxs4G}=;nqeBl&lxmws zzOt291mEl8`KD*l7NcPQt#vrTIBM{1-GMwqO~ti3=o48gILQ1`V kWm>t7Gl9S1=;zbyx_12V-r0R&A5~^ jrb)P>UwSD_{n qcwjNF-ql@+L5GxT_ju_NSqtOSN@Tn|c+cE=PV39GqHn@V~{4#xWAUB9?^6RfklV za*hhVi_~MNZtg(_kCr>Q*K+zwi8>gbD8I~Ljgs1&@H5hMjZJveGF`#D``K6Q*rj50 zSp8VK+w8jeP(2TQAlB-`%zBD$?L-6md39gx$4}NZm)L_8d)d6dJSA88+OAFM11XB) zp$O}c_Pq2VT|px;SVkwmhswJrnd_^<&(YKsr`6_u!iW~@9^RkpD(RdCnKb(ApX*2I z@4OCo^&hO}7R%|n=BOUpdc^^}sJ=Zh&bnxVKb7z2?9JFSV{uKhsXpcjk{n*ow6BWS zJntlMl=UY^Cs1ukj1+UJrYmS`<@a!Piyfs6l{OxEIVYD7>C1fin!$5!oxm${M_$Lx zBW+{9!zKMZn%8OTptjtqixxRuT{x> *PX086-E}$>+kOAk+q= z#u9I@x}+J~ Adb 37DrU;8=T=+H^bF&+`J-(W_HV6J{nf+? zB|owbSd5MQFL}H?TnLNb)JeQu#S?$kHNK!V_CwHTHx;1*otZLw>c*Ab@1%!hh0rSw zFO40HZGu*64Bb=tybSLJPYcVL^Nb6Ll*UJNr|L;}Fh))jQWBEyVK!*(bXynlxpdpP zs EUbfV@DWeqpj1Abs`70du0)E_?b^}MNy$5&K`s92QCh}<5<~H5Zw^KQWa`Z zqvyS2HH+y|k4C?Wl7-vA2)nCduMQ?CM>0R`R$jW=XTc(OYD(u2)c$~4 }$ND!KVwRg}JadCIxdw@#SqM#Ywv9S47! zH#AhnAT8)%iyEW5qJ|Z={BWiGb)VG9blKq_Q0qXSQnG;XTUV(=``VVaPa1qw^mrCf zM(){6PFHpFB)HvrlHIy{m7BXhCJY~I>l+L=e_c?e2>)4TtL~xqsJotf b zu6A85ZMD5F-`=`9hjoAY(PC*b)KI4;Ca;jpA@vb>s~s2UB)jI2c*6C25j{8Vzg2l{ z^h$bFbk944DZ@EcJyC Y}vCz`d`yEyHA2G^} IHn~{IL&s|NXEmKX7StV# z2?EbcXZ;QMOxB&=9FF45lGo ?U zvUd!POME4I_MQ!Sra07`@p&94ty}bmj$ZkFf&V}OUb;$4X9?i5XN++8U17bsh?pXq zAAb#hKpYyC@AjoBb$Qm*&gIL1F_k_%V2L$b+$v*{ct@z8sw}OC63zVi@+7zI@?K@8 zl;32A9( 3S0$w~?&D^yV@J#vBLy#O&k)J%1lO wV!;{{$q6p=+}ug6ilWbDzN~KZn`8|>tt~jx(&0lBy^up`dF1A~ zV&5ZHM+H;WBM%|2eU>~;WmBZYIKlYnbC2x6f;T_!s0h`Y-Jy<%kNr3 BHCG(c6YKpr3c2V}V=lzN!ShJ$ywr!qfA7)Q+ z{b(zWo#IB$s}miRzurH(ZpS}Oz>^tY=`y_h^2CW{`I-wRl&dFMk*QF5m&3tg#<<3O za&laD?pm7V7((ez3s*NKrH51ce6ek#!_=$~*WR4su9ZcLx!_>Eot2AUj$eU7`beUI z;Iv7(2JL#}+5_^dLPGLvm2N2uKTEq$^d1ctJ9~5APO)p>I@n#c8t&Tv!tbr9_pR#q z*#ZYwbp_W>`kA+xcWT0F# {>Qye>-c=cwq4Iv1j?lu11r1qi?&253Je1R!CobqGmlgfaY(vIsu0d~l zvmAptFG?u{?rcYx|3>6E%~*BSM2+xg pkTka7MN(*w#1jiZL*7}}* z?tdt7)%-({jrUAkpJjLUQQZ?^@ Oyj=Br{CWBkWG66 zx{w*33VLgsg)MqkJ~UVt8>&<@STCNnZqtfmDyu2;Rn8Ale6z!EOnS{+qDP+Gr4VqT zsl3|~NRE5iU0RXQ{h=_d2 @u{9>OLL#|2LoJEa=P)i99EyGAA3}naT5?ixE9hID}G++ zk(c!{G3hko)^(Gdt%;xhvrjC(qb;6Bq4uS|FX(f|Xw+Noa$&)xnyQkXG#aBT(y;OM zbhbJ7&RArQ>0#=jDFjE~?Mfm;BD1-W;^OI>QMp(ECUyjnR3RCpXxWdVMK |vC>$H zu5Nz55&MbRVn6MOjZJB+UzBhMr|yun>Vni9Ytd1~N82$k`taLHd0n@rH!tg1&Rli= zbzeB1{Z$huTb}QPO-b8=pzhwRvBw=1!Sa>3NKP*v<;l~$-e+?Hitt8vG=sGN<*zQi z#tB^A2Knl0Dz$cs=xEV~u@nBGmBvrMP=+IL;rSx34&2?kK#h?KyRV{!qaOQg-!9#a zY0 O8gd7mwMq4LoZHGD629i2v_<%*MKfBZ{n-K7J>#`yW*zfSMnsQp8F@6(^B_x^}g z*fL;zPh(eBo`o>P*T+6;)}tklRi4M&)MMy&)p}cJ>B!*;A4Z2BO{2Vm;OF!U_0+H= zy~o-0?BOIP6QRP0?=PFSJ&T7Wd|OSQ%Z~1IZWSDzIWL#{`<$?{3QOlH&1CP&o =xUXX~ZRg($(jtW)xg6Y247Nqx0fUAAY;*F>+4 z9{Lx1og1FKP`vZ?j_Ihi+s0h5ehzYVO8|)ot-Pb2ofTdYB|=bNs&uz03=cO9;q{~N zmv4XPtwgWW(ZBHN!@0 ND#Rv B_qE6xmeVm`xps1gSuBT(_!axiCDumc2%p5_fxRf&)_HCT zq(!;2$C1BkQ|6Ml9(uTtr&ZN&^)nnSf_fc@w`2=pI@q6}H!TT=Mhrxc#^Fwi>BZ+G zH 7xCz;c5~x?ro-0Uf%HT)Q(7}PNa|j7}M0AtjhU&IKA6qaTJbf=%&(M zkq5(}RH9Z(>kIfw*A}T<^5Ihpw>F82z#bl5-wNwZKB!5C)78xKqHE{2{w33u&pNY; zTdDGH{hogY19tqLMt=XnOo~RTR*uH1g7YS$srLt|w^&^t%{SR5<@3sS$~kPfQeOF4 z{i;ASnoqVYF?@NnNh4K>NLVO4P(QW4jt?vegHXYroPWY`z)ghhN@o$4wnA5lk}Q`< z>4)qj?EFK?a6gKtVTjj4KVOg1`sG=yBx#_EaxhK8OX%bxxJ{j}Qt7#GhKz-!cm9L< z=rCC!jLR4Pn)wF*&BwJ2RLf#k2}-CRp?5a7zK^FTFH7(uqrY_w>qKX4_R0(&h-HTr zh9t>wKKDyTc^x59{bAThDr9=OG3AF}3L{SmCRF@PT?aD1E^jDnwfhUU$uc6pc@-ys zcUW6STs>G9Xg(2MM;UG3s|;bA@jFF)m(41#FCXEvZ2VRmvNG#_RY8QGg 6GYV%TaS&vB;V~jdD| &@p`qaZG!-#Q67hHVPTzZKhJ16BrKES zK!9e0iB zDRAY)%75_6!O2~9*&d3B?){(YOo<$4mvs~4 zk0a8j?3A(6lV-i3Uzy~=d58DxH!`+ZZax_wb1+DlF)?=0*9w2BI=f_0`g(h3qV5?c zYP?0voOGx>lB_i)s={ueSJ4qj7|k8djn;L28(X^kFgOpM)TiTpgF4M~-rnl-$~6b1 zeMefMydLJ-!*!l2+Qa2?y&`9yF0u6D)#tjUlZ FX= zVaJc1Ce_Sm<6Mx+{@3Bs!yX#UJ#K!Ykqt$ad2TO0=&XFxwWEB*EB;Oh&nl)Jo_klY z-%Tanb?HOy(tJh=PkC+b_W#G+dxkaHZEK^kVnJg=I#B@;5Kxe&kcbTd5drB!R0IS> zKtx(dlrAC&2uO_xNGBpqYJ|{Rr1#!?LQUtJzH6=bU2E;V&pG?M_BrSK;iX)d=Xo-p zImWohnD@BH$jw1|Omy52bD@-gmVvlLu3O1TxU$8#f;`*U?{*{RxL}p@)?$7;Ixn!N z+$N?ec`w=$!lvwG=FEi^(k1byI|KhLPgRjaf>FZYxLeiJqt v+Ffl3`qu zP1J!ygGt@8Q;!8^oL&lA 9B!#WX zU~8*CWwJjw)kAq`h!A}?!Hqrac3ryVGoU1gPF0V%eB}Pz&0FG6BEr^HiHdW9}~6 zWp}kiG#f-392+1Wd@C3!a&GqLB!}P!)wEQFTgea3<|NE5R?2g}^i%|9FiJM?+sz|V zbI=-f{cFxq`|+O@zL@^jlaDqC&cY$;jJg2Sr7ClKq&T_})5v0*>qdy Kp+Dt(|EC1*8b7^ zufU(PO(zg1J(bxt&);#IQ=gp^Z;?L$oW9X$Ue16GCt9HU327w2-tJg~bJG<=i{jg% z(P7fU`s{DwTNMx&38K^9{f&7cX|3@}rm{#Vx1xury1GPuxNXGOlqUsUPjg~^ts;+7 ztn_~=ApjN4xXw?=`=`5RT3Jtl>1FwZLn?NsWePHH$ZH&lyl?gD!-vsl!$e6@EGBAN z0fm<0uI6%|o5;|Jj8H7#R_Tm7la{hvd-tf#MeV$IC`QJGV)JU`)1C@(S9klp*2X?O zkv|GAZ{++yUB6IpS)%l+ @^7S;wcXsyxJSf3Rv~R42%B zE5=zTV83qI-4pDEZ7qTmFIu_kEGZa!UkAIwBsK-nmgHD%Z&XWPrnm?3_R#Dr{-^Li zsr)+?XS}+(C7)Z~e)LwOK`4kr`)Y$w0z{xdJ1A9{E3@!yyMa{;f(v6uWOG#|wxM|| zIo~?pZ{)4I{mr-hlaoAy?I5eihAHD;CR=O1gp@`%V(klACaN>I(6I@b#`~WA0&6QB zEa*l$**>J=tn0-1t1M?M55;Rn`_u8MTXoMah_bjQU}DhE4J+X-=#Gddy?l`&xs|%w zrbvVNElgKPWcA~W@st*Q1I54%cBld6I$fQiPEiqhOKRN^cXu zyG#* 1@ky!OPv*vw zByoM&y$cmO>kAP74C@j_zhtAJAksIjCUve(S5sA|-FQedCwbT*lu`BhHdiRN?EdJ^ z^12QS)HRY*#GM-tX6h&Q53OAvn$pLYTFiMa5`$VPnwCd=&kJ>3Hfy_^0@rIW7}i0x zcr+AEcYMFAY=Ah7^K>N3!H%GA|9ZauvSZ5o25BR0udDI&BAgy7{XRKVpmIos8li@} zYiQ%pD8MO}mZ6w_KU2zgwWA=hacCVcZ@FnFSnDTUaEGQaxEYjwPN6*s7AWa#@3xj= z }6-7-sK#evi^jy zF#4B!W1D?!L^k)JHU8+jghsU$;gD6Pk*QMnO xvMXgD#2>; zjDkAAEwIMf$k$DLo8v7mAcjbFF3mg@mOQ$)=4(lrq_V^DQNU>}BO`+J&x3T5fjN z481rIByeyo+GOZ$ZBK%hztXi#YxH8ThT9HiSLLRTdhTLQ-8Y>7L Q%>Bk4=*4_Y(q(+V;8;=6HpCJez&XN$J|-lZ k$G8b&}vp{(T%RGN%A1bn@N*R!%Qei)s8vzhlgJ-JXkhCt6h29PfYN= z@2sOWRi>Aa)&8CE%O&+NzL~xaiE7zUE<4HFTbE#h|GBlQajY&K7k1^J7 ff&9j&UQIxcD)b1&PAJ?_Q&8o{j#9()|~B3 DumMcTMw!MF7ucI7l%kFxlT%rHlj zs~2MFUOo-bu-E3x;#&vLK0@x@hP+x|+=P!(uF#E8@|4s`YlE#_gc!yr ho8{PP`V-C=ZXV;ou4c|NC%m|1DLe(}}2EsH)@(B?eFyXydC zvS7u}jI(P)u}sr8rLUh7@<^^f{F=p)!9C)9OZy*&U4C-J5XcdV4^+-jnq@hB;s(>4 zeV1H^m&Np-i<#P`hOW8!AGDw}uGV#=a*f+9cNQ!OWpn>9y`>X!t80C5ra)P$CP9Sq zE^&5Pye%4aB{6gU-cZ%>GvoNhNBT17Aa1HlFj$sfL+0U*Wn1-2HjmGy>%9Az&>&tk z`}Skfo)(cOs66t!&6!{M+HikTWQ(jv+ebV7xdZ*Rp5}LXbeCV@uCE iU;S &R1Q0%28E#DCHJY7U$1%E9%Se$(r_qsCStU z!#MJ92{9w3WJ8IO$%j&>6hG742esa{SS!gn={P^ybf!ERwAap7PA=9BDO k}D}C_PKZOE31`Nq)?I!RWz`u0>C0 L*`)95Nhke`jy+4mo$=GvqhVNbET+_aEHm|FZ(g z@4b+=!kZPEtFLq5o5Cqtpyt^ug83}o8($a_%-BEqbXd^{rTyjW5w-K5D&AoFK5Rp> zcC}Gw<#! ftWHj^)QED >lc)1P9<{6UBa8C-UAVn6BnjU5_Bm=t@v`8P;R!;G`zSN9Z>gI>*WHM*WJNkIBX$SdN8)nnSy3hLtuE0*=uifpOWK}1w{ zbDm9cj+ktcI&|w|#^Qu%>Jm@zX8*-f>$B9C1?Bn`7ntn-8<2(_cY+c~s1o4mLg{4) zwOAz^wRl{vN=&|3>-*`P&`t-129B;bm7UEOFOfcGx5$0L6$})9b+!?^G^F+{g7}(s z0hFYVRGf}p$L!BfIQeVjoK8P-y|{-JbLYZ+*z(uQA2bYB7|LVQrXTwYrM9|IJtgIW zJzXOWf)nK8u4h`MT0 nI zI+qRV#wbM`k@sEiOuy4LwX<5g+dxY-?~dFX@4R%OUEk{MJU&-zN%*V!JRaJ;DQow| z#Pn`kS9w+Ru8H|C|?$kN1%^MHx|46S88$&JsEzv}g>|E3)~fN3!~IcT_Uv6$EXK zE8zM%!Hg)9KIR$B_9F+S%46};-2H_US`Eh(t073@L^6stCOu)KqEY{!u)f%8<1?;F zHAid{5M8N(?F8agGg0A(uL;jf%Mx3adK7CNZoTSho^YfRkB!ZD5_B91mGUC(4M?La zrrQu_&!M{|4uprPT&)JNZ72p_cg{`N`mmPDFM}$9&;5TMd+(ZD=@;3o?PMIs* ~FJt5J-Nw}stWsr8*eMj$MHXCN@es53( z72`5Ir}9$0iM9qeOEKz>QdWIs*`u~YEkB~y@6zj+lRZ;>dgHYQVQY_87YwpW5snB& z=k<`Jr_{l^01`!`${#OPiJ~zo$gu;8E$HLOYPb+C#K^zfIILg%w(vl>W%+Z)G77a; z_Hx0){%OZj$8xWGBlJ*RIl85);=)5Gfd_IgB6zUP6>JOLh%2~Do_9^F*KPX~L|-)F zwJ5NtPc&v^{B3=&bj?)G48ZOPhkudQdC=yfqJQrmTPcd4iG!bENUvUc|6HFe_eS7| z@lYd*a-0=?=fze2h$6|odxr~1sOvwuUpc#Xa(>UCrGV1y35=fA@JzN>tN55_AH8jD zWRmy`wu^$lL*DYq$8$d=Ke%a?X_8$n>X7*4o+|-zyZ+SrSv)pRsUk=3UAlKx$4CD7 zLMqJSTE63nF2lu^#xdDHt52)gezu9>SdxfJZaMY_R ZwBu zKX+rF7i^$JtGW~t%Eq|UuAO}0QEqQ56=jebJu|K7<8VYXvnZFuRwEURIEt1Urm`fu zd;0{Jy)_G)UwxoGl39aoe#-OcSwE~sP;`M3qV_{QvJ$>y;0b|E^^`?3c*Vuxq>iHN zZ*ZuIKOYM?sKWzeskK@G$o tXF0ZLHvh`uVB}+lN!`T2sI1E1R#FU(TbiSQn8id?lkYXXo~p2PwGnsr?YY zHR6#WcRyHRs-$O>&$LQb!7Q04op{$}YPKso$;5fcz7O^&a#|e5nG!RSyQoL5`}z+8 z=MF&sIcn&7onVX$EVvP!wf;^KXJvT*lApq_ygj>L-(}-IQ}RvWa P zh<}+<)Av~Y8Ws%~vt^L%PAjDfq0sV|v)^u6({mZ <23WzL_~RT TC=nuj$cli$P*BKQLc2XXbU@wCoc2w`=gk)V?Qv}W%M&yQ ag)KO?+b3{-#F!Nb1XZNbqU!P7tNJW8(3-u}lV4%Hb(@aD zD^^V(j6B@FxJ7m0M>@EVUFFQzbTJ|gr|vBnW9l#5#TVMOJup!>4~6FPF;(3O$9 ( zNaP)JdTKUWu?EerU3D-0lSSTC(D87ef;Jl}nD-)r(VhFgxU46_Bil95CQD0NJiu_J z-TUX0h;kUMU>oxC-mbJikIVK+R*ceq?RgLabRU$TLh6AGgdmWC_`m#7!9F+{Qx!!f zGT-%pF0a!ILt`OS{zr&~NyK{c3-fKrkPXI{r8*^xF@Pn23eZ2AN^-~WT$H@Z9;`hH z`_@@mU_wzER5pamgWIK*-#bo7GtNGOTD?o<#;pheYLB{e2B52*Er;Igz(2F^7rCp= z1+=XC@3HUs(P8(1>Eh-QGK0KmKwt@E;<^?YDY6tN6VjE5ICFE*9khpaOCL|yW}wLu zdB!%F qn@rcC1QQ z>>qkCjB+4p4oy{uw@3wK8LJ7uXs#my-P>?eCAEDW_2heA-|x?S0RuW@eiofKmD`3y zqpjw@s%B$Ev59+CL6@QcXe@O9wIg)l>fe9dL3O6z!8Ev%Q<**?tk(rhwMy*jY0!h> zzhU)r8oLsqtR;<0RoI*7OQRXoIBE$@@rlJN-JmYKOy#$jLWi1_4BU^NvgEGimE5uf zg%k|-vD~ZY<;vpJqCnL3R)xe8-Ke}TtZK6mhL#&@eKC|2?dKBMEOnCls1IEZgicNu zkgS5i%h;JIy)L`Kkg28O4{DwlzH8(8Na8UQ5kbv#8(b%G7m7$?N?k&%iWtyPv3lNM zKYX%+ZgpJh+c<|+bYm&~m #{DS>2uq zmIUVD{sly%NV~&KjqVf;Ydz`SGw$E%_aEPm!wCOvAle}{fu(bT3$0)8DJ8&GFaKS+ za1sbMlyrhu%?N3 WkK1@Jx5VqJ*XMX9^r zsYVz#tkfs6EDxt2htuyUiinEl7K#f;5{ >_Ax&H9^|VDKFbJzI-}uV @Whns#s &LAl^hXs9aVCXwd74Qa$OKV$xSS_fFZ)#5T7((?I6#YG-UkIod~5Mr^y;3M$N zUvZ^ytHO&TvYTbXHQ0qawg~e@CRVC{8ZWyIDdjaDwu<^RHPW{%9vibYXIJrzoIyX{ zas~SavudpLk4>WfpKlV}A6w)vYBJiC1^C&krVMK^>Prwb^h#iK9IO}t4)`&q5lJzX zKBcAGyfD|glN(OOO{{PrCbMcVg9xVlt~G%(ASH^>ieiQN^~|IOdhQ}2H v630 z+1LTReKlF?gsUykqzNd19dz|-%P!!T99zl}eCTSSP*2oTR+&}V?O^NY5TOXNWR#!O zL4#d@%{1TsE{|3EYB|IfoUELpgpY3#^2ob1$$rtPwe4Cb^z$kOWw?S5Ps3b9e(@|5 zrrN7@HUstHr9z`|w_M)PHZEs~o1DdJAt+8i#arP*DADQLoC-fBsGy-loo06E)%bZR z(-UmB6hXXL55EROV@(*hG_mmkcu}psB-U+JFCbjRgIJftO;yN%HQ5wrAiu!rKxfA( zb=KViRXO};a9xEmPROuS&}=}i?%g)zzwn*?z;`+sRK>s;LM+j?GTBXB9>&(@ 5 z!ZL_G^L441H@8%wzV&k8RNZ5Aif{Vc0_vG5JKU3@9Mf|i#-aYNqJ}*|>*v30<(^5* zUq;~%Huhh8@&8N9@n>9l$I298nT#+y{tMH1VGLWukm{<3FLqYITrr#2HfQM3xgGD9 zqwSN$iHF-cR=+HSWnFzJlOM;&gBX{@bi!Dmz_GN82~0W_PnBXEpTPwPoB;DTIO}*; zywM|@D~p=fJYe}$GlTf4VpF>GfK$+fh(H0YLNA!rQK?h=VwHY7-FT~@Yso{LSx0;< z>T%#Zpm_EE%vlp$SfPku8NN!6psjrqMmw=V0l6Zws>+?X`gT1|k$j(uPVLvDByODk zhgtpi9A_K!`Z`V=_?1vI5mq^I8=__fmtqd5Qjn7>$SRSEsTSCsAKe=s{5)+FYAef+ z3*ILSopO2jITsT5WbL(et)%8nXW7JChJm>${FCkUp%70>DgC7D$e4kbq>@9zncN2)L6^xC}meV5iQYrb Zh(% A$EM7H^p&kQKVqp-Vf=K64#4vE`ZvoOP$DG3^{!u^vqf`UAx7P 6KT-G2*8oFfS8pkSQ1y6Y8^Q{3-*Po)sT%%GijwcsHe|~f zJBA6RY9qVqW)iQ$Tv@Hi{GLka4hAQw$EVC_!v5QQ3CiW5lRY!mO|q&-x`feI-S{Fa z)yutS?15=z@vwDVPtDjlr)&Co(_ZCYB3sp#OJPP)mafKJhPvrjWYHm|tI8rQUXjh` zKx`ZYzmGs&f^`uIV)7As0BD}%1t|q^O%41b0}4_z3>cM-A*Z!C1avcjjMq6385xg5 zDE@=0+mNdxRAmI~Hlol0Nu(ga5P)aczEV~of~CCz6tZeyWay6&)*Jk)JBINA^K2B4 zC1ECEq&`HR3-bhqku Z`>3=G4`b5dHed-nz}$ZYAYx z3T^^kkfQZjoB-zGG0(0X|GCJOKJx6 ATEG737>!ZNOaY%OrF5{WelMIcD|wViJu-$N(DFu|w6+=( zQloo1;&w}Wt|9AV`# e)(aJz zG|y>%t+lmJvC2Ts+D|v6`}ErSvTMSrN$r|n!H&KL){?f)A!?|Sis9SKV3&qa
hI}8>^xNbbW-* zS|C|q7eZ<=A$@ls(olE+Ujoywzb_m6= zOePRi*kgPRO>mtrA#UMOy6)H5@HXL0yYgeECQ9F710n=7Vc;)QT%}aWdt<-4o~Z;I zIbSw*Zo*4^uA8%$WAfB_6|#3+sX3oHLe>sCGBb8qUUE#VIPz>zhPhQk!(GvaDoOP% z?a~*v&iVSZ;o$**@ta1m%#*O)m@yRqTk1rOu+d;}lal5( ztJn|3 zmLHK+Stm|n886;l0CS>TV_CjkKzU44i**Ar=JuPw_=PYU!Z;^^piG^7#7ZX1GY@8h zEF}iZ2*6aWQbd3y`B%ay_lz;*#`6TG+PFQ9(*Wp&$s(x7JYYfCea;#So-%hRnWOa5 zred9g9 ICCn Z4RyeS3E#7!9OrxbHr8q}Gog-27Sqy{NhW|H{$6Ucw z!^aUcaFW_E2SA`|KM?HrZy7#f>^sJCg?0{j164ej&v)WqFQ~%I^I~Btw;@1j?Fp-8 zWL+EFB?JQol7LSHcvfT90a^_QyKmB)ZncL2bld(IZQ|F!jK$9;Z9}HOz$Kt03qJL) z&W>%f^&k}-*sxTISoS5b?XkX~d3=Kyav4#pX)~kKHN0%N`212P@l4;cVSe(_=iGJ8 zl`6Bm@zpzlM%;hm4B;*mJkA;}N=Y3j9(!tlvFRvKemp |;%pmd%eZ(E_a`sxjGoUlktr4sl`q7xyst#2T; zPQ}$Rj9Dj-cVj+kC65sUQ=2y6>D_2?lq6|7@oeZg>@ECc`$}S6w5>;$Nu`+8SM9;0 zlt6|~-E1$MlPJR@e$=~CFP$wv>Fx`ywzPb(8opB`E!I1z7lQOTAFTSv#a7&g$XooZ zDH6yac7l6fXK_YVB53D-OW;*`fdvlMwuH@V9mo7C!L#;*=veR1kdpGktQ(?^g3Ijz z2E+HI8PS<=7APh S*DgHrOVmQgM_G!igY d?A{jX{Wu(|#~!um56-Hxe E=FMt$^R3AR%&UR26lzdND}3mAINEY)dj^@z6tau4RIDeu)lMU$K5T}x7z z+ 0AjD^Y0Nt5++btr9aqoiH zQ>o2#$s?3Z%m6-J7;_!_?0+-(o!fOyYD4XJmEhOkLDP<_|FczsHf*2m7?J=z!q`Ll znu}v7kSbCzP8 N {Pr0yOrO&Ll69{YgCXdUwy0z|vf-y8mYPa`<`bt5wO0C!&*4l0%S zc|&{Z7OWWa0aF!50h{SYNF|1H>bC*C#QgLByg-ZOWqH;uBLVzLu3*Qs36hYkeIP z!4?F-rc!anke+Nqn&ByaK=#v-9tAL$sFXKs{WgOWlp%nj$g9%o;(+CwA{pF>HE8-H z{LCPFily2ND}>hIeKC1^OUE>M?*bD0(eWKPp`zn0lF@cMlO4aOzrT18HS@|vfOCp^ zcNw}70&w2v5 `1FH%c1C<~Ro1I|qlE-s-0rP2_fr7gz}{ zPTla5 Cf;F+#Ur>_Ddz1PoJCu_A!^(L`2OGa1~8&D>DPlp9|dBlxAvIA59#+|sP5Yp>0 zA)FedB;`VqskP_t?AC9`SZ3EXOJ;E$6)q;4#{Pf}KHK4wC0(YT&gPG8>d2jR#U!Zm z`$cNK>#Bj9Y=jsx@KV=@Stn33-eV$f3J`mG8`}-9V>}!mH@~PF)-68~-kGBwKJwh7 zC(%D~!~OX|JFWbN)wIbUiesmkU81tH8^~89f~%GlS6NoVg*#FzA&e}y-!-Xr2N%Xi z1sIL41queLi66HicpZR!8Ff9IRNBvbY#_ESPT1MI(Giqet*gU|qgeITslXfTVS+L{ zv!J_=+& HSWoX)|LYJ+{A+z?Scq@E(z8JV8VV?En3+cx!qRXfZ>>ZxOj*Tx|H z*MXPiTyS9~CLvY$$cC-IDyq^?o&F*f#BRaMHNi#AoCQZT2+rK%ko|g1j t}GG76N!_@ -P&Gso>-l;?ga8-&XdeX@1SOPAk_g 7+tzd zycNbeCwny6MCo&X{k{D9M?3V$wa`W*gXvb4|MdAo*Ia6uaDp0zFrh4?o?DccSI{GM z#%p+3IQU$s#X}Cgs|O&Xkc-H6hP-|X3VTRDD=vBL cSYTWg@7h-5_&&ybOInbIKGKxJdf||VwF0|GudrH{Fujr47lq3Z|r}= z17Lrxehj%plJGxo86n4GH|0n@0Tow}lqLfbEBdReT*E2)$x!p%4s~u9U>BbRLDHbw zK}GHFX3^15V9=O@x=r6UQgpfHC2hi<-#t2KC0-;s#F@T0!jo;6Rl!DSZ*3h`IOR`> zYO6b8!*TQmI|SkYS;sl~Ats%D04M^4sP<6GS!;dgG`)1~qO-Af^8;&EQx@P5Fn*sq zEXB=eg%L{O}w52A*$Ws6? z5ixKYvsg(xiy~fWc8F}{pYy&jDaAG8V659-$6~wGvDfp-PmT^AyVCP(Izm`I&WgW- z$xSy1P$>Dqein9IrC&QO1}J=6P?WKuzicjJd=WMyJ}e$`ZCGwi`drZmn^$pDOOj*G zGX1%uV_p$6-t?lno%vyY@E{viXB;u(N=664G1y3zTSKiTU2 $?l6EJA*huXlAG=)=!(97`AD<`}WjVo{eBffBayYMs{53Pc zPyC6eN(g{dDc6etkYkGL1K{n?JU}u%u&l7wGB$b_f-Kd{G{8+AUqAI%dFcS;*E^;T zrsUj?^*>Km_IMCC%$f6$^ncU~|G7lPc8J`;x?LBHSv13g9SSvDVaL=jfX(_hNxyIy zYeAPGwTck?4S3hj?HWe&q5F`>SzNACnk30LiLPej1a+g6+)@(H1uUV0w_Q%e-(Q{i z5_}Q4KbGPdX=T_qk=dD+a>Z1H_i0!=**?{&n_wkN%8xiFY%vU46JKIUBn)6!ub-z# z0crFcklLLsTcP+G$upP$X1xKc`9XdgVt5hJ1S{T#e8LbaSuz0Y T9|yVh#mGQcnY@v 8~BW!?i-UGMT#e4`}?0NJI%fe)2fH?j&`IlVMXb5D8*XRa`dwn44;_b_qrzfT! z%`5)bAGL}@*?`K#gj8_*+;P`Z&9-dbn?~Kx#a@e`3m?BOv7UrrH*AB<(-)9hgfa>5 zu?X!)!lF+7j22~_t+9A<<@;u_i9ypF5v6X7zG>(6)cu3M2Lgt*joQ<>&Lh^{4Sc~f zt_s&|LwdkE&7EKPi7D6p4GV}}?ksJ@wZHf#yH-~=qrgX3Cba1LVay%2v~X|4 `NyADThA0BtC!HBu$g<76)>hDWVM?8XSXxilaJnkrLo$KGkUth}+ zs_Y4tHuQ%%4w%IXuW}>9u?n7a`f1c1isKmW^^vE;f|NI7<>Ek}WLhq&B D#@odJ-7nP71 O#uk(6d>Go 0&y%!Ltk(ws)DYtFQmIY!_sYRo-$)2OxX5 zBlm$BG8$S(hQQhr+?9tmLao;qmW=m3mzb`Ea;rY9V?TTt{>V!~cH RtSs!7g}V> zX*ADGoZ_OZ?HxC%?d=BYA1gn-nc z_Y!8-4%zg3X1(B|0R?} ^#qeT4fl+b-Y6Uw7oy`Yd zQ$xGH7P|I8mDE9H`PUTCjaH}q{R=X2hZbn|0a@mO8u4PQyuJGO)Mcv2Ixp-#CY_{k zY64K|^aT_ej}wU2shoC;UHOu6LyGCy9mVVR|851iRxobhpG<2L|NpLSJp%iWjY0%P zHKG?NnH^XGTNHTu0KISwu^&%4fuMro%zxVt=y`9hQc~?a;pujx7|2bxmmF3 8LE82+5qM}eTIE}oA(O};?jrlYU18XcXPeaYXUUPxY~{m- zTO;S2j+F%&`m(F-5(x_PTv%t{dmzEoH`OP3Am~_N#in|H|48VBe#Sy)gxd3==D C=2h3x3z&Kp&-tu;t=Z$9}{nGEC!lW83ZwjbpBm9bvYC z;p&d|C9+LZT#x6}?>4A|=v&V$MV~<-$QS}YREvZSMj^6?ZsJrgeX#D2 )GuG-?-<(#0Oezb<@?(0YgWeMQ4r+SiPy8>P+;@-d}KOiTmtQlf=tstFYdQi8AB? zlmiIDl3xm0oMhhfP7N|k!L757e3J vrHnjVGyfq{;o%AapuGS7W40~zK^Wc =RsqRf38j5r0BYRJeb3tgp# zu(#pd#*pVrfu+2mx*h$Xy4zr_+v&r!5+Z f{G zBaZd})D5tXXi)7xQG^a`YDrrBzRc+(q06 |&RKAm7Qy;#Aw97`IKakrtZZ z%J-HFADK{%dR+7s!oRF09K16x-%Ngy%42(k^~V_}>mU*SyVemA=*ls1vw;(3iGT}L zaps@6P!Ip@jsA@b6)>SqF7q|93Vm3bTkE1(YfD}^8MA44a|Xf8r dN>8d98rp#6{6wlTmQVLVH5DN#h)<1-$Jt^Z&(7(+3yOR%Yl*c?sB3JL zJXEXRvd8kYo>9pG+A{ZO&`6+V!r}Pzjkx^4bx&j%K0ZWa$H#FePI_9u^6&@=eIt_x zSB#rwagaLBja3+La2DoNbVj}f?Om5s=9d}7lBVx`ayvV0V?0M@OVH+->*j+8Aj=iI zGZD4eT|PS1wwodlt2feN7%`G-VoKOJh*R)?t6IGfr`~4n`%i52(&KH&ee7}|#Pd%A z)DB}h(|8+Fe)Ys({L6nf#$A=H3+Q^}VU|$zhUZRMQf=v3L!$K&x)L~u(0s(jEkcR1 zo)K~OT!Fg2`83&`6u?}PY@s)`Ba=#&% J1?58EBCqCt*KUzoPh5~-Jsm`+{AD` z4VK{FtV|o2)=*>0Rls#wO_1R(I$Eu|*yV`sbR}_kfipR(X~V$Gs`2qRS<>^alkqe@ zevg?`e@QwC%#b`bRVnYt^7UN>0VV|FgJTt4Zb#Q 0^!J z@)s1U3TxgMX~RG$jiZ_ZL{JZy!=c7EQ6;p}ym=B3bC9JM;LxcUB>dH1Y0ZvmlqN-n zv^74A-9&KBIyN~3>K~*y@^|6sVLRen`umXDX44wgZow+LST>!eEVEp6=7v7n>a6%4 zRc>BABs{eS#)UA)2OyzJO!KOF*12C3O;Lp*Oi2LRzJ3Ub+(mCl@cB?ccY(V4M_Fzr z1u|QZ^2>O&m+P+ )4&$r;d7w`(JFj;8wuq zTc@jciYAMRS)|KmT*O_AJbO6a@M~Eu^}4bvu3nrX64_aok)PRk>2g&=Q=8iQh9 4HE;u{{N^X>0uMb? !Cy2>$T-aK{*~awPz9(i!@VBBK&HF7z;OxQSkBg4vo>vf^9fHt*#j%Osp`TIE4^ z@ahng^}nOvZM-tG(&yY`!Y!_Yg!}i8u;I-y5C&pZ7Rwm`u=!U$H!=V;KSvgCpsBE7 z)cS6*7zQ{&ztJpX+g8b%jtbwxX3ou4*kN_O8lH|ViB+uQp`{Gs=bs05ra^YRX0u_C zOXrDC{{eL&pAYa%zxm=Cs}XyXd@|fR-SEMo?*oI?Lhk`Sd8MpsGg~#jFq)A;vHsEW ztRwGCih^E Y3vKH2BCcm~p^zJF;jj}trX$H-gk=L>aXVLk^H(CYA%9;0C&4FHT z4Q5Uo8Hb*jxmbK?MM<2BXanAB7!@)OJ#5Q5-eviVp^-E+7qvNa%w(|I+;P^y-Mcp! zYQyx{A4Hg#6QfwCS8m*0psm86r--&i3E#B*-H`E*w3_p!>2X7{K+4|tpRHYt-)}V8 z1rGP04O4|TDrQ|&h*7wFxA2V3MU%qCKwriCVL=>y8&Q=Y2QVHVRmnj;thjK?VIj## z{UFx*E1y)!S1tNgGBV5&`j+fF`=qUu
fP9g8YCuFPD;=zQL9LhA4`?dRtRV= z(w6u7r _ob*=B~A@D3ln=2<@UtB9<%KAKff*m(ajf?NvDpMZc zYaVVKEVz|lD!(G7h7MjR2odBBzgOe&Ys+G)s0DT$b&nzp5E#c+AH(#ELcYtu-X!~I z36b@F3RR1KewT;A5>@2b)S!x1EE~cj%Pb5j)n)326%G8^Q(1c7*k#tic_r@6)qVBZ zolXl#k?JzekB)aF-ly8ci|`9ocp8M4C=H_Z{F*se%yG{6jrmJA*mPqkIWUroc$4^L zY7i52P`!JQTXfgLrI~?ink0(vsiL5iiWe%-O%`h_R4A^;y4{yc>}9*Qu~5F6o8X)| z2S*df6OKd`&^!Krs#61WqPYJ^CqiN}k$_Me@n<~wv+VQ7zTAP-?s&;GtF@ES%PlI; zE56Rxy>;}hV5ru;#-Cn*H^b9)VNK6ktyaS_K4$%C^eveO-wjlo5L}2y<;M+=FJSlE zA6F|=eAS*4SCHkOBZxMZS#*)oDiPl>6Chu%5a^U~U8|$^RF~@$yNqa9L{R-B1crT) z4lvnkl*{ E7ZSnfz=+Sys&u(FI{DsuP?}^eOD@j z5i^7kNnevqyH98e%gDPZ{^PgA7xV^2kHkBUsG)=GY+pt2 NBYsbJCmsGeHHQ1+GbkU|y|-GD!g=5*sJOoEUF z@B+L`MWIkJ&J90-YZH0=EwifX3`BtInlw@BR_x||Y~MSO4+fgxZ(2-(u5F7G7a~8F z&$(c}D3?_5ZCv{g`ln=K5bS#o0rPIB-5n6iAK9*2;PbPRcD2|WC5=6ykw$=6Sl-yO zLi2ZC9ejRrMRUT)m =g_Xr6p1KI_C zKVdhi&R0ooLeorwttv)hI2|KD nkynwG*U1JI;WbmTLzyF c)@B8nIe#8C3!3sUU z#DWemVvh!72Bs|-gnvv}DlJ-r2OfE^kXC7Qa^aZQzpbvKeE^JI=@1#Aep&VX%Yz=2 zPT1MUa>O0V6CY5GXn!_7Z?hcT_=)f8!vlM?Yc@<_8K7LOuSn)HHcVUXdQW@@;8Tsj zC6Ws^8+9cTdE9z`lw9Ifa4I^;r@3+GB5TH-ndx7qR%t1h_z7pBi|@rC^42< Wk1Xa45Rg> g*PH0H@c8WRqY&%@C#!ZscPfJB|W_6^ti z!@v63UITXp*vS1=3TNiRY{j~V;rXya!Qss1Ix?$Vs{qc25+p&Z;^paAD&5)TvgUVh z+0&b1LqO;bO?VY_ZgCGM49+kA4%$WNXS&b!; j*k2`)5wa>rwC^FF;=mSd<&YPXd(v#0CUt#f@fwqeTiSrURUCME1R;ccq z@G1&|4yD) <_?e(tECcAK6m1ge6Jx}Scz z!&yvjV5r=Ln!tGcC7UBr_G;B%EB+V6+55nx(s_4e&%L@!(&=h~q1VOJxlVakvbAR% zk5Cj<>C5?IEo%vsAH6q)N&fGYKto_Y+0|a}S#jP&O`n!oNP|CXSH$-JYVX>^p-kKO zi$XbW7)&G)O4yP^4$aswEEc64BbIR-s@0-2qpZUqVo_M9$T{RN<$Ri9NwJ1B2GdN* zaWIbKG=1-W*Y~BZcK_Jx+wX9FUDy0IGtV{8`^ ;K)2 z*A=gge~PO$9^b)1K=<5nPeG$10)JAnqCJ{;#gbiiQhlRHRdgPonKJBIT+$`JT%AGC znuLR(6#O(cST|2y!Uhp~{n)zt^`Iijk<;>iM5e(Hw<1u8P~ORI>can{0zi>PcXZ zbvOy7#hRiIve;IfZh-d`t58dCA9vb00U1Iy?fdq?8{4}W;RK1w%*)?sU$CaF _$B7b1}t=pCN6K(yWecZ z^)Ne*P?xCZ9F{Q&T1}Zoo69pPnmQGYsy4-lx-IQYzfI|_l*YZmGDc-HGnne-esR>f zH4dW(lrHl+!D*sW=*{*fJ{te=?!}eKXAX%?3ECKP&))@h%Yizj;R04=%5zIIV7Dvm z w=?~7s*gt>=XgNN1A=ju(dglv;osnYVy3D-SL&gnZ zu3EQ0i6s9R&uPqitjh0*5naeHrsAe$xgjFEMTVjF_30>2U4m;zf6O-aeE5sI0r9hm z#R~gl1Q9zd(yQU>ny;eNqGYY~+0I&BP>=*9!_ -=VD}AMWKT#a6eJ9nd9dxl%X_QM?@Y&36~89c1^8AypG2j5@4cuXDuFVc^AdK zv+-#?Wr7_1>j`V_LS2Y*u-{hs@wrf;7hW1y1tRk1H`fhtRYWgGm1^P@?H6vBd9V6r zO!YaMK&dIV*-q&qwpdga^cQsB>YYqUcc!lM%cm+`RSi*H*7Xe8bG7)$`6Erfp{9YW z<}N(k-JQyGuxb(zCKsZ49=1}+2xyxN8;XcVADduFJ{s_2Nqdcc(F<|fV}QC~Y8?Pq z8X=PnlY3hv>&nt95VsBUJp)K^D9#B7siVPo1NS=2%eOvN#ar)F9bAU(illGa3sy&* zt%~v#xmP22%)! %Rb6K_h; 58(+-p~?9#e`X= zr=AeO;(c;10grN7L}*4%+2qg &1Gb#RwoJ8ep}`#17ek1<)Dv>niAArYLDJ^^lfr2Cp0NR z!J8}I)%sSzliOmj }M>097w6 z&C5jP5%Py0Jm^iJyWFk2em_}rMgP=7x5aK?9Bc?<=tU9Q M-?Ey@ zs^lBQrol4}!(@}mohFeGgm+n$o|eK_*p2XS58tc-XrPP()gVvhYQ=_o|&!98C}Q$jw$kml&G zYNM~xQL))~WFEFiSnRBQhgk1vra+cgqk1snIZKGS$4|mzlppN;uU+R+T4I=s`%cX5 zwm~h#-!(|1{(?-Em!53xtK0mLxs~L>;1l1{9%#3N;*MK!(f)i01hM4!Ets9jV>Mef z+u&lpJLbZU=Un+K=HgbJ#X|SD!}r&2<<* bmB*ANopra7=(lC~pC=B}yzt;2xtr-P7CW^!l7> zjfw78+8d{I>mTfGXy^kfn+XB3X;w+jNoSIPeb^7O4{)N4aFDJ4``77#^b>T;EiDzZZcEVqHR* 0O$ApP_0LF#l@fFbQ|l|IkUbO~mz4WhZFby>;^pzXAM>z;e=9P}a$jnxEUh z->hQ0pU3LX;3oz `WU*vV z_CtrWcrS1e3x3CB_<0DKyQ~Qq@J2m`EJ(K@K+kuCE0v-i_>+a)Rt}+i>)e9$aXp7z z*!|<-3xgcV!)$Gym
ooPn-}K2 znTHqK_wAn_D;czKc>3KvwW49;G0k`xxi`BK<(3UKEl6 wnn)@9qY2d zP@3p16*q 9#EH(_kY>8H6Rj7~ zhl2GQ2Cl#aTWI9#NZP^^dM&b2vWYZ0OG+3xvs%uYHOjS>KD2Pet4RsCI(W1GXaJRu zT?qnGt)X++S%@w2$1~4!!%FkqFTXgcZ>e8Y(^_;O1Q5&OoB9}@`D`!)@*6p87911r zjAr+vx4_gLkdK)2F WEiRtfPO|Y!q&-L G&<837QH~QA^FY>!ds7t>E&PaU-cByge_um57)-5= oOsw2SR4N!LX_y0of7E;Bp7f(P?D{;HNM%wv|!sx=UUBNl%m49R{W(#G2NIj zK#J6h0qKoLS;7G3^Y=cl%V#_IbN*p_mOC2(KMLpB#=O5W$a&}fe{iz+>3rsYYyj6= G(%%3LHU$Cz literal 0 HcmV?d00001 diff --git a/search/search_index.json b/search/search_index.json index 93af905b9..476f6cecc 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -107,7 +107,7 @@ }, { "location": "/deploy/baremetal/", - "text": "Bare-metal considerations\n\u00b6\n\n\nIn traditional \ncloud\n environments, where network load balancers are available on-demand, a single Kubernetes manifest\nsuffices to provide a single point of contact to the NGINX Ingress controller to external clients and, indirectly, to\nany application running inside the cluster. \nBare-metal\n environments lack this commodity, requiring a slightly\ndifferent setup to offer the same kind of access to external consumers.\n\n\n\n\n\n\nThe rest of this document describes a few recommended approaches to deploying the NGINX Ingress controller inside a\nKubernetes cluster running on bare-metal.\n\n\nOver a NodePort Service\n\u00b6\n\n\nDue to its simplicity, this is the setup a user will deploy by default when following the steps described in the\n\ninstallation guide\n.\n\n\n\n\nInfo\n\n\nA Service of type \nNodePort\n exposes, via the \nkube-proxy\n component, the \nsame unprivileged\n port (default:\n30000-32767) on every Kubernetes node, masters included. For more information, see \nServices\n.\n\n\n\n\nIn this configuration, the NGINX container remains isolated from the host network. As a result, it can safely bind to\nany port, including the standard HTTP ports 80 and 443. However, due to the container namespace isolation, a client\nlocated outside the cluster network (e.g. on the public internet) is not able to access Ingress hosts directly on ports\n80 and 443. Instead, the external client must append the NodePort allocated to the \ningress-nginx\n Service to HTTP\nrequests.\n\n\n\n\n\n\nExample\n\n\nGiven the NodePort \n30100\n allocated to the \ningress-nginx\n Service\n\n\n$\n kubectl -n ingress-nginx get svc\n\nNAME TYPE CLUSTER-IP PORT(S)\n\n\ndefault-http-backend ClusterIP 10.0.64.249 80/TCP\n\n\ningress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP\n\n\n\n\n\nand a Kubernetes node with the public IP address \n203.0.113.2\n (the external IP is added as an example, in most\nbare-metal environments this value is )\n\n\n$\n kubectl describe node \n\nNAME STATUS ROLES EXTERNAL-IP\n\n\nhost-1 Ready master 203.0.113.1\n\n\nhost-2 Ready node 203.0.113.2\n\n\nhost-3 Ready node 203.0.113.3\n\n\n\n\n\na client would reach an Ingress with \nhost\n:\n \nmyapp\n.\nexample\n.\ncom\n at \nhttp://myapp.example.com:30100\n, where the\nmyapp.example.com subdomain resolves to the 203.0.113.2 IP address.\n\n\n\n\n\n\nImpact on the host system\n\n\nWhile it may sound tempting to reconfigure the NodePort range using the \n--service-node-port-range\n API server flag\nto include unprivileged ports and be able to expose ports 80 and 443, doing so may result in unexpected issues\nincluding (but not limited to) the use of ports otherwise reserved to system daemons and the necessity to grant\n\nkube-proxy\n privileges it may otherwise not require.\n\n\nThis practice is therefore \ndiscouraged\n. See the other approaches proposed in this page for alternatives.\n\n\n\n\nThis approach has a few other limitations one ought to be aware of:\n\n\n\n\nSource IP address\n\n\n\n\nServices of type NodePort perform \nsource address translation\n by default. This means the source IP of a\nHTTP request is always \nthe IP address of the Kubernetes node that received the request\n from the perspective of\nNGINX.\n\n\nThe recommended way to preserve the source IP in a NodePort setup is to set the value of the \nexternalTrafficPolicy\n\nfield of the \ningress-nginx\n Service spec to \nLocal\n (\nexample\n).\n\n\n\n\nWarning\n\n\nThis setting effectively \ndrops packets\n sent to Kubernetes nodes which are not running any instance of the NGINX\nIngress controller. Consider \nassigning NGINX Pods to specific nodes\n in order to control on what nodes\nthe NGINX Ingress controller should be scheduled or not scheduled.\n\n\n\n\n\n\nExample\n\n\nIn a Kubernetes cluster composed of 3 nodes (the external IP is added as an example, in most bare-metal environments\nthis value is )\n\n\n$\n kubectl describe node \n\nNAME STATUS ROLES EXTERNAL-IP\n\n\nhost-1 Ready master 203.0.113.1\n\n\nhost-2 Ready node 203.0.113.2\n\n\nhost-3 Ready node 203.0.113.3\n\n\n\n\n\nwith a \nnginx-ingress-controller\n Deployment composed of 2 replicas\n\n\n$\n kubectl -n ingress-nginx get pod -o wide\n\nNAME READY STATUS IP NODE\n\n\ndefault-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2\n\n\nnginx-ingress-controller-cf9ff8c96-8vvf8 1/1 Running 172.17.0.3 host-3\n\n\nnginx-ingress-controller-cf9ff8c96-pxsds 1/1 Running 172.17.1.4 host-2\n\n\n\n\n\nRequests sent to \nhost-2\n and \nhost-3\n would be forwarded to NGINX and original client's IP would be preserved,\nwhile requests to \nhost-1\n would get dropped because there is no NGINX replica running on that node.\n\n\n\n\n\n\nIngress status\n\n\n\n\nBecause NodePort Services do not get a LoadBalancerIP assigned by definition, the NGINX Ingress controller \ndoes not\nupdate the status of Ingress objects it manages\n.\n\n\n$\n kubectl get ingress\n\nNAME HOSTS ADDRESS PORTS\n\n\ntest-ingress myapp.example.com 80\n\n\n\n\n\nDespite the fact there is no load balancer providing a public IP address to the NGINX Ingress controller, it is possible\nto force the status update of all managed Ingress objects by setting the \nexternalIPs\n field of the \ningress-nginx\n\nService.\n\n\n\n\nExample\n\n\nGiven the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal\nenvironments this value is )\n\n\n$\n kubectl describe node \n\nNAME STATUS ROLES EXTERNAL-IP\n\n\nhost-1 Ready master 203.0.113.1\n\n\nhost-2 Ready node 203.0.113.2\n\n\nhost-3 Ready node 203.0.113.3\n\n\n\n\n\none could edit the \ningress-nginx\n Service and add the following field to the object spec\n\n\nspec\n:\n\n \nexternalIPs\n:\n\n \n-\n \n203.0.113.1\n\n \n-\n \n203.0.113.2\n\n \n-\n \n203.0.113.3\n\n\n\n\n\nwhich would in turn be reflected on Ingress objects as follows:\n\n\n$\n kubectl get ingress -o wide\n\nNAME HOSTS ADDRESS PORTS\n\n\ntest-ingress myapp.example.com 203.0.113.1,203.0.113.2,203.0.113.3 80\n\n\n\n\n\n\n\n\n\nRedirects\n\n\n\n\nAs NGINX is \nnot aware of the port translation operated by the NodePort Service\n, backend applications are responsible\nfor generating redirect URLs that take into account the URL used by external clients, including the NodePort.\n\n\n\n\nExample\n\n\nRedirects generated by NGINX, for instance HTTP to HTTPS or \ndomain\n to \nwww.domain\n, are generated without\nNodePort:\n\n\n$\n curl http://myapp.example.com:30100\n`\n\n\nHTTP/1.1 308 Permanent Redirect\n\n\nServer: nginx/1.15.2\n\n\nLocation: https://myapp.example.com/ #-> missing NodePort in HTTPS redirect\n\n\n\n\n\n\n\nVia the host network\n\u00b6\n\n\nIn a setup where there is no external load balancer available but using NodePorts is not an option, one can configure\n\ningress-nginx\n Pods to use the network of the host they run on instead of a dedicated network namespace. The benefit of\nthis approach is that the NGINX Ingress controller can bind ports 80 and 443 directly to Kubernetes nodes' network\ninterfaces, without the extra network translation imposed by NodePort Services.\n\n\n\n\nNote\n\n\nThis approach does not leverage any Service object to expose the NGINX Ingress controller. If the \ningress-nginx\n\nService exists in the target cluster, it is \nrecommended to delete it\n.\n\n\n\n\nThis can be achieved by enabling the \nhostNetwork\n option in the Pods' spec.\n\n\ntemplate\n:\n\n \nspec\n:\n\n \nhostNetwork\n:\n \ntrue\n\n\n\n\n\n\n\nSecurity considerations\n\n\nEnabling this option \nexposes every system daemon to the NGINX Ingress controller\n on any network interface,\nincluding the host's loopback. Please evaluate the impact this may have on the security of your system carefully.\n\n\n\n\n\n\nExample\n\n\nConsider this \nnginx-ingress-controller\n Deployment composed of 2 replicas, NGINX Pods inherit from the IP address\nof their host instead of an internal Pod IP.\n\n\n$\n kubectl -n ingress-nginx get pod -o wide\n\nNAME READY STATUS IP NODE\n\n\ndefault-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2\n\n\nnginx-ingress-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3\n\n\nnginx-ingress-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2\n\n\n\n\n\n\n\nOne major limitation of this deployment approach is that only \na single NGINX Ingress controller Pod\n may be scheduled\non each cluster node, because binding the same port multiple times on the same network interface is technically\nimpossible. Pods that are unschedulable due to such situation fail with the following event:\n\n\n$\n kubectl -n ingress-nginx describe pod \n\n...\n\n\nEvents:\n\n\n Type Reason From Message\n\n\n ---- ------ ---- -------\n\n\n Warning FailedScheduling default-scheduler 0/3 nodes are available: 3 node(s) didn't have free ports for the requested pod ports.\n\n\n\n\n\nOne way to ensure only schedulable Pods are created is to deploy the NGINX Ingress controller as a \nDaemonSet\n instead\nof a traditional Deployment.\n\n\n\n\nInfo\n\n\nA DaemonSet schedules exactly one type of Pod per cluster node, masters included, unless a node is configured to\n\nrepel those Pods\n. For more information, see \nDaemonSet\n.\n\n\n\n\nBecause most properties of DaemonSet objects are identical to Deployment objects, this documentation page leaves the\nconfiguration of the corresponding manifest at the user's discretion.\n\n\n\n\nLike with NodePorts, this approach has a few quirks it is important to be aware of.\n\n\n\n\nDNS resolution\n\n\n\n\nPods configured with \nhostNetwork\n:\n \ntrue\n do not use the internal DNS resolver (i.e. \nkube-dns\n or \nCoreDNS\n), unless\ntheir \ndnsPolicy\n spec field is set to \nClusterFirstWithHostNet\n. Consider using this setting if NGINX is\nexpected to resolve internal names for any reason.\n\n\n\n\nIngress status\n\n\n\n\nBecause there is no Service exposing the NGINX Ingress controller in a configuration using the host network, the default\n\n--publish-service\n flag used in standard cloud setups \ndoes not apply\n and the status of all Ingress objects remains\nblank.\n\n\n$\n kubectl get ingress\n\nNAME HOSTS ADDRESS PORTS\n\n\ntest-ingress myapp.example.com 80\n\n\n\n\n\nInstead, and because bare-metal nodes usually don't have an ExternalIP, one has to enable the\n\n--report-node-internal-ip-address\n flag, which sets the status of all Ingress objects to the internal IP\naddress of all nodes running the NGINX Ingress controller.\n\n\n\n\nExample\n\n\nGiven a \nnginx-ingress-controller\n DaemonSet composed of 2 replicas\n\n\n$\n kubectl -n ingress-nginx get pod -o wide\n\nNAME READY STATUS IP NODE\n\n\ndefault-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2\n\n\nnginx-ingress-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3\n\n\nnginx-ingress-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2\n\n\n\n\n\nthe controller sets the status of all Ingress objects it manages to the following value:\n\n\n$\n kubectl get ingress -o wide\n\nNAME HOSTS ADDRESS PORTS\n\n\ntest-ingress myapp.example.com 203.0.113.2,203.0.113.3 80\n\n\n\n\n\n\n\n\n\nNote\n\n\nAlternatively, it is possible to override the address written to Ingress objects using the\n\n--publish-status-address\n flag. See \nCommand line arguments\n.\n\n\n\n\nUsing a self-provisioned edge\n\u00b6\n\n\nSimilarly to cloud environments, this deployment approach requires an edge network component providing a public\nentrypoint to the Kubernetes cluster. This edge component can be either hardware (e.g. vendor appliance) or software\n(e.g. \nHAproxy\n) and is usually managed outside of the Kubernetes landscape by operations teams.\n\n\nSuch deployment builds upon the NodePort Service described above in \nOver a NodePort Service\n,\nwith one significant difference: external clients do not access cluster nodes directly, only the edge component does.\nThis is particularly suitable for private Kubernetes clusters where none of the nodes has a public IP address.\n\n\nOn the edge side, the only prerequisite is to dedicate a public IP address that forwards all HTTP traffic to Kubernetes\nnodes and/or masters. Incoming traffic on TCP ports 80 and 443 is forwarded to the corresponding HTTP and HTTPS NodePort\non the target nodes as shown in the diagram below:", + "text": "Bare-metal considerations\n\u00b6\n\n\nIn traditional \ncloud\n environments, where network load balancers are available on-demand, a single Kubernetes manifest\nsuffices to provide a single point of contact to the NGINX Ingress controller to external clients and, indirectly, to\nany application running inside the cluster. \nBare-metal\n environments lack this commodity, requiring a slightly\ndifferent setup to offer the same kind of access to external consumers.\n\n\n\n\n\n\nThe rest of this document describes a few recommended approaches to deploying the NGINX Ingress controller inside a\nKubernetes cluster running on bare-metal.\n\n\nA pure software solution: MetalLB\n\u00b6\n\n\nMetalLB\n provides a network load-balancer implementation for Kubernetes clusters that do not run on a\nsupported cloud provider, effectively allowing the usage of LoadBalancer Services within any cluster.\n\n\nThis section demonstrates how to use the \nLayer 2 configuration mode\n of MetalLB together with the NGINX\nIngress controller in a Kubernetes cluster that has \npublicly accessible nodes\n. In this mode, one node attracts all\nthe traffic for the \ningress-nginx\n Service IP. See \nTraffic policies\n for more details.\n\n\n\n\n\n\nNote\n\n\nThe description of other supported configuration modes is off-scope for this document.\n\n\n\n\n\n\nWarning\n\n\nMetalLB is currently in \nbeta\n. Read about the \nProject maturity\n and make sure you inform\nyourself by reading the official documentation thoroughly.\n\n\n\n\nMetalLB can be deployed either with a simple Kubernetes manifest or with Helm. The rest of this example assumes MetalLB\nwas deployed following the \nInstallation\n instructions.\n\n\nMetalLB requires a pool of IP addresses in order to be able to take ownership of the \ningress-nginx\n Service. This pool\ncan be defined in a ConfigMap named \nconfig\n located in the same namespace as the MetalLB controller. In the simplest\npossible scenario, the pool is composed of the IP addresses of Kubernetes nodes, but IP addresses can also be handed out\nby a DHCP server.\n\n\n\n\nExample\n\n\nGiven the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal\nenvironments this value is )\n\n\n$\n kubectl describe node\n\nNAME STATUS ROLES EXTERNAL-IP\n\n\nhost-1 Ready master 203.0.113.1\n\n\nhost-2 Ready node 203.0.113.2\n\n\nhost-3 Ready node 203.0.113.3\n\n\n\n\n\nAfter creating the following ConfigMap, MetalLB takes ownership of one of the IP addresses in the pool and updates\nthe \nloadBalancer\n IP field of the \ningress-nginx\n Service accordingly.\n\n\napiVersion\n:\n \nv1\n\n\nkind\n:\n \nConfigMap\n\n\nmetadata\n:\n\n \nnamespace\n:\n \nmetallb-system\n\n \nname\n:\n \nconfig\n\n\ndata\n:\n\n \nconfig\n:\n \n|\n\n \naddress-pools:\n\n \n- name: default\n\n \nprotocol: layer2\n\n \naddresses:\n\n \n- 203.0.113.2-203.0.113.3\n\n\n\n\n\n$\n kubectl -n ingress-nginx get svc\n\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)\n\n\ndefault-http-backend ClusterIP 10.0.64.249 80/TCP\n\n\ningress-nginx LoadBalancer 10.0.220.217 203.0.113.3 80:30100/TCP,443:30101/TCP\n\n\n\n\n\n\n\nAs soon as MetalLB sets the external IP address of the \ningress-nginx\n LoadBalancer Service, the corresponding entries\nare created in the iptables NAT table and the node with the selected IP address starts responding to HTTP requests on\nthe ports configured in the LoadBalancer Service:\n\n\n$\n curl -D- http://203.0.113.3 -H \n'Host: myapp.example.com'\n\n\nHTTP/1.1 200 OK\n\n\nServer: nginx/1.15.2\n\n\n\n\n\n\n\nTip\n\n\nIn order to preserve the source IP address in HTTP requests sent to NGINX, it is necessary to use the \nLocal\n\ntraffic policy. Traffic policies are described in more details in \nTraffic policies\n as\nwell as in the next section.\n\n\n\n\nOver a NodePort Service\n\u00b6\n\n\nDue to its simplicity, this is the setup a user will deploy by default when following the steps described in the\n\ninstallation guide\n.\n\n\n\n\nInfo\n\n\nA Service of type \nNodePort\n exposes, via the \nkube-proxy\n component, the \nsame unprivileged\n port (default:\n30000-32767) on every Kubernetes node, masters included. For more information, see \nServices\n.\n\n\n\n\nIn this configuration, the NGINX container remains isolated from the host network. As a result, it can safely bind to\nany port, including the standard HTTP ports 80 and 443. However, due to the container namespace isolation, a client\nlocated outside the cluster network (e.g. on the public internet) is not able to access Ingress hosts directly on ports\n80 and 443. Instead, the external client must append the NodePort allocated to the \ningress-nginx\n Service to HTTP\nrequests.\n\n\n\n\n\n\nExample\n\n\nGiven the NodePort \n30100\n allocated to the \ningress-nginx\n Service\n\n\n$\n kubectl -n ingress-nginx get svc\n\nNAME TYPE CLUSTER-IP PORT(S)\n\n\ndefault-http-backend ClusterIP 10.0.64.249 80/TCP\n\n\ningress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP\n\n\n\n\n\nand a Kubernetes node with the public IP address \n203.0.113.2\n (the external IP is added as an example, in most\nbare-metal environments this value is )\n\n\n$\n kubectl describe node\n\nNAME STATUS ROLES EXTERNAL-IP\n\n\nhost-1 Ready master 203.0.113.1\n\n\nhost-2 Ready node 203.0.113.2\n\n\nhost-3 Ready node 203.0.113.3\n\n\n\n\n\na client would reach an Ingress with \nhost\n:\n \nmyapp\n.\nexample\n.\ncom\n at \nhttp://myapp.example.com:30100\n, where the\nmyapp.example.com subdomain resolves to the 203.0.113.2 IP address.\n\n\n\n\n\n\nImpact on the host system\n\n\nWhile it may sound tempting to reconfigure the NodePort range using the \n--service-node-port-range\n API server flag\nto include unprivileged ports and be able to expose ports 80 and 443, doing so may result in unexpected issues\nincluding (but not limited to) the use of ports otherwise reserved to system daemons and the necessity to grant\n\nkube-proxy\n privileges it may otherwise not require.\n\n\nThis practice is therefore \ndiscouraged\n. See the other approaches proposed in this page for alternatives.\n\n\n\n\nThis approach has a few other limitations one ought to be aware of:\n\n\n\n\nSource IP address\n\n\n\n\nServices of type NodePort perform \nsource address translation\n by default. This means the source IP of a\nHTTP request is always \nthe IP address of the Kubernetes node that received the request\n from the perspective of\nNGINX.\n\n\nThe recommended way to preserve the source IP in a NodePort setup is to set the value of the \nexternalTrafficPolicy\n\nfield of the \ningress-nginx\n Service spec to \nLocal\n (\nexample\n).\n\n\n\n\nWarning\n\n\nThis setting effectively \ndrops packets\n sent to Kubernetes nodes which are not running any instance of the NGINX\nIngress controller. Consider \nassigning NGINX Pods to specific nodes\n in order to control on what nodes\nthe NGINX Ingress controller should be scheduled or not scheduled.\n\n\n\n\n\n\nExample\n\n\nIn a Kubernetes cluster composed of 3 nodes (the external IP is added as an example, in most bare-metal environments\nthis value is )\n\n\n$\n kubectl describe node\n\nNAME STATUS ROLES EXTERNAL-IP\n\n\nhost-1 Ready master 203.0.113.1\n\n\nhost-2 Ready node 203.0.113.2\n\n\nhost-3 Ready node 203.0.113.3\n\n\n\n\n\nwith a \nnginx-ingress-controller\n Deployment composed of 2 replicas\n\n\n$\n kubectl -n ingress-nginx get pod -o wide\n\nNAME READY STATUS IP NODE\n\n\ndefault-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2\n\n\nnginx-ingress-controller-cf9ff8c96-8vvf8 1/1 Running 172.17.0.3 host-3\n\n\nnginx-ingress-controller-cf9ff8c96-pxsds 1/1 Running 172.17.1.4 host-2\n\n\n\n\n\nRequests sent to \nhost-2\n and \nhost-3\n would be forwarded to NGINX and original client's IP would be preserved,\nwhile requests to \nhost-1\n would get dropped because there is no NGINX replica running on that node.\n\n\n\n\n\n\nIngress status\n\n\n\n\nBecause NodePort Services do not get a LoadBalancerIP assigned by definition, the NGINX Ingress controller \ndoes not\nupdate the status of Ingress objects it manages\n.\n\n\n$\n kubectl get ingress\n\nNAME HOSTS ADDRESS PORTS\n\n\ntest-ingress myapp.example.com 80\n\n\n\n\n\nDespite the fact there is no load balancer providing a public IP address to the NGINX Ingress controller, it is possible\nto force the status update of all managed Ingress objects by setting the \nexternalIPs\n field of the \ningress-nginx\n\nService.\n\n\n\n\nWarning\n\n\nThere is more to setting \nexternalIPs\n than just enabling the NGINX Ingress controller to update the status of\nIngress objects. Please read about this option in the \nServices\n page of official Kubernetes\ndocumentation as well as the section about \nExternal IPs\n in this document for more information.\n\n\n\n\n\n\nExample\n\n\nGiven the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal\nenvironments this value is )\n\n\n$\n kubectl describe node\n\nNAME STATUS ROLES EXTERNAL-IP\n\n\nhost-1 Ready master 203.0.113.1\n\n\nhost-2 Ready node 203.0.113.2\n\n\nhost-3 Ready node 203.0.113.3\n\n\n\n\n\none could edit the \ningress-nginx\n Service and add the following field to the object spec\n\n\nspec\n:\n\n \nexternalIPs\n:\n\n \n-\n \n203.0.113.1\n\n \n-\n \n203.0.113.2\n\n \n-\n \n203.0.113.3\n\n\n\n\n\nwhich would in turn be reflected on Ingress objects as follows:\n\n\n$\n kubectl get ingress -o wide\n\nNAME HOSTS ADDRESS PORTS\n\n\ntest-ingress myapp.example.com 203.0.113.1,203.0.113.2,203.0.113.3 80\n\n\n\n\n\n\n\n\n\nRedirects\n\n\n\n\nAs NGINX is \nnot aware of the port translation operated by the NodePort Service\n, backend applications are responsible\nfor generating redirect URLs that take into account the URL used by external clients, including the NodePort.\n\n\n\n\nExample\n\n\nRedirects generated by NGINX, for instance HTTP to HTTPS or \ndomain\n to \nwww.domain\n, are generated without\nNodePort:\n\n\n$\n curl -D- http://myapp.example.com:30100\n`\n\n\nHTTP/1.1 308 Permanent Redirect\n\n\nServer: nginx/1.15.2\n\n\nLocation: https://myapp.example.com/ #-> missing NodePort in HTTPS redirect\n\n\n\n\n\n\n\nVia the host network\n\u00b6\n\n\nIn a setup where there is no external load balancer available but using NodePorts is not an option, one can configure\n\ningress-nginx\n Pods to use the network of the host they run on instead of a dedicated network namespace. The benefit of\nthis approach is that the NGINX Ingress controller can bind ports 80 and 443 directly to Kubernetes nodes' network\ninterfaces, without the extra network translation imposed by NodePort Services.\n\n\n\n\nNote\n\n\nThis approach does not leverage any Service object to expose the NGINX Ingress controller. If the \ningress-nginx\n\nService exists in the target cluster, it is \nrecommended to delete it\n.\n\n\n\n\nThis can be achieved by enabling the \nhostNetwork\n option in the Pods' spec.\n\n\ntemplate\n:\n\n \nspec\n:\n\n \nhostNetwork\n:\n \ntrue\n\n\n\n\n\n\n\nSecurity considerations\n\n\nEnabling this option \nexposes every system daemon to the NGINX Ingress controller\n on any network interface,\nincluding the host's loopback. Please evaluate the impact this may have on the security of your system carefully.\n\n\n\n\n\n\nExample\n\n\nConsider this \nnginx-ingress-controller\n Deployment composed of 2 replicas, NGINX Pods inherit from the IP address\nof their host instead of an internal Pod IP.\n\n\n$\n kubectl -n ingress-nginx get pod -o wide\n\nNAME READY STATUS IP NODE\n\n\ndefault-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2\n\n\nnginx-ingress-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3\n\n\nnginx-ingress-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2\n\n\n\n\n\n\n\nOne major limitation of this deployment approach is that only \na single NGINX Ingress controller Pod\n may be scheduled\non each cluster node, because binding the same port multiple times on the same network interface is technically\nimpossible. Pods that are unschedulable due to such situation fail with the following event:\n\n\n$\n kubectl -n ingress-nginx describe pod \n\n...\n\n\nEvents:\n\n\n Type Reason From Message\n\n\n ---- ------ ---- -------\n\n\n Warning FailedScheduling default-scheduler 0/3 nodes are available: 3 node(s) didn't have free ports for the requested pod ports.\n\n\n\n\n\nOne way to ensure only schedulable Pods are created is to deploy the NGINX Ingress controller as a \nDaemonSet\n instead\nof a traditional Deployment.\n\n\n\n\nInfo\n\n\nA DaemonSet schedules exactly one type of Pod per cluster node, masters included, unless a node is configured to\n\nrepel those Pods\n. For more information, see \nDaemonSet\n.\n\n\n\n\nBecause most properties of DaemonSet objects are identical to Deployment objects, this documentation page leaves the\nconfiguration of the corresponding manifest at the user's discretion.\n\n\n\n\nLike with NodePorts, this approach has a few quirks it is important to be aware of.\n\n\n\n\nDNS resolution\n\n\n\n\nPods configured with \nhostNetwork\n:\n \ntrue\n do not use the internal DNS resolver (i.e. \nkube-dns\n or \nCoreDNS\n), unless\ntheir \ndnsPolicy\n spec field is set to \nClusterFirstWithHostNet\n. Consider using this setting if NGINX is\nexpected to resolve internal names for any reason.\n\n\n\n\nIngress status\n\n\n\n\nBecause there is no Service exposing the NGINX Ingress controller in a configuration using the host network, the default\n\n--publish-service\n flag used in standard cloud setups \ndoes not apply\n and the status of all Ingress objects remains\nblank.\n\n\n$\n kubectl get ingress\n\nNAME HOSTS ADDRESS PORTS\n\n\ntest-ingress myapp.example.com 80\n\n\n\n\n\nInstead, and because bare-metal nodes usually don't have an ExternalIP, one has to enable the\n\n--report-node-internal-ip-address\n flag, which sets the status of all Ingress objects to the internal IP\naddress of all nodes running the NGINX Ingress controller.\n\n\n\n\nExample\n\n\nGiven a \nnginx-ingress-controller\n DaemonSet composed of 2 replicas\n\n\n$\n kubectl -n ingress-nginx get pod -o wide\n\nNAME READY STATUS IP NODE\n\n\ndefault-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2\n\n\nnginx-ingress-controller-5b4cf5fc6-7lg6c 1/1 Running 203.0.113.3 host-3\n\n\nnginx-ingress-controller-5b4cf5fc6-lzrls 1/1 Running 203.0.113.2 host-2\n\n\n\n\n\nthe controller sets the status of all Ingress objects it manages to the following value:\n\n\n$\n kubectl get ingress -o wide\n\nNAME HOSTS ADDRESS PORTS\n\n\ntest-ingress myapp.example.com 203.0.113.2,203.0.113.3 80\n\n\n\n\n\n\n\n\n\nNote\n\n\nAlternatively, it is possible to override the address written to Ingress objects using the\n\n--publish-status-address\n flag. See \nCommand line arguments\n.\n\n\n\n\nUsing a self-provisioned edge\n\u00b6\n\n\nSimilarly to cloud environments, this deployment approach requires an edge network component providing a public\nentrypoint to the Kubernetes cluster. This edge component can be either hardware (e.g. vendor appliance) or software\n(e.g. \nHAproxy\n) and is usually managed outside of the Kubernetes landscape by operations teams.\n\n\nSuch deployment builds upon the NodePort Service described above in \nOver a NodePort Service\n,\nwith one significant difference: external clients do not access cluster nodes directly, only the edge component does.\nThis is particularly suitable for private Kubernetes clusters where none of the nodes has a public IP address.\n\n\nOn the edge side, the only prerequisite is to dedicate a public IP address that forwards all HTTP traffic to Kubernetes\nnodes and/or masters. Incoming traffic on TCP ports 80 and 443 is forwarded to the corresponding HTTP and HTTPS NodePort\non the target nodes as shown in the diagram below:\n\n\n\n\nExternal IPs\n\u00b6\n\n\n\n\nSource IP address\n\n\nThis method does not allow preserving the source IP of HTTP requests in any manner, it is therefore \nnot\nrecommended\n to use it despite its apparent simplicity.\n\n\n\n\nThe \nexternalIPs\n Service option was previously mentioned in the \nNodePort\n section.\n\n\nAs per the \nServices\n page of the official Kubernetes documentation, the \nexternalIPs\n option causes\n\nkube-proxy\n to route traffic sent to arbitrary IP addresses \nand on the Service ports\n to the endpoints of that\nService. These IP addresses \nmust belong to the target node\n.\n\n\n\n\nExample\n\n\nGiven the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal\nenvironments this value is )\n\n\n$\n kubectl describe node\n\nNAME STATUS ROLES EXTERNAL-IP\n\n\nhost-1 Ready master 203.0.113.1\n\n\nhost-2 Ready node 203.0.113.2\n\n\nhost-3 Ready node 203.0.113.3\n\n\n\n\n\nand the following \ningress-nginx\n NodePort Service\n\n\n$\n kubectl -n ingress-nginx get svc\n\nNAME TYPE CLUSTER-IP PORT(S)\n\n\ningress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP\n\n\n\n\n\nOne could set the following external IPs in the Service spec, and NGINX would become available on both the NodePort\nand the Service port:\n\n\nspec\n:\n\n \nexternalIPs\n:\n\n \n-\n \n203.0.113.2\n\n \n-\n \n203.0.113.3\n\n\n\n\n\n$\n curl -D- http://myapp.example.com:30100\n\nHTTP/1.1 200 OK\n\n\nServer: nginx/1.15.2\n\n\n\n$\n curl -D- http://myapp.example.com\n\nHTTP/1.1 200 OK\n\n\nServer: nginx/1.15.2\n\n\n\n\n\nWe assume the myapp.example.com subdomain above resolves to both 203.0.113.2 and 203.0.113.3 IP addresses.", "title": "Bare-metal considerations" }, { @@ -115,9 +115,14 @@ "text": "In traditional cloud environments, where network load balancers are available on-demand, a single Kubernetes manifest\nsuffices to provide a single point of contact to the NGINX Ingress controller to external clients and, indirectly, to\nany application running inside the cluster. Bare-metal environments lack this commodity, requiring a slightly\ndifferent setup to offer the same kind of access to external consumers. The rest of this document describes a few recommended approaches to deploying the NGINX Ingress controller inside a\nKubernetes cluster running on bare-metal.", "title": "Bare-metal considerations" }, + { + "location": "/deploy/baremetal/#a-pure-software-solution-metallb", + "text": "MetalLB provides a network load-balancer implementation for Kubernetes clusters that do not run on a\nsupported cloud provider, effectively allowing the usage of LoadBalancer Services within any cluster. This section demonstrates how to use the Layer 2 configuration mode of MetalLB together with the NGINX\nIngress controller in a Kubernetes cluster that has publicly accessible nodes . In this mode, one node attracts all\nthe traffic for the ingress-nginx Service IP. See Traffic policies for more details. Note The description of other supported configuration modes is off-scope for this document. Warning MetalLB is currently in beta . Read about the Project maturity and make sure you inform\nyourself by reading the official documentation thoroughly. MetalLB can be deployed either with a simple Kubernetes manifest or with Helm. The rest of this example assumes MetalLB\nwas deployed following the Installation instructions. MetalLB requires a pool of IP addresses in order to be able to take ownership of the ingress-nginx Service. This pool\ncan be defined in a ConfigMap named config located in the same namespace as the MetalLB controller. In the simplest\npossible scenario, the pool is composed of the IP addresses of Kubernetes nodes, but IP addresses can also be handed out\nby a DHCP server. Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal\nenvironments this value is ) $ kubectl describe node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 After creating the following ConfigMap, MetalLB takes ownership of one of the IP addresses in the pool and updates\nthe loadBalancer IP field of the ingress-nginx Service accordingly. apiVersion : v1 kind : ConfigMap metadata : \n namespace : metallb-system \n name : config data : \n config : | \n address-pools: \n - name: default \n protocol: layer2 \n addresses: \n - 203.0.113.2-203.0.113.3 $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 80/TCP ingress-nginx LoadBalancer 10.0.220.217 203.0.113.3 80:30100/TCP,443:30101/TCP As soon as MetalLB sets the external IP address of the ingress-nginx LoadBalancer Service, the corresponding entries\nare created in the iptables NAT table and the node with the selected IP address starts responding to HTTP requests on\nthe ports configured in the LoadBalancer Service: $ curl -D- http://203.0.113.3 -H 'Host: myapp.example.com' HTTP/1.1 200 OK Server: nginx/1.15.2 Tip In order to preserve the source IP address in HTTP requests sent to NGINX, it is necessary to use the Local \ntraffic policy. Traffic policies are described in more details in Traffic policies as\nwell as in the next section.", + "title": "A pure software solution: MetalLB" + }, { "location": "/deploy/baremetal/#over-a-nodeport-service", - "text": "Due to its simplicity, this is the setup a user will deploy by default when following the steps described in the installation guide . Info A Service of type NodePort exposes, via the kube-proxy component, the same unprivileged port (default:\n30000-32767) on every Kubernetes node, masters included. For more information, see Services . In this configuration, the NGINX container remains isolated from the host network. As a result, it can safely bind to\nany port, including the standard HTTP ports 80 and 443. However, due to the container namespace isolation, a client\nlocated outside the cluster network (e.g. on the public internet) is not able to access Ingress hosts directly on ports\n80 and 443. Instead, the external client must append the NodePort allocated to the ingress-nginx Service to HTTP\nrequests. Example Given the NodePort 30100 allocated to the ingress-nginx Service $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 80/TCP ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP and a Kubernetes node with the public IP address 203.0.113.2 (the external IP is added as an example, in most\nbare-metal environments this value is ) $ kubectl describe node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 a client would reach an Ingress with host : myapp . example . com at http://myapp.example.com:30100 , where the\nmyapp.example.com subdomain resolves to the 203.0.113.2 IP address. Impact on the host system While it may sound tempting to reconfigure the NodePort range using the --service-node-port-range API server flag\nto include unprivileged ports and be able to expose ports 80 and 443, doing so may result in unexpected issues\nincluding (but not limited to) the use of ports otherwise reserved to system daemons and the necessity to grant kube-proxy privileges it may otherwise not require. This practice is therefore discouraged . See the other approaches proposed in this page for alternatives. This approach has a few other limitations one ought to be aware of: Source IP address Services of type NodePort perform source address translation by default. This means the source IP of a\nHTTP request is always the IP address of the Kubernetes node that received the request from the perspective of\nNGINX. The recommended way to preserve the source IP in a NodePort setup is to set the value of the externalTrafficPolicy \nfield of the ingress-nginx Service spec to Local ( example ). Warning This setting effectively drops packets sent to Kubernetes nodes which are not running any instance of the NGINX\nIngress controller. Consider assigning NGINX Pods to specific nodes in order to control on what nodes\nthe NGINX Ingress controller should be scheduled or not scheduled. Example In a Kubernetes cluster composed of 3 nodes (the external IP is added as an example, in most bare-metal environments\nthis value is ) $ kubectl describe node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 with a nginx-ingress-controller Deployment composed of 2 replicas $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 nginx-ingress-controller-cf9ff8c96-8vvf8 1/1 Running 172.17.0.3 host-3 nginx-ingress-controller-cf9ff8c96-pxsds 1/1 Running 172.17.1.4 host-2 Requests sent to host-2 and host-3 would be forwarded to NGINX and original client's IP would be preserved,\nwhile requests to host-1 would get dropped because there is no NGINX replica running on that node. Ingress status Because NodePort Services do not get a LoadBalancerIP assigned by definition, the NGINX Ingress controller does not\nupdate the status of Ingress objects it manages . $ kubectl get ingress NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 80 Despite the fact there is no load balancer providing a public IP address to the NGINX Ingress controller, it is possible\nto force the status update of all managed Ingress objects by setting the externalIPs field of the ingress-nginx \nService. Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal\nenvironments this value is ) $ kubectl describe node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 one could edit the ingress-nginx Service and add the following field to the object spec spec : \n externalIPs : \n - 203.0.113.1 \n - 203.0.113.2 \n - 203.0.113.3 which would in turn be reflected on Ingress objects as follows: $ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.1,203.0.113.2,203.0.113.3 80 Redirects As NGINX is not aware of the port translation operated by the NodePort Service , backend applications are responsible\nfor generating redirect URLs that take into account the URL used by external clients, including the NodePort. Example Redirects generated by NGINX, for instance HTTP to HTTPS or domain to www.domain , are generated without\nNodePort: $ curl http://myapp.example.com:30100 ` HTTP/1.1 308 Permanent Redirect Server: nginx/1.15.2 Location: https://myapp.example.com/ #-> missing NodePort in HTTPS redirect", + "text": "Due to its simplicity, this is the setup a user will deploy by default when following the steps described in the installation guide . Info A Service of type NodePort exposes, via the kube-proxy component, the same unprivileged port (default:\n30000-32767) on every Kubernetes node, masters included. For more information, see Services . In this configuration, the NGINX container remains isolated from the host network. As a result, it can safely bind to\nany port, including the standard HTTP ports 80 and 443. However, due to the container namespace isolation, a client\nlocated outside the cluster network (e.g. on the public internet) is not able to access Ingress hosts directly on ports\n80 and 443. Instead, the external client must append the NodePort allocated to the ingress-nginx Service to HTTP\nrequests. Example Given the NodePort 30100 allocated to the ingress-nginx Service $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) default-http-backend ClusterIP 10.0.64.249 80/TCP ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP and a Kubernetes node with the public IP address 203.0.113.2 (the external IP is added as an example, in most\nbare-metal environments this value is ) $ kubectl describe node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 a client would reach an Ingress with host : myapp . example . com at http://myapp.example.com:30100 , where the\nmyapp.example.com subdomain resolves to the 203.0.113.2 IP address. Impact on the host system While it may sound tempting to reconfigure the NodePort range using the --service-node-port-range API server flag\nto include unprivileged ports and be able to expose ports 80 and 443, doing so may result in unexpected issues\nincluding (but not limited to) the use of ports otherwise reserved to system daemons and the necessity to grant kube-proxy privileges it may otherwise not require. This practice is therefore discouraged . See the other approaches proposed in this page for alternatives. This approach has a few other limitations one ought to be aware of: Source IP address Services of type NodePort perform source address translation by default. This means the source IP of a\nHTTP request is always the IP address of the Kubernetes node that received the request from the perspective of\nNGINX. The recommended way to preserve the source IP in a NodePort setup is to set the value of the externalTrafficPolicy \nfield of the ingress-nginx Service spec to Local ( example ). Warning This setting effectively drops packets sent to Kubernetes nodes which are not running any instance of the NGINX\nIngress controller. Consider assigning NGINX Pods to specific nodes in order to control on what nodes\nthe NGINX Ingress controller should be scheduled or not scheduled. Example In a Kubernetes cluster composed of 3 nodes (the external IP is added as an example, in most bare-metal environments\nthis value is ) $ kubectl describe node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 with a nginx-ingress-controller Deployment composed of 2 replicas $ kubectl -n ingress-nginx get pod -o wide NAME READY STATUS IP NODE default-http-backend-7c5bc89cc9-p86md 1/1 Running 172.17.1.1 host-2 nginx-ingress-controller-cf9ff8c96-8vvf8 1/1 Running 172.17.0.3 host-3 nginx-ingress-controller-cf9ff8c96-pxsds 1/1 Running 172.17.1.4 host-2 Requests sent to host-2 and host-3 would be forwarded to NGINX and original client's IP would be preserved,\nwhile requests to host-1 would get dropped because there is no NGINX replica running on that node. Ingress status Because NodePort Services do not get a LoadBalancerIP assigned by definition, the NGINX Ingress controller does not\nupdate the status of Ingress objects it manages . $ kubectl get ingress NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 80 Despite the fact there is no load balancer providing a public IP address to the NGINX Ingress controller, it is possible\nto force the status update of all managed Ingress objects by setting the externalIPs field of the ingress-nginx \nService. Warning There is more to setting externalIPs than just enabling the NGINX Ingress controller to update the status of\nIngress objects. Please read about this option in the Services page of official Kubernetes\ndocumentation as well as the section about External IPs in this document for more information. Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal\nenvironments this value is ) $ kubectl describe node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 one could edit the ingress-nginx Service and add the following field to the object spec spec : \n externalIPs : \n - 203.0.113.1 \n - 203.0.113.2 \n - 203.0.113.3 which would in turn be reflected on Ingress objects as follows: $ kubectl get ingress -o wide NAME HOSTS ADDRESS PORTS test-ingress myapp.example.com 203.0.113.1,203.0.113.2,203.0.113.3 80 Redirects As NGINX is not aware of the port translation operated by the NodePort Service , backend applications are responsible\nfor generating redirect URLs that take into account the URL used by external clients, including the NodePort. Example Redirects generated by NGINX, for instance HTTP to HTTPS or domain to www.domain , are generated without\nNodePort: $ curl -D- http://myapp.example.com:30100 ` HTTP/1.1 308 Permanent Redirect Server: nginx/1.15.2 Location: https://myapp.example.com/ #-> missing NodePort in HTTPS redirect", "title": "Over a NodePort Service" }, { @@ -130,6 +135,11 @@ "text": "Similarly to cloud environments, this deployment approach requires an edge network component providing a public\nentrypoint to the Kubernetes cluster. This edge component can be either hardware (e.g. vendor appliance) or software\n(e.g. HAproxy ) and is usually managed outside of the Kubernetes landscape by operations teams. Such deployment builds upon the NodePort Service described above in Over a NodePort Service ,\nwith one significant difference: external clients do not access cluster nodes directly, only the edge component does.\nThis is particularly suitable for private Kubernetes clusters where none of the nodes has a public IP address. On the edge side, the only prerequisite is to dedicate a public IP address that forwards all HTTP traffic to Kubernetes\nnodes and/or masters. Incoming traffic on TCP ports 80 and 443 is forwarded to the corresponding HTTP and HTTPS NodePort\non the target nodes as shown in the diagram below:", "title": "Using a self-provisioned edge" }, + { + "location": "/deploy/baremetal/#external-ips", + "text": "Source IP address This method does not allow preserving the source IP of HTTP requests in any manner, it is therefore not\nrecommended to use it despite its apparent simplicity. The externalIPs Service option was previously mentioned in the NodePort section. As per the Services page of the official Kubernetes documentation, the externalIPs option causes kube-proxy to route traffic sent to arbitrary IP addresses and on the Service ports to the endpoints of that\nService. These IP addresses must belong to the target node . Example Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal\nenvironments this value is ) $ kubectl describe node NAME STATUS ROLES EXTERNAL-IP host-1 Ready master 203.0.113.1 host-2 Ready node 203.0.113.2 host-3 Ready node 203.0.113.3 and the following ingress-nginx NodePort Service $ kubectl -n ingress-nginx get svc NAME TYPE CLUSTER-IP PORT(S) ingress-nginx NodePort 10.0.220.217 80:30100/TCP,443:30101/TCP One could set the following external IPs in the Service spec, and NGINX would become available on both the NodePort\nand the Service port: spec : \n externalIPs : \n - 203.0.113.2 \n - 203.0.113.3 $ curl -D- http://myapp.example.com:30100 HTTP/1.1 200 OK Server: nginx/1.15.2 $ curl -D- http://myapp.example.com HTTP/1.1 200 OK Server: nginx/1.15.2 We assume the myapp.example.com subdomain above resolves to both 203.0.113.2 and 203.0.113.3 IP addresses.", + "title": "External IPs" + }, { "location": "/deploy/rbac/", "text": "Role Based Access Control (RBAC)\n\u00b6\n\n\nOverview\n\u00b6\n\n\nThis example applies to nginx-ingress-controllers being deployed in an environment with RBAC enabled.\n\n\nRole Based Access Control is comprised of four layers:\n\n\n\n\nClusterRole\n - permissions assigned to a role that apply to an entire cluster\n\n\nClusterRoleBinding\n - binding a ClusterRole to a specific account\n\n\nRole\n - permissions assigned to a role that apply to a specific namespace\n\n\nRoleBinding\n - binding a Role to a specific account\n\n\n\n\nIn order for RBAC to be applied to an nginx-ingress-controller, that controller\nshould be assigned to a \nServiceAccount\n. That \nServiceAccount\n should be\nbound to the \nRole\ns and \nClusterRole\ns defined for the nginx-ingress-controller.\n\n\nService Accounts created in this example\n\u00b6\n\n\nOne ServiceAccount is created in this example, \nnginx-ingress-serviceaccount\n.\n\n\nPermissions Granted in this example\n\u00b6\n\n\nThere are two sets of permissions defined in this example. Cluster-wide\npermissions defined by the \nClusterRole\n named \nnginx-ingress-clusterrole\n, and\nnamespace specific permissions defined by the \nRole\n named \nnginx-ingress-role\n.\n\n\nCluster Permissions\n\u00b6\n\n\nThese permissions are granted in order for the nginx-ingress-controller to be\nable to function as an ingress across the cluster. These permissions are\ngranted to the ClusterRole named \nnginx-ingress-clusterrole\n\n\n\n\nconfigmaps\n, \nendpoints\n, \nnodes\n, \npods\n, \nsecrets\n: list, watch\n\n\nnodes\n: get\n\n\nservices\n, \ningresses\n: get, list, watch\n\n\nevents\n: create, patch\n\n\ningresses/status\n: update\n\n\n\n\nNamespace Permissions\n\u00b6\n\n\nThese permissions are granted specific to the nginx-ingress namespace. These\npermissions are granted to the Role named \nnginx-ingress-role\n\n\n\n\nconfigmaps\n, \npods\n, \nsecrets\n: get\n\n\nendpoints\n: get\n\n\n\n\nFurthermore to support leader-election, the nginx-ingress-controller needs to\nhave access to a \nconfigmap\n using the resourceName \ningress-controller-leader-nginx\n\n\n\n\nNote that resourceNames can NOT be used to limit requests using the \u201ccreate\u201d\nverb because authorizers only have access to information that can be obtained\nfrom the request URL, method, and headers (resource names in a \u201ccreate\u201d request\nare part of the request body).\n\n\n\n\n\n\nconfigmaps\n: get, update (for resourceName \ningress-controller-leader-nginx\n)\n\n\nconfigmaps\n: create\n\n\n\n\nThis resourceName is the concatenation of the \nelection-id\n and the\n\ningress-class\n as defined by the ingress-controller, which defaults to:\n\n\n\n\nelection-id\n: \ningress-controller-leader\n\n\ningress-class\n: \nnginx\n\n\nresourceName\n : \n - \n\n\n\n\nPlease adapt accordingly if you overwrite either parameter when launching the\nnginx-ingress-controller.\n\n\nBindings\n\u00b6\n\n\nThe ServiceAccount \nnginx-ingress-serviceaccount\n is bound to the Role\n\nnginx-ingress-role\n and the ClusterRole \nnginx-ingress-clusterrole\n.\n\n\nThe serviceAccountName associated with the containers in the deployment must\nmatch the serviceAccount. The namespace references in the Deployment metadata, \ncontainer arguments, and POD_NAMESPACE should be in the nginx-ingress namespace.", @@ -202,7 +212,7 @@ }, { "location": "/user-guide/nginx-configuration/annotations/", - "text": "Annotations\n\u00b6\n\n\nYou can add these Kubernetes annotations to specific Ingress objects to customize their behavior.\n\n\n\n\nTip\n\n\nAnnotation keys and values can only be strings.\nOther types, such as boolean or numeric values must be quoted,\ni.e. \n\"true\"\n, \n\"false\"\n, \n\"100\"\n.\n\n\n\n\n\n\nNote\n\n\nThe annotation prefix can be changed using the\n\n--annotations-prefix\n command line argument\n,\nbut the default is \nnginx.ingress.kubernetes.io\n, as described in the\ntable below.\n\n\n\n\n\n\n\n\n\n\nName\n\n\ntype\n\n\n\n\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/add-base-url\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/app-root\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/affinity\n\n\ncookie\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-realm\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-secret\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-type\n\n\nbasic or digest\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-tls-secret\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-tls-verify-depth\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-tls-verify-client\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-tls-error-page\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-url\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/backend-protocol\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/base-url-scheme\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/client-body-buffer-size\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/configuration-snippet\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/default-backend\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/enable-cors\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-allow-origin\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-allow-methods\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-allow-headers\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-allow-credentials\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-max-age\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/force-ssl-redirect\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/from-to-www-redirect\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/grpc-backend\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/limit-connections\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/limit-rps\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/permanent-redirect\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/permanent-redirect-code\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-body-size\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-cookie-domain\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-connect-timeout\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-send-timeout\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-read-timeout\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-next-upstream\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-next-upstream-tries\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-request-buffering\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-redirect-from\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-redirect-to\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/rewrite-log\n\n\nURI\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/rewrite-target\n\n\nURI\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/secure-backends\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/secure-verify-ca-secret\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/server-alias\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/server-snippet\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/service-upstream\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/session-cookie-name\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/session-cookie-hash\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/ssl-redirect\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/ssl-passthrough\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/upstream-max-fails\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/upstream-fail-timeout\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/upstream-hash-by\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/load-balance\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/upstream-vhost\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/whitelist-source-range\n\n\nCIDR\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-buffering\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-buffer-size\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/ssl-ciphers\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/connection-proxy-header\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/enable-access-log\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/lua-resty-waf\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/lua-resty-waf-debug\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/lua-resty-waf-ignore-rulesets\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/lua-resty-waf-extra-rules\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/enable-influxdb\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/influxdb-measurement\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/influxdb-port\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/influxdb-host\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/influxdb-server-name\n\n\nstring\n\n\n\n\n\n\n\n\nRewrite\n\u00b6\n\n\nIn some scenarios the exposed URL in the backend service differs from the specified path in the Ingress rule. Without a rewrite any request will return 404.\nSet the annotation \nnginx.ingress.kubernetes.io/rewrite-target\n to the path expected by the service.\n\n\nIf the application contains relative links it is possible to add an additional annotation \nnginx.ingress.kubernetes.io/add-base-url\n that will prepend a \nbase\n tag\n in the header of the returned HTML from the backend.\n\n\nIf the scheme of \nbase\n tag\n need to be specific, set the annotation \nnginx.ingress.kubernetes.io/base-url-scheme\n to the scheme such as \nhttp\n and \nhttps\n.\n\n\nIf the Application Root is exposed in a different path and needs to be redirected, set the annotation \nnginx.ingress.kubernetes.io/app-root\n to redirect requests for \n/\n.\n\n\n\n\nExample\n\n\nPlease check the \nrewrite\n example.\n\n\n\n\nSession Affinity\n\u00b6\n\n\nThe annotation \nnginx.ingress.kubernetes.io/affinity\n enables and sets the affinity type in all Upstreams of an Ingress. This way, a request will always be directed to the same upstream server.\nThe only affinity type available for NGINX is \ncookie\n.\n\n\n\n\nExample\n\n\nPlease check the \naffinity\n example.\n\n\n\n\nCookie affinity\n\u00b6\n\n\nIf you use the \ncookie\n affinity type you can also specify the name of the cookie that will be used to route the requests with the annotation \nnginx.ingress.kubernetes.io/session-cookie-name\n. The default is to create a cookie named 'INGRESSCOOKIE'.\n\n\nIn case of NGINX the annotation \nnginx.ingress.kubernetes.io/session-cookie-hash\n defines which algorithm will be used to hash the used upstream. Default value is \nmd5\n and possible values are \nmd5\n, \nsha1\n and \nindex\n.\n\n\n\n\nAttention\n\n\nThe \nindex\n option is not an actual hash; an in-memory index is used instead, which has less overhead.\nHowever, with \nindex\n, matching against a changing upstream server list is inconsistent.\nSo, at reload, if upstream servers have changed, index values are not guaranteed to correspond to the same server as before!\n\nUse \nindex\n with caution\n and only if you need to!\n\n\n\n\nIn NGINX this feature is implemented by the third party module \nnginx-sticky-module-ng\n. The workflow used to define which upstream server will be used is explained \nhere\n\n\nAuthentication\n\u00b6\n\n\nIs possible to add authentication adding additional annotations in the Ingress rule. The source of the authentication is a secret that contains usernames and passwords inside the key \nauth\n.\n\n\nThe annotations are:\n\nnginx.ingress.kubernetes.io/auth-type: [basic|digest]\n\n\n\nIndicates the \nHTTP Authentication Type: Basic or Digest Access Authentication\n.\n\n\nnginx.ingress.kubernetes.io/auth-secret: secretName\n\n\n\n\nThe name of the Secret that contains the usernames and passwords which are granted access to the \npath\ns defined in the Ingress rules.\nThis annotation also accepts the alternative form \"namespace/secretName\", in which case the Secret lookup is performed in the referenced namespace instead of the Ingress namespace.\n\n\nnginx.ingress.kubernetes.io/auth-realm: \"realm string\"\n\n\n\n\n\n\nExample\n\n\nPlease check the \nauth\n example.\n\n\n\n\nCustom NGINX upstream checks\n\u00b6\n\n\nNGINX exposes some flags in the \nupstream configuration\n that enable the configuration of each server in the upstream. The Ingress controller allows custom \nmax_fails\n and \nfail_timeout\n parameters in a global context using \nupstream-max-fails\n and \nupstream-fail-timeout\n in the NGINX ConfigMap or in a particular Ingress rule. \nupstream-max-fails\n defaults to 0. This means NGINX will respect the container's \nreadinessProbe\n if it is defined. If there is no probe and no values for \nupstream-max-fails\n NGINX will continue to send traffic to the container.\n\n\n\n\nTip\n\n\nWith the default configuration NGINX will not health check your backends. Whenever the endpoints controller notices a readiness probe failure, that pod's IP will be removed from the list of endpoints. This will trigger the NGINX controller to also remove it from the upstreams.**\n\n\n\n\nTo use custom values in an Ingress rule define these annotations:\n\n\nnginx.ingress.kubernetes.io/upstream-max-fails\n: number of unsuccessful attempts to communicate with the server that should occur in the duration set by the \nupstream-fail-timeout\n parameter to consider the server unavailable.\n\n\nnginx.ingress.kubernetes.io/upstream-fail-timeout\n: time in seconds during which the specified number of unsuccessful attempts to communicate with the server should occur to consider the server unavailable. This is also the period of time the server will be considered unavailable.\n\n\nIn NGINX, backend server pools are called \"\nupstreams\n\". Each upstream contains the endpoints for a service. An upstream is created for each service that has Ingress rules defined.\n\n\n\n\nAttention\n\n\nAll Ingress rules using the same service will use the same upstream.\n\nOnly one of the Ingress rules should define annotations to configure the upstream servers.\n\n\n\n\n\n\nExample\n\n\nPlease check the \ncustom upstream check\n example.\n\n\n\n\nCustom NGINX upstream hashing\n\u00b6\n\n\nNGINX supports load balancing by client-server mapping based on \nconsistent hashing\n for a given key. The key can contain text, variables or any combination thereof. This feature allows for request stickiness other than client IP or cookies. The \nketama\n consistent hashing method will be used which ensures only a few keys would be remapped to different servers on upstream group changes.\n\n\nTo enable consistent hashing for a backend:\n\n\nnginx.ingress.kubernetes.io/upstream-hash-by\n: the nginx variable, text value or any combination thereof to use for consistent hashing. For example \nnginx.ingress.kubernetes.io/upstream-hash-by: \"$request_uri\"\n to consistently hash upstream requests by the current request URI.\n\n\nCustom NGINX load balancing\n\u00b6\n\n\nThis is similar to (https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/configmap.md#load-balance) but configures load balancing algorithm per ingress.\n\n\n\n\nNote that \nnginx.ingress.kubernetes.io/upstream-hash-by\n takes preference over this. If this and \nnginx.ingress.kubernetes.io/upstream-hash-by\n are not set then we fallback to using globally configured load balancing algorithm.\n\n\n\n\nCustom NGINX upstream vhost\n\u00b6\n\n\nThis configuration setting allows you to control the value for host in the following statement: \nproxy_set_header Host $host\n, which forms part of the location block. This is useful if you need to call the upstream server by something other than \n$host\n.\n\n\nClient Certificate Authentication\n\u00b6\n\n\nIt is possible to enable Client Certificate Authentication using additional annotations in Ingress Rule.\n\n\nThe annotations are:\n\n\n\n\nnginx.ingress.kubernetes.io/auth-tls-secret: secretName\n:\n The name of the Secret that contains the full Certificate Authority chain \nca.crt\n that is enabled to authenticate against this Ingress.\n This annotation also accepts the alternative form \"namespace/secretName\", in which case the Secret lookup is performed in the referenced namespace instead of the Ingress namespace.\n\n\nnginx.ingress.kubernetes.io/auth-tls-verify-depth\n:\n The validation depth between the provided client certificate and the Certification Authority chain.\n\n\nnginx.ingress.kubernetes.io/auth-tls-verify-client\n:\n Enables verification of client certificates.\n\n\nnginx.ingress.kubernetes.io/auth-tls-error-page\n:\n The URL/Page that user should be redirected in case of a Certificate Authentication Error\n\n\nnginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream\n:\n Indicates if the received certificates should be passed or not to the upstream server. By default this is disabled.\n\n\n\n\n\n\nExample\n\n\nPlease check the \nclient-certs\n example.\n\n\n\n\n\n\nAttention\n\n\nTLS with Client Authentication is \nnot\n possible in Cloudflare and might result in unexpected behavior.\n\n\nCloudflare only allows Authenticated Origin Pulls and is required to use their own certificate: \nhttps://blog.cloudflare.com/protecting-the-origin-with-tls-authenticated-origin-pulls/\n\n\nOnly Authenticated Origin Pulls are allowed and can be configured by following their tutorial: \nhttps://support.cloudflare.com/hc/en-us/articles/204494148-Setting-up-NGINX-to-use-TLS-Authenticated-Origin-Pulls\n\n\n\n\nConfiguration snippet\n\u00b6\n\n\nUsing this annotation you can add additional configuration to the NGINX location. For example:\n\n\nnginx.ingress.kubernetes.io/configuration-snippet\n:\n \n|\n\n \nmore_set_headers \"Request-Id: $req_id\";\n\n\n\n\n\nDefault Backend\n\u00b6\n\n\nThe ingress controller requires a \ndefault backend\n.\nThis service handles the response when the service in the Ingress rule does not have endpoints.\nThis is a global configuration for the ingress controller. In some cases could be required to return a custom content or format. In this scenario we can use the annotation \nnginx.ingress.kubernetes.io/default-backend: \n to specify a custom default backend.\n\n\nEnable CORS\n\u00b6\n\n\nTo enable Cross-Origin Resource Sharing (CORS) in an Ingress rule, add the annotation\n\nnginx.ingress.kubernetes.io/enable-cors: \"true\"\n. This will add a section in the server\nlocation enabling this functionality.\n\n\nCORS can be controlled with the following annotations:\n\n\n\n\nnginx.ingress.kubernetes.io/cors-allow-methods\n\n controls which methods are accepted. This is a multi-valued field, separated by ',' and\n accepts only letters (upper and lower case).\n\n\nDefault: \nGET, PUT, POST, DELETE, PATCH, OPTIONS\n\n\n\n\nExample: \nnginx.ingress.kubernetes.io/cors-allow-methods: \"PUT, GET, POST, OPTIONS\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-allow-headers\n\n controls which headers are accepted. This is a multi-valued field, separated by ',' and accepts letters,\n numbers, _ and -.\n\n\n\n\nDefault: \nDNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization\n\n\n\n\nExample: \nnginx.ingress.kubernetes.io/cors-allow-headers: \"X-Forwarded-For, X-app123-XPTO\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-allow-origin\n\n controls what's the accepted Origin for CORS.\n This is a single field value, with the following format: \nhttp(s)://origin-site.com\n or \nhttp(s)://origin-site.com:port\n\n\n\n\nDefault: \n*\n\n\n\n\nExample: \nnginx.ingress.kubernetes.io/cors-allow-origin: \"https://origin-site.com:4443\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-allow-credentials\n\n controls if credentials can be passed during CORS operations.\n\n\n\n\nDefault: \ntrue\n\n\n\n\nExample: \nnginx.ingress.kubernetes.io/cors-allow-credentials: \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-max-age\n\n controls how long preflight requests can be cached.\n Default: \n1728000\n\n Example: \nnginx.ingress.kubernetes.io/cors-max-age: 600\n\n\n\n\n\n\n\n\nNote\n\n\nFor more information please see \nhttps://enable-cors.org\n \n\n\n\n\nServer Alias\n\u00b6\n\n\nTo add Server Aliases to an Ingress rule add the annotation \nnginx.ingress.kubernetes.io/server-alias: \" \"\n.\nThis will create a server with the same configuration, but a different \nserver_name\n as the provided host.\n\n\n\n\nNote\n\n\nA server-alias name cannot conflict with the hostname of an existing server. If it does the server-alias annotation will be ignored.\nIf a server-alias is created and later a new server with the same hostname is created,\nthe new server configuration will take place over the alias configuration.\n\n\n\n\nFor more information please see \nthe \nserver_name\n documentation\n.\n\n\nServer snippet\n\u00b6\n\n\nUsing the annotation \nnginx.ingress.kubernetes.io/server-snippet\n it is possible to add custom configuration in the server configuration block.\n\n\napiVersion\n:\n \nextensions/v1beta1\n\n\nkind\n:\n \nIngress\n\n\nmetadata\n:\n\n \nannotations\n:\n\n \nnginx.ingress.kubernetes.io/server-snippet\n:\n \n|\n\n\nset $agentflag 0;\n\n\n\nif ($http_user_agent ~* \"(Mobile)\" ){\n\n \nset $agentflag 1;\n\n\n}\n\n\n\nif ( $agentflag = 1 ) {\n\n \nreturn 301 https://m.example.com;\n\n\n}\n\n\n\n\n\n\n\nAttention\n\n\nThis annotation can be used only once per host.\n\n\n\n\nClient Body Buffer Size\n\u00b6\n\n\nSets buffer size for reading client request body per location. In case the request body is larger than the buffer,\nthe whole body or only its part is written to a temporary file. By default, buffer size is equal to two memory pages.\nThis is 8K on x86, other 32-bit platforms, and x86-64. It is usually 16K on other 64-bit platforms. This annotation is\napplied to each location provided in the ingress rule.\n\n\n\n\nNote\n\n\nThe annotation value must be given in a format understood by Nginx.\n\n\n\n\n\n\nExample\n\n\n\n\nnginx.ingress.kubernetes.io/client-body-buffer-size: \"1000\"\n # 1000 bytes\n\n\nnginx.ingress.kubernetes.io/client-body-buffer-size: 1k\n # 1 kilobyte\n\n\nnginx.ingress.kubernetes.io/client-body-buffer-size: 1K\n # 1 kilobyte\n\n\nnginx.ingress.kubernetes.io/client-body-buffer-size: 1m\n # 1 megabyte\n\n\nnginx.ingress.kubernetes.io/client-body-buffer-size: 1M\n # 1 megabyte\n\n\n\n\n\n\nFor more information please see \nhttp://nginx.org\n\n\nExternal Authentication\n\u00b6\n\n\nTo use an existing service that provides authentication the Ingress rule can be annotated with \nnginx.ingress.kubernetes.io/auth-url\n to indicate the URL where the HTTP request should be sent.\n\n\nnginx.ingress.kubernetes.io/auth-url\n:\n \n\"URL\n \nto\n \nthe\n \nauthentication\n \nservice\"\n\n\n\n\n\nAdditionally it is possible to set:\n\n\n\n\nnginx.ingress.kubernetes.io/auth-method\n:\n \n \n to specify the HTTP method to use.\n\n\nnginx.ingress.kubernetes.io/auth-signin\n:\n \n \n to specify the location of the error page.\n\n\nnginx.ingress.kubernetes.io/auth-response-headers\n:\n \n \n to specify headers to pass to backend once authentication request completes.\n\n\nnginx.ingress.kubernetes.io/auth-request-redirect\n:\n \n \n to specify the X-Auth-Request-Redirect header value.\n\n\n\n\n\n\nExample\n\n\nPlease check the \nexternal-auth\n example.\n\n\n\n\nRate limiting\n\u00b6\n\n\nThese annotations define a limit on the connections that can be opened by a single client IP address.\nThis can be used to mitigate \nDDoS Attacks\n.\n\n\n\n\nnginx.ingress.kubernetes.io/limit-connections\n: number of concurrent connections allowed from a single IP address.\n\n\nnginx.ingress.kubernetes.io/limit-rps\n: number of connections that may be accepted from a given IP each second.\n\n\nnginx.ingress.kubernetes.io/limit-rpm\n: number of connections that may be accepted from a given IP each minute.\n\n\nnginx.ingress.kubernetes.io/limit-rate-after\n: sets the initial amount after which the further transmission of a response to a client will be rate limited.\n\n\nnginx.ingress.kubernetes.io/limit-rate\n: rate of request that accepted from a client each second.\n\n\n\n\nYou can specify the client IP source ranges to be excluded from rate-limiting through the \nnginx.ingress.kubernetes.io/limit-whitelist\n annotation. The value is a comma separated list of CIDRs.\n\n\nIf you specify multiple annotations in a single Ingress rule, \nlimit-rpm\n, and then \nlimit-rps\n takes precedence.\n\n\nThe annotation \nnginx.ingress.kubernetes.io/limit-rate\n, \nnginx.ingress.kubernetes.io/limit-rate-after\n define a limit the rate of response transmission to a client. The rate is specified in bytes per second. The zero value disables rate limiting. The limit is set per a request, and so if a client simultaneously opens two connections, the overall rate will be twice as much as the specified limit.\n\n\nTo configure this setting globally for all Ingress rules, the \nlimit-rate-after\n and \nlimit-rate\n value may be set in the \nNGINX ConfigMap\n. if you set the value in ingress annotation will cover global setting.\n\n\nPermanent Redirect\n\u00b6\n\n\nThis annotation allows to return a permanent redirect instead of sending data to the upstream. For example \nnginx.ingress.kubernetes.io/permanent-redirect: https://www.google.com\n would redirect everything to Google.\n\n\nPermanent Redirect Code\n\u00b6\n\n\nThis annotation allows you to modify the status code used for permanent redirects. For example \nnginx.ingress.kubernetes.io/permanent-redirect-code: '308'\n would return your permanent-redirect with a 308.\n\n\nSSL Passthrough\n\u00b6\n\n\nThe annotation \nnginx.ingress.kubernetes.io/ssl-passthrough\n allows to configure TLS termination in the pod and not in NGINX.\n\n\n\n\nAttention\n\n\nUsing the annotation \nnginx.ingress.kubernetes.io/ssl-passthrough\n invalidates all the other available annotations.\nThis is because SSL Passthrough works on level 4 of the OSI stack (TCP), not on the HTTP/HTTPS level.\n\n\n\n\n\n\nAttention\n\n\nThe use of this annotation requires the flag \n--enable-ssl-passthrough\n (By default it is disabled).\n\n\n\n\nSecure backends DEPRECATED (since 0.18.0)\n\u00b6\n\n\nPlease use \nnginx.ingress.kubernetes.io/backend-protocol: \"HTTPS\"\n\n\nBy default NGINX uses plain HTTP to reach the services.\nAdding the annotation \nnginx.ingress.kubernetes.io/secure-backends: \"true\"\n in the Ingress rule changes the protocol to HTTPS.\nIf you want to validate the upstream against a specific certificate, you can create a secret with it and reference the secret with the annotation \nnginx.ingress.kubernetes.io/secure-verify-ca-secret\n.\n\n\n\n\nAttention\n\n\nNote that if an invalid or non-existent secret is given,\nthe ingress controller will ignore the \nsecure-backends\n annotation.\n\n\n\n\nService Upstream\n\u00b6\n\n\nBy default the NGINX ingress controller uses a list of all endpoints (Pod IP/port) in the NGINX upstream configuration.\n\n\nThe \nnginx.ingress.kubernetes.io/service-upstream\n annotation disables that behavior and instead uses a single upstream in NGINX, the service's Cluster IP and port.\n\n\nThis can be desirable for things like zero-downtime deployments as it reduces the need to reload NGINX configuration when Pods come up and down. See issue \n#257\n.\n\n\nKnown Issues\n\u00b6\n\n\nIf the \nservice-upstream\n annotation is specified the following things should be taken into consideration:\n\n\n\n\nSticky Sessions will not work as only round-robin load balancing is supported.\n\n\nThe \nproxy_next_upstream\n directive will not have any effect meaning on error the request will not be dispatched to another upstream.\n\n\n\n\nServer-side HTTPS enforcement through redirect\n\u00b6\n\n\nBy default the controller redirects (308) to HTTPS if TLS is enabled for that ingress.\nIf you want to disable this behavior globally, you can use \nssl-redirect: \"false\"\n in the NGINX \nconfig map\n.\n\n\nTo configure this feature for specific ingress resources, you can use the \nnginx.ingress.kubernetes.io/ssl-redirect: \"false\"\n\nannotation in the particular resource.\n\n\nWhen using SSL offloading outside of cluster (e.g. AWS ELB) it may be useful to enforce a redirect to HTTPS\neven when there is no TLS certificate available.\nThis can be achieved by using the \nnginx.ingress.kubernetes.io/force-ssl-redirect: \"true\"\n annotation in the particular resource.\n\n\nRedirect from/to www.\n\u00b6\n\n\nIn some scenarios is required to redirect from \nwww.domain.com\n to \ndomain.com\n or vice versa.\nTo enable this feature use the annotation \nnginx.ingress.kubernetes.io/from-to-www-redirect: \"true\"\n\n\n\n\nAttention\n\n\nIf at some point a new Ingress is created with a host equal to one of the options (like \ndomain.com\n) the annotation will be omitted.\n\n\n\n\nWhitelist source range\n\u00b6\n\n\nYou can specify allowed client IP source ranges through the \nnginx.ingress.kubernetes.io/whitelist-source-range\n annotation.\nThe value is a comma separated list of \nCIDRs\n, e.g. \n10.0.0.0/24,172.10.0.1\n.\n\n\nTo configure this setting globally for all Ingress rules, the \nwhitelist-source-range\n value may be set in the \nNGINX ConfigMap\n.\n\n\n\n\nNote\n\n\nAdding an annotation to an Ingress rule overrides any global restriction.\n\n\n\n\nCustom timeouts\n\u00b6\n\n\nUsing the configuration configmap it is possible to set the default global timeout for connections to the upstream servers.\nIn some scenarios is required to have different values. To allow this we provide annotations that allows this customization:\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-connect-timeout\n\n\nnginx.ingress.kubernetes.io/proxy-send-timeout\n\n\nnginx.ingress.kubernetes.io/proxy-read-timeout\n\n\nnginx.ingress.kubernetes.io/proxy-next-upstream\n\n\nnginx.ingress.kubernetes.io/proxy-next-upstream-tries\n\n\nnginx.ingress.kubernetes.io/proxy-request-buffering\n\n\n\n\nProxy redirect\n\u00b6\n\n\nWith the annotations \nnginx.ingress.kubernetes.io/proxy-redirect-from\n and \nnginx.ingress.kubernetes.io/proxy-redirect-to\n it is possible to\nset the text that should be changed in the \nLocation\n and \nRefresh\n header fields of a proxied server response (http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_redirect)\n\n\nSetting \"off\" or \"default\" in the annotation \nnginx.ingress.kubernetes.io/proxy-redirect-from\n disables \nnginx.ingress.kubernetes.io/proxy-redirect-to\n,\notherwise, both annotations must be used in unison. Note that each annotation must be a string without spaces.\n\n\nBy default the value of each annotation is \"off\".\n\n\nCustom max body size\n\u00b6\n\n\nFor NGINX, an 413 error will be returned to the client when the size in a request exceeds the maximum allowed size of the client request body. This size can be configured by the parameter \nclient_max_body_size\n.\n\n\nTo configure this setting globally for all Ingress rules, the \nproxy-body-size\n value may be set in the \nNGINX ConfigMap\n.\nTo use custom values in an Ingress rule define these annotation:\n\n\nnginx.ingress.kubernetes.io/proxy-body-size\n:\n \n8m\n\n\n\n\n\nProxy cookie domain\n\u00b6\n\n\nSets a text that \nshould be changed in the domain attribute\n of the \"Set-Cookie\" header fields of a proxied server response.\n\n\nTo configure this setting globally for all Ingress rules, the \nproxy-cookie-domain\n value may be set in the \nNGINX ConfigMap\n.\n\n\nProxy buffering\n\u00b6\n\n\nEnable or disable proxy buffering \nproxy_buffering\n.\nBy default proxy buffering is disabled in the NGINX config.\n\n\nTo configure this setting globally for all Ingress rules, the \nproxy-buffering\n value may be set in the \nNGINX ConfigMap\n.\nTo use custom values in an Ingress rule define these annotation:\n\n\nnginx.ingress.kubernetes.io/proxy-buffering\n:\n \n\"on\"\n\n\n\n\n\nProxy buffer size\n\u00b6\n\n\nSets the size of the buffer \nproxy_buffer_size\n used for reading the first part of the response received from the proxied server.\nBy default proxy buffer size is set as \"4k\"\n\n\nTo configure this setting globally, set \nproxy-buffer-size\n in \nNGINX ConfigMap\n. To use custom values in an Ingress rule, define this annotation:\n\nnginx.ingress.kubernetes.io/proxy-buffer-size\n:\n \n\"8k\"\n\n\n\n\nSSL ciphers\n\u00b6\n\n\nSpecifies the \nenabled ciphers\n.\n\n\nUsing this annotation will set the \nssl_ciphers\n directive at the server level. This configuration is active for all the paths in the host.\n\n\nnginx.ingress.kubernetes.io/ssl-ciphers\n:\n \n\"ALL:!aNULL:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP\"\n\n\n\n\n\nConnection proxy header\n\u00b6\n\n\nUsing this annotation will override the default connection header set by NGINX.\nTo use custom values in an Ingress rule, define the annotation:\n\n\nnginx.ingress.kubernetes.io/connection-proxy-header\n:\n \n\"keep-alive\"\n\n\n\n\n\nEnable Access Log\n\u00b6\n\n\nAccess logs are enabled by default, but in some scenarios access logs might be required to be disabled for a given\ningress. To do this, use the annotation:\n\n\nnginx.ingress.kubernetes.io/enable-access-log\n:\n \n\"false\"\n\n\n\n\n\nEnable Rewrite Log\n\u00b6\n\n\nRewrite logs are not enabled by default. In some scenarios it could be required to enable NGINX rewrite logs.\nNote that rewrite logs are sent to the error_log file at the notice level. To enable this feature use the annotation:\n\n\nnginx.ingress.kubernetes.io/enable-rewrite-log\n:\n \n\"true\"\n\n\n\n\n\nLua Resty WAF\n\u00b6\n\n\nUsing \nlua-resty-waf-*\n annotations we can enable and control the \nlua-resty-waf\n\nWeb Application Firewall per location.\n\n\nFollowing configuration will enable the WAF for the paths defined in the corresponding ingress:\n\n\nnginx.ingress.kubernetes.io/lua-resty-waf\n:\n \n\"active\"\n\n\n\n\n\nIn order to run it in debugging mode you can set \nnginx.ingress.kubernetes.io/lua-resty-waf-debug\n to \n\"true\"\n in addition to the above configuration.\nThe other possible values for \nnginx.ingress.kubernetes.io/lua-resty-waf\n are \ninactive\n and \nsimulate\n.\nIn \ninactive\n mode WAF won't do anything, whereas in \nsimulate\n mode it will log a warning message if there's a matching WAF rule for given request. This is useful to debug a rule and eliminate possible false positives before fully deploying it.\n\n\nlua-resty-waf\n comes with predefined set of rules \nhttps://github.com/p0pr0ck5/lua-resty-waf/tree/84b4f40362500dd0cb98b9e71b5875cb1a40f1ad/rules\n that covers ModSecurity CRS.\nYou can use \nnginx.ingress.kubernetes.io/lua-resty-waf-ignore-rulesets\n to ignore a subset of those rulesets. For an example:\n\n\nnginx.ingress.kubernetes.io/lua-resty-waf-ignore-rulesets\n:\n \n\"41000_sqli,\n \n42000_xss\"\n\n\n\n\n\nwill ignore the two mentioned rulesets.\n\n\nIt is also possible to configure custom WAF rules per ingress using the \nnginx.ingress.kubernetes.io/lua-resty-waf-extra-rules\n annotation. For an example the following snippet will configure a WAF rule to deny requests with query string value that contains word \nfoo\n:\n\n\nnginx.ingress.kubernetes.io/lua-resty-waf-extra-rules\n:\n \n'[=[\n \n{\n \n\"access\":\n \n[\n \n{\n \n\"actions\":\n \n{\n \n\"disrupt\"\n \n:\n \n\"DENY\"\n \n},\n \n\"id\":\n \n10001,\n \n\"msg\":\n \n\"my\n \ncustom\n \nrule\",\n \n\"operator\":\n \n\"STR_CONTAINS\",\n \n\"pattern\":\n \n\"foo\",\n \n\"vars\":\n \n[\n \n{\n \n\"parse\":\n \n[\n \n\"values\",\n \n1\n \n],\n \n\"type\":\n \n\"REQUEST_ARGS\"\n \n}\n \n]\n \n}\n \n],\n \n\"body_filter\":\n \n[],\n \n\"header_filter\":[]\n \n}\n \n]=]'\n\n\n\n\n\nFor details on how to write WAF rules, please refer to \nhttps://github.com/p0pr0ck5/lua-resty-waf\n.\n\n\ngRPC backend DEPRECATED (since 0.18.0)\n\u00b6\n\n\nPlease use \nnginx.ingress.kubernetes.io/backend-protocol: \"GRPC\"\n or \nnginx.ingress.kubernetes.io/backend-protocol: \"GRPCS\"\n\n\nSince NGINX 1.13.10 it is possible to expose \ngRPC services natively\n\n\nYou only need to add the annotation \nnginx.ingress.kubernetes.io/grpc-backend: \"true\"\n to enable this feature.\nAdditionally, if the gRPC service requires TLS, add \nnginx.ingress.kubernetes.io/secure-backends: \"true\"\n.\n\n\n\n\nAttention\n\n\nThis feature requires HTTP2 to work which means we need to expose this service using HTTPS.\nExposing a gRPC service using HTTP is not supported.\n\n\n\n\nInfluxDB\n\u00b6\n\n\nUsing \ninfluxdb-*\n annotations we can monitor requests passing through a Location by sending them to an InfluxDB backend exposing the UDP socket\nusing the \nnginx-influxdb-module\n.\n\n\nnginx.ingress.kubernetes.io/enable-influxdb\n:\n \n\"true\"\n\n\nnginx.ingress.kubernetes.io/influxdb-measurement\n:\n \n\"nginx-reqs\"\n\n\nnginx.ingress.kubernetes.io/influxdb-port\n:\n \n\"8089\"\n\n\nnginx.ingress.kubernetes.io/influxdb-host\n:\n \n\"127.0.0.1\"\n\n\nnginx.ingress.kubernetes.io/influxdb-server-name\n:\n \n\"nginx-ingress\"\n\n\n\n\n\nFor the \ninfluxdb-host\n parameter you have two options:\n\n\n\n\nUse an InfluxDB server configured with the \nUDP protocol\n enabled. \n\n\nDeploy Telegraf as a sidecar proxy to the Ingress controller configured to listen UDP with the \nsocket listener input\n and to write using\nanyone of the \noutputs plugins\n like InfluxDB, Apache Kafka,\nPrometheus, etc.. (recommended)\n\n\n\n\nIt's important to remember that there's no DNS resolver at this stage so you will have to configure\nan ip address to \nnginx.ingress.kubernetes.io/influxdb-host\n. If you deploy Influx or Telegraf as sidecar (another container in the same pod) this becomes straightforward since you can directly use \n127.0.0.1\n.\n\n\nBackend Protocol\n\u00b6\n\n\nUsing \nbackend-protocol\n annotations is possible to indicate how NGINX should communicate with the backend service.\nValid Values: HTTP, HTTPS, GRPC, GRPCS and AJP\n\n\nBy default NGINX uses \nHTTP\n.\n\n\nExample:\n\n\nnginx.ingress.kubernetes.io/backend-protocol\n:\n \n\"HTTPS\"", + "text": "Annotations\n\u00b6\n\n\nYou can add these Kubernetes annotations to specific Ingress objects to customize their behavior.\n\n\n\n\nTip\n\n\nAnnotation keys and values can only be strings.\nOther types, such as boolean or numeric values must be quoted,\ni.e. \n\"true\"\n, \n\"false\"\n, \n\"100\"\n.\n\n\n\n\n\n\nNote\n\n\nThe annotation prefix can be changed using the\n\n--annotations-prefix\n command line argument\n,\nbut the default is \nnginx.ingress.kubernetes.io\n, as described in the\ntable below.\n\n\n\n\n\n\n\n\n\n\nName\n\n\ntype\n\n\n\n\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/add-base-url\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/app-root\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/affinity\n\n\ncookie\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-realm\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-secret\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-type\n\n\nbasic or digest\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-tls-secret\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-tls-verify-depth\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-tls-verify-client\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-tls-error-page\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/auth-url\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/backend-protocol\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/base-url-scheme\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/client-body-buffer-size\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/configuration-snippet\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/default-backend\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/enable-cors\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-allow-origin\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-allow-methods\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-allow-headers\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-allow-credentials\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-max-age\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/force-ssl-redirect\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/from-to-www-redirect\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/grpc-backend\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/limit-connections\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/limit-rps\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/permanent-redirect\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/permanent-redirect-code\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-body-size\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-cookie-domain\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-connect-timeout\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-send-timeout\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-read-timeout\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-next-upstream\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-next-upstream-tries\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-request-buffering\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-redirect-from\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-redirect-to\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/rewrite-log\n\n\nURI\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/rewrite-target\n\n\nURI\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/secure-backends\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/secure-verify-ca-secret\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/server-alias\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/server-snippet\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/service-upstream\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/session-cookie-name\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/session-cookie-hash\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/ssl-redirect\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/ssl-passthrough\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/upstream-max-fails\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/upstream-fail-timeout\n\n\nnumber\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/upstream-hash-by\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/load-balance\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/upstream-vhost\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/whitelist-source-range\n\n\nCIDR\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-buffering\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/proxy-buffer-size\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/ssl-ciphers\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/connection-proxy-header\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/enable-access-log\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/lua-resty-waf\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/lua-resty-waf-debug\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/lua-resty-waf-ignore-rulesets\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/lua-resty-waf-extra-rules\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/enable-influxdb\n\n\n\"true\" or \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/influxdb-measurement\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/influxdb-port\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/influxdb-host\n\n\nstring\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/influxdb-server-name\n\n\nstring\n\n\n\n\n\n\n\n\nRewrite\n\u00b6\n\n\nIn some scenarios the exposed URL in the backend service differs from the specified path in the Ingress rule. Without a rewrite any request will return 404.\nSet the annotation \nnginx.ingress.kubernetes.io/rewrite-target\n to the path expected by the service.\n\n\nIf the application contains relative links it is possible to add an additional annotation \nnginx.ingress.kubernetes.io/add-base-url\n that will prepend a \nbase\n tag\n in the header of the returned HTML from the backend.\n\n\nIf the scheme of \nbase\n tag\n need to be specific, set the annotation \nnginx.ingress.kubernetes.io/base-url-scheme\n to the scheme such as \nhttp\n and \nhttps\n.\n\n\nIf the Application Root is exposed in a different path and needs to be redirected, set the annotation \nnginx.ingress.kubernetes.io/app-root\n to redirect requests for \n/\n.\n\n\n\n\nExample\n\n\nPlease check the \nrewrite\n example.\n\n\n\n\nSession Affinity\n\u00b6\n\n\nThe annotation \nnginx.ingress.kubernetes.io/affinity\n enables and sets the affinity type in all Upstreams of an Ingress. This way, a request will always be directed to the same upstream server.\nThe only affinity type available for NGINX is \ncookie\n.\n\n\n\n\nExample\n\n\nPlease check the \naffinity\n example.\n\n\n\n\nCookie affinity\n\u00b6\n\n\nIf you use the \ncookie\n affinity type you can also specify the name of the cookie that will be used to route the requests with the annotation \nnginx.ingress.kubernetes.io/session-cookie-name\n. The default is to create a cookie named 'INGRESSCOOKIE'.\n\n\nIn case of NGINX the annotation \nnginx.ingress.kubernetes.io/session-cookie-hash\n defines which algorithm will be used to hash the used upstream. Default value is \nmd5\n and possible values are \nmd5\n, \nsha1\n and \nindex\n.\n\n\n\n\nAttention\n\n\nThe \nindex\n option is not an actual hash; an in-memory index is used instead, which has less overhead.\nHowever, with \nindex\n, matching against a changing upstream server list is inconsistent.\nSo, at reload, if upstream servers have changed, index values are not guaranteed to correspond to the same server as before!\n\nUse \nindex\n with caution\n and only if you need to!\n\n\n\n\nIn NGINX this feature is implemented by the third party module \nnginx-sticky-module-ng\n. The workflow used to define which upstream server will be used is explained \nhere\n\n\nAuthentication\n\u00b6\n\n\nIs possible to add authentication adding additional annotations in the Ingress rule. The source of the authentication is a secret that contains usernames and passwords inside the key \nauth\n.\n\n\nThe annotations are:\n\nnginx.ingress.kubernetes.io/auth-type: [basic|digest]\n\n\n\nIndicates the \nHTTP Authentication Type: Basic or Digest Access Authentication\n.\n\n\nnginx.ingress.kubernetes.io/auth-secret: secretName\n\n\n\n\nThe name of the Secret that contains the usernames and passwords which are granted access to the \npath\ns defined in the Ingress rules.\nThis annotation also accepts the alternative form \"namespace/secretName\", in which case the Secret lookup is performed in the referenced namespace instead of the Ingress namespace.\n\n\nnginx.ingress.kubernetes.io/auth-realm: \"realm string\"\n\n\n\n\n\n\nExample\n\n\nPlease check the \nauth\n example.\n\n\n\n\nCustom NGINX upstream checks\n\u00b6\n\n\nNGINX exposes some flags in the \nupstream configuration\n that enable the configuration of each server in the upstream. The Ingress controller allows custom \nmax_fails\n and \nfail_timeout\n parameters in a global context using \nupstream-max-fails\n and \nupstream-fail-timeout\n in the NGINX ConfigMap or in a particular Ingress rule. \nupstream-max-fails\n defaults to 0. This means NGINX will respect the container's \nreadinessProbe\n if it is defined. If there is no probe and no values for \nupstream-max-fails\n NGINX will continue to send traffic to the container.\n\n\n\n\nTip\n\n\nWith the default configuration NGINX will not health check your backends. Whenever the endpoints controller notices a readiness probe failure, that pod's IP will be removed from the list of endpoints. This will trigger the NGINX controller to also remove it from the upstreams.**\n\n\n\n\nTo use custom values in an Ingress rule define these annotations:\n\n\nnginx.ingress.kubernetes.io/upstream-max-fails\n: number of unsuccessful attempts to communicate with the server that should occur in the duration set by the \nupstream-fail-timeout\n parameter to consider the server unavailable.\n\n\nnginx.ingress.kubernetes.io/upstream-fail-timeout\n: time in seconds during which the specified number of unsuccessful attempts to communicate with the server should occur to consider the server unavailable. This is also the period of time the server will be considered unavailable.\n\n\nIn NGINX, backend server pools are called \"\nupstreams\n\". Each upstream contains the endpoints for a service. An upstream is created for each service that has Ingress rules defined.\n\n\n\n\nAttention\n\n\nAll Ingress rules using the same service will use the same upstream.\n\nOnly one of the Ingress rules should define annotations to configure the upstream servers.\n\n\n\n\n\n\nExample\n\n\nPlease check the \ncustom upstream check\n example.\n\n\n\n\nCustom NGINX upstream hashing\n\u00b6\n\n\nNGINX supports load balancing by client-server mapping based on \nconsistent hashing\n for a given key. The key can contain text, variables or any combination thereof. This feature allows for request stickiness other than client IP or cookies. The \nketama\n consistent hashing method will be used which ensures only a few keys would be remapped to different servers on upstream group changes.\n\n\nTo enable consistent hashing for a backend:\n\n\nnginx.ingress.kubernetes.io/upstream-hash-by\n: the nginx variable, text value or any combination thereof to use for consistent hashing. For example \nnginx.ingress.kubernetes.io/upstream-hash-by: \"$request_uri\"\n to consistently hash upstream requests by the current request URI.\n\n\nCustom NGINX load balancing\n\u00b6\n\n\nThis is similar to (https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/configmap.md#load-balance) but configures load balancing algorithm per ingress.\n\n\n\n\nNote that \nnginx.ingress.kubernetes.io/upstream-hash-by\n takes preference over this. If this and \nnginx.ingress.kubernetes.io/upstream-hash-by\n are not set then we fallback to using globally configured load balancing algorithm.\n\n\n\n\nCustom NGINX upstream vhost\n\u00b6\n\n\nThis configuration setting allows you to control the value for host in the following statement: \nproxy_set_header Host $host\n, which forms part of the location block. This is useful if you need to call the upstream server by something other than \n$host\n.\n\n\nClient Certificate Authentication\n\u00b6\n\n\nIt is possible to enable Client Certificate Authentication using additional annotations in Ingress Rule.\n\n\nThe annotations are:\n\n\n\n\nnginx.ingress.kubernetes.io/auth-tls-secret: secretName\n:\n The name of the Secret that contains the full Certificate Authority chain \nca.crt\n that is enabled to authenticate against this Ingress.\n This annotation also accepts the alternative form \"namespace/secretName\", in which case the Secret lookup is performed in the referenced namespace instead of the Ingress namespace.\n\n\nnginx.ingress.kubernetes.io/auth-tls-verify-depth\n:\n The validation depth between the provided client certificate and the Certification Authority chain.\n\n\nnginx.ingress.kubernetes.io/auth-tls-verify-client\n:\n Enables verification of client certificates.\n\n\nnginx.ingress.kubernetes.io/auth-tls-error-page\n:\n The URL/Page that user should be redirected in case of a Certificate Authentication Error\n\n\nnginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream\n:\n Indicates if the received certificates should be passed or not to the upstream server. By default this is disabled.\n\n\n\n\n\n\nExample\n\n\nPlease check the \nclient-certs\n example.\n\n\n\n\n\n\nAttention\n\n\nTLS with Client Authentication is \nnot\n possible in Cloudflare and might result in unexpected behavior.\n\n\nCloudflare only allows Authenticated Origin Pulls and is required to use their own certificate: \nhttps://blog.cloudflare.com/protecting-the-origin-with-tls-authenticated-origin-pulls/\n\n\nOnly Authenticated Origin Pulls are allowed and can be configured by following their tutorial: \nhttps://support.cloudflare.com/hc/en-us/articles/204494148-Setting-up-NGINX-to-use-TLS-Authenticated-Origin-Pulls\n\n\n\n\nConfiguration snippet\n\u00b6\n\n\nUsing this annotation you can add additional configuration to the NGINX location. For example:\n\n\nnginx.ingress.kubernetes.io/configuration-snippet\n:\n \n|\n\n \nmore_set_headers \"Request-Id: $req_id\";\n\n\n\n\n\nDefault Backend\n\u00b6\n\n\nThe ingress controller requires a \ndefault backend\n.\nThis service handles the response when the service in the Ingress rule does not have endpoints.\nThis is a global configuration for the ingress controller. In some cases could be required to return a custom content or format. In this scenario we can use the annotation \nnginx.ingress.kubernetes.io/default-backend: \n to specify a custom default backend.\n\n\nEnable CORS\n\u00b6\n\n\nTo enable Cross-Origin Resource Sharing (CORS) in an Ingress rule, add the annotation\n\nnginx.ingress.kubernetes.io/enable-cors: \"true\"\n. This will add a section in the server\nlocation enabling this functionality.\n\n\nCORS can be controlled with the following annotations:\n\n\n\n\nnginx.ingress.kubernetes.io/cors-allow-methods\n\n controls which methods are accepted. This is a multi-valued field, separated by ',' and\n accepts only letters (upper and lower case).\n\n\nDefault: \nGET, PUT, POST, DELETE, PATCH, OPTIONS\n\n\n\n\nExample: \nnginx.ingress.kubernetes.io/cors-allow-methods: \"PUT, GET, POST, OPTIONS\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-allow-headers\n\n controls which headers are accepted. This is a multi-valued field, separated by ',' and accepts letters,\n numbers, _ and -.\n\n\n\n\nDefault: \nDNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization\n\n\n\n\nExample: \nnginx.ingress.kubernetes.io/cors-allow-headers: \"X-Forwarded-For, X-app123-XPTO\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-allow-origin\n\n controls what's the accepted Origin for CORS.\n This is a single field value, with the following format: \nhttp(s)://origin-site.com\n or \nhttp(s)://origin-site.com:port\n\n\n\n\nDefault: \n*\n\n\n\n\nExample: \nnginx.ingress.kubernetes.io/cors-allow-origin: \"https://origin-site.com:4443\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-allow-credentials\n\n controls if credentials can be passed during CORS operations.\n\n\n\n\nDefault: \ntrue\n\n\n\n\nExample: \nnginx.ingress.kubernetes.io/cors-allow-credentials: \"false\"\n\n\n\n\n\n\nnginx.ingress.kubernetes.io/cors-max-age\n\n controls how long preflight requests can be cached.\n Default: \n1728000\n\n Example: \nnginx.ingress.kubernetes.io/cors-max-age: 600\n\n\n\n\n\n\n\n\nNote\n\n\nFor more information please see \nhttps://enable-cors.org\n \n\n\n\n\nServer Alias\n\u00b6\n\n\nTo add Server Aliases to an Ingress rule add the annotation \nnginx.ingress.kubernetes.io/server-alias: \"