From 08149a7a216c98e877bd1b474ee6770947067d3e Mon Sep 17 00:00:00 2001 From: chentao1596 Date: Fri, 20 Jan 2017 10:38:31 +0800 Subject: [PATCH 01/32] fix wrong link(change titile) --- controllers/nginx/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/controllers/nginx/configuration.md b/controllers/nginx/configuration.md index d38e30af4..2c2136bbc 100644 --- a/controllers/nginx/configuration.md +++ b/controllers/nginx/configuration.md @@ -154,7 +154,7 @@ The annotations `ingress.kubernetes.io/limit-connections` and `ingress.kubernete If you specify both annotations in a single Ingress rule, `limit-rps` takes precedence. -### Secure upstreams +### Secure backends By default NGINX uses `http` to reach the services. Adding the annotation `ingress.kubernetes.io/secure-backends: "true"` in the Ingress rule changes the protocol to `https`. From 7bcdef0505d898d6b184d54ce7dce81c901cada4 Mon Sep 17 00:00:00 2001 From: chentao1596 Date: Fri, 20 Jan 2017 10:49:40 +0800 Subject: [PATCH 02/32] adjust some improper punctuations --- controllers/nginx/configuration.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/controllers/nginx/configuration.md b/controllers/nginx/configuration.md index 2c2136bbc..e27dfa936 100644 --- a/controllers/nginx/configuration.md +++ b/controllers/nginx/configuration.md @@ -21,8 +21,8 @@ There are 3 ways to customize NGINX: -1. [ConfigMap](#allowed-parameters-in-configuration-configmap): create a stand alone ConfigMap, use this if you want a different global configuration -2. [annotations](#annotations): use this if you want a specific configuration for the site defined in the Ingress rule +1. [ConfigMap](#allowed-parameters-in-configuration-configmap): create a stand alone ConfigMap, use this if you want a different global configuration. +2. [annotations](#annotations): use this if you want a specific configuration for the site defined in the Ingress rule. 3. custom template: when more specific settings are required, like [open_file_cache](http://nginx.org/en/docs/http/ngx_http_core_module.html#open_file_cache), custom [log_format](http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format), adjust [listen](http://nginx.org/en/docs/http/ngx_http_core_module.html#listen) options as `rcvbuf` or when is not possible to change an through the ConfigMap. @@ -97,7 +97,7 @@ Please check the [custom upstream check](examples/custom-upstream-check/README.m ### Authentication -Is possible to add authentication adding additional annotations in the Ingress rule. The source of the authentication is a secret that contains usernames and passwords inside the the key `auth` +Is possible to add authentication adding additional annotations in the Ingress rule. The source of the authentication is a secret that contains usernames and passwords inside the the key `auth`. The annotations are: @@ -161,9 +161,9 @@ By default NGINX uses `http` to reach the services. Adding the annotation `ingre ### Server-side HTTPS enforcement through redirect -By default the controller redirects (301) to `HTTPS` if TLS is enabled for that ingress. If you want to disable that behaviour globally, you can use `ssl-redirect: "false"` in the NGINX config map +By default the controller redirects (301) to `HTTPS` if TLS is enabled for that ingress. If you want to disable that behaviour globally, you can use `ssl-redirect: "false"` in the NGINX config map. -To configure this feature for specific ingress resources, you can use the `ingress.kubernetes.io/ssl-redirect: "false"` annotation in the particular resource +To configure this feature for specific ingress resources, you can use the `ingress.kubernetes.io/ssl-redirect: "false"` annotation in the particular resource. ### Whitelist source range @@ -177,7 +177,7 @@ To configure this setting globally for all Ingress rules, the `whitelist-source- Please check the [whitelist](examples/whitelist/README.md) example. -### **Allowed parameters in configuration ConfigMap:** +### **Allowed parameters in configuration ConfigMap** **body-size:** Sets the maximum allowed size of the client request body. See NGINX [client_max_body_size](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size). @@ -222,7 +222,7 @@ http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout **max-worker-connections:** Sets the maximum number of simultaneous connections that can be opened by each [worker process](http://nginx.org/en/docs/ngx_core_module.html#worker_connections). -**proxy-buffer-size:** Sets the size of the buffer used for [reading the first part of the response](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) received from the proxied server. This part usually contains a small response header.` +**proxy-buffer-size:** Sets the size of the buffer used for [reading the first part of the response](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) received from the proxied server. This part usually contains a small response header. **proxy-connect-timeout:** Sets the timeout for [establishing a connection with a proxied server](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout). It should be noted that this timeout cannot usually exceed 75 seconds. @@ -249,7 +249,7 @@ http://nginx.org/en/docs/hash.html -**map-hash-bucket-size:** Sets the bucket size for the [map variables hash tables](http://nginx.org/en/docs/http/ngx_http_map_module.html#map_hash_bucket_size). The details of setting up hash tables are provided in a separate [document](http://nginx.org/en/docs/hash.html) +**map-hash-bucket-size:** Sets the bucket size for the [map variables hash tables](http://nginx.org/en/docs/http/ngx_http_map_module.html#map_hash_bucket_size). The details of setting up hash tables are provided in a separate [document](http://nginx.org/en/docs/hash.html). **ssl-buffer-size:** Sets the size of the [SSL buffer](http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_buffer_size) used for sending data. @@ -268,7 +268,7 @@ The recommendation above prioritizes algorithms that provide perfect [forward se Please check the [Mozilla SSL Configuration Generator](https://mozilla.github.io/server-side-tls/ssl-config-generator/). -**ssl-dh-param:** sets the Base64 string that contains Diffie-Hellman key to help with "Perfect Forward Secrecy." +**ssl-dh-param:** sets the Base64 string that contains Diffie-Hellman key to help with "Perfect Forward Secrecy". https://www.openssl.org/docs/manmaster/apps/dhparam.html https://wiki.mozilla.org/Security/Server_Side_TLS#DHE_handshake_and_dhparam http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_dhparam From b0c26195944680225db061237698692748297219 Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Fri, 20 Jan 2017 18:53:32 -0300 Subject: [PATCH 03/32] Add annotation to allow custom body sizes --- controllers/nginx/configuration.md | 2 +- controllers/nginx/pkg/config/config.go | 6 +----- .../nginx/rootfs/etc/nginx/template/nginx.tmpl | 6 +++--- core/pkg/ingress/annotations/proxy/main.go | 15 +++++++++++---- core/pkg/ingress/controller/controller.go | 1 + core/pkg/ingress/defaults/main.go | 5 +++++ 6 files changed, 22 insertions(+), 13 deletions(-) diff --git a/controllers/nginx/configuration.md b/controllers/nginx/configuration.md index e27dfa936..9dd2121cd 100644 --- a/controllers/nginx/configuration.md +++ b/controllers/nginx/configuration.md @@ -179,7 +179,7 @@ Please check the [whitelist](examples/whitelist/README.md) example. ### **Allowed parameters in configuration ConfigMap** -**body-size:** Sets the maximum allowed size of the client request body. See NGINX [client_max_body_size](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size). +**proxy-body-size:** Sets the maximum allowed size of the client request body. See NGINX [client_max_body_size](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size). **custom-http-errors:** Enables which HTTP codes should be passed for processing with the [error_page directive](http://nginx.org/en/docs/http/ngx_http_core_module.html#error_page). diff --git a/controllers/nginx/pkg/config/config.go b/controllers/nginx/pkg/config/config.go index 9cf9b987a..afd03ef04 100644 --- a/controllers/nginx/pkg/config/config.go +++ b/controllers/nginx/pkg/config/config.go @@ -78,10 +78,6 @@ var ( type Configuration struct { defaults.Backend `json:",squash"` - // http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size - // Sets the maximum allowed size of the client request body - BodySize string `json:"body-size,omitempty"` - // EnableDynamicTLSRecords enables dynamic TLS record sizes // https://blog.cloudflare.com/optimizing-tls-over-tcp-to-reduce-latency // By default this is enabled @@ -225,7 +221,6 @@ type Configuration struct { // NewDefault returns the default nginx configuration func NewDefault() Configuration { cfg := Configuration{ - BodySize: bodySize, EnableDynamicTLSRecords: true, EnableSPDY: false, ErrorLogLevel: errorLevel, @@ -253,6 +248,7 @@ func NewDefault() Configuration { VtsStatusZoneSize: "10m", UseHTTP2: true, Backend: defaults.Backend{ + ProxyBodySize: bodySize, ProxyConnectTimeout: 5, ProxyReadTimeout: 60, ProxySendTimeout: 60, diff --git a/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl b/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl index 437057745..7c77f52e6 100644 --- a/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl +++ b/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl @@ -70,9 +70,7 @@ http { gzip_proxied any; {{ end }} - server_tokens {{ if $cfg.ShowServerTokens }}on{{ else }}off{{ end }}; - - client_max_body_size "{{ $cfg.BodySize }}"; + server_tokens {{ if $cfg.ShowServerTokens }}on{{ else }}off{{ end }}; log_format upstreaminfo '{{ if $cfg.UseProxyProtocol }}$proxy_protocol_addr{{ else }}$remote_addr{{ end }} - ' '[$proxy_add_x_forwarded_for] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" ' @@ -281,6 +279,8 @@ http { {{ template "CORS" }} {{ end }} + client_max_body_size "{{ $location.Proxy.BodySize }}"; + proxy_set_header Host $host; # Pass Real IP diff --git a/core/pkg/ingress/annotations/proxy/main.go b/core/pkg/ingress/annotations/proxy/main.go index 97f3dddc0..c1ee7e47b 100644 --- a/core/pkg/ingress/annotations/proxy/main.go +++ b/core/pkg/ingress/annotations/proxy/main.go @@ -24,6 +24,7 @@ import ( ) const ( + bodySize = "ingress.kubernetes.io/proxy-body-size" connect = "ingress.kubernetes.io/proxy-connect-timeout" send = "ingress.kubernetes.io/proxy-send-timeout" read = "ingress.kubernetes.io/proxy-read-timeout" @@ -32,6 +33,7 @@ const ( // Configuration returns the proxy timeout to use in the upstream server/s type Configuration struct { + BodySize string `json:"bodySize"` ConnectTimeout int `json:"conectTimeout"` SendTimeout int `json:"sendTimeout"` ReadTimeout int `json:"readTimeout"` @@ -66,10 +68,15 @@ func (a proxy) Parse(ing *extensions.Ingress) (interface{}, error) { rt = defBackend.ProxyReadTimeout } - bs, err := parser.GetStringAnnotation(bufferSize, ing) - if err != nil || bs == "" { - bs = defBackend.ProxyBufferSize + bufs, err := parser.GetStringAnnotation(bufferSize, ing) + if err != nil || bufs == "" { + bufs = defBackend.ProxyBufferSize } - return &Configuration{ct, st, rt, bs}, nil + bs, err := parser.GetStringAnnotation(bodySize, ing) + if err != nil || bs == "" { + bs = defBackend.ProxyBodySize + } + + return &Configuration{bs, ct, st, rt, bufs}, nil } diff --git a/core/pkg/ingress/controller/controller.go b/core/pkg/ingress/controller/controller.go index 87cddf59c..a5f391778 100644 --- a/core/pkg/ingress/controller/controller.go +++ b/core/pkg/ingress/controller/controller.go @@ -788,6 +788,7 @@ func (ic *GenericController) createServers(data []interface{}, upstreams map[str bdef := ic.GetDefaultBackend() ngxProxy := proxy.Configuration{ + BodySize: bdef.ProxyBodySize, ConnectTimeout: bdef.ProxyConnectTimeout, SendTimeout: bdef.ProxySendTimeout, ReadTimeout: bdef.ProxyReadTimeout, diff --git a/core/pkg/ingress/defaults/main.go b/core/pkg/ingress/defaults/main.go index 88a817d93..12badd1b2 100644 --- a/core/pkg/ingress/defaults/main.go +++ b/core/pkg/ingress/defaults/main.go @@ -6,12 +6,17 @@ import "net" // The reason of this requirements is the annotations are generic. If some implementation do not supports // one or more annotations it just can provides defaults type Backend struct { + // enables which HTTP codes should be passed for processing with the error_page directive // http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_intercept_errors // http://nginx.org/en/docs/http/ngx_http_core_module.html#error_page // By default this is disabled CustomHTTPErrors []int `json:"custom-http-errors,-"` + // http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size + // Sets the maximum allowed size of the client request body + ProxyBodySize string `json:"proxy-body-size"` + // Defines a timeout for establishing a connection with a proxied server. // It should be noted that this timeout cannot usually exceed 75 seconds. // http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout From 6cd20734c61144b8fcb8d743c31e6ffb447bfddf Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Fri, 20 Jan 2017 19:01:37 -0300 Subject: [PATCH 04/32] Add flag to allow switch off the update of Ingress status --- core/pkg/ingress/controller/controller.go | 24 ++++++++++++++++------- core/pkg/ingress/controller/launch.go | 4 ++++ 2 files changed, 21 insertions(+), 7 deletions(-) diff --git a/core/pkg/ingress/controller/controller.go b/core/pkg/ingress/controller/controller.go index a5f391778..b82fae771 100644 --- a/core/pkg/ingress/controller/controller.go +++ b/core/pkg/ingress/controller/controller.go @@ -128,6 +128,8 @@ type Configuration struct { // Backend is the particular implementation to be used. // (for instance NGINX) Backend ingress.Controller + + UpdateStatus bool } // newIngressController creates an Ingress controller @@ -257,11 +259,15 @@ func newIngressController(config *Configuration) *GenericController { cache.ResourceEventHandlerFuncs{}, cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}) - ic.syncStatus = status.NewStatusSyncer(status.Config{ - Client: config.Client, - PublishService: ic.cfg.PublishService, - IngressLister: ic.ingLister, - }) + if config.UpdateStatus { + ic.syncStatus = status.NewStatusSyncer(status.Config{ + Client: config.Client, + PublishService: ic.cfg.PublishService, + IngressLister: ic.ingLister, + }) + } else { + glog.Warning("Update of ingress status is disabled (flag --update-status=false was specified)") + } ic.annotations = newAnnotationExtractor(ic) @@ -970,7 +976,9 @@ func (ic GenericController) Stop() error { close(ic.stopCh) go ic.syncQueue.Shutdown() go ic.secretQueue.Shutdown() - ic.syncStatus.Shutdown() + if ic.syncStatus != nil { + ic.syncStatus.Shutdown() + } return nil } @@ -990,7 +998,9 @@ func (ic GenericController) Start() { go ic.secretQueue.Run(5*time.Second, ic.stopCh) go ic.syncQueue.Run(5*time.Second, ic.stopCh) - go ic.syncStatus.Run(ic.stopCh) + if ic.syncStatus != nil { + go ic.syncStatus.Run(ic.stopCh) + } <-ic.stopCh } diff --git a/core/pkg/ingress/controller/launch.go b/core/pkg/ingress/controller/launch.go index 95e50c877..eecc9b39e 100644 --- a/core/pkg/ingress/controller/launch.go +++ b/core/pkg/ingress/controller/launch.go @@ -73,6 +73,9 @@ func NewIngressController(backend ingress.Controller) *GenericController { defHealthzURL = flags.String("health-check-path", "/healthz", `Defines the URL to be used as health check inside in the default server in NGINX.`) + + updateStatus = flags.Bool("update-status", true, `Indicates if the + ingress controller should update the Ingress status IP/hostname. Default is true`) ) flags.AddGoFlagSet(flag.CommandLine) @@ -134,6 +137,7 @@ func NewIngressController(backend ingress.Controller) *GenericController { os.MkdirAll(ingress.DefaultSSLDirectory, 0655) config := &Configuration{ + UpdateStatus: *updateStatus, Client: kubeClient, ResyncPeriod: *resyncPeriod, DefaultService: *defaultSvc, From 87322b84bac0078e341d786780ccdc78b17e6d62 Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Fri, 20 Jan 2017 19:14:59 -0300 Subject: [PATCH 05/32] Add support for custom header sizes --- controllers/nginx/pkg/config/config.go | 13 +++++++++++++ .../nginx/rootfs/etc/nginx/template/nginx.tmpl | 3 +++ 2 files changed, 16 insertions(+) diff --git a/controllers/nginx/pkg/config/config.go b/controllers/nginx/pkg/config/config.go index afd03ef04..d7ad8aff8 100644 --- a/controllers/nginx/pkg/config/config.go +++ b/controllers/nginx/pkg/config/config.go @@ -83,6 +83,11 @@ type Configuration struct { // By default this is enabled EnableDynamicTLSRecords bool `json:"enable-dynamic-tls-records"` + // ClientHeaderBufferSize allows to configure a custom buffer + // size for reading client request header + // http://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_buffer_size + ClientHeaderBufferSize string `json:"client-header-buffer-size"` + // EnableSPDY enables spdy and use ALPN and NPN to advertise the availability of the two protocols // https://blog.cloudflare.com/open-sourcing-our-nginx-http-2-spdy-code // By default this is enabled @@ -128,6 +133,12 @@ type Configuration struct { // http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout KeepAlive int `json:"keep-alive,omitempty"` + // LargeClientHeaderBuffers Sets the maximum number and size of buffers used for reading + // large client request header. + // http://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers + // Default: 4 8k + LargeClientHeaderBuffers string `json:"large-client-header-buffers"` + // Maximum number of simultaneous connections that can be opened by each worker process // http://nginx.org/en/docs/ngx_core_module.html#worker_connections MaxWorkerConnections int `json:"max-worker-connections,omitempty"` @@ -221,6 +232,7 @@ type Configuration struct { // NewDefault returns the default nginx configuration func NewDefault() Configuration { cfg := Configuration{ + ClientHeaderBufferSize: "1k", EnableDynamicTLSRecords: true, EnableSPDY: false, ErrorLogLevel: errorLevel, @@ -229,6 +241,7 @@ func NewDefault() Configuration { HSTSMaxAge: hstsMaxAge, GzipTypes: gzipTypes, KeepAlive: 75, + LargeClientHeaderBuffers: "4 8k", MaxWorkerConnections: 16384, MapHashBucketSize: 64, ProxyRealIPCIDR: defIPCIDR, diff --git a/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl b/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl index 7c77f52e6..7e4979956 100644 --- a/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl +++ b/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl @@ -54,6 +54,9 @@ http { keepalive_timeout {{ $cfg.KeepAlive }}s; + client_header_buffer_size {{ $cfg.ClientHeaderBufferSize }}; + large_client_header_buffers {{ $cfg.LargeClientHeaderBuffers }}; + types_hash_max_size 2048; server_names_hash_max_size {{ $cfg.ServerNameHashMaxSize }}; server_names_hash_bucket_size {{ $cfg.ServerNameHashBucketSize }}; From e5b02b609fc7e4f89b4f825072b4a8604b5c1ece Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Thu, 19 Jan 2017 02:05:46 -0300 Subject: [PATCH 06/32] Use protobuf instead of rest to connect to apiserver host --- core/pkg/ingress/controller/launch.go | 77 +++++-- docs/troubleshooting.md | 293 ++++++++++++++++++++++++++ 2 files changed, 356 insertions(+), 14 deletions(-) create mode 100644 docs/troubleshooting.md diff --git a/core/pkg/ingress/controller/launch.go b/core/pkg/ingress/controller/launch.go index 95e50c877..a5eb54563 100644 --- a/core/pkg/ingress/controller/launch.go +++ b/core/pkg/ingress/controller/launch.go @@ -15,10 +15,10 @@ import ( "github.com/spf13/pflag" "k8s.io/kubernetes/pkg/api" - clientset "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset" - "k8s.io/kubernetes/pkg/client/restclient" + client "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset" + "k8s.io/kubernetes/pkg/client/unversioned/clientcmd" + clientcmdapi "k8s.io/kubernetes/pkg/client/unversioned/clientcmd/api" "k8s.io/kubernetes/pkg/healthz" - kubectl_util "k8s.io/kubernetes/pkg/kubectl/cmd/util" "k8s.io/ingress/core/pkg/ingress" "k8s.io/ingress/core/pkg/k8s" @@ -29,6 +29,12 @@ func NewIngressController(backend ingress.Controller) *GenericController { var ( flags = pflag.NewFlagSet("", pflag.ExitOnError) + apiserverHost = flags.String("apiserver-host", "", "The address of the Kubernetes Apiserver "+ + "to connect to in the format of protocol://address:port, e.g., "+ + "http://localhost:8080. If not specified, the assumption is that the binary runs inside a "+ + "Kubernetes cluster and local discovery is attempted.") + kubeConfigFile = flags.String("kubeconfig", "", "Path to kubeconfig file with authorization and master location information.") + defaultSvc = flags.String("default-backend-service", "", `Service used to serve a 404 page for the default backend. Takes the form namespace/name. The controller uses the first node port of this Service for @@ -77,7 +83,6 @@ func NewIngressController(backend ingress.Controller) *GenericController { flags.AddGoFlagSet(flag.CommandLine) flags.Parse(os.Args) - clientConfig := kubectl_util.DefaultClientConfig(flags) flag.Set("logtostderr", "true") @@ -91,17 +96,9 @@ func NewIngressController(backend ingress.Controller) *GenericController { glog.Fatalf("Please specify --default-backend-service") } - kubeconfig, err := restclient.InClusterConfig() + kubeClient, err := createApiserverClient(*apiserverHost, *kubeConfigFile) if err != nil { - kubeconfig, err = clientConfig.ClientConfig() - if err != nil { - glog.Fatalf("error configuring the client: %v", err) - } - } - - kubeClient, err := clientset.NewForConfig(kubeconfig) - if err != nil { - glog.Fatalf("failed to create client: %v", err) + handleFatalInitError(err) } _, err = k8s.IsValidService(kubeClient, *defaultSvc) @@ -186,3 +183,55 @@ func registerHandlers(enableProfiling bool, port int, ic *GenericController) { } glog.Fatal(server.ListenAndServe()) } + +const ( + // High enough QPS to fit all expected use cases. QPS=0 is not set here, because + // client code is overriding it. + defaultQPS = 1e6 + // High enough Burst to fit all expected use cases. Burst=0 is not set here, because + // client code is overriding it. + defaultBurst = 1e6 +) + +// createApiserverClient creates new Kubernetes Apiserver client. When kubeconfig or apiserverHost param is empty +// the function assumes that it is running inside a Kubernetes cluster and attempts to +// discover the Apiserver. Otherwise, it connects to the Apiserver specified. +// +// apiserverHost param is in the format of protocol://address:port/pathPrefix, e.g.http://localhost:8001. +// kubeConfig location of kubeconfig file +func createApiserverClient(apiserverHost string, kubeConfig string) (*client.Clientset, error) { + + clientConfig := clientcmd.NewNonInteractiveDeferredLoadingClientConfig( + &clientcmd.ClientConfigLoadingRules{ExplicitPath: kubeConfig}, + &clientcmd.ConfigOverrides{ClusterInfo: clientcmdapi.Cluster{Server: apiserverHost}}) + + cfg, err := clientConfig.ClientConfig() + if err != nil { + return nil, err + } + + cfg.QPS = defaultQPS + cfg.Burst = defaultBurst + cfg.ContentType = "application/vnd.kubernetes.protobuf" + + glog.Infof("Creating API server client for %s", cfg.Host) + + client, err := client.NewForConfig(cfg) + + if err != nil { + return nil, err + } + return client, nil +} + +/** + * Handles fatal init error that prevents server from doing any work. Prints verbose error + * message and quits the server. + */ +func handleFatalInitError(err error) { + glog.Fatalf("Error while initializing connection to Kubernetes apiserver. "+ + "This most likely means that the cluster is misconfigured (e.g., it has "+ + "invalid apiserver certificates or service accounts configuration). Reason: %s\n"+ + "Refer to the troubleshooting guide for more information: "+ + "https://github.com/kubernetes/ingress/blob/master/docs/troubleshooting.md", err) +} diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 000000000..6f474692d --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,293 @@ + + +# Troubleshooting + + +## Authentication to the Kubernetes API Server + + +A number of components are involved in the authentication process and the first step is to narrow +down the source of the problem, namely whether it is a problem with service authentication or with the kubeconfig file. +Both authentications must work: + +``` ++-------------+ service +------------+ +| | authentication | | ++ apiserver +<-------------------+ ingress | +| | | controller | ++-------------+ +------------+ + +``` + +__Service authentication__ + +The Ingress controller needs information from apiserver. Therefore, authentication is required, which can be achieved in two different ways: + +1. _Service Account:_ This is recommended, because nothing has to be configured. The Ingress controller will use information provided by the system to communicate with the API server. See 'Service Account' section for details. + +2. _Kubeconfig file:_ In some Kubernetes environments service accounts are not available. In this case a manual configuration is required. The Ingress controller binary can be started with the `--kubeconfig` flag. The value of the flag is a path to a file specifying how to connect to the API server. +The format of the file is identical to `~/.kube/config` which is used by kubectl to connect to the API server. See 'kubeconfig' section for details. + +In the diagram below you can see the full authentication flow with all options, starting with the browser +on the lower left hand side. +``` + +Kubernetes Workstation ++---------------------------------------------------+ +------------------+ +| | | | +| +-----------+ apiserver +------------+ | | +------------+ | +| | | proxy | | | | | | | +| | apiserver | | ingress | | | | ingress | | +| | | | controller | | | | controller | | +| | | | | | | | | | +| | | | | | | | | | +| | | service account/ | | | | | | | +| | | kubeconfig | | | | | | | +| | +<-------------------+ | | | | | | +| | | | | | | | | | +| +------+----+ kubeconfig +------+-----+ | | +------+-----+ | +| |<--------------------------------------------------------| | +| | | | ++---------------------------------------------------+ +------------------+ +``` + + +## Service Account +If using a service account to connect to the API server, Dashboard expects the file +`/var/run/secrets/kubernetes.io/serviceaccount/token` to be present. It provides a secret +token that is required to authenticate with the API server. + +Verify with the following commands: + +```shell +# start a container that contains curl +$ kubectl run test --image=tutum/curl -- sleep 10000 + +# check that container is running +$ kubectl get pods +NAME READY STATUS RESTARTS AGE +test-701078429-s5kca 1/1 Running 0 16s + +# check if secret exists +$ kubectl exec test-701078429-s5kca ls /var/run/secrets/kubernetes.io/serviceaccount/ +ca.crt +namespace +token + +# get service IP of master +$ kubectl get services +NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE +kubernetes 10.0.0.1 443/TCP 1d + +# check base connectivity from cluster inside +$ kubectl exec test-701078429-s5kca -- curl -k https://10.0.0.1 +Unauthorized + +# connect using tokens +$ TOKEN_VALUE=$(kubectl exec test-701078429-s5kca -- cat /var/run/secrets/kubernetes.io/serviceaccount/token) +$ echo $TOKEN_VALUE +eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3Mi....9A +$ kubectl exec test-701078429-s5kca -- curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $TOKEN_VALUE" https://10.0.0.1 +{ + "paths": [ + "/api", + "/api/v1", + "/apis", + "/apis/apps", + "/apis/apps/v1alpha1", + "/apis/authentication.k8s.io", + "/apis/authentication.k8s.io/v1beta1", + "/apis/authorization.k8s.io", + "/apis/authorization.k8s.io/v1beta1", + "/apis/autoscaling", + "/apis/autoscaling/v1", + "/apis/batch", + "/apis/batch/v1", + "/apis/batch/v2alpha1", + "/apis/certificates.k8s.io", + "/apis/certificates.k8s.io/v1alpha1", + "/apis/extensions", + "/apis/extensions/v1beta1", + "/apis/policy", + "/apis/policy/v1alpha1", + "/apis/rbac.authorization.k8s.io", + "/apis/rbac.authorization.k8s.io/v1alpha1", + "/apis/storage.k8s.io", + "/apis/storage.k8s.io/v1beta1", + "/healthz", + "/healthz/ping", + "/logs", + "/metrics", + "/swaggerapi/", + "/ui/", + "/version" + ] +} +``` + +If it is not working, there are two possible reasons: + +1. The contents of the tokens is invalid. Find the secret name with `kubectl get secrets | grep service-account` and +delete it with `kubectl delete secret `. It will automatically be recreated. + +2. You have a non-standard Kubernetes installation and the file containing the token +may not be present. The API server will mount a volume containing this file, but +only if the API server is configured to use the ServiceAccount admission controller. +If you experience this error, verify that your API server is using the ServiceAccount +admission controller. If you are configuring the API server by hand, you can set +this with the `--admission-control` parameter. Please note that you should use +other admission controllers as well. Before configuring this option, you should +read about admission controllers. + +More information: + +* [User Guide: Service Accounts](http://kubernetes.io/docs/user-guide/service-accounts/) +* [Cluster Administrator Guide: Managing Service Accounts](http://kubernetes.io/docs/admin/service-accounts-admin/) + +## kubeconfig +If you want to use a kubeconfig file for authentication, create a deployment file similar to the one below: + +*Note:* the important part is the flag `--kubeconfig=/etc/kubernetes/kubeconfig.yaml`. + +``` +kind: Service +apiVersion: v1 +metadata: + name: nginx-default-backend + labels: + k8s-addon: ingress-nginx.addons.k8s.io +spec: + ports: + - port: 80 + targetPort: http + selector: + app: nginx-default-backend + +--- + +kind: Deployment +apiVersion: extensions/v1beta1 +metadata: + name: nginx-default-backend + labels: + k8s-addon: ingress-nginx.addons.k8s.io +spec: + replicas: 1 + template: + metadata: + labels: + k8s-addon: ingress-nginx.addons.k8s.io + app: nginx-default-backend + spec: + terminationGracePeriodSeconds: 60 + containers: + - name: default-http-backend + image: gcr.io/google_containers/defaultbackend:1.0 + livenessProbe: + httpGet: + path: /healthz + port: 8080 + scheme: HTTP + initialDelaySeconds: 30 + timeoutSeconds: 5 + resources: + limits: + cpu: 10m + memory: 20Mi + requests: + cpu: 10m + memory: 20Mi + ports: + - name: http + containerPort: 8080 + protocol: TCP + +--- + +kind: ConfigMap +apiVersion: v1 +metadata: + name: ingress-nginx + labels: + k8s-addon: ingress-nginx.addons.k8s.io + +--- + +kind: Service +apiVersion: v1 +metadata: + name: ingress-nginx + labels: + k8s-addon: ingress-nginx.addons.k8s.io +spec: + type: LoadBalancer + selector: + app: ingress-nginx + ports: + - name: http + port: 80 + targetPort: http + - name: https + port: 443 + targetPort: https + +--- + +kind: Deployment +apiVersion: extensions/v1beta1 +metadata: + name: ingress-nginx + labels: + k8s-addon: ingress-nginx.addons.k8s.io +spec: + replicas: 1 + template: + metadata: + labels: + app: ingress-nginx + k8s-addon: ingress-nginx.addons.k8s.io + spec: + terminationGracePeriodSeconds: 60 + containers: + - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0 + name: ingress-nginx + imagePullPolicy: Always + ports: + - name: http + containerPort: 80 + protocol: TCP + - name: https + containerPort: 443 + protocol: TCP + livenessProbe: + httpGet: + path: /healthz + port: 10254 + scheme: HTTP + initialDelaySeconds: 30 + timeoutSeconds: 5 + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + args: + - /nginx-ingress-controller + - --default-backend-service=$(POD_NAMESPACE)/nginx-default-backend + - --configmap=$(POD_NAMESPACE)/ingress-nginx + - --kubeconfig=/etc/kubernetes/kubeconfig.yaml + volumes: + - name: "kubeconfig" + hostPath: + path: "/etc/kubernetes/" +``` \ No newline at end of file From db17db812d4752312beaa64a6c2a94cffc4f3bda Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Thu, 19 Jan 2017 00:07:54 -0300 Subject: [PATCH 07/32] Add annotation for port in redirect --- .../annotations/portinredirect/main.go | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 core/pkg/ingress/annotations/portinredirect/main.go diff --git a/core/pkg/ingress/annotations/portinredirect/main.go b/core/pkg/ingress/annotations/portinredirect/main.go new file mode 100644 index 000000000..c76de0940 --- /dev/null +++ b/core/pkg/ingress/annotations/portinredirect/main.go @@ -0,0 +1,41 @@ +/* +Copyright 2016 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package cors + +import ( + "k8s.io/kubernetes/pkg/apis/extensions" + + "k8s.io/ingress/core/pkg/ingress/annotations/parser" +) + +const ( + annotation = "ingress.kubernetes.io/port-in-redirect" +) + +type portInRedirect struct { +} + +// NewParser creates a new port in redirect annotation parser +func NewParser() parser.IngressAnnotation { + return portInRedirect{} +} + +// Parse parses the annotations contained in the ingress +// rule used to indicate if the redirects must +func (a portInRedirect) Parse(ing *extensions.Ingress) (interface{}, error) { + return parser.GetBoolAnnotation(annotation, ing) +} From 4bdc7d87ffb343d6fe2042362cc8819689ef73c0 Mon Sep 17 00:00:00 2001 From: chentao1596 Date: Fri, 20 Jan 2017 17:06:44 +0800 Subject: [PATCH 08/32] add unit test cases for annotations.cors --- .../pkg/ingress/annotations/cors/main_test.go | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 core/pkg/ingress/annotations/cors/main_test.go diff --git a/core/pkg/ingress/annotations/cors/main_test.go b/core/pkg/ingress/annotations/cors/main_test.go new file mode 100644 index 000000000..61521f301 --- /dev/null +++ b/core/pkg/ingress/annotations/cors/main_test.go @@ -0,0 +1,62 @@ +/* +Copyright 2017 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package cors + +import ( + "testing" + + "k8s.io/kubernetes/pkg/api" + "k8s.io/kubernetes/pkg/apis/extensions" +) + +const ( + notCorsAnnotation = "ingress.kubernetes.io/enable-not-cors" +) + +func TestParse(t *testing.T) { + ap := NewParser() + if ap == nil { + t.Fatalf("expected a parser.IngressAnnotation but returned nil") + } + + testCases := []struct { + annotations map[string]string + expected bool + }{ + {map[string]string{annotation: "true"}, true}, + {map[string]string{annotation: "false"}, false}, + {map[string]string{notCorsAnnotation: "true"}, false}, + {map[string]string{}, false}, + {nil, false}, + } + + ing := &extensions.Ingress{ + ObjectMeta: api.ObjectMeta{ + Name: "foo", + Namespace: api.NamespaceDefault, + }, + Spec: extensions.IngressSpec{}, + } + + for _, testCase := range testCases { + ing.SetAnnotations(testCase.annotations) + result, _ := ap.Parse(ing) + if result != testCase.expected { + t.Errorf("expected %t but returned %t, annotations: %s", testCase.expected, result, testCase.annotations) + } + } +} From 3df139cb565d8c662ff9f88527127be77701e936 Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Fri, 20 Jan 2017 19:37:59 -0300 Subject: [PATCH 09/32] Add configuration and annotation for port_in_redirect --- controllers/nginx/pkg/cmd/controller/nginx.go | 42 +++++- controllers/nginx/pkg/config/config.go | 1 + controllers/nginx/pkg/template/configmap.go | 26 ++-- .../nginx/pkg/template/configmap_test.go | 29 ++--- .../rootfs/etc/nginx/template/nginx.tmpl | 2 + core/pkg/ingress/annotations/parser/main.go | 4 +- .../annotations/portinredirect/main.go | 17 ++- .../annotations/portinredirect/main_test.go | 120 ++++++++++++++++++ core/pkg/ingress/controller/annotations.go | 24 ++-- core/pkg/ingress/controller/controller.go | 38 +++--- core/pkg/ingress/defaults/main.go | 4 + core/pkg/ingress/errors/errors.go | 4 +- core/pkg/ingress/errors/errors_test.go | 2 +- core/pkg/ingress/types.go | 8 +- 14 files changed, 249 insertions(+), 72 deletions(-) create mode 100644 core/pkg/ingress/annotations/portinredirect/main_test.go diff --git a/controllers/nginx/pkg/cmd/controller/nginx.go b/controllers/nginx/pkg/cmd/controller/nginx.go index 1700ce555..9e2bb64d2 100644 --- a/controllers/nginx/pkg/cmd/controller/nginx.go +++ b/controllers/nginx/pkg/cmd/controller/nginx.go @@ -29,6 +29,8 @@ import ( "time" "github.com/golang/glog" + "github.com/mitchellh/mapstructure" + "k8s.io/kubernetes/pkg/api" "k8s.io/ingress/controllers/nginx/pkg/config" @@ -58,7 +60,10 @@ func newNGINXController() ingress.Controller { if ngx == "" { ngx = binary } - n := NGINXController{binary: ngx} + n := NGINXController{ + binary: ngx, + configmap: &api.ConfigMap{}, + } var onChange func() onChange = func() { @@ -87,13 +92,15 @@ Error loading new template : %v go n.Start() - return n + return ingress.Controller(&n) } // NGINXController ... type NGINXController struct { t *ngx_template.Template + configmap *api.ConfigMap + binary string } @@ -170,11 +177,31 @@ func (n NGINXController) Reload(data []byte) ([]byte, bool, error) { // BackendDefaults returns the nginx defaults func (n NGINXController) BackendDefaults() defaults.Backend { + if n.configmap == nil { + d := config.NewDefault() + return d.Backend + } + + return n.backendDefaults() +} + +func (n *NGINXController) backendDefaults() defaults.Backend { d := config.NewDefault() + config := &mapstructure.DecoderConfig{ + Metadata: nil, + WeaklyTypedInput: true, + Result: &d, + TagName: "json", + } + decoder, err := mapstructure.NewDecoder(config) + if err != nil { + glog.Warningf("unexpected error merging defaults: %v", err) + } + decoder.Decode(n.configmap.Data) return d.Backend } -// IsReloadRequired check if the new configuration file is different +// isReloadRequired check if the new configuration file is different // from the current one. func (n NGINXController) isReloadRequired(data []byte) bool { in, err := os.Open(cfgPath) @@ -249,6 +276,11 @@ Error: %v return nil } +// SetConfig ... +func (n *NGINXController) SetConfig(cmap *api.ConfigMap) { + n.configmap = cmap +} + // OnUpdate is called by syncQueue in https://github.com/aledbf/ingress-controller/blob/master/pkg/ingress/controller/controller.go#L82 // periodically to keep the configuration in sync. // @@ -257,7 +289,7 @@ Error: %v // write the configuration file // returning nill implies the backend will be reloaded. // if an error is returned means requeue the update -func (n NGINXController) OnUpdate(cmap *api.ConfigMap, ingressCfg ingress.Configuration) ([]byte, error) { +func (n *NGINXController) OnUpdate(ingressCfg ingress.Configuration) ([]byte, error) { var longestName int var serverNames int for _, srv := range ingressCfg.Servers { @@ -267,7 +299,7 @@ func (n NGINXController) OnUpdate(cmap *api.ConfigMap, ingressCfg ingress.Config } } - cfg := ngx_template.ReadConfig(cmap) + cfg := ngx_template.ReadConfig(n.configmap.Data) // NGINX cannot resize the has tables used to store server names. // For this reason we check if the defined size defined is correct diff --git a/controllers/nginx/pkg/config/config.go b/controllers/nginx/pkg/config/config.go index d7ad8aff8..c3dc11331 100644 --- a/controllers/nginx/pkg/config/config.go +++ b/controllers/nginx/pkg/config/config.go @@ -270,6 +270,7 @@ func NewDefault() Configuration { CustomHTTPErrors: []int{}, WhitelistSourceRange: []string{}, SkipAccessLogURLs: []string{}, + UsePortInRedirects: false, }, } diff --git a/controllers/nginx/pkg/template/configmap.go b/controllers/nginx/pkg/template/configmap.go index 3617988ad..701d5a238 100644 --- a/controllers/nginx/pkg/template/configmap.go +++ b/controllers/nginx/pkg/template/configmap.go @@ -23,8 +23,6 @@ import ( "github.com/golang/glog" "github.com/mitchellh/mapstructure" - "k8s.io/kubernetes/pkg/api" - "k8s.io/ingress/controllers/nginx/pkg/config" "k8s.io/ingress/core/pkg/net/dns" ) @@ -36,13 +34,21 @@ const ( ) // ReadConfig obtains the configuration defined by the user merged with the defaults. -func ReadConfig(conf *api.ConfigMap) config.Configuration { +func ReadConfig(src map[string]string) config.Configuration { + conf := map[string]string{} + if src != nil { + // we need to copy the configmap data because the content is altered + for k, v := range src { + conf[k] = v + } + } + errors := make([]int, 0) skipUrls := make([]string, 0) whitelist := make([]string, 0) - if val, ok := conf.Data[customHTTPErrors]; ok { - delete(conf.Data, customHTTPErrors) + if val, ok := conf[customHTTPErrors]; ok { + delete(conf, customHTTPErrors) for _, i := range strings.Split(val, ",") { j, err := strconv.Atoi(i) if err != nil { @@ -52,12 +58,12 @@ func ReadConfig(conf *api.ConfigMap) config.Configuration { } } } - if val, ok := conf.Data[skipAccessLogUrls]; ok { - delete(conf.Data, skipAccessLogUrls) + if val, ok := conf[skipAccessLogUrls]; ok { + delete(conf, skipAccessLogUrls) skipUrls = strings.Split(val, ",") } - if val, ok := conf.Data[whitelistSourceRange]; ok { - delete(conf.Data, whitelistSourceRange) + if val, ok := conf[whitelistSourceRange]; ok { + delete(conf, whitelistSourceRange) whitelist = append(whitelist, strings.Split(val, ",")...) } @@ -84,7 +90,7 @@ func ReadConfig(conf *api.ConfigMap) config.Configuration { if err != nil { glog.Warningf("unexpected error merging defaults: %v", err) } - err = decoder.Decode(conf.Data) + err = decoder.Decode(conf) if err != nil { glog.Warningf("unexpected error merging defaults: %v", err) } diff --git a/controllers/nginx/pkg/template/configmap_test.go b/controllers/nginx/pkg/template/configmap_test.go index 028fb2168..2e4c43af2 100644 --- a/controllers/nginx/pkg/template/configmap_test.go +++ b/controllers/nginx/pkg/template/configmap_test.go @@ -23,7 +23,6 @@ import ( "k8s.io/ingress/controllers/nginx/pkg/config" "k8s.io/ingress/core/pkg/net/dns" - "k8s.io/kubernetes/pkg/api" ) func TestFilterErrors(t *testing.T) { @@ -34,17 +33,15 @@ func TestFilterErrors(t *testing.T) { } func TestMergeConfigMapToStruct(t *testing.T) { - conf := &api.ConfigMap{ - Data: map[string]string{ - "custom-http-errors": "300,400,demo", - "proxy-read-timeout": "1", - "proxy-send-timeout": "2", - "skip-access-log-urls": "/log,/demo,/test", - "use-proxy-protocol": "true", - "use-gzip": "true", - "enable-dynamic-tls-records": "false", - "gzip-types": "text/html", - }, + conf := map[string]string{ + "custom-http-errors": "300,400,demo", + "proxy-read-timeout": "1", + "proxy-send-timeout": "2", + "skip-access-log-urls": "/log,/demo,/test", + "use-proxy-protocol": "true", + "use-gzip": "true", + "enable-dynamic-tls-records": "false", + "gzip-types": "text/html", } def := config.NewDefault() def.CustomHTTPErrors = []int{300, 400} @@ -68,7 +65,7 @@ func TestMergeConfigMapToStruct(t *testing.T) { def = config.NewDefault() def.Resolver = h - to = ReadConfig(&api.ConfigMap{}) + to = ReadConfig(map[string]string{}) if diff := pretty.Compare(to, def); diff != "" { t.Errorf("unexpected diff: (-got +want)\n%s", diff) } @@ -76,10 +73,8 @@ func TestMergeConfigMapToStruct(t *testing.T) { def = config.NewDefault() def.Resolver = h def.WhitelistSourceRange = []string{"1.1.1.1/32"} - to = ReadConfig(&api.ConfigMap{ - Data: map[string]string{ - "whitelist-source-range": "1.1.1.1/32", - }, + to = ReadConfig(map[string]string{ + "whitelist-source-range": "1.1.1.1/32", }) if diff := pretty.Compare(to, def); diff != "" { diff --git a/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl b/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl index 7e4979956..5f10d030b 100644 --- a/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl +++ b/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl @@ -251,6 +251,8 @@ http { deny all; {{ end }} + port_in_redirect {{ if $location.UsePortInRedirects }}on{{ else }}off{{ end }}; + {{ if not (empty $authPath) }} # this location requires authentication auth_request {{ $authPath }}; diff --git a/core/pkg/ingress/annotations/parser/main.go b/core/pkg/ingress/annotations/parser/main.go index eb505ae74..bff6f2210 100644 --- a/core/pkg/ingress/annotations/parser/main.go +++ b/core/pkg/ingress/annotations/parser/main.go @@ -36,7 +36,7 @@ func (a ingAnnotations) parseBool(name string) (bool, error) { if ok { b, err := strconv.ParseBool(val) if err != nil { - return false, errors.NewInvalidAnnotationContent(name) + return false, errors.NewInvalidAnnotationContent(name, val) } return b, nil } @@ -56,7 +56,7 @@ func (a ingAnnotations) parseInt(name string) (int, error) { if ok { i, err := strconv.Atoi(val) if err != nil { - return 0, errors.NewInvalidAnnotationContent(name) + return 0, errors.NewInvalidAnnotationContent(name, val) } return i, nil } diff --git a/core/pkg/ingress/annotations/portinredirect/main.go b/core/pkg/ingress/annotations/portinredirect/main.go index c76de0940..1b16e8dcd 100644 --- a/core/pkg/ingress/annotations/portinredirect/main.go +++ b/core/pkg/ingress/annotations/portinredirect/main.go @@ -14,28 +14,35 @@ See the License for the specific language governing permissions and limitations under the License. */ -package cors +package portinredirect import ( "k8s.io/kubernetes/pkg/apis/extensions" "k8s.io/ingress/core/pkg/ingress/annotations/parser" + "k8s.io/ingress/core/pkg/ingress/resolver" ) const ( - annotation = "ingress.kubernetes.io/port-in-redirect" + annotation = "ingress.kubernetes.io/use-port-in-redirects" ) type portInRedirect struct { + backendResolver resolver.DefaultBackend } // NewParser creates a new port in redirect annotation parser -func NewParser() parser.IngressAnnotation { - return portInRedirect{} +func NewParser(db resolver.DefaultBackend) parser.IngressAnnotation { + return portInRedirect{db} } // Parse parses the annotations contained in the ingress // rule used to indicate if the redirects must func (a portInRedirect) Parse(ing *extensions.Ingress) (interface{}, error) { - return parser.GetBoolAnnotation(annotation, ing) + up, err := parser.GetBoolAnnotation(annotation, ing) + if err != nil { + return a.backendResolver.GetDefaultBackend().UsePortInRedirects, nil + } + + return up, nil } diff --git a/core/pkg/ingress/annotations/portinredirect/main_test.go b/core/pkg/ingress/annotations/portinredirect/main_test.go new file mode 100644 index 000000000..cb6403fd1 --- /dev/null +++ b/core/pkg/ingress/annotations/portinredirect/main_test.go @@ -0,0 +1,120 @@ +/* +Copyright 2016 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package portinredirect + +import ( + "testing" + + "k8s.io/kubernetes/pkg/api" + "k8s.io/kubernetes/pkg/apis/extensions" + "k8s.io/kubernetes/pkg/util/intstr" + + "fmt" + + "k8s.io/ingress/core/pkg/ingress/defaults" +) + +func buildIngress() *extensions.Ingress { + defaultBackend := extensions.IngressBackend{ + ServiceName: "default-backend", + ServicePort: intstr.FromInt(80), + } + + return &extensions.Ingress{ + ObjectMeta: api.ObjectMeta{ + Name: "foo", + Namespace: api.NamespaceDefault, + }, + Spec: extensions.IngressSpec{ + Backend: &extensions.IngressBackend{ + ServiceName: "default-backend", + ServicePort: intstr.FromInt(80), + }, + Rules: []extensions.IngressRule{ + { + Host: "foo.bar.com", + IngressRuleValue: extensions.IngressRuleValue{ + HTTP: &extensions.HTTPIngressRuleValue{ + Paths: []extensions.HTTPIngressPath{ + { + Path: "/foo", + Backend: defaultBackend, + }, + }, + }, + }, + }, + }, + }, + } +} + +type mockBackend struct { + usePortInRedirects bool +} + +func (m mockBackend) GetDefaultBackend() defaults.Backend { + return defaults.Backend{UsePortInRedirects: m.usePortInRedirects} +} + +func TestPortInRedirect(t *testing.T) { + tests := []struct { + title string + usePort *bool + def bool + exp bool + }{ + {"false - default false", newFalse(), false, false}, + {"false - default true", newFalse(), true, false}, + {"no annotation - default false", nil, false, false}, + {"no annotation - default true", nil, true, true}, + {"true - default true", newTrue(), true, true}, + } + + for _, test := range tests { + ing := buildIngress() + + data := map[string]string{} + if test.usePort != nil { + data[annotation] = fmt.Sprintf("%v", *test.usePort) + } + ing.SetAnnotations(data) + + i, err := NewParser(mockBackend{test.def}).Parse(ing) + if err != nil { + t.Errorf("unexpected error parsing a valid") + } + p, ok := i.(bool) + if !ok { + t.Errorf("expected a bool type") + } + + if p != test.exp { + t.Errorf("%v: expected \"%v\" but \"%v\" was returned", test.title, test.exp, p) + } + } +} + +func newTrue() *bool { + b := true + return &b +} + +func newFalse() *bool { + b := false + return &b +} diff --git a/core/pkg/ingress/controller/annotations.go b/core/pkg/ingress/controller/annotations.go index 00f62f7f5..c1eb09fbd 100644 --- a/core/pkg/ingress/controller/annotations.go +++ b/core/pkg/ingress/controller/annotations.go @@ -28,6 +28,7 @@ import ( "k8s.io/ingress/core/pkg/ingress/annotations/healthcheck" "k8s.io/ingress/core/pkg/ingress/annotations/ipwhitelist" "k8s.io/ingress/core/pkg/ingress/annotations/parser" + "k8s.io/ingress/core/pkg/ingress/annotations/portinredirect" "k8s.io/ingress/core/pkg/ingress/annotations/proxy" "k8s.io/ingress/core/pkg/ingress/annotations/ratelimit" "k8s.io/ingress/core/pkg/ingress/annotations/rewrite" @@ -50,17 +51,18 @@ type annotationExtractor struct { func newAnnotationExtractor(cfg extractorConfig) annotationExtractor { return annotationExtractor{ map[string]parser.IngressAnnotation{ - "BasicDigestAuth": auth.NewParser(auth.AuthDirectory, cfg), - "ExternalAuth": authreq.NewParser(), - "CertificateAuth": authtls.NewParser(cfg), - "EnableCORS": cors.NewParser(), - "HealthCheck": healthcheck.NewParser(cfg), - "Whitelist": ipwhitelist.NewParser(cfg), - "Proxy": proxy.NewParser(cfg), - "RateLimit": ratelimit.NewParser(), - "Redirect": rewrite.NewParser(cfg), - "SecureUpstream": secureupstream.NewParser(), - "SSLPassthrough": sslpassthrough.NewParser(), + "BasicDigestAuth": auth.NewParser(auth.AuthDirectory, cfg), + "ExternalAuth": authreq.NewParser(), + "CertificateAuth": authtls.NewParser(cfg), + "EnableCORS": cors.NewParser(), + "HealthCheck": healthcheck.NewParser(cfg), + "Whitelist": ipwhitelist.NewParser(cfg), + "UsePortInRedirects": portinredirect.NewParser(cfg), + "Proxy": proxy.NewParser(cfg), + "RateLimit": ratelimit.NewParser(), + "Redirect": rewrite.NewParser(cfg), + "SecureUpstream": secureupstream.NewParser(), + "SSLPassthrough": sslpassthrough.NewParser(), }, } } diff --git a/core/pkg/ingress/controller/controller.go b/core/pkg/ingress/controller/controller.go index b82fae771..29b2c30d3 100644 --- a/core/pkg/ingress/controller/controller.go +++ b/core/pkg/ingress/controller/controller.go @@ -223,10 +223,22 @@ func newIngressController(config *Configuration) *GenericController { } mapEventHandler := cache.ResourceEventHandlerFuncs{ + AddFunc: func(obj interface{}) { + upCmap := obj.(*api.ConfigMap) + mapKey := fmt.Sprintf("%s/%s", upCmap.Namespace, upCmap.Name) + if mapKey == ic.cfg.ConfigMapName { + glog.V(2).Infof("adding configmap %v to backend", mapKey) + ic.cfg.Backend.SetConfig(upCmap) + } + }, UpdateFunc: func(old, cur interface{}) { if !reflect.DeepEqual(old, cur) { upCmap := cur.(*api.ConfigMap) mapKey := fmt.Sprintf("%s/%s", upCmap.Namespace, upCmap.Name) + if mapKey == ic.cfg.ConfigMapName { + glog.V(2).Infof("updating configmap backend (%v)", mapKey) + ic.cfg.Backend.SetConfig(upCmap) + } // updates to configuration configmaps can trigger an update if mapKey == ic.cfg.ConfigMapName || mapKey == ic.cfg.TCPConfigMapName || mapKey == ic.cfg.UDPConfigMapName { ic.recorder.Eventf(upCmap, api.EventTypeNormal, "UPDATE", fmt.Sprintf("ConfigMap %v", mapKey)) @@ -310,8 +322,14 @@ func (ic GenericController) GetSecret(name string) (*api.Secret, error) { } func (ic *GenericController) getConfigMap(ns, name string) (*api.ConfigMap, error) { - // TODO: check why ic.mapLister.Store.GetByKey(mapKey) is not stable (random content) - return ic.cfg.Client.ConfigMaps(ns).Get(name) + s, exists, err := ic.mapLister.Store.GetByKey(fmt.Sprintf("%v/%v", ns, name)) + if err != nil { + return nil, err + } + if !exists { + return nil, fmt.Errorf("configmap %v was not found", name) + } + return s.(*api.ConfigMap), nil } // sync collects all the pieces required to assemble the configuration file and @@ -329,20 +347,6 @@ func (ic *GenericController) sync(key interface{}) error { return fmt.Errorf("deferring sync till endpoints controller has synced") } - // by default no custom configuration - cfg := &api.ConfigMap{} - - if ic.cfg.ConfigMapName != "" { - // search for custom configmap (defined in main args) - var err error - ns, name, _ := k8s.ParseNameNS(ic.cfg.ConfigMapName) - cfg, err = ic.getConfigMap(ns, name) - if err != nil { - // requeue - return fmt.Errorf("unexpected error searching configmap %v: %v", ic.cfg.ConfigMapName, err) - } - } - upstreams, servers := ic.getBackendServers() var passUpstreams []*ingress.SSLPassthroughBackend for _, server := range servers { @@ -362,7 +366,7 @@ func (ic *GenericController) sync(key interface{}) error { } } - data, err := ic.cfg.Backend.OnUpdate(cfg, ingress.Configuration{ + data, err := ic.cfg.Backend.OnUpdate(ingress.Configuration{ Backends: upstreams, Servers: servers, TCPEndpoints: ic.getTCPServices(), diff --git a/core/pkg/ingress/defaults/main.go b/core/pkg/ingress/defaults/main.go index 12badd1b2..ba56bc7c9 100644 --- a/core/pkg/ingress/defaults/main.go +++ b/core/pkg/ingress/defaults/main.go @@ -49,6 +49,10 @@ type Backend struct { // Enables or disables the redirect (301) to the HTTPS port SSLRedirect bool `json:"ssl-redirect"` + // Enables or disables the specification of port in redirects + // Default: false + UsePortInRedirects bool `json:"use-port-in-redirects"` + // Number of unsuccessful attempts to communicate with the server that should happen in the // duration set by the fail_timeout parameter to consider the server unavailable // http://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream diff --git a/core/pkg/ingress/errors/errors.go b/core/pkg/ingress/errors/errors.go index 924753963..8db16c780 100644 --- a/core/pkg/ingress/errors/errors.go +++ b/core/pkg/ingress/errors/errors.go @@ -37,9 +37,9 @@ var ( ) // NewInvalidAnnotationContent returns a new InvalidContent error -func NewInvalidAnnotationContent(name string) error { +func NewInvalidAnnotationContent(name string, val interface{}) error { return InvalidContent{ - Name: fmt.Sprintf("the annotation %v does not contains a valid value", name), + Name: fmt.Sprintf("the annotation %v does not contains a valid value (%v)", name, val), } } diff --git a/core/pkg/ingress/errors/errors_test.go b/core/pkg/ingress/errors/errors_test.go index c0bd52406..957f10be3 100644 --- a/core/pkg/ingress/errors/errors_test.go +++ b/core/pkg/ingress/errors/errors_test.go @@ -38,7 +38,7 @@ func TestInvalidContent(t *testing.T) { if IsInvalidContent(ErrMissingAnnotations) { t.Error("expected false") } - err := NewInvalidAnnotationContent("demo") + err := NewInvalidAnnotationContent("demo", "") if !IsInvalidContent(err) { t.Error("expected true") } diff --git a/core/pkg/ingress/types.go b/core/pkg/ingress/types.go index dc8080987..4891995e7 100644 --- a/core/pkg/ingress/types.go +++ b/core/pkg/ingress/types.go @@ -68,7 +68,6 @@ type Controller interface { // Notifications of type Add, Update and Delete: // https://github.com/kubernetes/kubernetes/blob/master/pkg/client/cache/controller.go#L164 // - // ConfigMap content of --configmap // Configuration returns the translation from Ingress rules containing // information about all the upstreams (service endpoints ) "virtual" // servers (FQDN) and all the locations inside each server. Each @@ -79,7 +78,9 @@ type Controller interface { // // The returned configuration is then passed to test, and then to reload // if there is no errors. - OnUpdate(*api.ConfigMap, Configuration) ([]byte, error) + OnUpdate(Configuration) ([]byte, error) + // ConfigMap content of --configmap + SetConfig(*api.ConfigMap) // BackendDefaults returns the minimum settings required to configure the // communication to endpoints BackendDefaults() defaults.Backend @@ -233,6 +234,9 @@ type Location struct { // external authentication // +optional CertificateAuth resolver.AuthSSLCert `json:"certificateAuth,omitempty"` + // UsePortInRedirects indicates if redirects must specify the port + // +optional + UsePortInRedirects bool `json:"use-port-in-redirects"` } // SSLPassthroughBackend describes a SSL upstream server configured From 50297c8f47a60be8dd55491fbbb5195bcf82736c Mon Sep 17 00:00:00 2001 From: Tang Le Date: Mon, 23 Jan 2017 10:01:51 +0800 Subject: [PATCH 10/32] Fix issue for ratelimit Signed-off-by: Tang Le --- controllers/nginx/pkg/template/template.go | 10 +++++----- core/pkg/ingress/annotations/ratelimit/main.go | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/controllers/nginx/pkg/template/template.go b/controllers/nginx/pkg/template/template.go index 1f1251237..18373bda0 100644 --- a/controllers/nginx/pkg/template/template.go +++ b/controllers/nginx/pkg/template/template.go @@ -346,10 +346,10 @@ func buildRateLimitZones(input interface{}) []string { } if loc.RateLimit.RPS.Limit > 0 { - zone := fmt.Sprintf("limit_conn_zone $binary_remote_addr zone=%v:%vm rate=%vr/s;", - loc.RateLimit.Connections.Name, - loc.RateLimit.Connections.SharedSize, - loc.RateLimit.Connections.Limit) + zone := fmt.Sprintf("limit_req_zone $binary_remote_addr zone=%v:%vm rate=%vr/s;", + loc.RateLimit.RPS.Name, + loc.RateLimit.RPS.SharedSize, + loc.RateLimit.RPS.Limit) zones = append(zones, zone) } } @@ -376,7 +376,7 @@ func buildRateLimit(input interface{}) []string { if loc.RateLimit.RPS.Limit > 0 { limit := fmt.Sprintf("limit_req zone=%v burst=%v nodelay;", - loc.RateLimit.Connections.Name, loc.RateLimit.Connections.Burst) + loc.RateLimit.RPS.Name, loc.RateLimit.RPS.Burst) limits = append(limits, limit) } diff --git a/core/pkg/ingress/annotations/ratelimit/main.go b/core/pkg/ingress/annotations/ratelimit/main.go index 83ce4d2c0..f3ca328b7 100644 --- a/core/pkg/ingress/annotations/ratelimit/main.go +++ b/core/pkg/ingress/annotations/ratelimit/main.go @@ -91,7 +91,7 @@ func (a ratelimit) Parse(ing *extensions.Ingress) (interface{}, error) { RPS: Zone{ Name: fmt.Sprintf("%v_rps", zoneName), Limit: rps, - Burst: conn * defBurst, + Burst: rps * defBurst, SharedSize: defSharedSize, }, }, nil From 59903592ac04d2df7a2a2b64a1d9fc7ce77d7c2b Mon Sep 17 00:00:00 2001 From: chentao1596 Date: Mon, 23 Jan 2017 11:23:06 +0800 Subject: [PATCH 11/32] Prefect unit test cases for annotation.proxy --- .../ingress/annotations/proxy/main_test.go | 48 +++++++++++++++++-- 1 file changed, 45 insertions(+), 3 deletions(-) diff --git a/core/pkg/ingress/annotations/proxy/main_test.go b/core/pkg/ingress/annotations/proxy/main_test.go index 6b39a2d12..8671c42a4 100644 --- a/core/pkg/ingress/annotations/proxy/main_test.go +++ b/core/pkg/ingress/annotations/proxy/main_test.go @@ -65,7 +65,14 @@ type mockBackend struct { } func (m mockBackend) GetDefaultBackend() defaults.Backend { - return defaults.Backend{UpstreamFailTimeout: 1} + return defaults.Backend{ + UpstreamFailTimeout: 1, + ProxyConnectTimeout: 10, + ProxySendTimeout: 15, + ProxyReadTimeout: 20, + ProxyBufferSize: "10k", + ProxyBodySize: "3k", + } } func TestProxy(t *testing.T) { @@ -76,15 +83,16 @@ func TestProxy(t *testing.T) { data[send] = "2" data[read] = "3" data[bufferSize] = "1k" + data[bodySize] = "2k" ing.SetAnnotations(data) i, err := NewParser(mockBackend{}).Parse(ing) if err != nil { - t.Errorf("unexpected error parsing a valid") + t.Fatalf("unexpected error parsing a valid") } p, ok := i.(*Configuration) if !ok { - t.Errorf("expected a Configuration type") + t.Fatalf("expected a Configuration type") } if p.ConnectTimeout != 1 { t.Errorf("expected 1 as connect-timeout but returned %v", p.ConnectTimeout) @@ -98,4 +106,38 @@ func TestProxy(t *testing.T) { if p.BufferSize != "1k" { t.Errorf("expected 1k as buffer-size but returned %v", p.BufferSize) } + if p.BodySize != "2k" { + t.Errorf("expected 2k as body-size but returned %v", p.BodySize) + } +} + +func TestProxyWithNoAnnotation(t *testing.T) { + ing := buildIngress() + + data := map[string]string{} + ing.SetAnnotations(data) + + i, err := NewParser(mockBackend{}).Parse(ing) + if err != nil { + t.Fatalf("unexpected error parsing a valid") + } + p, ok := i.(*Configuration) + if !ok { + t.Fatalf("expected a Configuration type") + } + if p.ConnectTimeout != 10 { + t.Errorf("expected 10 as connect-timeout but returned %v", p.ConnectTimeout) + } + if p.SendTimeout != 15 { + t.Errorf("expected 15 as send-timeout but returned %v", p.SendTimeout) + } + if p.ReadTimeout != 20 { + t.Errorf("expected 20 as read-timeout but returned %v", p.ReadTimeout) + } + if p.BufferSize != "10k" { + t.Errorf("expected 10k as buffer-size but returned %v", p.BufferSize) + } + if p.BodySize != "3k" { + t.Errorf("expected 3k as body-size but returned %v", p.BodySize) + } } From 6c8792d80a9309d36f10ee5590e51ccf84e67d51 Mon Sep 17 00:00:00 2001 From: Peter Sutherland Date: Sun, 15 Jan 2017 22:12:02 +0000 Subject: [PATCH 12/32] Add whitelist-source-range to config map docs --- controllers/nginx/configuration.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/controllers/nginx/configuration.md b/controllers/nginx/configuration.md index 9dd2121cd..fc79f29f5 100644 --- a/controllers/nginx/configuration.md +++ b/controllers/nginx/configuration.md @@ -318,6 +318,9 @@ The default mime type list to compress is: `application/atom+xml application/jav **use-proxy-protocol:** Enables or disables the [PROXY protocol](https://www.nginx.com/resources/admin-guide/proxy-protocol/) to receive client connection (real IP address) information passed through proxy servers and load balancers such as HAProxy and Amazon Elastic Load Balancer (ELB). +**whitelist-source-range:** Sets the default whitelisted IPs for each `server` block. This can be overwritten by an annotation on an Ingress rule. See [ngx_http_access_module](http://nginx.org/en/docs/http/ngx_http_access_module.html). + + **worker-processes:** Sets the number of [worker processes](http://nginx.org/en/docs/ngx_core_module.html#worker_processes). The default of "auto" means number of available CPU cores. @@ -355,6 +358,7 @@ The following table shows the options, the default value and a description. |use-gzip|"true"| |use-http2|"true"| |vts-status-zone-size|10m| +|whitelist-source-range|permit all| |worker-processes|number of CPUs| From e665072eaa2af13f566ebe44b50ac2864fe341c3 Mon Sep 17 00:00:00 2001 From: Peter Sutherland Date: Sun, 15 Jan 2017 22:56:44 +0000 Subject: [PATCH 13/32] Document more parameters and list defaults --- controllers/nginx/configuration.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/controllers/nginx/configuration.md b/controllers/nginx/configuration.md index fc79f29f5..dec787ba7 100644 --- a/controllers/nginx/configuration.md +++ b/controllers/nginx/configuration.md @@ -188,9 +188,15 @@ Setting at least one code also enables [proxy_intercept_errors](http://nginx.org Example usage: `custom-http-errors: 404,415` +**enable-dynamic-tls-records:** Enables dynamically sized TLS records to improve time-to-first-byte. Enabled by default. See [CloudFlare's blog](https://blog.cloudflare.com/optimizing-tls-over-tcp-to-reduce-latency) for more information. + + **enable-sticky-sessions:** Enables sticky sessions using cookies. This is provided by [nginx-sticky-module-ng](https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng) module. +**enable-spdy:** Enables the SPDY protocol. + + **enable-vts-status:** Allows the replacement of the default status page with a third party module named [nginx-module-vts](https://github.com/vozlt/nginx-module-vts). @@ -248,6 +254,8 @@ http://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_bucket http://nginx.org/en/docs/hash.html +**server-tokens:** Send NGINX Server header in responses and display NGINX version in error pages. Enabled by default. + **map-hash-bucket-size:** Sets the bucket size for the [map variables hash tables](http://nginx.org/en/docs/http/ngx_http_map_module.html#map_hash_bucket_size). The details of setting up hash tables are provided in a separate [document](http://nginx.org/en/docs/hash.html). @@ -332,14 +340,17 @@ The following table shows the options, the default value and a description. |---------------------------|------| |body-size|1m| |custom-http-errors|" "| +|enable-dynamic-tls-records|"true"| +|enable-spdy|"true"| |enable-sticky-sessions|"false"| |enable-vts-status|"false"| |error-log-level|notice| -|gzip-types|| +|gzip-types|see use-gzip description above| |hsts|"true"| |hsts-include-subdomains|"true"| |hsts-max-age|"15724800"| |keep-alive|"75"| +|map-hash-bucket-size|"64"| |max-worker-connections|"16384"| |proxy-connect-timeout|"5"| |proxy-read-timeout|"60"| @@ -348,8 +359,10 @@ The following table shows the options, the default value and a description. |retry-non-idempotent|"false"| |server-name-hash-bucket-size|"64"| |server-name-hash-max-size|"512"| +|server-tokens|"true"| |ssl-buffer-size|4k| |ssl-ciphers|| +|ssl-dh-param|value from openssl| |ssl-protocols|TLSv1 TLSv1.1 TLSv1.2| |ssl-session-cache|"true"| |ssl-session-cache-size|10m| From 8fae080cce679ca88c083b437ec89b4eec5fe4c5 Mon Sep 17 00:00:00 2001 From: Peter Sutherland Date: Mon, 23 Jan 2017 14:50:52 +0000 Subject: [PATCH 14/32] Remove SPDY documentation as it is broken --- controllers/nginx/configuration.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/controllers/nginx/configuration.md b/controllers/nginx/configuration.md index dec787ba7..776644b6f 100644 --- a/controllers/nginx/configuration.md +++ b/controllers/nginx/configuration.md @@ -194,9 +194,6 @@ Example usage: `custom-http-errors: 404,415` **enable-sticky-sessions:** Enables sticky sessions using cookies. This is provided by [nginx-sticky-module-ng](https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng) module. -**enable-spdy:** Enables the SPDY protocol. - - **enable-vts-status:** Allows the replacement of the default status page with a third party module named [nginx-module-vts](https://github.com/vozlt/nginx-module-vts). @@ -341,7 +338,6 @@ The following table shows the options, the default value and a description. |body-size|1m| |custom-http-errors|" "| |enable-dynamic-tls-records|"true"| -|enable-spdy|"true"| |enable-sticky-sessions|"false"| |enable-vts-status|"false"| |error-log-level|notice| From c0aca1833ab6ec21f05b3d05013bc6e1e1daba0e Mon Sep 17 00:00:00 2001 From: Tang Le Date: Tue, 24 Jan 2017 16:19:28 +0800 Subject: [PATCH 15/32] Fix rate limit issue when more than 2 servers enabled in ingress Signed-off-by: Tang Le --- controllers/nginx/pkg/template/template.go | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/controllers/nginx/pkg/template/template.go b/controllers/nginx/pkg/template/template.go index 18373bda0..cc21dccac 100644 --- a/controllers/nginx/pkg/template/template.go +++ b/controllers/nginx/pkg/template/template.go @@ -26,6 +26,8 @@ import ( "strings" text_template "text/template" + "k8s.io/kubernetes/pkg/util/sets" + "github.com/golang/glog" "k8s.io/ingress/controllers/nginx/pkg/config" @@ -328,11 +330,11 @@ func buildProxyPass(b interface{}, loc interface{}) string { // rate limiting of request. Each Ingress rule could have up to two zones, one // for connection limit by IP address and other for limiting request per second func buildRateLimitZones(input interface{}) []string { - zones := []string{} + zones := sets.String{} servers, ok := input.([]*ingress.Server) if !ok { - return zones + return zones.List() } for _, server := range servers { @@ -342,7 +344,9 @@ func buildRateLimitZones(input interface{}) []string { zone := fmt.Sprintf("limit_conn_zone $binary_remote_addr zone=%v:%vm;", loc.RateLimit.Connections.Name, loc.RateLimit.Connections.SharedSize) - zones = append(zones, zone) + if !zones.Has(zone) { + zones.Insert(zone) + } } if loc.RateLimit.RPS.Limit > 0 { @@ -350,12 +354,14 @@ func buildRateLimitZones(input interface{}) []string { loc.RateLimit.RPS.Name, loc.RateLimit.RPS.SharedSize, loc.RateLimit.RPS.Limit) - zones = append(zones, zone) + if !zones.Has(zone) { + zones.Insert(zone) + } } } } - return zones + return zones.List() } // buildRateLimit produces an array of limit_req to be used inside the Path of From a930b29e4155bb268c1e5dc61870fe6fdc6aa2a1 Mon Sep 17 00:00:00 2001 From: Ricardo Pchevuzinske Katz Date: Tue, 24 Jan 2017 11:21:49 -0200 Subject: [PATCH 16/32] Changes the SSL Temp file to something inside the same SSL Directory --- core/pkg/net/ssl/ssl.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/pkg/net/ssl/ssl.go b/core/pkg/net/ssl/ssl.go index 14a6fbef3..1ac6d2fa2 100644 --- a/core/pkg/net/ssl/ssl.go +++ b/core/pkg/net/ssl/ssl.go @@ -36,7 +36,7 @@ func AddOrUpdateCertAndKey(name string, cert, key, ca []byte) (*ingress.SSLCert, pemName := fmt.Sprintf("%v.pem", name) pemFileName := fmt.Sprintf("%v/%v", ingress.DefaultSSLDirectory, pemName) - tempPemFile, err := ioutil.TempFile("", pemName) + tempPemFile, err := ioutil.TempFile(ingress.DefaultSSLDirectory, pemName) if err != nil { return nil, fmt.Errorf("could not create temp pem file %v: %v", tempPemFile.Name(), err) } From f5706d1d74242c4a13076c2bb8b683967ade811c Mon Sep 17 00:00:00 2001 From: chentao1596 Date: Tue, 24 Jan 2017 11:08:46 +0800 Subject: [PATCH 17/32] prefect unit test cases for core.pkg.ingress.controller.util --- core/pkg/ingress/controller/util_test.go | 111 +++++++++++++++++++++++ 1 file changed, 111 insertions(+) diff --git a/core/pkg/ingress/controller/util_test.go b/core/pkg/ingress/controller/util_test.go index ba19ea830..370714533 100644 --- a/core/pkg/ingress/controller/util_test.go +++ b/core/pkg/ingress/controller/util_test.go @@ -19,10 +19,25 @@ package controller import ( "testing" + "k8s.io/ingress/core/pkg/ingress" + "k8s.io/ingress/core/pkg/ingress/annotations/auth" + "k8s.io/ingress/core/pkg/ingress/annotations/authreq" + "k8s.io/ingress/core/pkg/ingress/annotations/ipwhitelist" + "k8s.io/ingress/core/pkg/ingress/annotations/proxy" + "k8s.io/ingress/core/pkg/ingress/annotations/ratelimit" + "k8s.io/ingress/core/pkg/ingress/annotations/rewrite" + "k8s.io/ingress/core/pkg/ingress/resolver" "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/apis/extensions" + "reflect" ) +type fakeError struct{} + +func (fe *fakeError) Error() string { + return "fakeError" +} + func TestIsValidClass(t *testing.T) { ing := &extensions.Ingress{ ObjectMeta: api.ObjectMeta{ @@ -48,3 +63,99 @@ func TestIsValidClass(t *testing.T) { t.Errorf("Expected invalid class but %v returned", b) } } + +func TestIsHostValid(t *testing.T) { + fkCert := &ingress.SSLCert{ + CAFileName: "foo", + PemFileName: "foo.cr", + PemSHA: "perha", + CN: []string{ + "*.cluster.local", "default.local", + }, + } + + fooTests := []struct { + cr *ingress.SSLCert + host string + er bool + }{ + {nil, "foo1.cluster.local", false}, + {fkCert, "foo1.cluster.local", true}, + {fkCert, "default.local", true}, + {fkCert, "foo2.cluster.local.t", false}, + {fkCert, "", false}, + } + + for _, foo := range fooTests { + r := isHostValid(foo.host, foo.cr) + if r != foo.er { + t.Errorf("Returned %v but expected %v for foo=%v", r, foo.er, foo) + } + } +} + +func TestMatchHostnames(t *testing.T) { + fooTests := []struct { + pattern string + host string + er bool + }{ + {"*.cluster.local.", "foo1.cluster.local.", true}, + {"foo1.cluster.local.", "foo2.cluster.local.", false}, + {"cluster.local.", "foo1.cluster.local.", false}, + {".", "foo1.cluster.local.", false}, + {"cluster.local.", ".", false}, + } + + for _, foo := range fooTests { + r := matchHostnames(foo.pattern, foo.host) + if r != foo.er { + t.Errorf("Returned %v but expected %v for foo=%v", r, foo.er, foo) + } + } +} + +func TestMergeLocationAnnotations(t *testing.T) { + // initial parameters + loc := ingress.Location{} + annotations := map[string]interface{}{ + "Path": "/checkpath", + "IsDefBackend": true, + "Backend": "foo_backend", + "BasicDigestAuth": auth.BasicDigest{}, + DeniedKeyName: &fakeError{}, + "EnableCORS": true, + "ExternalAuth": authreq.External{}, + "RateLimit": ratelimit.RateLimit{}, + "Redirect": rewrite.Redirect{}, + "Whitelist": ipwhitelist.SourceRange{}, + "Proxy": proxy.Configuration{}, + "CertificateAuth": resolver.AuthSSLCert{}, + "UsePortInRedirects": true, + } + + // create test table + type fooMergeLocationAnnotationsStruct struct { + fName string + er interface{} + } + fooTests := []fooMergeLocationAnnotationsStruct{} + for name, value := range annotations { + fva := fooMergeLocationAnnotationsStruct{name, value} + fooTests = append(fooTests, fva) + } + + // execute test + mergeLocationAnnotations(&loc, annotations) + + // check result + for _, foo := range fooTests { + fv := reflect.ValueOf(loc).FieldByName(foo.fName).Interface() + if !reflect.DeepEqual(fv, foo.er) { + t.Errorf("Returned %v but expected %v for the field %s", fv, foo.er, foo.fName) + } + } + if _, ok := annotations[DeniedKeyName]; ok { + t.Errorf("%s should be removed after mergeLocationAnnotations", DeniedKeyName) + } +} From 0245868808b37888321a8f9297ea466a5801ee37 Mon Sep 17 00:00:00 2001 From: chentao1596 Date: Wed, 25 Jan 2017 10:17:45 +0800 Subject: [PATCH 18/32] prefect unit test cases for core.pkg.ingress.controller.annotations --- .../ingress/controller/annotations_test.go | 89 +++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/core/pkg/ingress/controller/annotations_test.go b/core/pkg/ingress/controller/annotations_test.go index 687830288..f577920db 100644 --- a/core/pkg/ingress/controller/annotations_test.go +++ b/core/pkg/ingress/controller/annotations_test.go @@ -27,6 +27,13 @@ import ( "k8s.io/ingress/core/pkg/ingress/resolver" ) +const ( + annotation_secureUpstream = "ingress.kubernetes.io/secure-backends" + annotation_upsMaxFails = "ingress.kubernetes.io/upstream-max-fails" + annotation_upsFailTimeout = "ingress.kubernetes.io/upstream-fail-timeout" + annotation_passthrough = "ingress.kubernetes.io/ssl-passthrough" +) + type mockCfg struct { } @@ -90,3 +97,85 @@ func buildIngress() *extensions.Ingress { }, } } + +func TestSecureUpstream(t *testing.T) { + ec := newAnnotationExtractor(mockCfg{}) + ing := buildIngress() + + fooAnns := []struct { + annotations map[string]string + er bool + }{ + {map[string]string{annotation_secureUpstream: "true"}, true}, + {map[string]string{annotation_secureUpstream: "false"}, false}, + {map[string]string{annotation_secureUpstream + "_no": "true"}, false}, + {map[string]string{}, false}, + {nil, false}, + } + + for _, foo := range fooAnns { + ing.SetAnnotations(foo.annotations) + r := ec.SecureUpstream(ing) + if r != foo.er { + t.Errorf("Returned %v but expected %v", r, foo.er) + } + } +} + +func TestHealthCheck(t *testing.T) { + ec := newAnnotationExtractor(mockCfg{}) + ing := buildIngress() + + fooAnns := []struct { + annotations map[string]string + eumf int + euft int + }{ + {map[string]string{annotation_upsMaxFails: "3", annotation_upsFailTimeout: "10"}, 3, 10}, + {map[string]string{annotation_upsMaxFails: "3"}, 3, 0}, + {map[string]string{annotation_upsFailTimeout: "10"}, 0, 10}, + {map[string]string{}, 0, 0}, + {nil, 0, 0}, + } + + for _, foo := range fooAnns { + ing.SetAnnotations(foo.annotations) + r := ec.HealthCheck(ing) + if r == nil { + t.Errorf("Returned nil but expected a healthcheck.Upstream") + continue + } + + if r.FailTimeout != foo.euft { + t.Errorf("Returned %d but expected %d for FailTimeout", r.FailTimeout, foo.euft) + } + + if r.MaxFails != foo.eumf { + t.Errorf("Returned %d but expected %d for MaxFails", r.MaxFails, foo.eumf) + } + } +} + +func TestSSLPassthrough(t *testing.T) { + ec := newAnnotationExtractor(mockCfg{}) + ing := buildIngress() + + fooAnns := []struct { + annotations map[string]string + er bool + }{ + {map[string]string{annotation_passthrough: "true"}, true}, + {map[string]string{annotation_passthrough: "false"}, false}, + {map[string]string{annotation_passthrough + "_no": "true"}, false}, + {map[string]string{}, false}, + {nil, false}, + } + + for _, foo := range fooAnns { + ing.SetAnnotations(foo.annotations) + r := ec.SSLPassthrough(ing) + if r != foo.er { + t.Errorf("Returned %v but expected %v", r, foo.er) + } + } +} From 93c712867ab83524a15ca9427f20cb83a8c8fa5c Mon Sep 17 00:00:00 2001 From: chentao1596 Date: Wed, 25 Jan 2017 10:59:46 +0800 Subject: [PATCH 19/32] add unit test cases for core.pkg.ingress.sort_ingress --- core/pkg/ingress/sort_ingress_test.go | 379 ++++++++++++++++++++++++++ 1 file changed, 379 insertions(+) create mode 100644 core/pkg/ingress/sort_ingress_test.go diff --git a/core/pkg/ingress/sort_ingress_test.go b/core/pkg/ingress/sort_ingress_test.go new file mode 100644 index 000000000..52ceb7029 --- /dev/null +++ b/core/pkg/ingress/sort_ingress_test.go @@ -0,0 +1,379 @@ +/* +Copyright 2017 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package ingress + +import ( + "testing" + + "k8s.io/kubernetes/pkg/api" +) + +func buildBackendByNameServers() BackendByNameServers { + return []*Backend{ + { + Name: "foo1", + Secure: true, + Endpoints: []Endpoint{}, + }, + { + Name: "foo2", + Secure: false, + Endpoints: []Endpoint{}, + }, + { + Name: "foo3", + Secure: true, + Endpoints: []Endpoint{}, + }, + } +} + +func TestBackendByNameServersLen(t *testing.T) { + fooTests := []struct { + backends BackendByNameServers + el int + }{ + {[]*Backend{}, 0}, + {buildBackendByNameServers(), 3}, + {nil, 0}, + } + + for _, fooTest := range fooTests { + r := fooTest.backends.Len() + if r != fooTest.el { + t.Errorf("returned %v but expected %v for the len of BackendByNameServers: %v", r, fooTest.el, fooTest.backends) + } + } +} + +func TestBackendByNameServersSwap(t *testing.T) { + fooTests := []struct { + backends BackendByNameServers + i int + j int + }{ + {buildBackendByNameServers(), 0, 1}, + {buildBackendByNameServers(), 2, 1}, + } + + for _, fooTest := range fooTests { + fooi := fooTest.backends[fooTest.i] + fooj := fooTest.backends[fooTest.j] + fooTest.backends.Swap(fooTest.i, fooTest.j) + if fooi.Name != fooTest.backends[fooTest.j].Name || fooj.Name != fooTest.backends[fooTest.i].Name { + t.Errorf("failed to swap for ByNameServers, foo: %v", fooTest) + } + } +} + +func TestBackendByNameServersLess(t *testing.T) { + fooTests := []struct { + backends BackendByNameServers + i int + j int + er bool + }{ + // order by name + {buildBackendByNameServers(), 0, 2, true}, + {buildBackendByNameServers(), 1, 0, false}, + } + + for _, fooTest := range fooTests { + r := fooTest.backends.Less(fooTest.i, fooTest.j) + if r != fooTest.er { + t.Errorf("returned %v but expected %v for the foo: %v", r, fooTest.er, fooTest) + } + } +} + +func buildEndpointByAddrPort() EndpointByAddrPort { + return []Endpoint{ + { + Address: "127.0.0.1", + Port: "8080", + MaxFails: 3, + FailTimeout: 10, + }, + { + Address: "127.0.0.1", + Port: "8081", + MaxFails: 3, + FailTimeout: 10, + }, + { + Address: "127.0.0.1", + Port: "8082", + MaxFails: 3, + FailTimeout: 10, + }, + { + Address: "127.0.0.2", + Port: "8082", + MaxFails: 3, + FailTimeout: 10, + }, + } +} + +func TestEndpointByAddrPortLen(t *testing.T) { + fooTests := []struct { + endpoints EndpointByAddrPort + el int + }{ + {[]Endpoint{}, 0}, + {buildEndpointByAddrPort(), 4}, + {nil, 0}, + } + + for _, fooTest := range fooTests { + r := fooTest.endpoints.Len() + if r != fooTest.el { + t.Errorf("returned %v but expected %v for the len of EndpointByAddrPort: %v", r, fooTest.el, fooTest.endpoints) + } + } +} + +func TestEndpointByAddrPortSwap(t *testing.T) { + fooTests := []struct { + endpoints EndpointByAddrPort + i int + j int + }{ + {buildEndpointByAddrPort(), 0, 1}, + {buildEndpointByAddrPort(), 2, 1}, + } + + for _, fooTest := range fooTests { + fooi := fooTest.endpoints[fooTest.i] + fooj := fooTest.endpoints[fooTest.j] + fooTest.endpoints.Swap(fooTest.i, fooTest.j) + if fooi.Port != fooTest.endpoints[fooTest.j].Port || + fooi.Address != fooTest.endpoints[fooTest.j].Address || + fooj.Port != fooTest.endpoints[fooTest.i].Port || + fooj.Address != fooTest.endpoints[fooTest.i].Address { + t.Errorf("failed to swap for EndpointByAddrPort, foo: %v", fooTest) + } + } +} + +func TestEndpointByAddrPortLess(t *testing.T) { + fooTests := []struct { + endpoints EndpointByAddrPort + i int + j int + er bool + }{ + // 1) order by name + // 2) order by port(if the name is the same one) + {buildEndpointByAddrPort(), 0, 1, true}, + {buildEndpointByAddrPort(), 2, 1, false}, + {buildEndpointByAddrPort(), 2, 3, true}, + } + + for _, fooTest := range fooTests { + r := fooTest.endpoints.Less(fooTest.i, fooTest.j) + if r != fooTest.er { + t.Errorf("returned %v but expected %v for the foo: %v", r, fooTest.er, fooTest) + } + } +} + +func buildServerByName() ServerByName { + return []*Server{ + { + Hostname: "foo1", + SSLPassthrough: true, + SSLCertificate: "foo1_cert", + SSLPemChecksum: "foo1_pem", + Locations: []*Location{}, + }, + { + Hostname: "foo2", + SSLPassthrough: true, + SSLCertificate: "foo2_cert", + SSLPemChecksum: "foo2_pem", + Locations: []*Location{}, + }, + { + Hostname: "foo3", + SSLPassthrough: false, + SSLCertificate: "foo3_cert", + SSLPemChecksum: "foo3_pem", + Locations: []*Location{}, + }, + { + Hostname: "_", + SSLPassthrough: false, + SSLCertificate: "foo4_cert", + SSLPemChecksum: "foo4_pem", + Locations: []*Location{}, + }, + } +} + +func TestServerByNameLen(t *testing.T) { + fooTests := []struct { + servers ServerByName + el int + }{ + {[]*Server{}, 0}, + {buildServerByName(), 4}, + {nil, 0}, + } + + for _, fooTest := range fooTests { + r := fooTest.servers.Len() + if r != fooTest.el { + t.Errorf("returned %v but expected %v for the len of ServerByName: %v", r, fooTest.el, fooTest.servers) + } + } +} + +func TestServerByNameSwap(t *testing.T) { + fooTests := []struct { + servers ServerByName + i int + j int + }{ + {buildServerByName(), 0, 1}, + {buildServerByName(), 2, 1}, + } + + for _, fooTest := range fooTests { + fooi := fooTest.servers[fooTest.i] + fooj := fooTest.servers[fooTest.j] + fooTest.servers.Swap(fooTest.i, fooTest.j) + if fooi.Hostname != fooTest.servers[fooTest.j].Hostname || + fooj.Hostname != fooTest.servers[fooTest.i].Hostname { + t.Errorf("failed to swap for ServerByName, foo: %v", fooTest) + } + } +} + +func TestServerByNameLess(t *testing.T) { + fooTests := []struct { + servers ServerByName + i int + j int + er bool + }{ + {buildServerByName(), 0, 1, true}, + {buildServerByName(), 2, 1, false}, + {buildServerByName(), 2, 3, false}, + } + + for _, fooTest := range fooTests { + r := fooTest.servers.Less(fooTest.i, fooTest.j) + if r != fooTest.er { + t.Errorf("returned %v but expected %v for the foo: %v", r, fooTest.er, fooTest) + } + } +} + +func buildLocationByPath() LocationByPath { + return []*Location{ + { + Path: "a", + IsDefBackend: true, + Backend: "a_back", + }, + { + Path: "b", + IsDefBackend: true, + Backend: "b_back", + }, + { + Path: "c", + IsDefBackend: true, + Backend: "c_back", + }, + } +} + +func TestLocationByPath(t *testing.T) { + fooTests := []struct { + locations LocationByPath + el int + }{ + {[]*Location{}, 0}, + {buildLocationByPath(), 3}, + {nil, 0}, + } + + for _, fooTest := range fooTests { + r := fooTest.locations.Len() + if r != fooTest.el { + t.Errorf("returned %v but expected %v for the len of LocationByPath: %v", r, fooTest.el, fooTest.locations) + } + } +} + +func TestLocationByPathSwap(t *testing.T) { + fooTests := []struct { + locations LocationByPath + i int + j int + }{ + {buildLocationByPath(), 0, 1}, + {buildLocationByPath(), 2, 1}, + } + + for _, fooTest := range fooTests { + fooi := fooTest.locations[fooTest.i] + fooj := fooTest.locations[fooTest.j] + fooTest.locations.Swap(fooTest.i, fooTest.j) + if fooi.Path != fooTest.locations[fooTest.j].Path || + fooj.Path != fooTest.locations[fooTest.i].Path { + t.Errorf("failed to swap for LocationByPath, foo: %v", fooTest) + } + } +} + +func TestLocationByPathLess(t *testing.T) { + fooTests := []struct { + locations LocationByPath + i int + j int + er bool + }{ + // sorts location by path in descending order + {buildLocationByPath(), 0, 1, false}, + {buildLocationByPath(), 2, 1, true}, + } + + for _, fooTest := range fooTests { + r := fooTest.locations.Less(fooTest.i, fooTest.j) + if r != fooTest.er { + t.Errorf("returned %v but expected %v for the foo: %v", r, fooTest.er, fooTest) + } + } +} + +func TestGetObjectKindForSSLCert(t *testing.T) { + fk := &SSLCert{ + ObjectMeta: api.ObjectMeta{}, + CAFileName: "ca_file", + PemFileName: "pemfile", + PemSHA: "pem_sha", + CN: []string{}, + } + + r := fk.GetObjectKind() + if r == nil { + t.Errorf("Returned nil but expected a valid ObjectKind") + } +} From 96df5b3d55f6bccded24226c08cac96c1f0297be Mon Sep 17 00:00:00 2001 From: Justin Ryan Date: Wed, 25 Jan 2017 09:52:50 -0500 Subject: [PATCH 20/32] Clarify usage of Ingress backend.servicePort --- controllers/gce/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/controllers/gce/README.md b/controllers/gce/README.md index 77ed0e153..aa1d6bd8d 100644 --- a/controllers/gce/README.md +++ b/controllers/gce/README.md @@ -51,7 +51,7 @@ __Lines 5-7__: Ingress Spec has all the information needed to configure a GCE L7 __Lines 8-9__: Each http rule contains the following information: A host (eg: foo.bar.com, defaults to `*` in this example), a list of paths (eg: `/hostless`) each of which has an associated backend (`test:80`). Both the `host` and `path` must match the content of an incoming request before the L7 directs traffic to the `backend`. -__Lines 10-12__: A `backend` is a service:port combination. It selects a group of pods capable of servicing traffic sent to the path specified in the parent rule. +__Lines 10-12__: A `backend` is a service:port combination. It selects a group of pods capable of servicing traffic sent to the path specified in the parent rule. The `port` is the desired `spec.ports[*].port` from the Service Spec -- Note, though, that the L7 actually directs traffic to the corresponding `NodePort`. __Global Prameters__: For the sake of simplicity the example Ingress has no global parameters. However, one can specify a default backend (see examples below) in the absence of which requests that don't match a path in the spec are sent to the default backend of glbc. Though glbc doesn't support HTTPS yet, security configs would also be global. From 08eda50ebb6c72cedbd2e4ac9bda58258e9d4235 Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Wed, 25 Jan 2017 15:16:31 -0300 Subject: [PATCH 21/32] Update nginx to 1.11.9 --- controllers/nginx/rootfs/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/controllers/nginx/rootfs/Dockerfile b/controllers/nginx/rootfs/Dockerfile index 10c833e37..74ee4f493 100644 --- a/controllers/nginx/rootfs/Dockerfile +++ b/controllers/nginx/rootfs/Dockerfile @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -FROM gcr.io/google_containers/nginx-slim:0.12 +FROM gcr.io/google_containers/nginx-slim:0.13 RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y \ diffutils \ From bc810d8eef382e8f9bceb0dfcc780e945ad1d708 Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Wed, 25 Jan 2017 23:08:40 -0300 Subject: [PATCH 22/32] Fix TLS does not get updated when changed --- core/pkg/ingress/controller/controller.go | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/core/pkg/ingress/controller/controller.go b/core/pkg/ingress/controller/controller.go index 29b2c30d3..650fcc2a5 100644 --- a/core/pkg/ingress/controller/controller.go +++ b/core/pkg/ingress/controller/controller.go @@ -26,6 +26,7 @@ import ( "time" "github.com/golang/glog" + "github.com/kylelemons/godebug/pretty" "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/apis/extensions" @@ -178,6 +179,7 @@ func newIngressController(config *Configuration) *GenericController { ic.syncQueue.Enqueue(obj) }, UpdateFunc: func(old, cur interface{}) { + oldIng := old.(*extensions.Ingress) curIng := cur.(*extensions.Ingress) if !IsValidClass(curIng, config.IngressClass) { return @@ -186,6 +188,24 @@ func newIngressController(config *Configuration) *GenericController { if !reflect.DeepEqual(old, cur) { upIng := cur.(*extensions.Ingress) ic.recorder.Eventf(upIng, api.EventTypeNormal, "UPDATE", fmt.Sprintf("Ingress %s/%s", upIng.Namespace, upIng.Name)) + // the referenced secret is different? + if diff := pretty.Compare(curIng.Spec.TLS, oldIng.Spec.TLS); diff != "" { + for _, secretName := range curIng.Spec.TLS { + secKey := fmt.Sprintf("%v/%v", curIng.Namespace, secretName.SecretName) + go func() { + glog.Infof("TLS section in ingress %v/%v changed (secret is now %v)", upIng.Namespace, upIng.Name, secKey) + // we need to wait until the ingress store is updated + time.Sleep(10 * time.Second) + key, err := ic.GetSecret(secKey) + if err != nil { + glog.Errorf("unexpected error: %v", err) + } + if key != nil { + ic.secretQueue.Enqueue(key) + } + }() + } + } ic.syncQueue.Enqueue(cur) } }, From ec67f83305f188dd9753a0d9f731314de973f9ab Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Thu, 26 Jan 2017 00:10:33 -0300 Subject: [PATCH 23/32] Refactoring sysctlFSFileMax helper --- controllers/nginx/pkg/cmd/controller/utils.go | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/controllers/nginx/pkg/cmd/controller/utils.go b/controllers/nginx/pkg/cmd/controller/utils.go index c6e1979f9..3a3b04823 100644 --- a/controllers/nginx/pkg/cmd/controller/utils.go +++ b/controllers/nginx/pkg/cmd/controller/utils.go @@ -20,6 +20,7 @@ import ( "io/ioutil" "os" "os/exec" + "syscall" "k8s.io/kubernetes/pkg/util/sysctl" @@ -42,14 +43,14 @@ func sysctlSomaxconn() int { // sysctlFSFileMax returns the value of fs.file-max, i.e. // maximum number of open file descriptors func sysctlFSFileMax() int { - maxConns, err := sysctl.New().GetSysctl("fs/file-max") + var rLimit syscall.Rlimit + err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rLimit) if err != nil { - glog.Errorf("unexpected error reading system maximum number of open file descriptors (fs.file-max): %v", err) + glog.Errorf("unexpected error reading system maximum number of open file descriptors (RLIMIT_NOFILE): %v", err) // returning 0 means don't render the value return 0 } - - return maxConns + return int(rLimit.Max) } func diff(b1, b2 []byte) ([]byte, error) { From 2baa1def46e42d81062b8d7faff69f8becf6f3f0 Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Thu, 26 Jan 2017 10:33:29 -0300 Subject: [PATCH 24/32] Add initialization of proxy variable --- controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl b/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl index 5f10d030b..3fd2da9df 100644 --- a/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl +++ b/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl @@ -244,6 +244,8 @@ http { {{ end }} location {{ $path }} { + set $proxy_upstream_name "{{ $location.Backend }}"; + {{ if isLocationAllowed $location }} {{ if gt (len $location.Whitelist.CIDR) 0 }} {{ range $ip := $location.Whitelist.CIDR }} @@ -319,7 +321,6 @@ http { proxy_set_header Accept-Encoding ""; {{ end }} - set $proxy_upstream_name "{{ $location.Backend }}"; {{ buildProxyPass $backends $location }} {{ else }} #{{ $location.Denied }} From cc1413261ff2ae1dcc8191a5abb1d2b32eba8f73 Mon Sep 17 00:00:00 2001 From: Ricardo Pchevuzinske Katz Date: Thu, 26 Jan 2017 16:51:55 -0200 Subject: [PATCH 25/32] Allows the usage of Default SSL Cert --- .../rootfs/etc/nginx/template/nginx.tmpl | 6 +++- core/pkg/ingress/controller/controller.go | 28 +++++++++++++++++-- 2 files changed, 31 insertions(+), 3 deletions(-) diff --git a/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl b/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl index 5f10d030b..1b9f12d3f 100644 --- a/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl +++ b/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl @@ -203,7 +203,11 @@ http { server_name {{ $server.Hostname }}; listen [::]:80{{ if $cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ if eq $index 0 }} ipv6only=off{{end}}{{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $backlogSize }}{{end}}; {{/* Listen on 442 because port 443 is used in the stream section */}} - {{ if not (empty $server.SSLCertificate) }}listen 442 {{ if $cfg.UseProxyProtocol }}proxy_protocol{{ end }} ssl {{ if $cfg.UseHTTP2 }}http2{{ end }}; + {{ if not (empty $server.SSLCertificate) }}listen 442 {{ if $cfg.UseProxyProtocol }}proxy_protocol{{ end }} {{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $backlogSize }}{{end}} ssl {{ if $cfg.UseHTTP2 }}http2{{ end }}; + {{/* comment PEM sha is required to detect changes in the generated configuration and force a reload */}} + # PEM sha: {{ $server.SSLPemChecksum }} + ssl_certificate {{ $server.SSLCertificate }}; + {{/* comment PEM sha is required to detect changes in the generated configuration and force a reload */}} # PEM sha: {{ $server.SSLPemChecksum }} ssl_certificate {{ $server.SSLCertificate }}; diff --git a/core/pkg/ingress/controller/controller.go b/core/pkg/ingress/controller/controller.go index 650fcc2a5..146a00aa3 100644 --- a/core/pkg/ingress/controller/controller.go +++ b/core/pkg/ingress/controller/controller.go @@ -47,6 +47,7 @@ import ( "k8s.io/ingress/core/pkg/ingress/resolver" "k8s.io/ingress/core/pkg/ingress/status" "k8s.io/ingress/core/pkg/k8s" + ssl "k8s.io/ingress/core/pkg/net/ssl" local_strings "k8s.io/ingress/core/pkg/strings" "k8s.io/ingress/core/pkg/task" ) @@ -827,9 +828,30 @@ func (ic *GenericController) createServers(data []interface{}, upstreams map[str dun := ic.getDefaultUpstream().Name + // This adds the Default Certificate to Default Backend and also for vhosts missing the secret + var defaultPemFileName, defaultPemSHA string + defaultCertificate, err := ic.getPemCertificate(ic.cfg.DefaultSSLCertificate) + // If no default Certificate was supplied, tries to generate a new dumb one + if err != nil { + var cert *ingress.SSLCert + defCert, defKey := ssl.GetFakeSSLCert() + cert, err = ssl.AddOrUpdateCertAndKey("system-snake-oil-certificate", defCert, defKey, []byte{}) + if err != nil { + glog.Fatalf("Error generating self signed certificate: %v", err) + } else { + defaultPemFileName = cert.PemFileName + defaultPemSHA = cert.PemSHA + } + } else { + defaultPemFileName = defaultCertificate.PemFileName + defaultPemSHA = defaultCertificate.PemSHA + } + // default server servers[defServerName] = &ingress.Server{ - Hostname: defServerName, + Hostname: defServerName, + SSLCertificate: defaultPemFileName, + SSLPemChecksum: defaultPemSHA, Locations: []*ingress.Location{ { Path: rootLocation, @@ -899,7 +921,9 @@ func (ic *GenericController) createServers(data []interface{}, upstreams map[str servers[host].SSLPemChecksum = cert.PemSHA } } else { - glog.Warningf("secret %v does not exists", key) + + servers[host].SSLCertificate = defaultPemFileName + servers[host].SSLPemChecksum = defaultPemSHA } } From c3ac5624296c2490e502ae88ccdef19917213650 Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Fri, 27 Jan 2017 17:50:59 -0300 Subject: [PATCH 26/32] Fix template error --- controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl b/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl index eb5ee0cca..2c4d3cc2e 100644 --- a/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl +++ b/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl @@ -203,11 +203,7 @@ http { server_name {{ $server.Hostname }}; listen [::]:80{{ if $cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ if eq $index 0 }} ipv6only=off{{end}}{{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $backlogSize }}{{end}}; {{/* Listen on 442 because port 443 is used in the stream section */}} - {{ if not (empty $server.SSLCertificate) }}listen 442 {{ if $cfg.UseProxyProtocol }}proxy_protocol{{ end }} {{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $backlogSize }}{{end}} ssl {{ if $cfg.UseHTTP2 }}http2{{ end }}; - {{/* comment PEM sha is required to detect changes in the generated configuration and force a reload */}} - # PEM sha: {{ $server.SSLPemChecksum }} - ssl_certificate {{ $server.SSLCertificate }}; - + {{ if not (empty $server.SSLCertificate) }}listen 442 {{ if $cfg.UseProxyProtocol }}proxy_protocol{{ end }} {{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $backlogSize }}{{end}} ssl {{ if $cfg.UseHTTP2 }}http2{{ end }}; {{/* comment PEM sha is required to detect changes in the generated configuration and force a reload */}} # PEM sha: {{ $server.SSLPemChecksum }} ssl_certificate {{ $server.SSLCertificate }}; From 2887daaf788b26bdffa2a4d5e7d5527d8119d189 Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Thu, 26 Jan 2017 17:00:06 -0300 Subject: [PATCH 27/32] Release 0.9.0 --- controllers/nginx/Changelog.md | 92 ++++++++++++++++++- controllers/nginx/Makefile | 2 +- .../kubeadm/nginx-ingress-controller.yaml | 2 +- .../nginx/nginx-ingress-controller.yaml | 2 +- 4 files changed, 94 insertions(+), 4 deletions(-) diff --git a/controllers/nginx/Changelog.md b/controllers/nginx/Changelog.md index 60fd919fe..cabaa38db 100644 --- a/controllers/nginx/Changelog.md +++ b/controllers/nginx/Changelog.md @@ -2,6 +2,96 @@ Changelog ### 0.9 +**Temporal Image:** `quay.io/aledbf/nginx-ingress-controller:0.9.0` + +*New Features:* + +- SSL Passthrough +- New Flag `--publish-service` that set the Service fronting the ingress controllers +- Ingress status shows the correct IP/hostname address without duplicates +- Custom body sizes per Ingress +- Prometheus metrics + + +*Breaking changes:* + +- Flag `--nginx-configmap` was replaced with `--configmap` +- Configmap field `body-size` was replaced with `proxy-body-size` + +*Changes:* + +- [X] [#184](https://github.com/kubernetes/ingress/pull/184) Fix template error +- [X] [#179](https://github.com/kubernetes/ingress/pull/179) Allows the usage of Default SSL Cert +- [X] [#178](https://github.com/kubernetes/ingress/pull/178) Add initialization of proxy variable +- [X] [#177](https://github.com/kubernetes/ingress/pull/177) Refactoring sysctlFSFileMax helper +- [X] [#176](https://github.com/kubernetes/ingress/pull/176) Fix TLS does not get updated when changed +- [X] [#174](https://github.com/kubernetes/ingress/pull/174) Update nginx to 1.11.9 +- [X] [#172](https://github.com/kubernetes/ingress/pull/172) add some unit test cases for some packages under folder "core.pkg.ingress" +- [X] [#168](https://github.com/kubernetes/ingress/pull/168) Changes the SSL Temp file to something inside the same SSL Directory +- [X] [#165](https://github.com/kubernetes/ingress/pull/165) Fix rate limit issue when more than 2 servers enabled in ingress +- [X] [#161](https://github.com/kubernetes/ingress/pull/161) Document some missing parameters and their defaults for NGINX controller +- [X] [#158](https://github.com/kubernetes/ingress/pull/158) prefect unit test cases for annotation.proxy +- [X] [#156](https://github.com/kubernetes/ingress/pull/156) Fix issue for ratelimit +- [X] [#154](https://github.com/kubernetes/ingress/pull/154) add unit test cases for core.pkg.ingress.annotations.cors +- [X] [#151](https://github.com/kubernetes/ingress/pull/151) Port in redirect +- [X] [#150](https://github.com/kubernetes/ingress/pull/150) Add support for custom header sizes +- [X] [#149](https://github.com/kubernetes/ingress/pull/149) Add flag to allow switch off the update of Ingress status +- [X] [#148](https://github.com/kubernetes/ingress/pull/148) Add annotation to allow custom body sizes +- [X] [#145](https://github.com/kubernetes/ingress/pull/145) fix wrong links and punctuations +- [X] [#144](https://github.com/kubernetes/ingress/pull/144) add unit test cases for core.pkg.k8s +- [X] [#143](https://github.com/kubernetes/ingress/pull/143) Use protobuf instead of rest to connect to apiserver host and add troubleshooting doc +- [X] [#142](https://github.com/kubernetes/ingress/pull/142) Use system fs.max-files as limits instead of hard-coded value +- [X] [#141](https://github.com/kubernetes/ingress/pull/141) Add reuse port and backlog to port 80 and 443 +- [X] [#138](https://github.com/kubernetes/ingress/pull/138) reference to const +- [X] [#136](https://github.com/kubernetes/ingress/pull/136) Add content and descriptions about nginx's configuration +- [X] [#135](https://github.com/kubernetes/ingress/pull/135) correct improper punctuation +- [X] [#134](https://github.com/kubernetes/ingress/pull/134) fix typo +- [X] [#133](https://github.com/kubernetes/ingress/pull/133) Add TCP and UDP services removed in migration +- [X] [#132](https://github.com/kubernetes/ingress/pull/132) Document nginx controller configuration tweaks +- [X] [#128](https://github.com/kubernetes/ingress/pull/128) Add tests and godebug to compare structs +- [X] [#126](https://github.com/kubernetes/ingress/pull/126) change the type of imagePullPolicy +- [X] [#123](https://github.com/kubernetes/ingress/pull/123) Add resolver configuration to nginx +- [X] [#119](https://github.com/kubernetes/ingress/pull/119) add unit test case for annotations.service +- [X] [#115](https://github.com/kubernetes/ingress/pull/115) add default_server to listen statement for default backend +- [X] [#114](https://github.com/kubernetes/ingress/pull/114) fix typo +- [X] [#113](https://github.com/kubernetes/ingress/pull/113) Add condition of enqueue and unit test cases for task.Queue +- [X] [#108](https://github.com/kubernetes/ingress/pull/108) annotations: print error and skip if malformed +- [X] [#107](https://github.com/kubernetes/ingress/pull/107) fix some wrong links of examples which to be used for nginx +- [X] [#103](https://github.com/kubernetes/ingress/pull/103) Update the nginx controller manifests +- [X] [#101](https://github.com/kubernetes/ingress/pull/101) Add unit test for strings.StringInSlice +- [X] [#99](https://github.com/kubernetes/ingress/pull/99) Update nginx to 1.11.8 +- [X] [#97](https://github.com/kubernetes/ingress/pull/97) Fix gofmt +- [X] [#96](https://github.com/kubernetes/ingress/pull/96) Fix typo PassthrougBackends -> PassthroughBackends +- [X] [#95](https://github.com/kubernetes/ingress/pull/95) Deny location mapping in case of specific errors +- [X] [#94](https://github.com/kubernetes/ingress/pull/94) Add support to disable server_tokens directive +- [X] [#93](https://github.com/kubernetes/ingress/pull/93) Fix sort for catch all server +- [X] [#92](https://github.com/kubernetes/ingress/pull/92) Refactoring of nginx configuration deserialization +- [X] [#91](https://github.com/kubernetes/ingress/pull/91) Fix x-forwarded-port mapping +- [X] [#90](https://github.com/kubernetes/ingress/pull/90) fix the wrong link to build/test/release +- [X] [#89](https://github.com/kubernetes/ingress/pull/89) fix the wrong links to the examples and developer documentation +- [X] [#88](https://github.com/kubernetes/ingress/pull/88) Fix multiple tls hosts sharing the same secretName +- [X] [#86](https://github.com/kubernetes/ingress/pull/86) Update X-Forwarded-Port +- [X] [#82](https://github.com/kubernetes/ingress/pull/82) Fix incorrect X-Forwarded-Port for TLS +- [X] [#81](https://github.com/kubernetes/ingress/pull/81) Do not push containers to remote repo as part of test-e2e +- [X] [#78](https://github.com/kubernetes/ingress/pull/78) Fix #76: hardcode X-Forwarded-Port due to SSL Passthrough +- [X] [#77](https://github.com/kubernetes/ingress/pull/77) Add support for IPV6 in dns resolvers +- [X] [#66](https://github.com/kubernetes/ingress/pull/66) Start FAQ docs +- [X] [#65](https://github.com/kubernetes/ingress/pull/65) Support hostnames in Ingress status +- [X] [#64](https://github.com/kubernetes/ingress/pull/64) Sort whitelist list to avoid random orders +- [X] [#62](https://github.com/kubernetes/ingress/pull/62) Fix e2e make targets +- [X] [#61](https://github.com/kubernetes/ingress/pull/61) Ignore coverage profile files +- [X] [#58](https://github.com/kubernetes/ingress/pull/58) Fix "invalid port in upstream" on nginx controller +- [X] [#57](https://github.com/kubernetes/ingress/pull/57) Fix invalid port in upstream +- [X] [#54](https://github.com/kubernetes/ingress/pull/54) Expand developer docs +- [X] [#52](https://github.com/kubernetes/ingress/pull/52) fix typo in variable ProxyRealIPCIDR +- [X] [#44](https://github.com/kubernetes/ingress/pull/44) Bump nginx version to one higher than that in contrib +- [X] [#36](https://github.com/kubernetes/ingress/pull/36) Add nginx metrics to prometheus +- [X] [#34](https://github.com/kubernetes/ingress/pull/34) nginx: also listen on ipv6 +- [X] [#32](https://github.com/kubernetes/ingress/pull/32) Restart nginx if master process dies +- [X] [#31](https://github.com/kubernetes/ingress/pull/31) Add healthz checker +- [X] [#25](https://github.com/kubernetes/ingress/pull/25) Fix a data race in TestFileWatcher +- [X] [#12](https://github.com/kubernetes/ingress/pull/12) Split implementations from generic code +- [X] [#10](https://github.com/kubernetes/ingress/pull/10) Copy Ingress history from kubernetes/contrib - [X] [#1498](https://github.com/kubernetes/contrib/pull/1498) Refactoring of template handling - [X] [#1571](https://github.com/kubernetes/contrib/pull/1571) use POD_NAMESPACE as a namespace in cli parameters - [X] [#1591](https://github.com/kubernetes/contrib/pull/1591) Always listen on port 443, even without ingress rules @@ -81,4 +171,4 @@ Changelog - [X] [#1093](https://github.com/kubernetes/contrib/pull/1093) rate limiting - [X] [#1102](https://github.com/kubernetes/contrib/pull/1102) geolocation of traffic in stats - [X] [#884](https://github.com/kubernetes/contrib/issues/884) support services running ssl -- [X] [#930](https://github.com/kubernetes/contrib/issues/930) detect changes in configuration configmaps +- [X] [#930](https://github.com/kubernetes/contrib/issues/930) detect changes in configuration configmaps \ No newline at end of file diff --git a/controllers/nginx/Makefile b/controllers/nginx/Makefile index b6bcfe810..94c38a6f4 100644 --- a/controllers/nginx/Makefile +++ b/controllers/nginx/Makefile @@ -3,7 +3,7 @@ all: push BUILDTAGS= # Use the 0.0 tag for testing, it shouldn't clobber any release builds -RELEASE?=0.8.4 +RELEASE?=0.9.0 PREFIX?=gcr.io/google_containers/nginx-ingress-controller GOOS?=linux diff --git a/examples/deployment/nginx/kubeadm/nginx-ingress-controller.yaml b/examples/deployment/nginx/kubeadm/nginx-ingress-controller.yaml index e5c532a9b..5c3edddda 100644 --- a/examples/deployment/nginx/kubeadm/nginx-ingress-controller.yaml +++ b/examples/deployment/nginx/kubeadm/nginx-ingress-controller.yaml @@ -71,7 +71,7 @@ spec: hostNetwork: true terminationGracePeriodSeconds: 60 containers: - - image: gcr.io/google_containers/nginx-ingress-controller:0.8.3 + - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0 name: nginx-ingress-controller readinessProbe: httpGet: diff --git a/examples/deployment/nginx/nginx-ingress-controller.yaml b/examples/deployment/nginx/nginx-ingress-controller.yaml index 787b1c110..4760c2e4e 100644 --- a/examples/deployment/nginx/nginx-ingress-controller.yaml +++ b/examples/deployment/nginx/nginx-ingress-controller.yaml @@ -19,7 +19,7 @@ spec: # hostNetwork: true terminationGracePeriodSeconds: 60 containers: - - image: gcr.io/google_containers/nginx-ingress-controller:0.8.3 + - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0 name: nginx-ingress-controller readinessProbe: httpGet: From 7d709d5e93ee32339375fc0f5f864d17711bff5c Mon Sep 17 00:00:00 2001 From: bprashanth Date: Fri, 27 Jan 2017 14:22:44 -0800 Subject: [PATCH 28/32] Match named port between container and probe We were previous matching the target port with the readiness probe, and hence dropping the case where the container port and the probe had the same name, but the target port did not. --- controllers/gce/controller/util_test.go | 25 +++++++++++++++++++++++- controllers/gce/controller/utils.go | 26 +++++++++++++------------ 2 files changed, 38 insertions(+), 13 deletions(-) diff --git a/controllers/gce/controller/util_test.go b/controllers/gce/controller/util_test.go index fbc7f0022..a3bbbe120 100644 --- a/controllers/gce/controller/util_test.go +++ b/controllers/gce/controller/util_test.go @@ -109,6 +109,29 @@ func TestProbeGetter(t *testing.T) { } } +func TestProbeGetterNamedPort(t *testing.T) { + cm := NewFakeClusterManager(DefaultClusterUID) + lbc := newLoadBalancerController(t, cm, "") + nodePortToHealthCheck := map[int64]string{ + 3001: "/healthz", + } + addPods(lbc, nodePortToHealthCheck, api.NamespaceDefault) + for _, p := range lbc.podLister.Indexer.List() { + pod := p.(*api.Pod) + pod.Spec.Containers[0].Ports[0].Name = "test" + pod.Spec.Containers[0].ReadinessProbe.Handler.HTTPGet.Port = intstr.IntOrString{Type: intstr.String, StrVal: "test"} + } + for p, exp := range nodePortToHealthCheck { + got, err := lbc.tr.HealthCheck(p) + if err != nil { + t.Errorf("Failed to get health check for node port %v: %v", p, err) + } else if got.RequestPath != exp { + t.Errorf("Wrong health check for node port %v, got %v expected %v", p, got.RequestPath, exp) + } + } + +} + func TestProbeGetterCrossNamespace(t *testing.T) { cm := NewFakeClusterManager(DefaultClusterUID) lbc := newLoadBalancerController(t, cm, "") @@ -191,7 +214,7 @@ func addPods(lbc *LoadBalancerController, nodePortToHealthCheck map[int64]string Spec: api.PodSpec{ Containers: []api.Container{ { - Ports: []api.ContainerPort{{ContainerPort: 80}}, + Ports: []api.ContainerPort{{Name: "test", ContainerPort: 80}}, ReadinessProbe: &api.Probe{ Handler: api.Handler{ HTTPGet: &api.HTTPGetAction{ diff --git a/controllers/gce/controller/utils.go b/controllers/gce/controller/utils.go index b86e811f0..617ce5fad 100644 --- a/controllers/gce/controller/utils.go +++ b/controllers/gce/controller/utils.go @@ -410,14 +410,6 @@ func (t *GCETranslator) ListZones() ([]string, error) { return zones.List(), nil } -// isPortEqual compares the given IntOrString ports -func isPortEqual(port, targetPort intstr.IntOrString) bool { - if targetPort.Type == intstr.Int { - return port.IntVal == targetPort.IntVal - } - return port.StrVal == targetPort.StrVal -} - // geHTTPProbe returns the http readiness probe from the first container // that matches targetPort, from the set of pods matching the given labels. func (t *GCETranslator) getHTTPProbe(svc api.Service, targetPort intstr.IntOrString) (*api.Probe, error) { @@ -443,11 +435,21 @@ func (t *GCETranslator) getHTTPProbe(svc api.Service, targetPort intstr.IntOrStr continue } for _, p := range c.Ports { - cPort := intstr.IntOrString{IntVal: p.ContainerPort, StrVal: p.Name} - if isPortEqual(cPort, targetPort) { - if isPortEqual(c.ReadinessProbe.Handler.HTTPGet.Port, targetPort) { - return c.ReadinessProbe, nil + if targetPort.Type == intstr.Int && targetPort.IntVal == p.ContainerPort || + targetPort.Type == intstr.String && targetPort.StrVal == p.Name { + + readinessProbePort := c.ReadinessProbe.Handler.HTTPGet.Port + switch readinessProbePort.Type { + case intstr.Int: + if readinessProbePort.IntVal == p.ContainerPort { + return c.ReadinessProbe, nil + } + case intstr.String: + if readinessProbePort.StrVal == p.Name { + return c.ReadinessProbe, nil + } } + glog.Infof("%v: found matching targetPort on container %v, but not on readinessProbe (%+v)", logStr, c.Name, c.ReadinessProbe.Handler.HTTPGet.Port) } From f75ef4a576d4245b1298ea75e6cdc40657c2395c Mon Sep 17 00:00:00 2001 From: bprashanth Date: Fri, 27 Jan 2017 17:51:00 -0800 Subject: [PATCH 29/32] Add a tls-termination example --- examples/PREREQUISITES.md | 131 ++++++++++++++++++ examples/README.md | 4 +- examples/health-checks/gce.md | 3 + examples/http-svc.yaml | 35 +++++ examples/tls-termination/gce-tls-ingress.yaml | 15 ++ examples/tls-termination/gce.md | 78 +++++++++++ .../tls-termination/nginx-tls-ingress.yaml | 18 +++ examples/tls-termination/nginx.md | 74 ++++++++++ 8 files changed, 357 insertions(+), 1 deletion(-) create mode 100644 examples/PREREQUISITES.md create mode 100644 examples/health-checks/gce.md create mode 100644 examples/http-svc.yaml create mode 100644 examples/tls-termination/gce-tls-ingress.yaml create mode 100644 examples/tls-termination/gce.md create mode 100644 examples/tls-termination/nginx-tls-ingress.yaml create mode 100644 examples/tls-termination/nginx.md diff --git a/examples/PREREQUISITES.md b/examples/PREREQUISITES.md new file mode 100644 index 000000000..bbb066ee4 --- /dev/null +++ b/examples/PREREQUISITES.md @@ -0,0 +1,131 @@ +# Prerequisites + +Many of the examples in this directory have common prerequisites. + +## Deploying a controller + +Unless you're running on a cloudprovider that supports Ingress out of the box +(eg: GCE/GKE), you will need to deploy a controller. You can do so following +[these instructions](/examples/deployment). + +## Firewall rules + +If you're using a bare-metal controller (eg the nginx ingress controller), you +will need to create a firewall rule that targets port 80/443 on the specific VMs +the nginx controller is running on. On cloudproviders, the respective backend +will auto-create firewall rules for your Ingress. + +## TLS certificates + +Unless otherwise mentioned, the TLS secret used in examples is a 2048 bit RSA +key/cert pair with an arbitrarily chosen hostname, created as follows + +```console +$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginxsvc/O=nginxsvc" +Generating a 2048 bit RSA private key +......................................................................................................................................+++ +....................................................................+++ +writing new private key to 'tls.key' +----- + +$ kubectl create secret tls tls-secret --key tls.key --cert tls.crt +secret "tls-secret" created +``` + +## Test HTTP Service + +All examples that require a test HTTP Service use the standard echoheaders pod, +which you can deploy as follows + +```console +$ kubectl create -f http-svc.yaml +service "http-svc" created +replicationcontroller "http-svc" created + +$ kubectl get po +NAME READY STATUS RESTARTS AGE +echoheaders-p1t3t 1/1 Running 0 1d + +$ kubectl get svc +NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE +echoheaders 10.0.122.116 80/TCP 1d +``` + +You can test that the HTTP Service works by exposing it temporarily +```console +$ kubectl patch svc echoheaders -p '{"spec":{"type": "LoadBalancer"}}' +"echoheaders" patched + +$ kubectl get svc echoheaders +NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE +echoheaders 10.0.122.116 80:32100/TCP 1d + +$ kubectl describe svc echoheaders +Name: echoheaders +Namespace: default +Labels: app=echoheaders +Selector: app=echoheaders +Type: LoadBalancer +IP: 10.0.122.116 +LoadBalancer Ingress: 108.59.87.136 +Port: http 80/TCP +NodePort: http 32100/TCP +Endpoints: 10.180.1.6:8080 +Session Affinity: None +Events: + FirstSeen LastSeen Count From SubObjectPath Type Reason Message + --------- -------- ----- ---- ------------- -------- ------ ------- + 1m 1m 1 {service-controller } Normal Type ClusterIP -> LoadBalancer + 1m 1m 1 {service-controller } Normal CreatingLoadBalancer Creating load balancer + 16s 16s 1 {service-controller } Normal CreatedLoadBalancer Created load balancer + +$ curl 108.59.87.126 +CLIENT VALUES: +client_address=10.240.0.3 +command=GET +real path=/ +query=nil +request_version=1.1 +request_uri=http://108.59.87.136:8080/ + +SERVER VALUES: +server_version=nginx: 1.9.11 - lua: 10001 + +HEADERS RECEIVED: +accept=*/* +host=108.59.87.136 +user-agent=curl/7.46.0 +BODY: +-no body in request- + +$ kubectl patch svc echoheaders -p '{"spec":{"type": "NodePort"}}' +"echoheaders" patched +``` + +## Ingress Class + +If you have multiple Ingress controllers in a single cluster, you can pick one +by specifying the `ingress.class` annotation, eg creating an Ingress with an +annotation like + +```yaml +metadata: + name: foo + annotations: + kubernetes.io/ingress.class: "gce" +``` + +will target the GCE controller, forcing the nginx controller to ignore it, while +an annotation like + +```yaml +metadata: + name: foo + annotations: + kubernetes.io/ingress.class: "nginx" +``` + +will target the nginx controller, forcing the GCE controller to ignore it. + +__Note__: Deploying multiple ingress controller and not specifying the +annotation will result in both controllers fighting to satisfy the Ingress. diff --git a/examples/README.md b/examples/README.md index 78e0adf65..2a00a24d7 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,6 +1,8 @@ # Ingress examples -A catalog of examples on how to run, configure and scale Ingress. +This directory contains a catalog of examples on how to run, configure and +scale Ingress. Please review the [prerequisities](prerequisites.md) before +trying them. ## Basic cross platform diff --git a/examples/health-checks/gce.md b/examples/health-checks/gce.md new file mode 100644 index 000000000..9d3831a6f --- /dev/null +++ b/examples/health-checks/gce.md @@ -0,0 +1,3 @@ +# Health checks for the GCE controller + +Placeholder diff --git a/examples/http-svc.yaml b/examples/http-svc.yaml new file mode 100644 index 000000000..c76d21d01 --- /dev/null +++ b/examples/http-svc.yaml @@ -0,0 +1,35 @@ +apiVersion: v1 +kind: Service +metadata: + name: http-svc + labels: + app: http-svc +spec: + type: NodePort + ports: + - port: 80 + # This port needs to be available on all nodes in the cluster + nodePort: 30301 + targetPort: 8080 + protocol: TCP + name: http + selector: + app: http-svc +--- +apiVersion: v1 +kind: ReplicationController +metadata: + name: http-svc +spec: + replicas: 2 + template: + metadata: + labels: + app: http-svc + spec: + containers: + - name: http-svc + image: gcr.io/google_containers/echoserver:1.3 + ports: + - containerPort: 8080 + diff --git a/examples/tls-termination/gce-tls-ingress.yaml b/examples/tls-termination/gce-tls-ingress.yaml new file mode 100644 index 000000000..705a17d36 --- /dev/null +++ b/examples/tls-termination/gce-tls-ingress.yaml @@ -0,0 +1,15 @@ +apiVersion: extensions/v1beta1 +kind: Ingress +metadata: + name: test + annotations: + kubernetes.io/ingress.class: "gce" +spec: + tls: + # This assumes tls-secret exists. + - secretName: tls-secret + backend: + # This assumes http-svc exists and routes to healthy endpoints. + serviceName: http-svc + servicePort: 80 + diff --git a/examples/tls-termination/gce.md b/examples/tls-termination/gce.md new file mode 100644 index 000000000..ef2b99544 --- /dev/null +++ b/examples/tls-termination/gce.md @@ -0,0 +1,78 @@ +# TLS termination + +This example demonstrates how to terminate TLS through the GCE Ingress controller. + +## Prerequisites + +You need a [TLS cert](/examples/PREREQUISITES.md#tls-certificates) and a [test HTTP service](/examples/PREREQUISITES.md#test-http-service) for this example. +You will also need to make sure you Ingress targets exactly one Ingress +controller by specifying the [ingress.class annotation](/examples/PREREQUISITES.md#ingress-class). + +## Deployment + +The following command instructs the controller to terminate traffic using +the provided TLS cert, and forward un-encrypted HTTP traffic to the test +HTTP service. + +```console +$ kubectl create -f gce-tls-ingress.yaml +``` + +## Validation + +You can confirm that the Ingress works. + +```console +$ kubectl describe ing gce-test +Name: gce-test +Namespace: default +Address: 35.186.221.137 +Default backend: http-svc:80 (10.180.1.9:8080,10.180.3.6:8080) +TLS: + tls-secret terminates +Rules: + Host Path Backends + ---- ---- -------- + * * http-svc:80 (10.180.1.9:8080,10.180.3.6:8080) +Annotations: + target-proxy: k8s-tp-default-gce-test--32658fa96c080068 + url-map: k8s-um-default-gce-test--32658fa96c080068 + backends: {"k8s-be-30301--32658fa96c080068":"Unknown"} + forwarding-rule: k8s-fw-default-gce-test--32658fa96c080068 + https-forwarding-rule: k8s-fws-default-gce-test--32658fa96c080068 + https-target-proxy: k8s-tps-default-gce-test--32658fa96c080068 + static-ip: k8s-fw-default-gce-test--32658fa96c080068 +Events: + FirstSeen LastSeen Count From SubObjectPath Type Reason Message + --------- -------- ----- ---- ------------- -------- ------ ------- + 2m 2m 1 {loadbalancer-controller } Normal ADD default/gce-test + 1m 1m 1 {loadbalancer-controller } Normal CREATE ip: 35.186.221.137 + 1m 1m 3 {loadbalancer-controller } Normal Service default backend set to http-svc:30301 + +$ curl 35.186.221.137 -k +curl 35.186.221.137 -L +curl: (60) SSL certificate problem: self signed certificate +More details here: http://curl.haxx.se/docs/sslcerts.html + +$ curl 35.186.221.137 -kl +CLIENT VALUES: +client_address=10.240.0.3 +command=GET +real path=/ +query=nil +request_version=1.1 +request_uri=http://35.186.221.137:8080/ + +SERVER VALUES: +server_version=nginx: 1.9.11 - lua: 10001 + +HEADERS RECEIVED: +accept=*/* +connection=Keep-Alive +host=35.186.221.137 +user-agent=curl/7.46.0 +via=1.1 google +x-cloud-trace-context=bfa123130fd623989cca0192e43d9ba4/8610689379063045825 +x-forwarded-for=104.132.0.80, 35.186.221.137 +x-forwarded-proto=https +``` diff --git a/examples/tls-termination/nginx-tls-ingress.yaml b/examples/tls-termination/nginx-tls-ingress.yaml new file mode 100644 index 000000000..c73452dd6 --- /dev/null +++ b/examples/tls-termination/nginx-tls-ingress.yaml @@ -0,0 +1,18 @@ +apiVersion: extensions/v1beta1 +kind: Ingress +metadata: + name: nginx-test + annotations: + kubernetes.io/ingress.class: "nginx" +spec: + tls: + # This assumes tls-secret exists. + - secretName: tls-secret + rules: + - http: + paths: + - backend: + # This assumes http-svc exists and routes to healthy endpoints. + serviceName: http-svc + servicePort: 80 + diff --git a/examples/tls-termination/nginx.md b/examples/tls-termination/nginx.md new file mode 100644 index 000000000..b116cad09 --- /dev/null +++ b/examples/tls-termination/nginx.md @@ -0,0 +1,74 @@ +# TLS termination + +This example demonstrates how to terminate TLS through the nginx Ingress controller. + +## Prerequisites + +You need a [TLS cert](/examples/PREREQUISITES.md#tls-certificates) and a [test HTTP service](/examples/PREREQUISITES.md#test-http-service) for this example. +You will also need to make sure you Ingress targets exactly one Ingress +controller by specifying the [ingress.class annotation](/examples/PREREQUISITES.md#ingress-class). + +## Deployment + +The following command instructs the controller to terminate traffic using +the provided TLS cert, and forward un-encrypted HTTP traffic to the test +HTTP service. + +```console +$ kubectl create -f nginx-tls-ingress.yaml +``` + +## Validation + +You can confirm that the Ingress works. + +```console +$ kubectl describe ing nginx-test +Name: nginx-test +Namespace: default +Address: 104.198.183.6 +Default backend: default-http-backend:80 (10.180.0.4:8080,10.240.0.2:8080) +TLS: + tls-secret terminates +Rules: + Host Path Backends + ---- ---- -------- + * + http-svc:80 () +Annotations: +Events: + FirstSeen LastSeen Count From SubObjectPath Type Reason Message + --------- -------- ----- ---- ------------- -------- ------ ------- + 7s 7s 1 {nginx-ingress-controller } Normal CREATE default/nginx-test + 7s 7s 1 {nginx-ingress-controller } Normal UPDATE default/nginx-test + 7s 7s 1 {nginx-ingress-controller } Normal CREATE ip: 104.198.183.6 + 7s 7s 1 {nginx-ingress-controller } Warning MAPPING Ingress rule 'default/nginx-test' contains no path definition. Assuming / + +$ curl 104.198.183.6 -L +curl: (60) SSL certificate problem: self signed certificate +More details here: http://curl.haxx.se/docs/sslcerts.html + +$ curl 104.198.183.6 -Lk +CLIENT VALUES: +client_address=10.240.0.4 +command=GET +real path=/ +query=nil +request_version=1.1 +request_uri=http://35.186.221.137:8080/ + +SERVER VALUES: +server_version=nginx: 1.9.11 - lua: 10001 + +HEADERS RECEIVED: +accept=*/* +connection=Keep-Alive +host=35.186.221.137 +user-agent=curl/7.46.0 +via=1.1 google +x-cloud-trace-context=f708ea7e369d4514fc90d51d7e27e91d/13322322294276298106 +x-forwarded-for=104.132.0.80, 35.186.221.137 +x-forwarded-proto=https +BODY: + +``` From 3a37607138e3bbdb83dd393241d1fdd3a6232176 Mon Sep 17 00:00:00 2001 From: bprashanth Date: Mon, 30 Jan 2017 12:11:38 -0800 Subject: [PATCH 30/32] Change nginx controller image to 0.9.0-beta.1 --- controllers/nginx/Changelog.md | 16 ++++++++-------- controllers/nginx/Makefile | 6 +++--- .../nginx/kubeadm/nginx-ingress-controller.yaml | 2 +- .../nginx/nginx-ingress-controller.yaml | 2 +- 4 files changed, 13 insertions(+), 13 deletions(-) diff --git a/controllers/nginx/Changelog.md b/controllers/nginx/Changelog.md index cabaa38db..ef43ba96f 100644 --- a/controllers/nginx/Changelog.md +++ b/controllers/nginx/Changelog.md @@ -1,8 +1,8 @@ Changelog -### 0.9 +### 0.9-beta.1 -**Temporal Image:** `quay.io/aledbf/nginx-ingress-controller:0.9.0` +**Image:** `gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.1` *New Features:* @@ -33,7 +33,7 @@ Changelog - [X] [#158](https://github.com/kubernetes/ingress/pull/158) prefect unit test cases for annotation.proxy - [X] [#156](https://github.com/kubernetes/ingress/pull/156) Fix issue for ratelimit - [X] [#154](https://github.com/kubernetes/ingress/pull/154) add unit test cases for core.pkg.ingress.annotations.cors -- [X] [#151](https://github.com/kubernetes/ingress/pull/151) Port in redirect +- [X] [#151](https://github.com/kubernetes/ingress/pull/151) Port in redirect - [X] [#150](https://github.com/kubernetes/ingress/pull/150) Add support for custom header sizes - [X] [#149](https://github.com/kubernetes/ingress/pull/149) Add flag to allow switch off the update of Ingress status - [X] [#148](https://github.com/kubernetes/ingress/pull/148) Add annotation to allow custom body sizes @@ -43,7 +43,7 @@ Changelog - [X] [#142](https://github.com/kubernetes/ingress/pull/142) Use system fs.max-files as limits instead of hard-coded value - [X] [#141](https://github.com/kubernetes/ingress/pull/141) Add reuse port and backlog to port 80 and 443 - [X] [#138](https://github.com/kubernetes/ingress/pull/138) reference to const -- [X] [#136](https://github.com/kubernetes/ingress/pull/136) Add content and descriptions about nginx's configuration +- [X] [#136](https://github.com/kubernetes/ingress/pull/136) Add content and descriptions about nginx's configuration - [X] [#135](https://github.com/kubernetes/ingress/pull/135) correct improper punctuation - [X] [#134](https://github.com/kubernetes/ingress/pull/134) fix typo - [X] [#133](https://github.com/kubernetes/ingress/pull/133) Add TCP and UDP services removed in migration @@ -81,9 +81,9 @@ Changelog - [X] [#62](https://github.com/kubernetes/ingress/pull/62) Fix e2e make targets - [X] [#61](https://github.com/kubernetes/ingress/pull/61) Ignore coverage profile files - [X] [#58](https://github.com/kubernetes/ingress/pull/58) Fix "invalid port in upstream" on nginx controller -- [X] [#57](https://github.com/kubernetes/ingress/pull/57) Fix invalid port in upstream +- [X] [#57](https://github.com/kubernetes/ingress/pull/57) Fix invalid port in upstream - [X] [#54](https://github.com/kubernetes/ingress/pull/54) Expand developer docs -- [X] [#52](https://github.com/kubernetes/ingress/pull/52) fix typo in variable ProxyRealIPCIDR +- [X] [#52](https://github.com/kubernetes/ingress/pull/52) fix typo in variable ProxyRealIPCIDR - [X] [#44](https://github.com/kubernetes/ingress/pull/44) Bump nginx version to one higher than that in contrib - [X] [#36](https://github.com/kubernetes/ingress/pull/36) Add nginx metrics to prometheus - [X] [#34](https://github.com/kubernetes/ingress/pull/34) nginx: also listen on ipv6 @@ -147,7 +147,7 @@ Changelog - [X] [#1136](https://github.com/kubernetes/contrib/pull/1136) Fix nginx rewrite rule order - [X] [#1144](https://github.com/kubernetes/contrib/pull/1144) Add cidr whitelist support - [X] [#1230](https://github.com/kubernetes/contrib/pull/1130) Improve docs and examples -- [X] [#1258](https://github.com/kubernetes/contrib/pull/1258) Avoid sync without a reachable +- [X] [#1258](https://github.com/kubernetes/contrib/pull/1258) Avoid sync without a reachable - [X] [#1235](https://github.com/kubernetes/contrib/pull/1235) Fix stats by country in nginx status page - [X] [#1236](https://github.com/kubernetes/contrib/pull/1236) Update nginx to add dynamic TLS records and spdy - [X] [#1238](https://github.com/kubernetes/contrib/pull/1238) Add support for dynamic TLS records and spdy @@ -171,4 +171,4 @@ Changelog - [X] [#1093](https://github.com/kubernetes/contrib/pull/1093) rate limiting - [X] [#1102](https://github.com/kubernetes/contrib/pull/1102) geolocation of traffic in stats - [X] [#884](https://github.com/kubernetes/contrib/issues/884) support services running ssl -- [X] [#930](https://github.com/kubernetes/contrib/issues/930) detect changes in configuration configmaps \ No newline at end of file +- [X] [#930](https://github.com/kubernetes/contrib/issues/930) detect changes in configuration configmaps diff --git a/controllers/nginx/Makefile b/controllers/nginx/Makefile index 94c38a6f4..65eb4af98 100644 --- a/controllers/nginx/Makefile +++ b/controllers/nginx/Makefile @@ -3,7 +3,7 @@ all: push BUILDTAGS= # Use the 0.0 tag for testing, it shouldn't clobber any release builds -RELEASE?=0.9.0 +RELEASE?=0.9.0-beta.1 PREFIX?=gcr.io/google_containers/nginx-ingress-controller GOOS?=linux @@ -20,10 +20,10 @@ build: clean -ldflags "-s -w -X ${PKG}/pkg/version.RELEASE=${RELEASE} -X ${PKG}/pkg/version.COMMIT=${COMMIT} -X ${PKG}/pkg/version.REPO=${REPO_INFO}" \ -o rootfs/nginx-ingress-controller ${PKG}/pkg/cmd/controller -container: +container: build docker build -t $(PREFIX):$(RELEASE) rootfs -push: +push: container gcloud docker push $(PREFIX):$(RELEASE) fmt: diff --git a/examples/deployment/nginx/kubeadm/nginx-ingress-controller.yaml b/examples/deployment/nginx/kubeadm/nginx-ingress-controller.yaml index 5c3edddda..d25c7731d 100644 --- a/examples/deployment/nginx/kubeadm/nginx-ingress-controller.yaml +++ b/examples/deployment/nginx/kubeadm/nginx-ingress-controller.yaml @@ -71,7 +71,7 @@ spec: hostNetwork: true terminationGracePeriodSeconds: 60 containers: - - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0 + - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.1 name: nginx-ingress-controller readinessProbe: httpGet: diff --git a/examples/deployment/nginx/nginx-ingress-controller.yaml b/examples/deployment/nginx/nginx-ingress-controller.yaml index 4760c2e4e..7ee5e797e 100644 --- a/examples/deployment/nginx/nginx-ingress-controller.yaml +++ b/examples/deployment/nginx/nginx-ingress-controller.yaml @@ -19,7 +19,7 @@ spec: # hostNetwork: true terminationGracePeriodSeconds: 60 containers: - - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0 + - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.1 name: nginx-ingress-controller readinessProbe: httpGet: From 66c826f41fc9dd0e0ae9dc474e851f8c05fb7165 Mon Sep 17 00:00:00 2001 From: bprashanth Date: Mon, 30 Jan 2017 18:42:36 -0800 Subject: [PATCH 31/32] Add a custom controller example --- docs/catalog.md | 2 + docs/dev/devel.md | 1 + docs/faq/README.md | 2 +- examples/README.md | 6 ++ examples/custom-controller/Dockerfile | 20 ++++++ examples/custom-controller/Makefile | 39 ++++++++++ examples/custom-controller/README.md | 29 ++++++++ examples/custom-controller/deployment.yaml | 51 ++++++++++++++ examples/custom-controller/server.go | 82 ++++++++++++++++++++++ 9 files changed, 231 insertions(+), 1 deletion(-) create mode 100644 examples/custom-controller/Dockerfile create mode 100644 examples/custom-controller/Makefile create mode 100644 examples/custom-controller/README.md create mode 100644 examples/custom-controller/deployment.yaml create mode 100644 examples/custom-controller/server.go diff --git a/docs/catalog.md b/docs/catalog.md index 0b34df51b..6c1ccd43f 100644 --- a/docs/catalog.md +++ b/docs/catalog.md @@ -2,4 +2,6 @@ This is a non-comprehensive list of existing ingress controllers. +* [Dummy controller backend](/examples/custom-controller) + diff --git a/docs/dev/devel.md b/docs/dev/devel.md index d472638e7..e3d7c94c2 100644 --- a/docs/dev/devel.md +++ b/docs/dev/devel.md @@ -1,3 +1,4 @@ # Writing Ingress controllers This doc outlines the basic steps needed to write an Ingress controller. +If you want the tl;dr version, skip straight to the [example](/examples/custom-controller). diff --git a/docs/faq/README.md b/docs/faq/README.md index 53810643d..f7e4de16e 100644 --- a/docs/faq/README.md +++ b/docs/faq/README.md @@ -85,7 +85,7 @@ as well as in [this](/examples/pipeline) example. First check the [catalog](#is-there-a-catalog-of-existing-ingress-controllers), to make sure you really need to write one. -1. Write a [generic backend](https://github.com/kubernetes/ingress/blob/master/core/pkg/ingress/doc.go) +1. Write a [generic backend](/examples/custom-controller) 2. Keep it in your own repo, make sure it passes the [conformance suite](https://github.com/kubernetes/kubernetes/blob/master/test/e2e/ingress_utils.go#L112) 3. Submit an example(s) in the appropriate subdirectories [here](/examples/README.md) 4. Add it to the catalog diff --git a/examples/README.md b/examples/README.md index 2a00a24d7..01d842eb2 100644 --- a/examples/README.md +++ b/examples/README.md @@ -69,4 +69,10 @@ Websockets | websockets loadbalancing | nginx | Intermediate HTTP/2 | HTTP/2 loadbalancing | * | Intermediate Proxy protocol | leverage the proxy protocol for source IP | nginx | Advanced +## Custom controllers + +Name | Description | Platform | Complexity Level +-----| ----------- | ---------- | ---------------- +Dummy | A simple dummy controller that logs updates | * | Advanced + diff --git a/examples/custom-controller/Dockerfile b/examples/custom-controller/Dockerfile new file mode 100644 index 000000000..197ea92f7 --- /dev/null +++ b/examples/custom-controller/Dockerfile @@ -0,0 +1,20 @@ +# Copyright 2017[<0;55;12M The Kubernetes Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# TODO(ingress#191): Change this to something more appropriate, like busybox +From ubuntu:15.10 +MAINTAINER Prashanth B +RUN apt-get update && apt-get install ssl-cert -y +COPY server / +ENTRYPOINT ["/server"] diff --git a/examples/custom-controller/Makefile b/examples/custom-controller/Makefile new file mode 100644 index 000000000..a66cf01f6 --- /dev/null +++ b/examples/custom-controller/Makefile @@ -0,0 +1,39 @@ +# Copyright 2017 The Kubernetes Authors All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Build the default backend binary or image for amd64, arm, arm64 and ppc64le +# +# Usage: +# [PREFIX=gcr.io/google_containers/dummy-ingress-controller] [ARCH=amd64] [TAG=1.1] make (server|container|push) + +all: push + +TAG=0.1 +PREFIX?=bprashanth/dummy-ingress-controller +ARCH?=amd64 +GOLANG_VERSION=1.6 +TEMP_DIR:=$(shell mktemp -d) + +server: server.go + CGO_ENABLED=0 GOOS=linux GOARCH=$(ARCH) GOARM=6 godep go build -a -installsuffix cgo -ldflags '-w' -o server ./server.go + +container: server + docker build --pull -t $(PREFIX)-$(ARCH):$(TAG) . + +push: container + gcloud docker push $(PREFIX)-$(ARCH):$(TAG) + +clean: + rm -f server + diff --git a/examples/custom-controller/README.md b/examples/custom-controller/README.md new file mode 100644 index 000000000..f420ae780 --- /dev/null +++ b/examples/custom-controller/README.md @@ -0,0 +1,29 @@ +# Dummy controller + +This example contains the source code of a simple dummy controller. If you want +more details on the interface, or what the generic controller is actually doing, +please read [this doc](/docs/dev/devel.md). You can deploy the controller as +follows: + +```console +$ kubectl create -f deployment.yaml +service "default-backend" created +deployment "dummy-ingress-controller" created + +$ kubectl get po +NAME READY STATUS RESTARTS AGE +dummy-ingress-controller-3685541482-082nl 1/1 Running 0 10m + +$ kubectl logs dummy-ingress-controller-3685541482-082nl +I0131 02:29:02.462123 1 launch.go:92] &{dummy 0.0.0 git-00000000 git://foo.bar.com} +I0131 02:29:02.462513 1 launch.go:221] Creating API server client for https://10.0.0.1:443 +I0131 02:29:02.494571 1 launch.go:111] validated default/default-backend as the default backend +I0131 02:29:02.503180 1 controller.go:1038] starting Ingress controller +I0131 02:29:02.513528 1 leaderelection.go:247] lock is held by dummy-ingress-controller-3685541482-50jh0 and has not yet expired +W0131 02:29:03.510699 1 queue.go:87] requeuing kube-system/kube-scheduler, err deferring sync till endpoints controller has synced +W0131 02:29:03.514445 1 queue.go:87] requeuing kube-system/node-controller-token-826dl, err deferring sync till endpoints controller has synced +2017/01/31 02:29:12 Received OnUpdate notification +2017/01/31 02:29:12 upstream-default-backend: 10.180.1.20 +``` + + diff --git a/examples/custom-controller/deployment.yaml b/examples/custom-controller/deployment.yaml new file mode 100644 index 000000000..cf924b7f7 --- /dev/null +++ b/examples/custom-controller/deployment.yaml @@ -0,0 +1,51 @@ +apiVersion: v1 +kind: Service +metadata: + name: default-backend + namespace: default + labels: + name: default-backend + app: dummy-ingress-controller +spec: + ports: + - port: 80 + targetPort: 10254 + selector: + # Point back the the dummy controller's + # healthz port + app: dummy-ingress-controller +--- +apiVersion: extensions/v1beta1 +kind: Deployment +metadata: + name: dummy-ingress-controller + namespace: default + labels: + app: dummy-ingress-controller +spec: + selector: + matchLabels: + app: dummy-ingress-controller + template: + metadata: + labels: + app: dummy-ingress-controller + spec: + containers: + - name: server + image: bprashanth/dummy-ingress-controller-amd64:0.1 + imagePullPolicy: Always + ports: + - containerPort: 10254 + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + args: + - /server + - --default-backend-service=$(POD_NAMESPACE)/default-backend diff --git a/examples/custom-controller/server.go b/examples/custom-controller/server.go new file mode 100644 index 000000000..6f1dcb840 --- /dev/null +++ b/examples/custom-controller/server.go @@ -0,0 +1,82 @@ +package main + +import ( + "log" + "net/http" + "os/exec" + "strings" + + nginxconfig "k8s.io/ingress/controllers/nginx/pkg/config" + "k8s.io/ingress/core/pkg/ingress" + "k8s.io/ingress/core/pkg/ingress/controller" + "k8s.io/ingress/core/pkg/ingress/defaults" + "k8s.io/kubernetes/pkg/api" +) + +func main() { + dc := newDummyController() + ic := controller.NewIngressController(dc) + defer func() { + log.Printf("Shutting down ingress controller...") + ic.Stop() + }() + ic.Start() +} + +func newDummyController() ingress.Controller { + return &DummyController{} +} + +type DummyController struct{} + +func (dc DummyController) SetConfig(cfgMap *api.ConfigMap) { + log.Printf("Config map %+v", cfgMap) +} + +func (dc DummyController) Reload(data []byte) ([]byte, bool, error) { + out, err := exec.Command("echo", string(data)).CombinedOutput() + if err != nil { + log.Printf("Reloaded new config %s", out) + } else { + return out, false, err + } + return out, true, err +} + +func (dc DummyController) Test(file string) *exec.Cmd { + return exec.Command("echo", file) +} + +func (dc DummyController) OnUpdate(updatePayload ingress.Configuration) ([]byte, error) { + log.Printf("Received OnUpdate notification") + for _, b := range updatePayload.Backends { + eps := []string{} + for _, e := range b.Endpoints { + eps = append(eps, e.Address) + } + log.Printf("%v: %v", b.Name, strings.Join(eps, ", ")) + } + return []byte(``), nil +} + +func (dc DummyController) BackendDefaults() defaults.Backend { + // Just adopt nginx's default backend config + return nginxconfig.NewDefault().Backend +} + +func (n DummyController) Name() string { + return "dummy Controller" +} + +func (n DummyController) Check(_ *http.Request) error { + return nil +} + +func (dc DummyController) Info() *ingress.BackendInfo { + return &ingress.BackendInfo{ + Name: "dummy", + Release: "0.0.0", + Build: "git-00000000", + Repository: "git://foo.bar.com", + } +} From d002ca2f5ed08e8cf8756acae7329aa2ba536cc5 Mon Sep 17 00:00:00 2001 From: bprashanth Date: Tue, 31 Jan 2017 10:19:58 -0800 Subject: [PATCH 32/32] Move custom images out of contrib --- images/OWNERS | 5 + images/nginx-slim/Dockerfile | 29 ++++ images/nginx-slim/Makefile | 14 ++ images/nginx-slim/README.md | 24 +++ images/nginx-slim/build.sh | 223 ++++++++++++++++++++++++++++ images/nginx-slim/rc.yaml | 34 +++++ images/ubuntu-slim/Dockerfile | 5 + images/ubuntu-slim/Dockerfile.build | 45 ++++++ images/ubuntu-slim/Makefile | 21 +++ images/ubuntu-slim/README.md | 22 +++ images/ubuntu-slim/excludes | 10 ++ images/ubuntu-slim/runlevel | 3 + 12 files changed, 435 insertions(+) create mode 100644 images/OWNERS create mode 100644 images/nginx-slim/Dockerfile create mode 100644 images/nginx-slim/Makefile create mode 100644 images/nginx-slim/README.md create mode 100755 images/nginx-slim/build.sh create mode 100644 images/nginx-slim/rc.yaml create mode 100644 images/ubuntu-slim/Dockerfile create mode 100644 images/ubuntu-slim/Dockerfile.build create mode 100755 images/ubuntu-slim/Makefile create mode 100644 images/ubuntu-slim/README.md create mode 100644 images/ubuntu-slim/excludes create mode 100755 images/ubuntu-slim/runlevel diff --git a/images/OWNERS b/images/OWNERS new file mode 100644 index 000000000..79db51eae --- /dev/null +++ b/images/OWNERS @@ -0,0 +1,5 @@ +approvers: +- aledbf +reviewers: +- bprashanth +- aledbf diff --git a/images/nginx-slim/Dockerfile b/images/nginx-slim/Dockerfile new file mode 100644 index 000000000..089243ef0 --- /dev/null +++ b/images/nginx-slim/Dockerfile @@ -0,0 +1,29 @@ +# Copyright 2015 The Kubernetes Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +FROM gcr.io/google_containers/ubuntu-slim:0.6 + +COPY build.sh /tmp + +RUN /tmp/build.sh + +# Create symlinks to redirect nginx logs to stdout and stderr docker log collector +# This only works if nginx is started with CMD or ENTRYPOINT +RUN ln -sf /dev/stdout /var/log/nginx/access.log +RUN ln -sf /dev/stderr /var/log/nginx/error.log + +EXPOSE 80 443 + +CMD ["nginx", "-g", "daemon off;"] diff --git a/images/nginx-slim/Makefile b/images/nginx-slim/Makefile new file mode 100644 index 000000000..2c783c60d --- /dev/null +++ b/images/nginx-slim/Makefile @@ -0,0 +1,14 @@ +all: push + +# 0.0.0 shouldn't clobber any released builds +TAG = 0.13 +PREFIX = gcr.io/google_containers/nginx-slim + +container: + docker build --pull -t $(PREFIX):$(TAG) . + +push: container + gcloud docker push $(PREFIX):$(TAG) + +clean: + docker rmi -f $(PREFIX):$(TAG) || true diff --git a/images/nginx-slim/README.md b/images/nginx-slim/README.md new file mode 100644 index 000000000..8d93e3fa4 --- /dev/null +++ b/images/nginx-slim/README.md @@ -0,0 +1,24 @@ + +nginx 1.11.x base image using [ubuntu-slim](https://github.com/kubernetes/contrib/tree/master/images/ubuntu-slim) + +nginx [engine x] is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP proxy server. + +This custom nginx image contains: +- [lua](https://github.com/openresty/lua-nginx-module) support +- [stream](http://nginx.org/en/docs/stream/ngx_stream_core_module.html) tcp support for upstreams +- nginx stats [nginx-module-vts](https://github.com/vozlt/nginx-module-vts) +- [Dynamic TLS record sizing](https://blog.cloudflare.com/optimizing-tls-over-tcp-to-reduce-latency/) + + +**How to use this image:** +This image does provides a default configuration file with no backend servers. + +*Using docker* +``` +$ docker run -v /some/nginx.con:/etc/nginx/nginx.conf:ro gcr.io/google_containers/nginx-slim:0.12 +``` + +*Creating a replication controller* +``` +$ kubectl create -f ./rc.yaml +``` diff --git a/images/nginx-slim/build.sh b/images/nginx-slim/build.sh new file mode 100755 index 000000000..f05ae633a --- /dev/null +++ b/images/nginx-slim/build.sh @@ -0,0 +1,223 @@ +#!/bin/sh + +# Copyright 2015 The Kubernetes Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +set -e + +export NGINX_VERSION=1.11.9 +export NDK_VERSION=0.3.0 +export VTS_VERSION=0.1.11 +export SETMISC_VERSION=0.31 +export LUA_VERSION=0.10.7 +export STICKY_SESSIONS_VERSION=08a395c66e42 +export LUA_CJSON_VERSION=2.1.0.4 +export LUA_RESTY_HTTP_VERSION=0.07 +export LUA_UPSTREAM_VERSION=0.06 +export MORE_HEADERS_VERSION=0.32 +export NGINX_DIGEST_AUTH=7955af9c77598c697ac292811914ce1e2b3b824c +export NGINX_SUBSTITUTIONS=bc58cb11844bc42735bbaef7085ea86ace46d05b + +export BUILD_PATH=/tmp/build + +get_src() +{ + hash="$1" + url="$2" + f=$(basename "$url") + + curl -sSL "$url" -o "$f" + echo "$hash $f" | sha256sum -c - || exit 10 + tar xzf "$f" + rm -rf "$f" +} + +mkdir "$BUILD_PATH" +cd "$BUILD_PATH" + +# install required packages to build +apt-get update && apt-get install --no-install-recommends -y \ + bash \ + build-essential \ + curl ca-certificates \ + libgeoip1 \ + libgeoip-dev \ + patch \ + libpcre3 \ + libpcre3-dev \ + libssl-dev \ + zlib1g \ + zlib1g-dev \ + libaio1 \ + libaio-dev \ + luajit \ + openssl \ + libluajit-5.1 \ + libluajit-5.1-dev \ + linux-headers-generic || exit 1 + +# download, verify and extract the source files +get_src dc22b71f16b551705930544dc042f1ad1af2f9715f565187ec22c7a4b2625748 \ + "http://nginx.org/download/nginx-$NGINX_VERSION.tar.gz" + +get_src 88e05a99a8a7419066f5ae75966fb1efc409bad4522d14986da074554ae61619 \ + "https://github.com/simpl/ngx_devel_kit/archive/v$NDK_VERSION.tar.gz" + +get_src 97946a68937b50ab8637e1a90a13198fe376d801dc3e7447052e43c28e9ee7de \ + "https://github.com/openresty/set-misc-nginx-module/archive/v$SETMISC_VERSION.tar.gz" + +get_src 31db853251a631a6b6a0b96b10806c9c32eda3c3d08fe46a38ff944b22dba636 \ + "https://github.com/vozlt/nginx-module-vts/archive/v$VTS_VERSION.tar.gz" + +get_src c21c8937dcdd6fc2b6a955f929e3f4d1388610f47180e60126e6dcab06786f77 \ + "https://github.com/openresty/lua-nginx-module/archive/v$LUA_VERSION.tar.gz" + +get_src 5417991b6db4d46383da2d18f2fd46b93fafcebfe87ba87f7cfeac4c9bcb0224 \ + "https://github.com/openresty/lua-cjson/archive/$LUA_CJSON_VERSION.tar.gz" + +get_src 1c6aa06c9955397c94e9c3e0c0fba4e2704e85bee77b4512fb54ae7c25d58d86 \ + "https://github.com/pintsized/lua-resty-http/archive/v$LUA_RESTY_HTTP_VERSION.tar.gz" + +get_src c6d9dab8ea1fc997031007e2e8f47cced01417e203cd88d53a9fe9f6ae138720 \ + "https://github.com/openresty/headers-more-nginx-module/archive/v$MORE_HEADERS_VERSION.tar.gz" + +get_src 55475fe4f9e4b5220761269ccf0069ebb1ded61d7e7888f9c785c651cff3d141 \ + "https://github.com/openresty/lua-upstream-nginx-module/archive/v$LUA_UPSTREAM_VERSION.tar.gz" + +get_src 53e440737ed1aff1f09fae150219a45f16add0c8d6e84546cb7d80f73ebffd90 \ + "https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng/get/$STICKY_SESSIONS_VERSION.tar.gz" + +get_src 9b1d0075df787338bb607f14925886249bda60b6b3156713923d5d59e99a708b \ + "https://github.com/atomx/nginx-http-auth-digest/archive/$NGINX_DIGEST_AUTH.tar.gz" + +get_src 8eabbcd5950fdcc718bb0ef9165206c2ed60f67cd9da553d7bc3e6fe4e338461 \ + "https://github.com/yaoweibin/ngx_http_substitutions_filter_module/archive/$NGINX_SUBSTITUTIONS.tar.gz" + + +#https://blog.cloudflare.com/optimizing-tls-over-tcp-to-reduce-latency/ +curl -sSL -o nginx__dynamic_tls_records.patch https://raw.githubusercontent.com/cloudflare/sslconfig/master/patches/nginx__1.11.5_dynamic_tls_records.patch + +# build nginx +cd "$BUILD_PATH/nginx-$NGINX_VERSION" + +echo "Applying tls nginx patches..." +patch -p1 < $BUILD_PATH/nginx__dynamic_tls_records.patch + +./configure \ + --prefix=/usr/share/nginx \ + --conf-path=/etc/nginx/nginx.conf \ + --http-log-path=/var/log/nginx/access.log \ + --error-log-path=/var/log/nginx/error.log \ + --lock-path=/var/lock/nginx.lock \ + --pid-path=/run/nginx.pid \ + --http-client-body-temp-path=/var/lib/nginx/body \ + --http-fastcgi-temp-path=/var/lib/nginx/fastcgi \ + --http-proxy-temp-path=/var/lib/nginx/proxy \ + --http-scgi-temp-path=/var/lib/nginx/scgi \ + --http-uwsgi-temp-path=/var/lib/nginx/uwsgi \ + --with-debug \ + --with-pcre-jit \ + --with-http_ssl_module \ + --with-http_stub_status_module \ + --with-http_realip_module \ + --with-http_auth_request_module \ + --with-http_addition_module \ + --with-http_dav_module \ + --with-http_geoip_module \ + --with-http_gzip_static_module \ + --with-http_sub_module \ + --with-http_v2_module \ + --with-stream \ + --with-stream_ssl_module \ + --with-stream_ssl_preread_module \ + --with-threads \ + --with-file-aio \ + --without-mail_pop3_module \ + --without-mail_smtp_module \ + --without-mail_imap_module \ + --without-http_uwsgi_module \ + --without-http_scgi_module \ + --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' \ + --add-module="$BUILD_PATH/ngx_devel_kit-$NDK_VERSION" \ + --add-module="$BUILD_PATH/set-misc-nginx-module-$SETMISC_VERSION" \ + --add-module="$BUILD_PATH/nginx-module-vts-$VTS_VERSION" \ + --add-module="$BUILD_PATH/lua-nginx-module-$LUA_VERSION" \ + --add-module="$BUILD_PATH/headers-more-nginx-module-$MORE_HEADERS_VERSION" \ + --add-module="$BUILD_PATH/nginx-goodies-nginx-sticky-module-ng-$STICKY_SESSIONS_VERSION" \ + --add-module="$BUILD_PATH/nginx-http-auth-digest-$NGINX_DIGEST_AUTH" \ + --add-module="$BUILD_PATH/ngx_http_substitutions_filter_module-$NGINX_SUBSTITUTIONS" \ + --add-module="$BUILD_PATH/lua-upstream-nginx-module-$LUA_UPSTREAM_VERSION" || exit 1 \ + && make || exit 1 \ + && make install || exit 1 + +echo "Installing CJSON module" +cd "$BUILD_PATH/lua-cjson-$LUA_CJSON_VERSION" +make LUA_INCLUDE_DIR=/usr/include/luajit-2.0 && make install + +echo "Installing lua-resty-http module" +# copy lua module +cd "$BUILD_PATH/lua-resty-http-$LUA_RESTY_HTTP_VERSION" +sed -i 's/resty.http_headers/http_headers/' $BUILD_PATH/lua-resty-http-$LUA_RESTY_HTTP_VERSION/lib/resty/http.lua +cp $BUILD_PATH/lua-resty-http-$LUA_RESTY_HTTP_VERSION/lib/resty/http.lua /usr/local/lib/lua/5.1 +cp $BUILD_PATH/lua-resty-http-$LUA_RESTY_HTTP_VERSION/lib/resty/http_headers.lua /usr/local/lib/lua/5.1 + +echo "Cleaning..." + +cd / + +apt-mark unmarkauto \ + bash \ + curl ca-certificates \ + libgeoip1 \ + libpcre3 \ + zlib1g \ + libaio1 \ + luajit \ + libluajit-5.1-2 \ + xz-utils \ + geoip-bin \ + openssl + +apt-get remove -y --purge \ + build-essential \ + gcc-5 \ + cpp-5 \ + libgeoip-dev \ + libpcre3-dev \ + libssl-dev \ + zlib1g-dev \ + libaio-dev \ + libluajit-5.1-dev \ + linux-libc-dev \ + perl-modules-5.22 \ + linux-headers-generic + +apt-get autoremove -y + +mkdir -p /var/lib/nginx/body /usr/share/nginx/html + +mv /usr/share/nginx/sbin/nginx /usr/sbin + +rm -rf "$BUILD_PATH" +rm -Rf /usr/share/man /usr/share/doc +rm -rf /tmp/* /var/tmp/* +rm -rf /var/lib/apt/lists/* +rm -rf /var/cache/apt/archives/* + +# Download of GeoIP databases +curl -sSL -o /etc/nginx/GeoIP.dat.gz http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz \ + && curl -sSL -o /etc/nginx/GeoLiteCity.dat.gz http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz \ + && gunzip /etc/nginx/GeoIP.dat.gz \ + && gunzip /etc/nginx/GeoLiteCity.dat.gz diff --git a/images/nginx-slim/rc.yaml b/images/nginx-slim/rc.yaml new file mode 100644 index 000000000..248308d0f --- /dev/null +++ b/images/nginx-slim/rc.yaml @@ -0,0 +1,34 @@ +apiVersion: v1 +kind: Service +metadata: + name: nginxslimsvc + labels: + app: nginxslim +spec: + type: NodePort + ports: + - port: 80 + protocol: TCP + name: http + selector: + app: nginxslim +--- +apiVersion: v1 +kind: ReplicationController +metadata: + name: nginxslim +spec: + replicas: 1 + selector: + app: nginxslim + template: + metadata: + labels: + app: nginxslim + name: frontend + spec: + containers: + - name: nginxslim + image: gcr.io/google_containers/nginx-slim:0.12 + ports: + - containerPort: 80 diff --git a/images/ubuntu-slim/Dockerfile b/images/ubuntu-slim/Dockerfile new file mode 100644 index 000000000..05b6e8dd8 --- /dev/null +++ b/images/ubuntu-slim/Dockerfile @@ -0,0 +1,5 @@ +FROM scratch + +ADD rootfs.tar / + +CMD ["/bin/bash"] diff --git a/images/ubuntu-slim/Dockerfile.build b/images/ubuntu-slim/Dockerfile.build new file mode 100644 index 000000000..1a1844e27 --- /dev/null +++ b/images/ubuntu-slim/Dockerfile.build @@ -0,0 +1,45 @@ +FROM ubuntu:16.04 + +ENV DEBIAN_FRONTEND=noninteractive + +COPY excludes /etc/dpkg/dpkg.cfg.d/excludes + +RUN apt-get update \ + && apt-get dist-upgrade -y + +COPY runlevel /sbin/runlevel + +# hold required packages to avoid breaking the installation of packages +RUN apt-mark hold apt gnupg adduser passwd libsemanage1 + +# dpkg --get-selections | grep -v deinstall +RUN echo "Yes, do as I say!" | apt-get purge \ + libcap2-bin \ + libkmod2 \ + libsmartcols1 \ + libudev1 \ + tzdata + +# cleanup +RUN apt-get autoremove -y && \ + apt-get clean -y && \ + tar -czf /usr/share/copyrights.tar.gz /usr/share/common-licenses /usr/share/doc/*/copyright && \ + rm -rf \ + /usr/share/doc \ + /usr/share/man \ + /usr/share/info \ + /usr/share/locale \ + /var/lib/apt/lists/* \ + /var/log/* \ + /var/cache/debconf/* \ + /usr/share/common-licenses* \ + ~/.bashrc \ + /etc/systemd \ + /lib/lsb \ + /lib/udev \ + /usr/lib/x86_64-linux-gnu/gconv/IBM* \ + /usr/lib/x86_64-linux-gnu/gconv/EBC* && \ + mkdir -p /usr/share/man/man1 /usr/share/man/man2 \ + /usr/share/man/man3 /usr/share/man/man4 \ + /usr/share/man/man5 /usr/share/man/man6 \ + /usr/share/man/man7 /usr/share/man/man8 diff --git a/images/ubuntu-slim/Makefile b/images/ubuntu-slim/Makefile new file mode 100755 index 000000000..bf093cec1 --- /dev/null +++ b/images/ubuntu-slim/Makefile @@ -0,0 +1,21 @@ +all: push + +TAG ?= 0.6 +PREFIX ?= gcr.io/google_containers/ubuntu-slim +BUILD_IMAGE ?= ubuntu-build +TAR_FILE ?= rootfs.tar + +container: clean + docker build --pull -t $(BUILD_IMAGE) -f Dockerfile.build . + docker create --name $(BUILD_IMAGE) $(BUILD_IMAGE) + docker export $(BUILD_IMAGE) > $(TAR_FILE) + docker build --pull -t $(PREFIX):$(TAG) . + +push: container + docker push $(PREFIX):$(TAG) + +clean: + docker rmi -f $(PREFIX):$(TAG) || true + docker rmi -f $(BUILD_IMAGE) || true + docker rm -f $(BUILD_IMAGE) || true + rm -f $(TAR_FILE) diff --git a/images/ubuntu-slim/README.md b/images/ubuntu-slim/README.md new file mode 100644 index 000000000..b905af43b --- /dev/null +++ b/images/ubuntu-slim/README.md @@ -0,0 +1,22 @@ + +Small Ubuntu 16.04 docker image + +The size of this image is ~56MB (less than half than `ubuntu:16.04). +This is possible by the removal of packages that are not required in a container: +- dmsetup +- e2fsprogs +- init +- initscripts +- libcap2-bin +- libcryptsetup4 +- libdevmapper1.02.1 +- libkmod2 +- libsmartcols1 +- libudev1 +- mount +- procps +- systemd +- systemd-sysv +- tzdata +- udev +- util-linux diff --git a/images/ubuntu-slim/excludes b/images/ubuntu-slim/excludes new file mode 100644 index 000000000..d5af11a9d --- /dev/null +++ b/images/ubuntu-slim/excludes @@ -0,0 +1,10 @@ +path-exclude /usr/share/doc/* +path-include /usr/share/doc/*/copyright +path-exclude /usr/share/man/* +path-exclude /usr/share/groff/* +path-exclude /usr/share/info/* +path-exclude /usr/share/locale/* +path-include /usr/share/locale/en_US* +path-include /usr/share/locale/locale.alias +path-exclude /usr/share/i18n/locales/* +path-include /usr/share/i18n/locales/en_US* diff --git a/images/ubuntu-slim/runlevel b/images/ubuntu-slim/runlevel new file mode 100755 index 000000000..c52d3c26b --- /dev/null +++ b/images/ubuntu-slim/runlevel @@ -0,0 +1,3 @@ +#!/bin/sh + +exit 0