diff --git a/controllers/nginx/README.md b/controllers/nginx/README.md index dabfcdc9d..9a1e3a4fb 100644 --- a/controllers/nginx/README.md +++ b/controllers/nginx/README.md @@ -17,6 +17,7 @@ This is an nginx Ingress controller that uses [ConfigMap](https://kubernetes.io/ * [TCP Services](#exposing-tcp-services) * [UDP Services](#exposing-udp-services) * [Proxy Protocol](#proxy-protocol) +* [Opentracing](#opentracing) * [NGINX customization](configuration.md) * [Custom errors](#custom-errors) * [NGINX status page](#nginx-status-page) @@ -334,8 +335,8 @@ version to fully support Kube-Lego is nginx Ingress controller 0.8. ## Exposing TCP services -Ingress does not support TCP services (yet). For this reason this Ingress controller uses the flag `--tcp-services-configmap` to point to an existing config map where the key is the external port to use and the value is `::[PROXY]` -It is possible to use a number or the name of the port. The last field is optional. Adding `PROXY` in the last field we can enable Proxy Protocol in a TCP service. +Ingress does not support TCP services (yet). For this reason this Ingress controller uses the flag `--tcp-services-configmap` to point to an existing config map where the key is the external port to use and the value is `::[PROXY]:[PROXY]` +It is possible to use a number or the name of the port. The two last fields are optional. Adding `PROXY` in either or both of the two last fields we can use Proxy Protocol decoding (listen) and/or encoding (proxy_pass) in a TCP service (https://www.nginx.com/resources/admin-guide/proxy-protocol/). The next example shows how to expose the service `example-go` running in the namespace `default` in the port `8080` using the port `9000` ``` @@ -378,14 +379,63 @@ Amongst others [ELBs in AWS](http://docs.aws.amazon.com/ElasticLoadBalancing/lat Please check the [proxy-protocol](examples/proxy-protocol/) example +### Opentracing + +Using the third party module [rnburn/nginx-opentracing](https://github.com/rnburn/nginx-opentracing) the NGINX ingress controller can configure NGINX to enable [OpenTracing](http://opentracing.io) instrumentation. +By default this feature is disabled. + +To enable the instrumentation we just need to enable the instrumentation in the configuration configmap and set the host where we should send the traces. + +In the [aledbf/zipkin-js-example](https://github.com/aledbf/zipkin-js-example) github repository is possible to see a dockerized version of zipkin-js-example with the required Kubernetes descriptors. +To install the example and the zipkin collector we just need to run: + +``` +$ kubectl create -f https://raw.githubusercontent.com/aledbf/zipkin-js-example/kubernetes/kubernetes/zipkin.yaml +$ kubectl create -f https://raw.githubusercontent.com/aledbf/zipkin-js-example/kubernetes/kubernetes/deployment.yaml +``` + +Also we need to configure the NGINX controller configmap with the required values: + +``` +apiVersion: v1 +data: + enable-opentracing: "true" + zipkin-collector-host: zipkin.default.svc.cluster.local +kind: ConfigMap +metadata: + labels: + k8s-app: nginx-ingress-controller + name: nginx-custom-configuration +``` + +Using curl we can generate some traces: +``` +$ curl -v http://$(minikube ip)/api -H 'Host: zipkin-js-example' +$ curl -v http://$(minikube ip)/api -H 'Host: zipkin-js-example' +``` + +In the zipkin inteface we can see the details: + +![zipkin screenshot](docs/images/zipkin-demo.png "zipkin collector screenshot") ### Custom errors -In case of an error in a request the body of the response is obtained from the `default backend`. Each request to the default backend includes two headers: -- `X-Code` indicates the HTTP code -- `X-Format` the value of the `Accept` header +In case of an error in a request the body of the response is obtained from the `default backend`. +Each request to the default backend includes two headers: -Using this two headers is possible to use a custom backend service like [this one](https://github.com/aledbf/contrib/tree/nginx-debug-server/Ingress/images/nginx-error-server) that inspect each request and returns a custom error page with the format expected by the client. Please check the example [custom-errors](examples/custom-errors/README.md) +- `X-Code` indicates the HTTP code to be returned to the client. +- `X-Format` the value of the `Accept` header. + +**Important:** the custom backend must return the correct HTTP status code to be returned. NGINX do not changes the reponse from the custom default backend. + +Using this two headers is possible to use a custom backend service like [this one](https://github.com/kubernetes/ingress/tree/master/examples/customization/custom-errors/nginx) that inspect each request and returns a custom error page with the format expected by the client. Please check the example [custom-errors](examples/customization/custom-errors/nginx/README.md) + +NGINX sends aditional headers that can be used to build custom response: + +- X-Original-URI +- X-Namespace +- X-Ingress-Name +- X-Service-Name ### NGINX status page diff --git a/controllers/nginx/configuration.md b/controllers/nginx/configuration.md index 17299ba9d..517cea608 100644 --- a/controllers/nginx/configuration.md +++ b/controllers/nginx/configuration.md @@ -135,14 +135,14 @@ Please check the [auth](/examples/auth/basic/nginx/README.md) example. ### Certificate Authentication -It's possible to enable Certificate based authentication using additional annotations in Ingress Rule. +It's possible to enable Certificate-Based Authentication (Mutual Authentication) using additional annotations in Ingress Rule. The annotations are: ``` ingress.kubernetes.io/auth-tls-secret: secretName ``` -The name of the secret that contains the full Certificate Authority chain that is enabled to authenticate against this ingress. It's composed of namespace/secretName +The name of the secret that contains the full Certificate Authority chain `ca.crt` that is enabled to authenticate against this ingress. It's composed of namespace/secretName. ``` ingress.kubernetes.io/auth-tls-verify-depth @@ -487,6 +487,17 @@ The default mime type list to compress is: `application/atom+xml application/jav **bind-address:** Sets the addresses on which the server will accept requests instead of *. It should be noted that these addresses must exist in the runtime environment or the controller will crash loop. +**enable-opentracing:** enables the nginx Opentracing extension https://github.com/rnburn/nginx-opentracing +Default is "false" + +**zipkin-collector-host:** specifies the host to use when uploading traces. It must be a valid URL + +**zipkin-collector-port:** specifies the port to use when uploading traces +Default: 9411 + +**zipkin-service-name:** specifies the service name to use for any traces created +Default: nginx + ### Default configuration options The following table shows the options, the default value and a description. diff --git a/controllers/nginx/docs/images/zipkin-demo.png b/controllers/nginx/docs/images/zipkin-demo.png new file mode 100644 index 000000000..c83fcf236 Binary files /dev/null and b/controllers/nginx/docs/images/zipkin-demo.png differ diff --git a/controllers/nginx/pkg/cmd/controller/nginx.go b/controllers/nginx/pkg/cmd/controller/nginx.go index d11a857ca..f9e034800 100644 --- a/controllers/nginx/pkg/cmd/controller/nginx.go +++ b/controllers/nginx/pkg/cmd/controller/nginx.go @@ -35,6 +35,7 @@ import ( "github.com/spf13/pflag" proxyproto "github.com/armon/go-proxyproto" + "github.com/ncabatoff/process-exporter/proc" apiv1 "k8s.io/api/core/v1" extensions "k8s.io/api/extensions/v1beta1" @@ -62,7 +63,7 @@ const ( var ( tmplPath = "/etc/nginx/template/nginx.tmpl" cfgPath = "/etc/nginx/nginx.conf" - binary = "/usr/sbin/nginx" + nginxBinary = "/usr/sbin/nginx" defIngressClass = "nginx" ) @@ -72,7 +73,7 @@ var ( func newNGINXController() *NGINXController { ngx := os.Getenv("NGINX_BINARY") if ngx == "" { - ngx = binary + ngx = nginxBinary } h, err := dns.GetSystemNameServers() @@ -200,7 +201,26 @@ NGINX master process died (%v): %v break } conn.Close() - time.Sleep(1 * time.Second) + // kill nginx worker processes + fs, err := proc.NewFS("/proc") + procs, _ := fs.FS.AllProcs() + for _, p := range procs { + pn, err := p.Comm() + if err != nil { + glog.Errorf("unexpected error obtaining process information: %v", err) + continue + } + + if pn == "nginx" { + osp, err := os.FindProcess(p.PID) + if err != nil { + glog.Errorf("unexpected error obtaining process information: %v", err) + continue + } + osp.Signal(syscall.SIGQUIT) + } + } + time.Sleep(100 * time.Millisecond) } // restart a new nginx master process if the controller // is not being stopped @@ -710,6 +730,28 @@ func (n NGINXController) Check(_ *http.Request) error { if res.StatusCode != 200 { return fmt.Errorf("ingress controller is not healthy") } + + // check the nginx master process is running + fs, err := proc.NewFS("/proc") + if err != nil { + glog.Errorf("%v", err) + return err + } + f, err := ioutil.ReadFile("/run/nginx.pid") + if err != nil { + glog.Errorf("%v", err) + return err + } + pid, err := strconv.Atoi(strings.TrimRight(string(f), "\r\n")) + if err != nil { + return err + } + _, err = fs.NewProc(int(pid)) + if err != nil { + glog.Errorf("%v", err) + return err + } + return nil } diff --git a/controllers/nginx/pkg/template/template.go b/controllers/nginx/pkg/template/template.go index eca21b54a..111a7626c 100644 --- a/controllers/nginx/pkg/template/template.go +++ b/controllers/nginx/pkg/template/template.go @@ -22,7 +22,6 @@ import ( "encoding/json" "fmt" "net" - "net/url" "os" "os/exec" "strconv" @@ -151,7 +150,6 @@ var ( "serverConfig": func(all config.TemplateConfig, server *ingress.Server) interface{} { return struct{ First, Second interface{} }{all, server} }, - "buildAuthSignURL": buildAuthSignURL, "isValidClientBodyBufferSize": isValidClientBodyBufferSize, "buildForwardedFor": buildForwardedFor, "trustHTTPHeaders": trustHTTPHeaders, @@ -570,22 +568,6 @@ func buildNextUpstream(input interface{}) string { return strings.Join(nextUpstreamCodes, " ") } -func buildAuthSignURL(input interface{}) string { - s, ok := input.(string) - if !ok { - glog.Errorf("expected an 'string' type but %T was returned", input) - return "" - } - - u, _ := url.Parse(s) - q := u.Query() - if len(q) == 0 { - return fmt.Sprintf("%v?rd=$request_uri", s) - } - - return fmt.Sprintf("%v&rd=$request_uri", s) -} - // buildRandomUUID return a random string to be used in the template func buildRandomUUID() string { s := uuid.New() diff --git a/controllers/nginx/pkg/template/template_test.go b/controllers/nginx/pkg/template/template_test.go index 62d0342f4..9bed3d3ac 100644 --- a/controllers/nginx/pkg/template/template_test.go +++ b/controllers/nginx/pkg/template/template_test.go @@ -310,24 +310,6 @@ func TestBuildResolvers(t *testing.T) { } } -func TestBuildAuthSignURL(t *testing.T) { - urlOne := "http://google.com" - validUrlOne := "http://google.com?rd=$request_uri" - - urlTwo := "http://google.com?cat" - validUrlTwo := "http://google.com?cat&rd=$request_uri" - - authSignURLOne := buildAuthSignURL(urlOne) - if authSignURLOne != validUrlOne { - t.Errorf("Expected '%v' but returned '%v'", validUrlOne, authSignURLOne) - } - - authSignURLTwo := buildAuthSignURL(urlTwo) - if authSignURLTwo != validUrlTwo { - t.Errorf("Expected '%v' but returned '%v'", validUrlTwo, authSignURLTwo) - } -} - func TestBuildNextUpstream(t *testing.T) { nextUpstream := "timeout http_500 http_502 non_idempotent" validNextUpstream := "timeout http_500 http_502" diff --git a/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl b/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl index f15accd6c..5f15abf01 100644 --- a/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl +++ b/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl @@ -461,19 +461,22 @@ stream { } server { {{ range $address := $all.Cfg.BindAddressIpv4 }} - listen {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.UseProxyProtocol }} proxy_protocol{{ end }}; + listen {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }}; {{ else }} - listen {{ $tcpServer.Port }}{{ if $tcpServer.Backend.UseProxyProtocol }} proxy_protocol{{ end }}; + listen {{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }}; {{ end }} {{ if $IsIPV6Enabled }} {{ range $address := $all.Cfg.BindAddressIpv6 }} - listen {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.UseProxyProtocol }} proxy_protocol{{ end }}; + listen {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }}; {{ else }} - listen [::]:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.UseProxyProtocol }} proxy_protocol{{ end }}; + listen [::]:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }}; {{ end }} {{ end }} proxy_timeout {{ $cfg.ProxyStreamTimeout }}; proxy_pass tcp-{{ $tcpServer.Port }}-{{ $tcpServer.Backend.Namespace }}-{{ $tcpServer.Backend.Name }}-{{ $tcpServer.Backend.Port }}; + {{ if $tcpServer.Backend.ProxyProtocol.Encode }} + proxy_protocol on; + {{ end }} } {{ end }} @@ -514,6 +517,8 @@ stream { location @custom_{{ $errCode }} { internal; + proxy_intercept_errors off; + proxy_set_header X-Code {{ $errCode }}; proxy_set_header X-Format $http_accept; proxy_set_header X-Original-URI $request_uri; @@ -521,6 +526,7 @@ stream { proxy_set_header X-Ingress-Name $ingress_name; proxy_set_header X-Service-Name $service_name; + rewrite (.*) / break; proxy_pass http://upstream-default-backend; } {{ end }} @@ -626,6 +632,10 @@ stream { {{ end }} {{ end }} + {{ if not (empty $server.ServerSnippet) }} + {{ $server.ServerSnippet }} + {{ end }} + {{ range $location := $server.Locations }} {{ $path := buildLocation $location }} {{ $authPath := buildAuthLocation $location }} @@ -704,7 +714,7 @@ stream { {{ end }} {{ if not (empty $location.ExternalAuth.SigninURL) }} - error_page 401 = {{ buildAuthSignURL $location.ExternalAuth.SigninURL }}; + error_page 401 = $location.ExternalAuth.SigninURL; {{ end }} {{/* if the location contains a rate limit annotation, create one */}} @@ -763,6 +773,9 @@ stream { proxy_set_header X-Original-URI $request_uri; proxy_set_header X-Scheme $pass_access_scheme; + {{/* This header is used for external authentication */}} + proxy_set_header X-Auth-Request-Redirect $request_uri; + # mitigate HTTPoxy Vulnerability # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/ proxy_set_header Proxy ""; diff --git a/core/pkg/ingress/annotations/serversnippet/main.go b/core/pkg/ingress/annotations/serversnippet/main.go new file mode 100644 index 000000000..0062bfdd5 --- /dev/null +++ b/core/pkg/ingress/annotations/serversnippet/main.go @@ -0,0 +1,42 @@ +/* +Copyright 2016 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package serversnippet + +import ( + extensions "k8s.io/api/extensions/v1beta1" + + "k8s.io/ingress/core/pkg/ingress/annotations/parser" +) + +const ( + annotation = "ingress.kubernetes.io/server-snippet" +) + +type serverSnippet struct { +} + +// NewParser creates a new server snippet annotation parser +func NewParser() parser.IngressAnnotation { + return serverSnippet{} +} + +// Parse parses the annotations contained in the ingress rule +// used to indicate if the location/s contains a fragment of +// configuration to be included inside the paths of the rules +func (a serverSnippet) Parse(ing *extensions.Ingress) (interface{}, error) { + return parser.GetStringAnnotation(annotation, ing) +} diff --git a/core/pkg/ingress/annotations/serversnippet/main_test.go b/core/pkg/ingress/annotations/serversnippet/main_test.go new file mode 100644 index 000000000..1a4d51bed --- /dev/null +++ b/core/pkg/ingress/annotations/serversnippet/main_test.go @@ -0,0 +1,58 @@ +/* +Copyright 2017 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package serversnippet + +import ( + "testing" + + api "k8s.io/api/core/v1" + extensions "k8s.io/api/extensions/v1beta1" + meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +func TestParse(t *testing.T) { + ap := NewParser() + if ap == nil { + t.Fatalf("expected a parser.IngressAnnotation but returned nil") + } + + testCases := []struct { + annotations map[string]string + expected string + }{ + {map[string]string{annotation: "more_headers"}, "more_headers"}, + {map[string]string{annotation: "false"}, "false"}, + {map[string]string{}, ""}, + {nil, ""}, + } + + ing := &extensions.Ingress{ + ObjectMeta: meta_v1.ObjectMeta{ + Name: "foo", + Namespace: api.NamespaceDefault, + }, + Spec: extensions.IngressSpec{}, + } + + for _, testCase := range testCases { + ing.SetAnnotations(testCase.annotations) + result, _ := ap.Parse(ing) + if result != testCase.expected { + t.Errorf("expected %v but returned %v, annotations: %s", testCase.expected, result, testCase.annotations) + } + } +} diff --git a/core/pkg/ingress/controller/annotations.go b/core/pkg/ingress/controller/annotations.go index 4ae3ea163..40f623bcf 100644 --- a/core/pkg/ingress/controller/annotations.go +++ b/core/pkg/ingress/controller/annotations.go @@ -36,6 +36,7 @@ import ( "k8s.io/ingress/core/pkg/ingress/annotations/redirect" "k8s.io/ingress/core/pkg/ingress/annotations/rewrite" "k8s.io/ingress/core/pkg/ingress/annotations/secureupstream" + "k8s.io/ingress/core/pkg/ingress/annotations/serversnippet" "k8s.io/ingress/core/pkg/ingress/annotations/serviceupstream" "k8s.io/ingress/core/pkg/ingress/annotations/sessionaffinity" "k8s.io/ingress/core/pkg/ingress/annotations/snippet" @@ -83,6 +84,7 @@ func newAnnotationExtractor(cfg extractorConfig) annotationExtractor { "DefaultBackend": defaultbackend.NewParser(cfg), "UpstreamVhost": upstreamvhost.NewParser(), "VtsFilterKey": vtsfilterkey.NewParser(), + "ServerSnippet": serversnippet.NewParser(), }, } } @@ -128,6 +130,7 @@ const ( serverAlias = "Alias" clientBodyBufferSize = "ClientBodyBufferSize" certificateAuth = "CertificateAuth" + serverSnippet = "ServerSnippet" ) func (e *annotationExtractor) ServiceUpstream(ing *extensions.Ingress) bool { @@ -181,3 +184,8 @@ func (e *annotationExtractor) CertificateAuth(ing *extensions.Ingress) *authtls. secure := val.(*authtls.AuthSSLConfig) return secure } + +func (e *annotationExtractor) ServerSnippet(ing *extensions.Ingress) string { + val, _ := e.annotations[serverSnippet].Parse(ing) + return val.(string) +} diff --git a/core/pkg/ingress/controller/backend_ssl.go b/core/pkg/ingress/controller/backend_ssl.go index 98005f340..5791bcfa1 100644 --- a/core/pkg/ingress/controller/backend_ssl.go +++ b/core/pkg/ingress/controller/backend_ssl.go @@ -74,35 +74,47 @@ func (ic *GenericController) getPemCertificate(secretName string) (*ingress.SSLC cert, okcert := secret.Data[apiv1.TLSCertKey] key, okkey := secret.Data[apiv1.TLSPrivateKeyKey] - ca := secret.Data["ca.crt"] + // namespace/secretName -> namespace-secretName nsSecName := strings.Replace(secretName, "/", "-", -1) var s *ingress.SSLCert if okcert && okkey { - if cert == nil || key == nil { - return nil, fmt.Errorf("error retrieving cert or key from secret %v: %v", secretName, err) + if cert == nil { + return nil, fmt.Errorf("secret %v has no 'tls.crt'", secretName) } + if key == nil { + return nil, fmt.Errorf("secret %v has no 'tls.key'", secretName) + } + + // If 'ca.crt' is also present, it will allow this secret to be used in the + // 'ingress.kubernetes.io/auth-tls-secret' annotation s, err = ssl.AddOrUpdateCertAndKey(nsSecName, cert, key, ca) if err != nil { - return nil, fmt.Errorf("unexpected error creating pem file %v", err) + return nil, fmt.Errorf("unexpected error creating pem file: %v", err) } - glog.V(3).Infof("found certificate and private key, configuring %v as a TLS Secret (CN: %v)", secretName, s.CN) + + glog.V(3).Infof("found 'tls.crt' and 'tls.key', configuring %v as a TLS Secret (CN: %v)", secretName, s.CN) + if ca != nil { + glog.V(3).Infof("found 'ca.crt', secret %v can also be used for Certificate Authentication", secretName) + } + } else if ca != nil { - glog.V(3).Infof("found only ca.crt, configuring %v as an Certificate Authentication secret", secretName) s, err = ssl.AddCertAuth(nsSecName, ca) + if err != nil { - return nil, fmt.Errorf("unexpected error creating pem file %v", err) + return nil, fmt.Errorf("unexpected error creating pem file: %v", err) } + + // makes this secret in 'syncSecret' to be used for Certificate Authentication + // this does not enable Certificate Authentication + glog.V(3).Infof("found only 'ca.crt', configuring %v as an Certificate Authentication Secret", secretName) + } else { return nil, fmt.Errorf("no keypair or CA cert could be found in %v", secretName) } - if err != nil { - return nil, err - } - s.Name = secret.Name s.Namespace = secret.Namespace return s, nil diff --git a/core/pkg/ingress/controller/controller.go b/core/pkg/ingress/controller/controller.go index f11192785..37df71a03 100644 --- a/core/pkg/ingress/controller/controller.go +++ b/core/pkg/ingress/controller/controller.go @@ -238,17 +238,19 @@ func (ic GenericController) GetService(name string) (*apiv1.Service, error) { // sync collects all the pieces required to assemble the configuration file and // then sends the content to the backend (OnUpdate) receiving the populated // template as response reloading the backend if is required. -func (ic *GenericController) syncIngress(key interface{}) error { +func (ic *GenericController) syncIngress(item interface{}) error { ic.syncRateLimiter.Accept() if ic.syncQueue.IsShuttingDown() { return nil } - if name, ok := key.(string); ok { - if obj, exists, _ := ic.listers.Ingress.GetByKey(name); exists { - ing := obj.(*extensions.Ingress) - ic.readSecrets(ing) + if element, ok := item.(task.Element); ok { + if name, ok := element.Key.(string); ok { + if obj, exists, _ := ic.listers.Ingress.GetByKey(name); exists { + ing := obj.(*extensions.Ingress) + ic.readSecrets(ing) + } } } @@ -346,6 +348,7 @@ func (ic *GenericController) getStreamServices(configmapName string, proto apiv1 } var svcs []ingress.L4Service + var svcProxyProtocol ingress.ProxyProtocol // k -> port to expose // v -> /: for k, v := range configmap.Data { @@ -363,18 +366,22 @@ func (ic *GenericController) getStreamServices(configmapName string, proto apiv1 nsSvcPort := strings.Split(v, ":") if len(nsSvcPort) < 2 { - glog.Warningf("invalid format (namespace/name:port:[PROXY]) '%v'", k) + glog.Warningf("invalid format (namespace/name:port:[PROXY]:[PROXY]) '%v'", k) continue } nsName := nsSvcPort[0] svcPort := nsSvcPort[1] - useProxyProtocol := false + svcProxyProtocol.Decode = false + svcProxyProtocol.Encode = false // Proxy protocol is possible if the service is TCP - if len(nsSvcPort) == 3 && proto == apiv1.ProtocolTCP { - if strings.ToUpper(nsSvcPort[2]) == "PROXY" { - useProxyProtocol = true + if len(nsSvcPort) >= 3 && proto == apiv1.ProtocolTCP { + if len(nsSvcPort) >= 3 && strings.ToUpper(nsSvcPort[2]) == "PROXY" { + svcProxyProtocol.Decode = true + } + if len(nsSvcPort) == 4 && strings.ToUpper(nsSvcPort[3]) == "PROXY" { + svcProxyProtocol.Encode = true } } @@ -432,11 +439,11 @@ func (ic *GenericController) getStreamServices(configmapName string, proto apiv1 svcs = append(svcs, ingress.L4Service{ Port: externalPort, Backend: ingress.L4Backend{ - Name: svcName, - Namespace: svcNs, - Port: intstr.FromString(svcPort), - Protocol: proto, - UseProxyProtocol: useProxyProtocol, + Name: svcName, + Namespace: svcNs, + Port: intstr.FromString(svcPort), + Protocol: proto, + ProxyProtocol: svcProxyProtocol, }, Endpoints: endps, }) @@ -509,9 +516,13 @@ func (ic *GenericController) getBackendServers(ingresses []*extensions.Ingress) ca := ic.annotations.CertificateAuth(ing) if ca != nil { server.CertificateAuth = *ca + // It is possible that no CAFileName is found in the secret + if server.CertificateAuth.CAFileName == "" { + glog.V(3).Infof("secret %v does not contain 'ca.crt', mutual authentication not enabled - ingress rule %v/%v.", server.CertificateAuth.Secret, ing.Namespace, ing.Name) + } } } else { - glog.V(3).Infof("server %v already contains a muthual autentication configuration - ingress rule %v/%v", server.Hostname, ing.Namespace, ing.Name) + glog.V(3).Infof("server %v already contains a mutual authentication configuration - ingress rule %v/%v", server.Hostname, ing.Namespace, ing.Name) } for _, path := range rule.HTTP.Paths { @@ -671,7 +682,8 @@ func (ic *GenericController) getBackendServers(ingresses []*extensions.Ingress) return aUpstreams, aServers } -// GetAuthCertificate ... + +// GetAuthCertificate is used by the auth-tls annotations to get a cert from a secret func (ic GenericController) GetAuthCertificate(secretName string) (*resolver.AuthSSLCert, error) { if _, exists := ic.sslCertTracker.Get(secretName); !exists { ic.syncSecret(secretName) @@ -894,10 +906,12 @@ func (ic *GenericController) createServers(data []*extensions.Ingress, RequestBuffering: bdef.ProxyRequestBuffering, } + // generated on Start() with createDefaultSSLCertificate() defaultPemFileName := fakeCertificatePath defaultPemSHA := fakeCertificateSHA - // Tries to fetch the default Certificate. If it does not exists, generate a new self signed one. + // Tries to fetch the default Certificate from nginx configuration. + // If it does not exists, use the ones generated on Start() defaultCertificate, err := ic.getPemCertificate(ic.cfg.DefaultSSLCertificate) if err == nil { defaultPemFileName = defaultCertificate.PemFileName @@ -976,6 +990,7 @@ func (ic *GenericController) createServers(data []*extensions.Ingress, for _, ing := range data { // setup server-alias based on annotations aliasAnnotation := ic.annotations.Alias(ing) + srvsnippet := ic.annotations.ServerSnippet(ing) for _, rule := range ing.Spec.Rules { host := rule.Host @@ -991,6 +1006,17 @@ func (ic *GenericController) createServers(data []*extensions.Ingress, } } + //notifying the user that it has already been configured. + if servers[host].ServerSnippet != "" && srvsnippet != "" { + glog.Warningf("ingress %v/%v for host %v contains a Server Snippet section that it has already been configured.", + ing.Namespace, ing.Name, host) + } + + // only add a server snippet if the server does not have one previously configured + if servers[host].ServerSnippet == "" && srvsnippet != "" { + servers[host].ServerSnippet = srvsnippet + } + // only add a certificate if the server does not have one previously configured if servers[host].SSLCertificate != "" { continue @@ -1054,6 +1080,7 @@ func (ic *GenericController) createServers(data []*extensions.Ingress, servers[host].Alias = "" } } + return servers } diff --git a/core/pkg/ingress/resolver/main.go b/core/pkg/ingress/resolver/main.go index 9bb11db40..d4672aa76 100644 --- a/core/pkg/ingress/resolver/main.go +++ b/core/pkg/ingress/resolver/main.go @@ -54,7 +54,7 @@ type AuthSSLCert struct { Secret string `json:"secret"` // CAFileName contains the path to the secrets 'ca.crt' CAFileName string `json:"caFilename"` - // PemSHA contains the SHA1 hash of the 'tls.crt' value + // PemSHA contains the SHA1 hash of the 'ca.crt' or combinations of (tls.crt, tls.key, tls.crt) depending on certs in secret PemSHA string `json:"pemSha"` } diff --git a/core/pkg/ingress/types.go b/core/pkg/ingress/types.go index a24ed5c5b..e3a7ab34d 100644 --- a/core/pkg/ingress/types.go +++ b/core/pkg/ingress/types.go @@ -236,6 +236,10 @@ type Server struct { // CertificateAuth indicates the this server requires mutual authentication // +optional CertificateAuth authtls.AuthSSLConfig `json:"certificateAuth"` + + // ServerSnippet returns the snippet of server + // +optional + ServerSnippet string `json:"serverSnippet"` } // Location describes an URI inside a server. @@ -359,5 +363,11 @@ type L4Backend struct { Namespace string `json:"namespace"` Protocol apiv1.Protocol `json:"protocol"` // +optional - UseProxyProtocol bool `json:"useProxyProtocol"` + ProxyProtocol ProxyProtocol `json:"proxyProtocol"` +} + +// ProxyProtocol describes the proxy protocol configuration +type ProxyProtocol struct { + Decode bool `json:"decode"` + Encode bool `json:"encode"` } diff --git a/core/pkg/net/ssl/ssl.go b/core/pkg/net/ssl/ssl.go index e1cd5b0ff..535c5f5fe 100644 --- a/core/pkg/net/ssl/ssl.go +++ b/core/pkg/net/ssl/ssl.go @@ -271,7 +271,7 @@ func AddCertAuth(name string, ca []byte) (*ingress.SSLCert, error) { return nil, fmt.Errorf("could not write CA file %v: %v", caFileName, err) } - glog.V(3).Infof("Created CA Certificate for authentication: %v", caFileName) + glog.V(3).Infof("Created CA Certificate for Authentication: %v", caFileName) return &ingress.SSLCert{ CAFileName: caFileName, PemFileName: caFileName, diff --git a/core/pkg/task/queue.go b/core/pkg/task/queue.go index 977fdc2f6..9b05f1653 100644 --- a/core/pkg/task/queue.go +++ b/core/pkg/task/queue.go @@ -48,7 +48,8 @@ type Queue struct { lastSync int64 } -type element struct { +// Element represents one item of the queue +type Element struct { Key interface{} Timestamp int64 } @@ -72,7 +73,7 @@ func (t *Queue) Enqueue(obj interface{}) { glog.Errorf("%v", err) return } - t.queue.Add(element{ + t.queue.Add(Element{ Key: key, Timestamp: ts, }) @@ -99,7 +100,7 @@ func (t *Queue) worker() { } ts := time.Now().UnixNano() - item := key.(element) + item := key.(Element) if t.lastSync > item.Timestamp { glog.V(3).Infof("skipping %v sync (%v > %v)", item.Key, t.lastSync, item.Timestamp) t.queue.Forget(key) @@ -110,7 +111,7 @@ func (t *Queue) worker() { glog.V(3).Infof("syncing %v", item.Key) if err := t.sync(key); err != nil { glog.Warningf("requeuing %v, err %v", item.Key, err) - t.queue.AddRateLimited(element{ + t.queue.AddRateLimited(Element{ Key: item.Key, Timestamp: time.Now().UnixNano(), }) diff --git a/examples/PREREQUISITES.md b/examples/PREREQUISITES.md index 03f9f74a4..7c6bf2fd6 100644 --- a/examples/PREREQUISITES.md +++ b/examples/PREREQUISITES.md @@ -132,7 +132,12 @@ The final step is to create a secret with the content of this file. This secret the TLS Auth directive: ```console -$ kubectl create secret generic caingress --namespace=default --from-file=ca.crt +$ kubectl create secret generic caingress --namespace=default --from-file=ca.crt= +``` + +Note: You can also generate the CA Authentication Secret along with the TLS Secret by using: +```console +$ kubectl create secret generic caingress --namespace=default --from-file=ca.crt= --from-file=tls.crt= --from-file=tls.key= ``` ## Test HTTP Service diff --git a/examples/auth/client-certs/nginx/README.md b/examples/auth/client-certs/nginx/README.md index e8c9e83ac..d3da9d1a6 100644 --- a/examples/auth/client-certs/nginx/README.md +++ b/examples/auth/client-certs/nginx/README.md @@ -16,13 +16,12 @@ the child, except for the root, which has Issuer == Subject. * Client Cert: Certificate used by the clients to authenticate themselves with the loadbalancer/backends. - ## Prerequisites You need a valid CA File, composed of a group of valid enabled CAs. This MUST be in PEM Format. -The instructions are described [here](../../../PREREQUISITES.md#ca-authentication) +The instructions are described [here](../../../PREREQUISITES.md) -Also your ingress must be configured as a HTTPs/TLS Ingress. +Also your ingress must be configured as a HTTPS/TLS Ingress. ## Deployment @@ -51,8 +50,7 @@ Name: nginx-test Namespace: default Address: 104.198.183.6 Default backend: default-http-backend:80 (10.180.0.4:8080,10.240.0.2:8080) -TLS: - tls-secret terminates ingress.test.com +TLS: tls-secret terminates ingress.test.com Rules: Host Path Backends ---- ---- -------- @@ -79,13 +77,12 @@ Server: nginx/1.11.9 $ curl -I -k --key ~/user.key --cert ~/user.cer https://ingress.test.com HTTP/1.1 200 OK Server: nginx/1.11.9 - ``` You must use the full DNS name while testing, as NGINX relies on the Server Name (SNI) to select the correct Ingress to be used. - The curl version used here was ``curl 7.47.0`` ## Which certificate was used for authentication? -In your backend application you might want to know which certificate was used for authentication. For this purpose, we pass the full certificate in PEM format to the backend in the `ssl-client-cert` header. +In your backend application you might want to know which certificate was used for authentication. +For this purpose, we pass the full certificate in PEM format to the backend in the `ssl-client-cert` header. diff --git a/examples/auth/client-certs/nginx/nginx-tls-auth.yaml b/examples/auth/client-certs/nginx/nginx-tls-auth.yaml index ac03d9d7c..c3ef92a8e 100644 --- a/examples/auth/client-certs/nginx/nginx-tls-auth.yaml +++ b/examples/auth/client-certs/nginx/nginx-tls-auth.yaml @@ -21,6 +21,5 @@ spec: tls: - hosts: - ingress.test.com - # Create this cert as described in 'multi-tls' example - secretName: cert + secretName: tls-secret diff --git a/examples/customization/custom-errors/nginx/README.md b/examples/customization/custom-errors/nginx/README.md index 2f79388d5..c37240afb 100644 --- a/examples/customization/custom-errors/nginx/README.md +++ b/examples/customization/custom-errors/nginx/README.md @@ -1,4 +1,4 @@ -This example shows how is possible to use a custom backend to render custom error pages. The code of this example is located here [nginx-debug-server](https://github.com/aledbf/contrib/tree/nginx-debug-server) +This example shows how is possible to use a custom backend to render custom error pages. The code of this example is located here [custom-error-pages](https://github.com/kubernetes/ingress/tree/master/examples/customization/custom-errors/nginx) The idea is to use the headers `X-Code` and `X-Format` that NGINX pass to the backend in case of an error to find out the best existent representation of the response to be returned. i.e. if the request contains an `Accept` header of type `json` the error should be in that format and not in `html` (the default in NGINX). @@ -78,5 +78,3 @@ $ curl -v http://172.17.4.99/ -H 'Accept: application/json' * Connection #0 to host 172.17.4.99 left intact ``` - -By default the Ingress controller provides support for `html`, `json` and `XML`. diff --git a/images/custom-error-pages/Makefile b/images/custom-error-pages/Makefile new file mode 100644 index 000000000..96b3d3479 --- /dev/null +++ b/images/custom-error-pages/Makefile @@ -0,0 +1,110 @@ +all: push + +BUILDTAGS= + +# Use the 0.0 tag for testing, it shouldn't clobber any release builds +TAG?=0.1 +REGISTRY?=aledbf +GOOS?=linux +DOCKER?=gcloud docker -- +SED_I?=sed -i +GOHOSTOS ?= $(shell go env GOHOSTOS) + +PKG=k8s.io/ingress/images/custom-error-pages + +ifeq ($(GOHOSTOS),darwin) + SED_I=sed -i '' +endif + +REPO_INFO=$(shell git config --get remote.origin.url) + +ifndef COMMIT + COMMIT := git-$(shell git rev-parse --short HEAD) +endif + +ARCH ?= $(shell go env GOARCH) +GOARCH = ${ARCH} +DUMB_ARCH = ${ARCH} + +BASEIMAGE?=alpine:3.6 + +ALL_ARCH = amd64 arm arm64 ppc64le + +QEMUVERSION=v2.9.1 + +IMGNAME = custom-error-pages +IMAGE = $(REGISTRY)/$(IMGNAME) +MULTI_ARCH_IMG = $(IMAGE)-$(ARCH) + +ifeq ($(ARCH),arm) + QEMUARCH=arm + GOARCH=arm +endif +ifeq ($(ARCH),arm64) + QEMUARCH=aarch64 +endif +ifeq ($(ARCH),ppc64le) + QEMUARCH=ppc64le + GOARCH=ppc64le +endif +#ifeq ($(ARCH),s390x) +# QEMUARCH=s390x +#endif + +TEMP_DIR := $(shell mktemp -d) + +DOCKERFILE := $(TEMP_DIR)/rootfs/Dockerfile + +all: all-container + +sub-container-%: + $(MAKE) ARCH=$* build container + +sub-push-%: + $(MAKE) ARCH=$* push + +all-container: $(addprefix sub-container-,$(ALL_ARCH)) + +all-push: $(addprefix sub-push-,$(ALL_ARCH)) + +container: .container-$(ARCH) +.container-$(ARCH): + cp -r ./* $(TEMP_DIR) + $(SED_I) 's|BASEIMAGE|$(BASEIMAGE)|g' $(DOCKERFILE) + $(SED_I) "s|QEMUARCH|$(QEMUARCH)|g" $(DOCKERFILE) + +ifeq ($(ARCH),amd64) + # When building "normally" for amd64, remove the whole line, it has no part in the amd64 image + $(SED_I) "/CROSS_BUILD_/d" $(DOCKERFILE) +else + # When cross-building, only the placeholder "CROSS_BUILD_" should be removed + # Register /usr/bin/qemu-ARCH-static as the handler for ARM binaries in the kernel + $(DOCKER) run --rm --privileged multiarch/qemu-user-static:register --reset + curl -sSL https://github.com/multiarch/qemu-user-static/releases/download/$(QEMUVERSION)/x86_64_qemu-$(QEMUARCH)-static.tar.gz | tar -xz -C $(TEMP_DIR)/rootfs + $(SED_I) "s/CROSS_BUILD_//g" $(DOCKERFILE) +endif + + $(DOCKER) build -t $(MULTI_ARCH_IMG):$(TAG) $(TEMP_DIR)/rootfs + +ifeq ($(ARCH), amd64) + # This is for to maintain the backward compatibility + $(DOCKER) tag $(MULTI_ARCH_IMG):$(TAG) $(IMAGE):$(TAG) +endif + +push: .push-$(ARCH) +.push-$(ARCH): + $(DOCKER) push $(MULTI_ARCH_IMG):$(TAG) +ifeq ($(ARCH), amd64) + $(DOCKER) push $(IMAGE):$(TAG) +endif + +clean: + $(DOCKER) rmi -f $(MULTI_ARCH_IMG):$(TAG) || true + +build: clean + CGO_ENABLED=0 GOOS=${GOOS} GOARCH=${GOARCH} go build -a -installsuffix cgo \ + -ldflags "-s -w" \ + -o ${TEMP_DIR}/rootfs/custom-error-pages ${PKG}/... + +release: all-container all-push + echo "done" diff --git a/images/custom-error-pages/README.md b/images/custom-error-pages/README.md new file mode 100644 index 000000000..3ee67e5d9 --- /dev/null +++ b/images/custom-error-pages/README.md @@ -0,0 +1,2 @@ + +Example of Custom error pages for the NGINX Ingress controller diff --git a/images/custom-error-pages/main.go b/images/custom-error-pages/main.go new file mode 100644 index 000000000..88b709b6c --- /dev/null +++ b/images/custom-error-pages/main.go @@ -0,0 +1,94 @@ +/* +Copyright 2017 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "fmt" + "io" + "log" + "mime" + "net/http" + "os" + "strconv" +) + +const ( + FormatHeader = "X-Format" + + CodeHeader = "X-Code" + + ContentType = "Content-Type" +) + +func main() { + path := "/www" + if os.Getenv("PATH") != "" { + path = os.Getenv("PATH") + } + http.HandleFunc("/", errorHandler(path)) + http.ListenAndServe(fmt.Sprintf(":8080"), nil) +} + +func errorHandler(path string) func(http.ResponseWriter, *http.Request) { + return func(w http.ResponseWriter, r *http.Request) { + ext := "html" + + format := r.Header.Get(FormatHeader) + if format == "" { + format = "text/html" + log.Printf("forma not specified. Using %v\n", format) + } + + mediaType, _, _ := mime.ParseMediaType(format) + cext, err := mime.ExtensionsByType(mediaType) + if err != nil { + log.Printf("unexpected error reading media type extension: %v. Using %v\n", err, ext) + } else { + ext = cext[0] + } + w.Header().Set(ContentType, format) + + errCode := r.Header.Get(CodeHeader) + code, err := strconv.Atoi(errCode) + if err != nil { + code = 404 + log.Printf("unexpected error reading return code: %v. Using %v\n", err, code) + } + w.WriteHeader(code) + + file := fmt.Sprintf("%v/%v%v", path, code, ext) + f, err := os.Open(file) + if err != nil { + log.Printf("unexpected error opening file: %v\n", err) + scode := strconv.Itoa(code) + file := fmt.Sprintf("%v/%cxx%v", path, scode[0], ext) + f, err := os.Open(file) + if err != nil { + log.Printf("unexpected error opening file: %v\n", err) + http.NotFound(w, r) + return + } + defer f.Close() + log.Printf("serving custom error response for code %v and format %v from file %v\n", code, format, file) + io.Copy(w, f) + return + } + defer f.Close() + log.Printf("serving custom error response for code %v and format %v from file %v\n", code, format, file) + io.Copy(w, f) + } +} diff --git a/images/custom-error-pages/rootfs/Dockerfile b/images/custom-error-pages/rootfs/Dockerfile new file mode 100755 index 000000000..aaf60efb1 --- /dev/null +++ b/images/custom-error-pages/rootfs/Dockerfile @@ -0,0 +1,21 @@ +# Copyright 2017 The Kubernetes Authors. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +FROM BASEIMAGE + +CROSS_BUILD_COPY qemu-QEMUARCH-static /usr/bin/ + +COPY . / + +CMD ["/custom-error-pages"] diff --git a/images/custom-error-pages/rootfs/www/404.html b/images/custom-error-pages/rootfs/www/404.html new file mode 100644 index 000000000..0f00f93f5 --- /dev/null +++ b/images/custom-error-pages/rootfs/www/404.html @@ -0,0 +1 @@ +The page you're looking for could not be found. \ No newline at end of file diff --git a/images/custom-error-pages/rootfs/www/404.json b/images/custom-error-pages/rootfs/www/404.json new file mode 100644 index 000000000..36f3e753e --- /dev/null +++ b/images/custom-error-pages/rootfs/www/404.json @@ -0,0 +1 @@ +{ "message": "The page you're looking for could not be found" } \ No newline at end of file diff --git a/images/custom-error-pages/rootfs/www/4xx.html b/images/custom-error-pages/rootfs/www/4xx.html new file mode 100644 index 000000000..05ba01769 --- /dev/null +++ b/images/custom-error-pages/rootfs/www/4xx.html @@ -0,0 +1 @@ +4xx html \ No newline at end of file diff --git a/images/custom-error-pages/rootfs/www/4xx.json b/images/custom-error-pages/rootfs/www/4xx.json new file mode 100644 index 000000000..48246d04e --- /dev/null +++ b/images/custom-error-pages/rootfs/www/4xx.json @@ -0,0 +1 @@ +4xx json \ No newline at end of file diff --git a/images/custom-error-pages/rootfs/www/500.html b/images/custom-error-pages/rootfs/www/500.html new file mode 100644 index 000000000..68d7a7b52 --- /dev/null +++ b/images/custom-error-pages/rootfs/www/500.html @@ -0,0 +1 @@ +500 html \ No newline at end of file diff --git a/images/custom-error-pages/rootfs/www/500.json b/images/custom-error-pages/rootfs/www/500.json new file mode 100644 index 000000000..9cad530c1 --- /dev/null +++ b/images/custom-error-pages/rootfs/www/500.json @@ -0,0 +1 @@ +500 json \ No newline at end of file diff --git a/images/custom-error-pages/rootfs/www/5xx.html b/images/custom-error-pages/rootfs/www/5xx.html new file mode 100644 index 000000000..1bcb87780 --- /dev/null +++ b/images/custom-error-pages/rootfs/www/5xx.html @@ -0,0 +1 @@ +5xx html \ No newline at end of file diff --git a/images/custom-error-pages/rootfs/www/5xx.json b/images/custom-error-pages/rootfs/www/5xx.json new file mode 100644 index 000000000..4703b68c6 --- /dev/null +++ b/images/custom-error-pages/rootfs/www/5xx.json @@ -0,0 +1 @@ +5xx json \ No newline at end of file diff --git a/images/nginx-slim/Makefile b/images/nginx-slim/Makefile index 0bcce6696..0a5430822 100644 --- a/images/nginx-slim/Makefile +++ b/images/nginx-slim/Makefile @@ -13,8 +13,8 @@ # limitations under the License. # 0.0.0 shouldn't clobber any released builds -TAG = 0.25 -REGISTRY = gcr.io/google_containers +TAG ?= 0.25 +REGISTRY ?= gcr.io/google_containers ARCH ?= $(shell go env GOARCH) ALL_ARCH = amd64 arm arm64 ppc64le