Merge remote-tracking branch 'master/master' into refactor-template-headers

controllers/nginx/README.md

@@ -17,6 +17,7 @@ This is an nginx Ingress controller that uses [ConfigMap](https://kubernetes.io/
 * [TCP Services](#exposing-tcp-services)
 * [UDP Services](#exposing-udp-services)
 * [Proxy Protocol](#proxy-protocol)
+* [Opentracing](#opentracing)
 * [NGINX customization](configuration.md)
 * [Custom errors](#custom-errors)
 * [NGINX status page](#nginx-status-page)
@@ -334,8 +335,8 @@ version to fully support Kube-Lego is nginx Ingress controller 0.8.
 
 ## Exposing TCP services
 
-Ingress does not support TCP services (yet). For this reason this Ingress controller uses the flag `--tcp-services-configmap` to point to an existing config map where the key is the external port to use and the value is `<namespace/service name>:<service port>:[PROXY]`
+Ingress does not support TCP services (yet). For this reason this Ingress controller uses the flag `--tcp-services-configmap` to point to an existing config map where the key is the external port to use and the value is `<namespace/service name>:<service port>:[PROXY]:[PROXY]`
-It is possible to use a number or the name of the port. The last field is optional. Adding `PROXY` in the last field we can enable Proxy Protocol in a TCP service.
+It is possible to use a number or the name of the port. The two last fields are optional. Adding `PROXY` in either or both of the two last fields we can use Proxy Protocol decoding (listen) and/or encoding (proxy_pass) in a TCP service (https://www.nginx.com/resources/admin-guide/proxy-protocol/).
 
 The next example shows how to expose the service `example-go` running in the namespace `default` in the port `8080` using the port `9000`
 ```
@@ -378,14 +379,63 @@ Amongst others [ELBs in AWS](http://docs.aws.amazon.com/ElasticLoadBalancing/lat
 
 Please check the [proxy-protocol](examples/proxy-protocol/) example
 
+### Opentracing
+
+Using the third party module [rnburn/nginx-opentracing](https://github.com/rnburn/nginx-opentracing) the NGINX ingress controller can configure NGINX to enable [OpenTracing](http://opentracing.io) instrumentation.
+By default this feature is disabled.
+
+To enable the instrumentation we just need to enable the instrumentation in the configuration configmap and set the host where we should send the traces.
+
+In the [aledbf/zipkin-js-example](https://github.com/aledbf/zipkin-js-example) github repository is possible to see a dockerized version of zipkin-js-example with the required Kubernetes descriptors.
+To install the example and the zipkin collector we just need to run:
+
+```
+$ kubectl create -f https://raw.githubusercontent.com/aledbf/zipkin-js-example/kubernetes/kubernetes/zipkin.yaml
+$ kubectl create -f https://raw.githubusercontent.com/aledbf/zipkin-js-example/kubernetes/kubernetes/deployment.yaml
+```
+
+Also we need to configure the NGINX controller configmap with the required values:
+
+```
+apiVersion: v1
+data:
+  enable-opentracing: "true"
+  zipkin-collector-host: zipkin.default.svc.cluster.local
+kind: ConfigMap
+metadata:
+  labels:
+    k8s-app: nginx-ingress-controller
+  name: nginx-custom-configuration
+```
+
+Using curl we can generate some traces:
+```
+$ curl -v http://$(minikube ip)/api -H 'Host: zipkin-js-example'
+$ curl -v http://$(minikube ip)/api -H 'Host: zipkin-js-example'
+```
+
+In the zipkin inteface we can see the details:
+
+![zipkin screenshot](docs/images/zipkin-demo.png "zipkin collector screenshot")
 
 ### Custom errors
 
-In case of an error in a request the body of the response is obtained from the `default backend`. Each request to the default backend includes two headers: 
+In case of an error in a request the body of the response is obtained from the `default backend`.
-- `X-Code` indicates the HTTP code
+Each request to the default backend includes two headers: 
-- `X-Format` the value of the `Accept` header
 
-Using this two headers is possible to use a custom backend service like [this one](https://github.com/aledbf/contrib/tree/nginx-debug-server/Ingress/images/nginx-error-server) that inspect each request and returns a custom error page with the format expected by the client. Please check the example [custom-errors](examples/custom-errors/README.md)
+- `X-Code` indicates the HTTP code to be returned to the client.
+- `X-Format` the value of the `Accept` header.
+
+**Important:** the custom backend must return the correct HTTP status code to be returned. NGINX do not changes the reponse from the custom default backend.
+
+Using this two headers is possible to use a custom backend service like [this one](https://github.com/kubernetes/ingress/tree/master/examples/customization/custom-errors/nginx) that inspect each request and returns a custom error page with the format expected by the client. Please check the example [custom-errors](examples/customization/custom-errors/nginx/README.md)
+
+NGINX sends aditional headers that can be used to build custom response:
+
+- X-Original-URI
+- X-Namespace
+- X-Ingress-Name
+- X-Service-Name
 
 ### NGINX status page
 

controllers/nginx/configuration.md

@@ -135,14 +135,14 @@ Please check the [auth](/examples/auth/basic/nginx/README.md) example.
 
 ### Certificate Authentication
 
-It's possible to enable Certificate based authentication using additional annotations in Ingress Rule.
+It's possible to enable Certificate-Based Authentication (Mutual Authentication) using additional annotations in Ingress Rule.
 
 The annotations are:
 ```
 ingress.kubernetes.io/auth-tls-secret: secretName
 ```
 
-The name of the secret that contains the full Certificate Authority chain that is enabled to authenticate against this ingress. It's composed of namespace/secretName
+The name of the secret that contains the full Certificate Authority chain `ca.crt` that is enabled to authenticate against this ingress. It's composed of namespace/secretName.
 
 ```
 ingress.kubernetes.io/auth-tls-verify-depth
@@ -487,6 +487,17 @@ The default mime type list to compress is: `application/atom+xml application/jav
 
 **bind-address:** Sets the addresses on which the server will accept requests instead of *. It should be noted that these addresses must exist in the runtime environment or the controller will crash loop.
 
+**enable-opentracing:** enables the nginx Opentracing extension https://github.com/rnburn/nginx-opentracing
+Default is "false"
+
+**zipkin-collector-host:** specifies the host to use when uploading traces. It must be a valid URL
+
+**zipkin-collector-port:** specifies the port to use when uploading traces
+Default: 9411
+
+**zipkin-service-name:** specifies the service name to use for any traces created
+Default: nginx
+
 ### Default configuration options
 
 The following table shows the options, the default value and a description.

controllers/nginx/pkg/cmd/controller/nginx.go

@@ -35,6 +35,7 @@ import (
 	"github.com/spf13/pflag"
 
 	proxyproto "github.com/armon/go-proxyproto"
+	"github.com/ncabatoff/process-exporter/proc"
 	apiv1 "k8s.io/api/core/v1"
 	extensions "k8s.io/api/extensions/v1beta1"
 
@@ -62,7 +63,7 @@ const (
 var (
 	tmplPath        = "/etc/nginx/template/nginx.tmpl"
 	cfgPath         = "/etc/nginx/nginx.conf"
-	binary          = "/usr/sbin/nginx"
+	nginxBinary     = "/usr/sbin/nginx"
 	defIngressClass = "nginx"
 )
 
@@ -72,7 +73,7 @@ var (
 func newNGINXController() *NGINXController {
 	ngx := os.Getenv("NGINX_BINARY")
 	if ngx == "" {
-		ngx = binary
+		ngx = nginxBinary
 	}
 
 	h, err := dns.GetSystemNameServers()
@@ -200,7 +201,26 @@ NGINX master process died (%v): %v
 				break
 			}
 			conn.Close()
-			time.Sleep(1 * time.Second)
+			// kill nginx worker processes
+			fs, err := proc.NewFS("/proc")
+			procs, _ := fs.FS.AllProcs()
+			for _, p := range procs {
+				pn, err := p.Comm()
+				if err != nil {
+					glog.Errorf("unexpected error obtaining process information: %v", err)
+					continue
+				}
+
+				if pn == "nginx" {
+					osp, err := os.FindProcess(p.PID)
+					if err != nil {
+						glog.Errorf("unexpected error obtaining process information: %v", err)
+						continue
+					}
+					osp.Signal(syscall.SIGQUIT)
+				}
+			}
+			time.Sleep(100 * time.Millisecond)
 		}
 		// restart a new nginx master process if the controller
 		// is not being stopped
@@ -710,6 +730,28 @@ func (n NGINXController) Check(_ *http.Request) error {
 	if res.StatusCode != 200 {
 		return fmt.Errorf("ingress controller is not healthy")
 	}
+
+	// check the nginx master process is running
+	fs, err := proc.NewFS("/proc")
+	if err != nil {
+		glog.Errorf("%v", err)
+		return err
+	}
+	f, err := ioutil.ReadFile("/run/nginx.pid")
+	if err != nil {
+		glog.Errorf("%v", err)
+		return err
+	}
+	pid, err := strconv.Atoi(strings.TrimRight(string(f), "\r\n"))
+	if err != nil {
+		return err
+	}
+	_, err = fs.NewProc(int(pid))
+	if err != nil {
+		glog.Errorf("%v", err)
+		return err
+	}
+
 	return nil
 }
 

controllers/nginx/pkg/template/template.go

@@ -22,7 +22,6 @@ import (
 	"encoding/json"
 	"fmt"
 	"net"
-	"net/url"
 	"os"
 	"os/exec"
 	"strconv"
@@ -151,7 +150,6 @@ var (
 		"serverConfig": func(all config.TemplateConfig, server *ingress.Server) interface{} {
 			return struct{ First, Second interface{} }{all, server}
 		},
-		"buildAuthSignURL":            buildAuthSignURL,
 		"isValidClientBodyBufferSize": isValidClientBodyBufferSize,
 		"buildForwardedFor":           buildForwardedFor,
 		"trustHTTPHeaders":            trustHTTPHeaders,
@@ -570,22 +568,6 @@ func buildNextUpstream(input interface{}) string {
 	return strings.Join(nextUpstreamCodes, " ")
 }
 
-func buildAuthSignURL(input interface{}) string {
-	s, ok := input.(string)
-	if !ok {
-		glog.Errorf("expected an 'string' type but %T was returned", input)
-		return ""
-	}
-
-	u, _ := url.Parse(s)
-	q := u.Query()
-	if len(q) == 0 {
-		return fmt.Sprintf("%v?rd=$request_uri", s)
-	}
-
-	return fmt.Sprintf("%v&rd=$request_uri", s)
-}
-
 // buildRandomUUID return a random string to be used in the template
 func buildRandomUUID() string {
 	s := uuid.New()

controllers/nginx/pkg/template/template_test.go

@@ -310,24 +310,6 @@ func TestBuildResolvers(t *testing.T) {
 	}
 }
 
-func TestBuildAuthSignURL(t *testing.T) {
-	urlOne := "http://google.com"
-	validUrlOne := "http://google.com?rd=$request_uri"
-
-	urlTwo := "http://google.com?cat"
-	validUrlTwo := "http://google.com?cat&rd=$request_uri"
-
-	authSignURLOne := buildAuthSignURL(urlOne)
-	if authSignURLOne != validUrlOne {
-		t.Errorf("Expected '%v' but returned '%v'", validUrlOne, authSignURLOne)
-	}
-
-	authSignURLTwo := buildAuthSignURL(urlTwo)
-	if authSignURLTwo != validUrlTwo {
-		t.Errorf("Expected '%v' but returned '%v'", validUrlTwo, authSignURLTwo)
-	}
-}
-
 func TestBuildNextUpstream(t *testing.T) {
 	nextUpstream := "timeout http_500 http_502 non_idempotent"
 	validNextUpstream := "timeout http_500 http_502"

controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl

@@ -461,19 +461,22 @@ stream {
     }
     server {
         {{ range $address := $all.Cfg.BindAddressIpv4 }}
-        listen                  {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.UseProxyProtocol }} proxy_protocol{{ end }};
+        listen                  {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }};
         {{ else }}
-        listen                  {{ $tcpServer.Port }}{{ if $tcpServer.Backend.UseProxyProtocol }} proxy_protocol{{ end }};
+        listen                  {{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }};
         {{ end }}
         {{ if $IsIPV6Enabled }}
         {{ range $address := $all.Cfg.BindAddressIpv6 }}
-        listen                  {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.UseProxyProtocol }} proxy_protocol{{ end }};
+        listen                  {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }};
         {{ else }}
-        listen                  [::]:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.UseProxyProtocol }} proxy_protocol{{ end }};
+        listen                  [::]:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }};
         {{ end }}
         {{ end }}
         proxy_timeout           {{ $cfg.ProxyStreamTimeout }};
         proxy_pass              tcp-{{ $tcpServer.Port }}-{{ $tcpServer.Backend.Namespace }}-{{ $tcpServer.Backend.Name }}-{{ $tcpServer.Backend.Port }};
+        {{ if $tcpServer.Backend.ProxyProtocol.Encode }}
+        proxy_protocol          on;
+        {{ end }}
     }
 
     {{ end }}
@@ -514,6 +517,8 @@ stream {
         location @custom_{{ $errCode }} {
             internal;
 
+            proxy_intercept_errors off;
+
             proxy_set_header       X-Code             {{ $errCode }};
             proxy_set_header       X-Format           $http_accept;
             proxy_set_header       X-Original-URI     $request_uri;
@@ -521,6 +526,7 @@ stream {
             proxy_set_header       X-Ingress-Name     $ingress_name;
             proxy_set_header       X-Service-Name     $service_name;
 
+            rewrite                (.*) / break;
             proxy_pass             http://upstream-default-backend;
         }
         {{ end }}
@@ -626,6 +632,10 @@ stream {
         {{ end }}
         {{ end }}
 
+        {{ if not (empty $server.ServerSnippet) }}
+        {{ $server.ServerSnippet }}
+        {{ end }}
+        
         {{ range $location := $server.Locations }}
         {{ $path := buildLocation $location }}
         {{ $authPath := buildAuthLocation $location }}
@@ -704,7 +714,7 @@ stream {
             {{ end }}
 
             {{ if not (empty $location.ExternalAuth.SigninURL) }}
-            error_page 401 = {{ buildAuthSignURL $location.ExternalAuth.SigninURL }};
+            error_page 401 = $location.ExternalAuth.SigninURL;
             {{ end }}
 
             {{/* if the location contains a rate limit annotation, create one */}}
@@ -763,6 +773,9 @@ stream {
             proxy_set_header X-Original-URI         $request_uri;
             proxy_set_header X-Scheme               $pass_access_scheme;
 
+            {{/* This header is used for external authentication */}}
+            proxy_set_header X-Auth-Request-Redirect $request_uri;
+
             # mitigate HTTPoxy Vulnerability
             # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
             proxy_set_header Proxy                  "";

core/pkg/ingress/annotations/serversnippet/main.go

@@ -0,0 +1,42 @@
+/*
+Copyright 2016 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package serversnippet
+
+import (
+	extensions "k8s.io/api/extensions/v1beta1"
+
+	"k8s.io/ingress/core/pkg/ingress/annotations/parser"
+)
+
+const (
+	annotation = "ingress.kubernetes.io/server-snippet"
+)
+
+type serverSnippet struct {
+}
+
+// NewParser creates a new server snippet annotation parser
+func NewParser() parser.IngressAnnotation {
+	return serverSnippet{}
+}
+
+// Parse parses the annotations contained in the ingress rule
+// used to indicate if the location/s contains a fragment of
+// configuration to be included inside the paths of the rules
+func (a serverSnippet) Parse(ing *extensions.Ingress) (interface{}, error) {
+	return parser.GetStringAnnotation(annotation, ing)
+}

core/pkg/ingress/annotations/serversnippet/main_test.go

@@ -0,0 +1,58 @@
+/*
+Copyright 2017 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package serversnippet
+
+import (
+	"testing"
+
+	api "k8s.io/api/core/v1"
+	extensions "k8s.io/api/extensions/v1beta1"
+	meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+func TestParse(t *testing.T) {
+	ap := NewParser()
+	if ap == nil {
+		t.Fatalf("expected a parser.IngressAnnotation but returned nil")
+	}
+
+	testCases := []struct {
+		annotations map[string]string
+		expected    string
+	}{
+		{map[string]string{annotation: "more_headers"}, "more_headers"},
+		{map[string]string{annotation: "false"}, "false"},
+		{map[string]string{}, ""},
+		{nil, ""},
+	}
+
+	ing := &extensions.Ingress{
+		ObjectMeta: meta_v1.ObjectMeta{
+			Name:      "foo",
+			Namespace: api.NamespaceDefault,
+		},
+		Spec: extensions.IngressSpec{},
+	}
+
+	for _, testCase := range testCases {
+		ing.SetAnnotations(testCase.annotations)
+		result, _ := ap.Parse(ing)
+		if result != testCase.expected {
+			t.Errorf("expected %v but returned %v, annotations: %s", testCase.expected, result, testCase.annotations)
+		}
+	}
+}

core/pkg/ingress/controller/annotations.go

@@ -36,6 +36,7 @@ import (
 	"k8s.io/ingress/core/pkg/ingress/annotations/redirect"
 	"k8s.io/ingress/core/pkg/ingress/annotations/rewrite"
 	"k8s.io/ingress/core/pkg/ingress/annotations/secureupstream"
+	"k8s.io/ingress/core/pkg/ingress/annotations/serversnippet"
 	"k8s.io/ingress/core/pkg/ingress/annotations/serviceupstream"
 	"k8s.io/ingress/core/pkg/ingress/annotations/sessionaffinity"
 	"k8s.io/ingress/core/pkg/ingress/annotations/snippet"
@@ -83,6 +84,7 @@ func newAnnotationExtractor(cfg extractorConfig) annotationExtractor {
 			"DefaultBackend":       defaultbackend.NewParser(cfg),
 			"UpstreamVhost":        upstreamvhost.NewParser(),
 			"VtsFilterKey":         vtsfilterkey.NewParser(),
+			"ServerSnippet":        serversnippet.NewParser(),
 		},
 	}
 }
@@ -128,6 +130,7 @@ const (
 	serverAlias          = "Alias"
 	clientBodyBufferSize = "ClientBodyBufferSize"
 	certificateAuth      = "CertificateAuth"
+	serverSnippet        = "ServerSnippet"
 )
 
 func (e *annotationExtractor) ServiceUpstream(ing *extensions.Ingress) bool {
@@ -181,3 +184,8 @@ func (e *annotationExtractor) CertificateAuth(ing *extensions.Ingress) *authtls.
 	secure := val.(*authtls.AuthSSLConfig)
 	return secure
 }
+
+func (e *annotationExtractor) ServerSnippet(ing *extensions.Ingress) string {
+	val, _ := e.annotations[serverSnippet].Parse(ing)
+	return val.(string)
+}

core/pkg/ingress/controller/backend_ssl.go

@@ -74,35 +74,47 @@ func (ic *GenericController) getPemCertificate(secretName string) (*ingress.SSLC
 
 	cert, okcert := secret.Data[apiv1.TLSCertKey]
 	key, okkey := secret.Data[apiv1.TLSPrivateKeyKey]
-
 	ca := secret.Data["ca.crt"]
 
+	// namespace/secretName -> namespace-secretName
 	nsSecName := strings.Replace(secretName, "/", "-", -1)
 
 	var s *ingress.SSLCert
 	if okcert && okkey {
-		if cert == nil || key == nil {
+		if cert == nil {
-			return nil, fmt.Errorf("error retrieving cert or key from secret %v: %v", secretName, err)
+			return nil, fmt.Errorf("secret %v has no 'tls.crt'", secretName)
 		}
+		if key == nil {
+			return nil, fmt.Errorf("secret %v has no 'tls.key'", secretName)
+		}
+
+		// If 'ca.crt' is also present, it will allow this secret to be used in the
+		// 'ingress.kubernetes.io/auth-tls-secret' annotation
 		s, err = ssl.AddOrUpdateCertAndKey(nsSecName, cert, key, ca)
 		if err != nil {
-			return nil, fmt.Errorf("unexpected error creating pem file %v", err)
+			return nil, fmt.Errorf("unexpected error creating pem file: %v", err)
 		}
-		glog.V(3).Infof("found certificate and private key, configuring %v as a TLS Secret (CN: %v)", secretName, s.CN)
+
+		glog.V(3).Infof("found 'tls.crt' and 'tls.key', configuring %v as a TLS Secret (CN: %v)", secretName, s.CN)
+		if ca != nil {
+			glog.V(3).Infof("found 'ca.crt', secret %v can also be used for Certificate Authentication", secretName)
+		}
+
 	} else if ca != nil {
-		glog.V(3).Infof("found only ca.crt, configuring %v as an Certificate Authentication secret", secretName)
 		s, err = ssl.AddCertAuth(nsSecName, ca)
+
 		if err != nil {
-			return nil, fmt.Errorf("unexpected error creating pem file %v", err)
+			return nil, fmt.Errorf("unexpected error creating pem file: %v", err)
 		}
+
+		// makes this secret in 'syncSecret' to be used for Certificate Authentication
+		// this does not enable Certificate Authentication
+		glog.V(3).Infof("found only 'ca.crt', configuring %v as an Certificate Authentication Secret", secretName)
+
 	} else {
 		return nil, fmt.Errorf("no keypair or CA cert could be found in %v", secretName)
 	}
 
-	if err != nil {
-		return nil, err
-	}
-
 	s.Name = secret.Name
 	s.Namespace = secret.Namespace
 	return s, nil

core/pkg/ingress/controller/controller.go

@@ -238,17 +238,19 @@ func (ic GenericController) GetService(name string) (*apiv1.Service, error) {
 // sync collects all the pieces required to assemble the configuration file and
 // then sends the content to the backend (OnUpdate) receiving the populated
 // template as response reloading the backend if is required.
-func (ic *GenericController) syncIngress(key interface{}) error {
+func (ic *GenericController) syncIngress(item interface{}) error {
 	ic.syncRateLimiter.Accept()
 
 	if ic.syncQueue.IsShuttingDown() {
 		return nil
 	}
 
-	if name, ok := key.(string); ok {
+	if element, ok := item.(task.Element); ok {
-		if obj, exists, _ := ic.listers.Ingress.GetByKey(name); exists {
+		if name, ok := element.Key.(string); ok {
-			ing := obj.(*extensions.Ingress)
+			if obj, exists, _ := ic.listers.Ingress.GetByKey(name); exists {
-			ic.readSecrets(ing)
+				ing := obj.(*extensions.Ingress)
+				ic.readSecrets(ing)
+			}
 		}
 	}
 
@@ -346,6 +348,7 @@ func (ic *GenericController) getStreamServices(configmapName string, proto apiv1
 	}
 
 	var svcs []ingress.L4Service
+	var svcProxyProtocol ingress.ProxyProtocol
 	// k -> port to expose
 	// v -> <namespace>/<service name>:<port from service to be used>
 	for k, v := range configmap.Data {
@@ -363,18 +366,22 @@ func (ic *GenericController) getStreamServices(configmapName string, proto apiv1
 
 		nsSvcPort := strings.Split(v, ":")
 		if len(nsSvcPort) < 2 {
-			glog.Warningf("invalid format (namespace/name:port:[PROXY]) '%v'", k)
+			glog.Warningf("invalid format (namespace/name:port:[PROXY]:[PROXY]) '%v'", k)
 			continue
 		}
 
 		nsName := nsSvcPort[0]
 		svcPort := nsSvcPort[1]
-		useProxyProtocol := false
+		svcProxyProtocol.Decode = false
+		svcProxyProtocol.Encode = false
 
 		// Proxy protocol is possible if the service is TCP
-		if len(nsSvcPort) == 3 && proto == apiv1.ProtocolTCP {
+		if len(nsSvcPort) >= 3 && proto == apiv1.ProtocolTCP {
-			if strings.ToUpper(nsSvcPort[2]) == "PROXY" {
+			if len(nsSvcPort) >= 3 && strings.ToUpper(nsSvcPort[2]) == "PROXY" {
-				useProxyProtocol = true
+				svcProxyProtocol.Decode = true
+			}
+			if len(nsSvcPort) == 4 && strings.ToUpper(nsSvcPort[3]) == "PROXY" {
+				svcProxyProtocol.Encode = true
 			}
 		}
 
@@ -432,11 +439,11 @@ func (ic *GenericController) getStreamServices(configmapName string, proto apiv1
 		svcs = append(svcs, ingress.L4Service{
 			Port: externalPort,
 			Backend: ingress.L4Backend{
-				Name:             svcName,
+				Name:          svcName,
-				Namespace:        svcNs,
+				Namespace:     svcNs,
-				Port:             intstr.FromString(svcPort),
+				Port:          intstr.FromString(svcPort),
-				Protocol:         proto,
+				Protocol:      proto,
-				UseProxyProtocol: useProxyProtocol,
+				ProxyProtocol: svcProxyProtocol,
 			},
 			Endpoints: endps,
 		})
@@ -509,9 +516,13 @@ func (ic *GenericController) getBackendServers(ingresses []*extensions.Ingress)
 				ca := ic.annotations.CertificateAuth(ing)
 				if ca != nil {
 					server.CertificateAuth = *ca
+					// It is possible that no CAFileName is found in the secret
+					if server.CertificateAuth.CAFileName == "" {
+						glog.V(3).Infof("secret %v does not contain 'ca.crt', mutual authentication not enabled - ingress rule %v/%v.", server.CertificateAuth.Secret, ing.Namespace, ing.Name)
+					}
 				}
 			} else {
-				glog.V(3).Infof("server %v already contains a muthual autentication configuration - ingress rule %v/%v", server.Hostname, ing.Namespace, ing.Name)
+				glog.V(3).Infof("server %v already contains a mutual authentication configuration - ingress rule %v/%v", server.Hostname, ing.Namespace, ing.Name)
 			}
 
 			for _, path := range rule.HTTP.Paths {
@@ -671,7 +682,8 @@ func (ic *GenericController) getBackendServers(ingresses []*extensions.Ingress)
 	return aUpstreams, aServers
 }
 
-// GetAuthCertificate ...
+
+// GetAuthCertificate is used by the auth-tls annotations to get a cert from a secret
 func (ic GenericController) GetAuthCertificate(secretName string) (*resolver.AuthSSLCert, error) {
 	if _, exists := ic.sslCertTracker.Get(secretName); !exists {
 		ic.syncSecret(secretName)
@@ -894,10 +906,12 @@ func (ic *GenericController) createServers(data []*extensions.Ingress,
 		RequestBuffering: bdef.ProxyRequestBuffering,
 	}
 
+	// generated on Start() with createDefaultSSLCertificate()
 	defaultPemFileName := fakeCertificatePath
 	defaultPemSHA := fakeCertificateSHA
 
-	// Tries to fetch the default Certificate. If it does not exists, generate a new self signed one.
+	// Tries to fetch the default Certificate from nginx configuration.
+	// If it does not exists, use the ones generated on Start()
 	defaultCertificate, err := ic.getPemCertificate(ic.cfg.DefaultSSLCertificate)
 	if err == nil {
 		defaultPemFileName = defaultCertificate.PemFileName
@@ -976,6 +990,7 @@ func (ic *GenericController) createServers(data []*extensions.Ingress,
 	for _, ing := range data {
 		// setup server-alias based on annotations
 		aliasAnnotation := ic.annotations.Alias(ing)
+		srvsnippet := ic.annotations.ServerSnippet(ing)
 
 		for _, rule := range ing.Spec.Rules {
 			host := rule.Host
@@ -991,6 +1006,17 @@ func (ic *GenericController) createServers(data []*extensions.Ingress,
 				}
 			}
 
+			//notifying the user that it has already been configured.
+			if servers[host].ServerSnippet != "" && srvsnippet != "" {
+				glog.Warningf("ingress %v/%v for host %v contains a Server Snippet section that it has already been configured.",
+					ing.Namespace, ing.Name, host)
+			}
+
+			// only add a server snippet if the server does not have one previously configured
+			if servers[host].ServerSnippet == "" && srvsnippet != "" {
+				servers[host].ServerSnippet = srvsnippet
+			}
+
 			// only add a certificate if the server does not have one previously configured
 			if servers[host].SSLCertificate != "" {
 				continue
@@ -1054,6 +1080,7 @@ func (ic *GenericController) createServers(data []*extensions.Ingress,
 			servers[host].Alias = ""
 		}
 	}
+
 	return servers
 }
 

core/pkg/ingress/resolver/main.go

@@ -54,7 +54,7 @@ type AuthSSLCert struct {
 	Secret string `json:"secret"`
 	// CAFileName contains the path to the secrets 'ca.crt'
 	CAFileName string `json:"caFilename"`
-	// PemSHA contains the SHA1 hash of the 'tls.crt' value
+	// PemSHA contains the SHA1 hash of the 'ca.crt' or combinations of (tls.crt, tls.key, tls.crt) depending on certs in secret
 	PemSHA string `json:"pemSha"`
 }
 

core/pkg/ingress/types.go

@@ -236,6 +236,10 @@ type Server struct {
 	// CertificateAuth indicates the this server requires mutual authentication
 	// +optional
 	CertificateAuth authtls.AuthSSLConfig `json:"certificateAuth"`
+
+	// ServerSnippet returns the snippet of server
+	// +optional
+	ServerSnippet string `json:"serverSnippet"`
 }
 
 // Location describes an URI inside a server.
@@ -359,5 +363,11 @@ type L4Backend struct {
 	Namespace string             `json:"namespace"`
 	Protocol  apiv1.Protocol     `json:"protocol"`
 	// +optional
-	UseProxyProtocol bool `json:"useProxyProtocol"`
+	ProxyProtocol ProxyProtocol `json:"proxyProtocol"`
+}
+
+// ProxyProtocol describes the proxy protocol configuration
+type ProxyProtocol struct {
+	Decode bool `json:"decode"`
+	Encode bool `json:"encode"`
 }

core/pkg/net/ssl/ssl.go

@@ -271,7 +271,7 @@ func AddCertAuth(name string, ca []byte) (*ingress.SSLCert, error) {
 		return nil, fmt.Errorf("could not write CA file %v: %v", caFileName, err)
 	}
 
-	glog.V(3).Infof("Created CA Certificate for authentication: %v", caFileName)
+	glog.V(3).Infof("Created CA Certificate for Authentication: %v", caFileName)
 	return &ingress.SSLCert{
 		CAFileName:  caFileName,
 		PemFileName: caFileName,

core/pkg/task/queue.go

@@ -48,7 +48,8 @@ type Queue struct {
 	lastSync int64
 }
 
-type element struct {
+// Element represents one item of the queue
+type Element struct {
 	Key       interface{}
 	Timestamp int64
 }
@@ -72,7 +73,7 @@ func (t *Queue) Enqueue(obj interface{}) {
 		glog.Errorf("%v", err)
 		return
 	}
-	t.queue.Add(element{
+	t.queue.Add(Element{
 		Key:       key,
 		Timestamp: ts,
 	})
@@ -99,7 +100,7 @@ func (t *Queue) worker() {
 		}
 		ts := time.Now().UnixNano()
 
-		item := key.(element)
+		item := key.(Element)
 		if t.lastSync > item.Timestamp {
 			glog.V(3).Infof("skipping %v sync (%v > %v)", item.Key, t.lastSync, item.Timestamp)
 			t.queue.Forget(key)
@@ -110,7 +111,7 @@ func (t *Queue) worker() {
 		glog.V(3).Infof("syncing %v", item.Key)
 		if err := t.sync(key); err != nil {
 			glog.Warningf("requeuing %v, err %v", item.Key, err)
-			t.queue.AddRateLimited(element{
+			t.queue.AddRateLimited(Element{
 				Key:       item.Key,
 				Timestamp: time.Now().UnixNano(),
 			})

examples/PREREQUISITES.md

@@ -132,7 +132,12 @@ The final step is to create a secret with the content of this file. This secret
 the TLS Auth directive:
 
 ```console
-$ kubectl create secret generic caingress --namespace=default --from-file=ca.crt
+$ kubectl create secret generic caingress --namespace=default --from-file=ca.crt=<ca.crt>
+```
+
+Note: You can also generate the CA Authentication Secret along with the TLS Secret by using:
+```console
+$ kubectl create secret generic caingress --namespace=default --from-file=ca.crt=<ca.crt> --from-file=tls.crt=<tls.crt> --from-file=tls.key=<tls.key>
 ```
 
 ## Test HTTP Service

examples/auth/client-certs/nginx/README.md

@@ -16,13 +16,12 @@ the child, except for the root, which has Issuer == Subject.
 
 * Client Cert: Certificate used by the clients to authenticate themselves with the loadbalancer/backends.
 
-
 ## Prerequisites
 
 You need a valid CA File, composed of a group of valid enabled CAs. This MUST be in PEM Format.
-The instructions are described [here](../../../PREREQUISITES.md#ca-authentication)
+The instructions are described [here](../../../PREREQUISITES.md)
 
-Also your ingress must be configured as a HTTPs/TLS Ingress.
+Also your ingress must be configured as a HTTPS/TLS Ingress.
 
 ## Deployment
 
@@ -51,8 +50,7 @@ Name:			nginx-test
 Namespace:		default
 Address:		104.198.183.6
 Default backend:	default-http-backend:80 (10.180.0.4:8080,10.240.0.2:8080)
-TLS:
+TLS: tls-secret terminates ingress.test.com
-  tls-secret terminates ingress.test.com
 Rules:
   Host	Path	Backends
   ----	----	--------
@@ -79,13 +77,12 @@ Server: nginx/1.11.9
 $ curl -I -k --key ~/user.key --cert ~/user.cer https://ingress.test.com 
 HTTP/1.1 200 OK
 Server: nginx/1.11.9
-
 ```
 
 You must use the full DNS name while testing, as NGINX relies on the Server Name (SNI) to select the correct Ingress to be used.
-
 The curl version used here was ``curl 7.47.0``
 
 ## Which certificate was used for authentication?
 
-In your backend application you might want to know which certificate was used for authentication. For this purpose, we pass the full certificate in PEM format to the backend in the `ssl-client-cert` header.
+In your backend application you might want to know which certificate was used for authentication.
+For this purpose, we pass the full certificate in PEM format to the backend in the `ssl-client-cert` header.

examples/auth/client-certs/nginx/nginx-tls-auth.yaml

@@ -21,6 +21,5 @@ spec:
   tls:
   - hosts:
     - ingress.test.com
-    # Create this cert as described in 'multi-tls' example
+    secretName: tls-secret
-    secretName: cert
 

examples/customization/custom-errors/nginx/README.md

@@ -1,4 +1,4 @@
-This example shows how is possible to use a custom backend to render custom error pages. The code of this example is located here [nginx-debug-server](https://github.com/aledbf/contrib/tree/nginx-debug-server)
+This example shows how is possible to use a custom backend to render custom error pages. The code of this example is located here [custom-error-pages](https://github.com/kubernetes/ingress/tree/master/examples/customization/custom-errors/nginx)
 
 
 The idea is to use the headers `X-Code` and `X-Format` that NGINX pass to the backend in case of an error to find out the best existent representation of the response to be returned. i.e. if the request contains an `Accept` header of type `json` the error should be in that format and not in `html` (the default in NGINX).
@@ -78,5 +78,3 @@ $ curl -v http://172.17.4.99/ -H 'Accept: application/json'
 
 * Connection #0 to host 172.17.4.99 left intact
 ```
-
-By default the Ingress controller provides support for `html`, `json` and `XML`.

images/custom-error-pages/Makefile

@@ -0,0 +1,110 @@
+all: push
+
+BUILDTAGS=
+
+# Use the 0.0 tag for testing, it shouldn't clobber any release builds
+TAG?=0.1
+REGISTRY?=aledbf
+GOOS?=linux
+DOCKER?=gcloud docker --
+SED_I?=sed -i
+GOHOSTOS ?= $(shell go env GOHOSTOS)
+
+PKG=k8s.io/ingress/images/custom-error-pages
+
+ifeq ($(GOHOSTOS),darwin)
+  SED_I=sed -i ''
+endif
+
+REPO_INFO=$(shell git config --get remote.origin.url)
+
+ifndef COMMIT
+  COMMIT := git-$(shell git rev-parse --short HEAD)
+endif
+
+ARCH ?= $(shell go env GOARCH)
+GOARCH = ${ARCH}
+DUMB_ARCH = ${ARCH}
+
+BASEIMAGE?=alpine:3.6
+
+ALL_ARCH = amd64 arm arm64 ppc64le
+
+QEMUVERSION=v2.9.1
+
+IMGNAME = custom-error-pages
+IMAGE = $(REGISTRY)/$(IMGNAME)
+MULTI_ARCH_IMG = $(IMAGE)-$(ARCH)
+
+ifeq ($(ARCH),arm)
+    QEMUARCH=arm
+	GOARCH=arm
+endif
+ifeq ($(ARCH),arm64)
+        QEMUARCH=aarch64
+endif
+ifeq ($(ARCH),ppc64le)
+	QEMUARCH=ppc64le
+	GOARCH=ppc64le
+endif
+#ifeq ($(ARCH),s390x)
+#        QEMUARCH=s390x
+#endif
+
+TEMP_DIR := $(shell mktemp -d)
+
+DOCKERFILE := $(TEMP_DIR)/rootfs/Dockerfile
+
+all: all-container
+
+sub-container-%:
+	$(MAKE) ARCH=$* build container
+
+sub-push-%:
+	$(MAKE) ARCH=$* push
+
+all-container: $(addprefix sub-container-,$(ALL_ARCH))
+
+all-push: $(addprefix sub-push-,$(ALL_ARCH))
+
+container: .container-$(ARCH)
+.container-$(ARCH):
+	cp -r ./* $(TEMP_DIR)
+	$(SED_I) 's|BASEIMAGE|$(BASEIMAGE)|g' $(DOCKERFILE)
+	$(SED_I) "s|QEMUARCH|$(QEMUARCH)|g" $(DOCKERFILE)
+
+ifeq ($(ARCH),amd64)
+	# When building "normally" for amd64, remove the whole line, it has no part in the amd64 image
+	$(SED_I) "/CROSS_BUILD_/d" $(DOCKERFILE)
+else
+	# When cross-building, only the placeholder "CROSS_BUILD_" should be removed
+	# Register /usr/bin/qemu-ARCH-static as the handler for ARM binaries in the kernel
+	$(DOCKER) run --rm --privileged multiarch/qemu-user-static:register --reset
+	curl -sSL https://github.com/multiarch/qemu-user-static/releases/download/$(QEMUVERSION)/x86_64_qemu-$(QEMUARCH)-static.tar.gz | tar -xz -C $(TEMP_DIR)/rootfs
+	$(SED_I) "s/CROSS_BUILD_//g" $(DOCKERFILE)
+endif
+
+	$(DOCKER) build -t $(MULTI_ARCH_IMG):$(TAG) $(TEMP_DIR)/rootfs
+
+ifeq ($(ARCH), amd64)
+	# This is for to maintain the backward compatibility
+	$(DOCKER) tag $(MULTI_ARCH_IMG):$(TAG) $(IMAGE):$(TAG)
+endif
+
+push: .push-$(ARCH)
+.push-$(ARCH):
+	$(DOCKER) push $(MULTI_ARCH_IMG):$(TAG)
+ifeq ($(ARCH), amd64)
+	$(DOCKER) push $(IMAGE):$(TAG)
+endif
+
+clean:
+	$(DOCKER) rmi -f $(MULTI_ARCH_IMG):$(TAG) || true
+
+build: clean
+	CGO_ENABLED=0 GOOS=${GOOS} GOARCH=${GOARCH} go build -a -installsuffix cgo \
+		-ldflags "-s -w" \
+		-o ${TEMP_DIR}/rootfs/custom-error-pages ${PKG}/...
+
+release: all-container all-push
+	echo "done"

images/custom-error-pages/README.md

@@ -0,0 +1,2 @@
+
+Example of Custom error pages for the NGINX Ingress controller

images/custom-error-pages/main.go

@@ -0,0 +1,94 @@
+/*
+Copyright 2017 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package main
+
+import (
+	"fmt"
+	"io"
+	"log"
+	"mime"
+	"net/http"
+	"os"
+	"strconv"
+)
+
+const (
+	FormatHeader = "X-Format"
+
+	CodeHeader = "X-Code"
+
+	ContentType = "Content-Type"
+)
+
+func main() {
+	path := "/www"
+	if os.Getenv("PATH") != "" {
+		path = os.Getenv("PATH")
+	}
+	http.HandleFunc("/", errorHandler(path))
+	http.ListenAndServe(fmt.Sprintf(":8080"), nil)
+}
+
+func errorHandler(path string) func(http.ResponseWriter, *http.Request) {
+	return func(w http.ResponseWriter, r *http.Request) {
+		ext := "html"
+
+		format := r.Header.Get(FormatHeader)
+		if format == "" {
+			format = "text/html"
+			log.Printf("forma not specified. Using %v\n", format)
+		}
+
+		mediaType, _, _ := mime.ParseMediaType(format)
+		cext, err := mime.ExtensionsByType(mediaType)
+		if err != nil {
+			log.Printf("unexpected error reading media type extension: %v. Using %v\n", err, ext)
+		} else {
+			ext = cext[0]
+		}
+		w.Header().Set(ContentType, format)
+
+		errCode := r.Header.Get(CodeHeader)
+		code, err := strconv.Atoi(errCode)
+		if err != nil {
+			code = 404
+			log.Printf("unexpected error reading return code: %v. Using %v\n", err, code)
+		}
+		w.WriteHeader(code)
+
+		file := fmt.Sprintf("%v/%v%v", path, code, ext)
+		f, err := os.Open(file)
+		if err != nil {
+			log.Printf("unexpected error opening file: %v\n", err)
+			scode := strconv.Itoa(code)
+			file := fmt.Sprintf("%v/%cxx%v", path, scode[0], ext)
+			f, err := os.Open(file)
+			if err != nil {
+				log.Printf("unexpected error opening file: %v\n", err)
+				http.NotFound(w, r)
+				return
+			}
+			defer f.Close()
+			log.Printf("serving custom error response for code %v and format %v from file %v\n", code, format, file)
+			io.Copy(w, f)
+			return
+		}
+		defer f.Close()
+		log.Printf("serving custom error response for code %v and format %v from file %v\n", code, format, file)
+		io.Copy(w, f)
+	}
+}

images/custom-error-pages/rootfs/Dockerfile

@@ -0,0 +1,21 @@
+# Copyright 2017 The Kubernetes Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM BASEIMAGE
+
+CROSS_BUILD_COPY qemu-QEMUARCH-static /usr/bin/
+
+COPY . /
+
+CMD ["/custom-error-pages"]

images/custom-error-pages/rootfs/www/404.html

@@ -0,0 +1 @@
+<span>The page you're looking for could not be found.</span>

images/custom-error-pages/rootfs/www/404.json

@@ -0,0 +1 @@
+{ "message": "The page you're looking for could not be found" }

images/custom-error-pages/rootfs/www/4xx.html

@@ -0,0 +1 @@
+4xx html

images/custom-error-pages/rootfs/www/4xx.json

@@ -0,0 +1 @@
+4xx json

images/custom-error-pages/rootfs/www/500.html

@@ -0,0 +1 @@
+500 html

images/custom-error-pages/rootfs/www/500.json

@@ -0,0 +1 @@
+500 json

images/custom-error-pages/rootfs/www/5xx.html

@@ -0,0 +1 @@
+5xx html

images/custom-error-pages/rootfs/www/5xx.json

@@ -0,0 +1 @@
+5xx json

images/nginx-slim/Makefile

@@ -13,8 +13,8 @@
 # limitations under the License.
 
 # 0.0.0 shouldn't clobber any released builds
-TAG = 0.25
+TAG ?= 0.25
-REGISTRY = gcr.io/google_containers
+REGISTRY ?= gcr.io/google_containers
 ARCH ?= $(shell go env GOARCH)
 ALL_ARCH = amd64 arm arm64 ppc64le