DevFW-CICD/ingress-nginx-helm
14
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge pull request #258 from rikatz/nginx-sticky-annotations

Browse source 
Nginx sticky annotations
This commit is contained in:
Prashanth B 2017-02-17 05:27:18 +05:30 committed by GitHub
commit 698c08402a
10 changed files with 406 additions and 10 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

20
controllers/nginx/configuration.md
View file

@ -52,6 +52,9 @@ The following annotations are supported:
|[ingress.kubernetes.io/upstream-max-fails](#custom-nginx-upstream-checks)|number|
|[ingress.kubernetes.io/upstream-fail-timeout](#custom-nginx-upstream-checks)|number|
|[ingress.kubernetes.io/whitelist-source-range](#whitelist-source-range)|CIDR|
|[ingress.kubernetes.io/affinity](#session-affinity)|true or false|
|[ingress.kubernetes.io/session-cookie-name](#cookie-affinity)|string|
|[ingress.kubernetes.io/session-cookie-hash](#cookie-affinity)|string|
@ -179,7 +182,22 @@ To configure this setting globally for all Ingress rules, the `whitelist-source-
*Note:* Adding an annotation to an Ingress rule overrides any global restriction.
Please check the [whitelist](examples/whitelist/README.md) example.
Please check the [whitelist](examples/affinity/cookie/nginx/README.md) example.
### Session Affinity
The annotation `ingress.kubernetes.io/affinity` enables and sets the affinity type in all Upstreams of an Ingress. This way, a request will always be directed to the same upstream server.
#### Cookie affinity
If you use the ``cookie`` type you can also specify the name of the cookie that will be used to route the requests with the annotation `ingress.kubernetes.io/session-cookie-name`. The default is to create a cookie named 'route'.
In case of NGINX the annotation `ingress.kubernetes.io/session-cookie-hash` defines which algorithm will be used to 'hash' the used upstream. Default value is `md5` and possible values are `md5`, `sha1` and `index`.
The `index` option is not hashed, an in-memory index is used instead, it's quicker and the overhead is shorter Warning: the matching against upstream servers list is inconsistent. So, at reload, if upstreams servers has changed, index values are not guaranted to correspond to the same server as before! USE IT WITH CAUTION and only if you need to!
In NGINX this feature is implemented by the third party module [nginx-sticky-module-ng](https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng). The workflow used to define which upstream server will be used is explained [here]https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng/raw/08a395c66e425540982c00482f55034e1fee67b6/docs/sticky.pdf
### **Allowed parameters in configuration ConfigMap**

4
controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl
View file

@ -185,8 +185,8 @@ http {
{{range $name, $upstream := $backends}}
upstream {{$upstream.Name}} {
{{ if $cfg.EnableStickySessions }}
sticky hash=sha1 httponly;
{{ if eq $upstream.SessionAffinity.AffinityType "cookie" }}
sticky hash={{$upstream.SessionAffinity.CookieSessionAffinity.Hash}} name={{$upstream.SessionAffinity.CookieSessionAffinity.Name}} httponly;
{{ else }}
least_conn;
{{ end }}

118
core/pkg/ingress/annotations/sessionaffinity/main.go Normal file
View file

@ -0,0 +1,118 @@
/*
Copyright 2016 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package sessionaffinity
import (
"regexp"
"github.com/golang/glog"
"k8s.io/kubernetes/pkg/apis/extensions"
"k8s.io/ingress/core/pkg/ingress/annotations/parser"
)
const (
annotationAffinityType = "ingress.kubernetes.io/affinity"
// If a cookie with this name exists,
// its value is used as an index into the list of available backends.
annotationAffinityCookieName = "ingress.kubernetes.io/session-cookie-name"
defaultAffinityCookieName = "INGRESSCOOKIE"
// This is the algorithm used by nginx to generate a value for the session cookie, if
// one isn't supplied and affintiy is set to "cookie".
annotationAffinityCookieHash = "ingress.kubernetes.io/session-cookie-hash"
defaultAffinityCookieHash = "md5"
)
var (
affinityCookieHashRegex = regexp.MustCompile(`^(index|md5|sha1)$`)
)
// AffinityConfig describes the per ingress session affinity config
type AffinityConfig struct {
// The type of affinity that will be used
AffinityType string `json:"type"`
CookieConfig
}
// CookieConfig describes the Config of cookie type affinity
type CookieConfig struct {
// The name of the cookie that will be used in case of cookie affinity type.
Name string `json:"name"`
// The hash that will be used to encode the cookie in case of cookie affinity type
Hash string `json:"hash"`
}
// CookieAffinityParse gets the annotation values related to Cookie Affinity
// It also sets default values when no value or incorrect value is found
func CookieAffinityParse(ing *extensions.Ingress) *CookieConfig {
sn, err := parser.GetStringAnnotation(annotationAffinityCookieName, ing)
if err != nil || sn == "" {
glog.V(3).Infof("Ingress %v: No value found in annotation %v. Using the default %v", ing.Name, annotationAffinityCookieName, defaultAffinityCookieName)
sn = defaultAffinityCookieName
}
sh, err := parser.GetStringAnnotation(annotationAffinityCookieHash, ing)
if err != nil || !affinityCookieHashRegex.MatchString(sh) {
glog.V(3).Infof("Invalid or no annotation value found in Ingress %v: %v. Setting it to default %v", ing.Name, annotationAffinityCookieHash, defaultAffinityCookieHash)
sh = defaultAffinityCookieHash
}
return &CookieConfig{
Name: sn,
Hash: sh,
}
}
// NewParser creates a new Affinity annotation parser
func NewParser() parser.IngressAnnotation {
return affinity{}
}
type affinity struct {
}
// ParseAnnotations parses the annotations contained in the ingress
// rule used to configure the affinity directives
func (a affinity) Parse(ing *extensions.Ingress) (interface{}, error) {
var cookieAffinityConfig *CookieConfig
cookieAffinityConfig = &CookieConfig{}
// Check the type of affinity that will be used
at, err := parser.GetStringAnnotation(annotationAffinityType, ing)
if err != nil {
at = ""
}
switch at {
case "cookie":
cookieAffinityConfig = CookieAffinityParse(ing)
default:
glog.V(3).Infof("No default affinity was found for Ingress %v", ing.Name)
}
return &AffinityConfig{
AffinityType: at,
CookieConfig: *cookieAffinityConfig,
}, nil
}

88
core/pkg/ingress/annotations/sessionaffinity/main_test.go Normal file
View file

@ -0,0 +1,88 @@
/*
Copyright 2016 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package sessionaffinity
import (
"testing"
"k8s.io/kubernetes/pkg/api"
"k8s.io/kubernetes/pkg/apis/extensions"
"k8s.io/kubernetes/pkg/util/intstr"
)
func buildIngress() *extensions.Ingress {
defaultBackend := extensions.IngressBackend{
ServiceName: "default-backend",
ServicePort: intstr.FromInt(80),
}
return &extensions.Ingress{
ObjectMeta: api.ObjectMeta{
Name: "foo",
Namespace: api.NamespaceDefault,
},
Spec: extensions.IngressSpec{
Backend: &extensions.IngressBackend{
ServiceName: "default-backend",
ServicePort: intstr.FromInt(80),
},
Rules: []extensions.IngressRule{
{
Host: "foo.bar.com",
IngressRuleValue: extensions.IngressRuleValue{
HTTP: &extensions.HTTPIngressRuleValue{
Paths: []extensions.HTTPIngressPath{
{
Path: "/foo",
Backend: defaultBackend,
},
},
},
},
},
},
},
}
}
func TestIngressAffinityCookieConfig(t *testing.T) {
ing := buildIngress()
data := map[string]string{}
data[annotationAffinityType] = "cookie"
data[annotationAffinityCookieHash] = "sha123"
data[annotationAffinityCookieName] = "INGRESSCOOKIE"
ing.SetAnnotations(data)
affin, _ := NewParser().Parse(ing)
nginxAffinity, ok := affin.(*AffinityConfig)
if !ok {
t.Errorf("expected a Config type")
}
if nginxAffinity.AffinityType != "cookie" {
t.Errorf("expected cookie as sticky-type but returned %v", nginxAffinity.AffinityType)
}
if nginxAffinity.CookieConfig.Hash != "md5" {
t.Errorf("expected md5 as sticky-hash but returned %v", nginxAffinity.CookieConfig.Hash)
}
if nginxAffinity.CookieConfig.Name != "INGRESSCOOKIE" {
t.Errorf("expected route as sticky-name but returned %v", nginxAffinity.CookieConfig.Name)
}
}

14
core/pkg/ingress/controller/annotations.go
View file

@ -33,6 +33,7 @@ import (
"k8s.io/ingress/core/pkg/ingress/annotations/ratelimit"
"k8s.io/ingress/core/pkg/ingress/annotations/rewrite"
"k8s.io/ingress/core/pkg/ingress/annotations/secureupstream"
"k8s.io/ingress/core/pkg/ingress/annotations/sessionaffinity"
"k8s.io/ingress/core/pkg/ingress/annotations/sslpassthrough"
"k8s.io/ingress/core/pkg/ingress/errors"
"k8s.io/ingress/core/pkg/ingress/resolver"
@ -62,6 +63,7 @@ func newAnnotationExtractor(cfg extractorConfig) annotationExtractor {
"RateLimit": ratelimit.NewParser(),
"Redirect": rewrite.NewParser(cfg),
"SecureUpstream": secureupstream.NewParser(),
"SessionAffinity": sessionaffinity.NewParser(),
"SSLPassthrough": sslpassthrough.NewParser(),
},
}
@ -96,9 +98,10 @@ func (e *annotationExtractor) Extract(ing *extensions.Ingress) map[string]interf
}
const (
secureUpstream = "SecureUpstream"
healthCheck = "HealthCheck"
sslPassthrough = "SSLPassthrough"
secureUpstream = "SecureUpstream"
healthCheck = "HealthCheck"
sslPassthrough = "SSLPassthrough"
sessionAffinity = "SessionAffinity"
)
func (e *annotationExtractor) SecureUpstream(ing *extensions.Ingress) bool {
@ -115,3 +118,8 @@ func (e *annotationExtractor) SSLPassthrough(ing *extensions.Ingress) bool {
val, _ := e.annotations[sslPassthrough].Parse(ing)
return val.(bool)
}
func (e *annotationExtractor) SessionAffinity(ing *extensions.Ingress) *sessionaffinity.AffinityConfig {
val, _ := e.annotations[sessionAffinity].Parse(ing)
return val.(*sessionaffinity.AffinityConfig)
}

47
core/pkg/ingress/controller/annotations_test.go
View file

@ -28,10 +28,13 @@ import (
)
const (
annotationSecureUpstream = "ingress.kubernetes.io/secure-backends"
annotationUpsMaxFails = "ingress.kubernetes.io/upstream-max-fails"
annotationUpsFailTimeout = "ingress.kubernetes.io/upstream-fail-timeout"
annotationPassthrough = "ingress.kubernetes.io/ssl-passthrough"
annotationSecureUpstream = "ingress.kubernetes.io/secure-backends"
annotationUpsMaxFails = "ingress.kubernetes.io/upstream-max-fails"
annotationUpsFailTimeout = "ingress.kubernetes.io/upstream-fail-timeout"
annotationPassthrough = "ingress.kubernetes.io/ssl-passthrough"
annotationAffinityType = "ingress.kubernetes.io/affinity"
annotationAffinityCookieName = "ingress.kubernetes.io/session-cookie-name"
annotationAffinityCookieHash = "ingress.kubernetes.io/session-cookie-hash"
)
type mockCfg struct {
@ -179,3 +182,39 @@ func TestSSLPassthrough(t *testing.T) {
}
}
}
func TestAffinitySession(t *testing.T) {
ec := newAnnotationExtractor(mockCfg{})
ing := buildIngress()
fooAnns := []struct {
annotations map[string]string
affinitytype string
hash string
name string
}{
{map[string]string{annotationAffinityType: "cookie", annotationAffinityCookieHash: "md5", annotationAffinityCookieName: "route"}, "cookie", "md5", "route"},
{map[string]string{annotationAffinityType: "cookie", annotationAffinityCookieHash: "xpto", annotationAffinityCookieName: "route1"}, "cookie", "md5", "route1"},
{map[string]string{annotationAffinityType: "cookie", annotationAffinityCookieHash: "", annotationAffinityCookieName: ""}, "cookie", "md5", "INGRESSCOOKIE"},
{map[string]string{}, "", "", ""},
{nil, "", "", ""},
}
for _, foo := range fooAnns {
ing.SetAnnotations(foo.annotations)
r := ec.SessionAffinity(ing)
t.Logf("Testing pass %v %v %v", foo.affinitytype, foo.hash, foo.name)
if r == nil {
t.Errorf("Returned nil but expected a SessionAffinity.AffinityConfig")
continue
}
if r.CookieConfig.Hash != foo.hash {
t.Errorf("Returned %v but expected %v for Hash", r.CookieConfig.Hash, foo.hash)
}
if r.CookieConfig.Name != foo.name {
t.Errorf("Returned %v but expected %v for Name", r.CookieConfig.Name, foo.name)
}
}
}

9
core/pkg/ingress/controller/controller.go
View file

@ -723,6 +723,7 @@ func (ic *GenericController) createUpstreams(data []interface{}) map[string]*ing
secUpstream := ic.annotations.SecureUpstream(ing)
hz := ic.annotations.HealthCheck(ing)
affinity := ic.annotations.SessionAffinity(ing)
var defBackend string
if ing.Spec.Backend != nil {
@ -762,6 +763,14 @@ func (ic *GenericController) createUpstreams(data []interface{}) map[string]*ing
if !upstreams[name].Secure {
upstreams[name].Secure = secUpstream
}
if upstreams[name].SessionAffinity.AffinityType == "" {
upstreams[name].SessionAffinity.AffinityType = affinity.AffinityType
if affinity.AffinityType == "cookie" {
upstreams[name].SessionAffinity.CookieSessionAffinity.Name = affinity.CookieConfig.Name
upstreams[name].SessionAffinity.CookieSessionAffinity.Hash = affinity.CookieConfig.Hash
}
}
svcKey := fmt.Sprintf("%v/%v", ing.GetNamespace(), path.Backend.ServiceName)
endp, err := ic.serviceEndpoints(svcKey, path.Backend.ServicePort.String(), hz)
if err != nil {

20
core/pkg/ingress/types.go
View file

@ -154,6 +154,26 @@ type Backend struct {
Secure bool `json:"secure"`
// Endpoints contains the list of endpoints currently running
Endpoints []Endpoint `json:"endpoints"`
// StickySession contains the StickyConfig object with stickness configuration
SessionAffinity SessionAffinityConfig
}
// SessionAffinityConfig describes different affinity configurations for new sessions.
// Once a session is mapped to a backend based on some affinity setting, it
// retains that mapping till the backend goes down, or the ingress controller
// restarts. Exactly one of these values will be set on the upstream, since multiple
// affinity values are incompatible. Once set, the backend makes no guarantees
// about honoring updates.
type SessionAffinityConfig struct {
AffinityType string `json:"name"`
CookieSessionAffinity CookieSessionAffinity
}
// CookieSessionAffinity defines the structure used in Affinity configured by Cookies.
type CookieSessionAffinity struct {
Name string `json:"name"`
Hash string `json:"hash"`
}
// Endpoint describes a kubernetes endpoint in a backend

77
examples/affinity/cookie/nginx/README.md Normal file
View file

@ -0,0 +1,77 @@
# Sticky Session
This example demonstrates how to achieve session affinity using cookies
## Prerequisites
You will need to make sure you Ingress targets exactly one Ingress
controller by specifying the [ingress.class annotation](/examples/PREREQUISITES.md#ingress-class),
and that you have an ingress controller [running](/examples/deployment) in your cluster.
You will also need to deploy multiple replicas of your application that show up as endpoints for the Service referenced in the Ingress object, to test session stickyness.
Using a deployment with only one replica doesn't set the 'sticky' cookie.
## Deployment
Session stickyness is achieved through 3 annotations on the Ingress, as shown in the [example](sticky-ingress.yaml).
|Name|Description|Values|
| --- | --- | --- |
|ingress.kubernetes.io/affinity|Sets the affinity type|string (in NGINX only ``cookie`` is possible|
|ingress.kubernetes.io/session-cookie-name|Name of the cookie that will be used|string (default to route)|
|ingress.kubernetes.io/session-cookie-hash|Type of hash that will be used in cookie value|sha1/md5/index|
You can create the ingress to test this
```console
$ kubectl create -f sticky-ingress.yaml
```
## Validation
You can confirm that the Ingress works.
```console
$ kubectl describe ing nginx-test
Name: nginx-test
Namespace: default
Address:
Default backend: default-http-backend:80 (10.180.0.4:8080,10.240.0.2:8080)
Rules:
Host Path Backends
---- ---- --------
stickyingress.example.com
/ nginx-service:80 (<none>)
Annotations:
affinity: cookie
session-cookie-hash: sha1
session-cookie-name: route
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
7s 7s 1 {nginx-ingress-controller } Normal CREATE default/nginx-test
$ curl -I http://stickyingress.example.com
HTTP/1.1 200 OK
Server: nginx/1.11.9
Date: Fri, 10 Feb 2017 14:11:12 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Set-Cookie: route=a9907b79b248140b56bb13723f72b67697baac3d; Path=/; HttpOnly
Last-Modified: Tue, 24 Jan 2017 14:02:19 GMT
ETag: "58875e6b-264"
Accept-Ranges: bytes
```
In the example above, you can see a line containing the 'Set-Cookie: route' setting the right defined stickness cookie.
This cookie is created by NGINX containing the hash of the used upstream in that request.
If the user changes this cookie, NGINX creates a new one and redirect the user to another upstream.
If the backend pool grows up NGINX will keep sending the requests through the same server of the first request, even if it's overloaded.
When the backend server is removed, the requests are then re-routed to another upstream server and NGINX creates a new cookie, as the previous hash became invalid.
When you have more than one Ingress Object pointing to the same Service, but one containing affinity configuration and other don't, the first created Ingress will be used.
This means that you can face the situation that you've configured Session Affinity in one Ingress and it doesn't reflects in NGINX configuration, because there is another Ingress Object pointing to the same service that doesn't configure this.

19
examples/affinity/cookie/nginx/sticky-ingress.yaml Normal file
View file

@ -0,0 +1,19 @@
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-test
annotations:
kubernetes.io/ingress.class: "nginx"
ingress.kubernetes.io/affinity: "cookie"
ingress.kubernetes.io/session-cookie-name: "route"
ingress.kubernetes.io/session-cookie-hash: "sha1"
spec:
rules:
- host: stickyingress.example.com
http:
paths:
- backend:
serviceName: nginx-service
servicePort: 80
path: /