DevFW-CICD/ingress-nginx-helm
16
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge branch 'master' into nginx-0.9.0-beta.8.serversnippet

Browse source
This commit is contained in:
decker 2017-09-14 11:47:06 +08:00 committed by GitHub
commit 5ff11f54f6
1956 changed files with 267614 additions and 352615 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
.gitignore vendored
View file

@ -2,6 +2,9 @@
._*
.DS_Store
# intellij files
.idea/*
# Eclipse files
.classpath
.project

24
.travis.yml
View file

@ -9,7 +9,7 @@ notifications:
email: true
go:
- 1.8.1
- 1.8.3
go_import_path: k8s.io/ingress
@ -19,15 +19,19 @@ env:
# docker run --rm caktux/travis-cli encrypt key=value -r kubernetes/ingress
- RELEASE="ci-${TRAVIS_BUILD_ID}"
install:
- go get github.com/golang/lint/golint
- go get github.com/mattn/goveralls
- go get github.com/modocache/gover
- if ! go get github.com/golang/tools/cmd/cover; then go get golang.org/x/tools/cmd/cover; fi
before_script:
- export PATH=$PATH:$PWD/hack/e2e-internal/
script:
- make fmt lint vet cover
#- make test-e2e
jobs:
include:
- stage: Static Check
script:
- go get github.com/golang/lint/golint
- make fmt lint vet
- stage: Coverage
script:
- go get github.com/mattn/goveralls
- go get github.com/modocache/gover
- if ! go get github.com/golang/tools/cmd/cover; then go get golang.org/x/tools/cmd/cover; fi
- make cover
#- make test-e2e

6
CONTRIBUTING.md
View file

@ -19,14 +19,14 @@ Follow either of the two links above to access the appropriate CLA and instructi
If you're new to the project and want to help, but don't know where to start, we have a semi-curated list of issues that should not need deep knowledge of the system. [Have a look and see if anything sounds interesting](https://github.com/kubernetes/ingress/issues?utf8=%E2%9C%93&q=is%3Aopen%20is%3Aissue%20label%3A%22help+wanted%22). Alternatively, read some of the docs on other controllers and try to write your own, file and fix any/all issues that come up, including gaps in documentation!
## Contributing A Patch
## Contributing a Patch
1. If you haven't already done so, sign a Contributor License Agreement (see details above).
1. Read the [Ingress development guide](docs/dev/README.md)
1. Read the [Ingress development guide](docs/dev/README.md).
1. Fork the desired repo, develop and test your code changes.
1. Submit a pull request.
All changes must be code reviewed. Coding conventions and standards are explained in the official [developer docs](https://github.com/kubernetes/kubernetes/tree/master/docs/devel). Expect reviewers to request that you avoid common [go style mistakes](https://github.com/golang/go/wiki/CodeReviewComments) in your PRs.
All changes must be code reviewed. Coding conventions and standards are explained in the official [developer docs](https://github.com/kubernetes/community/tree/master/contributors/devel). Expect reviewers to request that you avoid common [go style mistakes](https://github.com/golang/go/wiki/CodeReviewComments) in your PRs.
### Merge Approval

1445
Godeps/Godeps.json generated
View file

File diff suppressed because it is too large Load diff

8
Makefile
View file

@ -61,11 +61,15 @@ controllers:
.PHONY: docker-build
docker-build:
make -C controllers/nginx container
make -C controllers/nginx all-container
.PHONY: docker-push
docker-push:
make -C controllers/nginx push
make -C controllers/nginx all-push
.PHONE: release
release:
make -C controllers/nginx release
.PHONY: ginkgo
ginkgo:

1
README.md
View file

@ -3,6 +3,7 @@
[![Build Status](https://travis-ci.org/kubernetes/ingress.svg?branch=master)](https://travis-ci.org/kubernetes/ingress)
[![Coverage Status](https://coveralls.io/repos/github/kubernetes/ingress/badge.svg?branch=master)](https://coveralls.io/github/kubernetes/ingress)
[![Go Report Card](https://goreportcard.com/badge/github.com/kubernetes/ingress)](https://goreportcard.com/report/github.com/kubernetes/ingress)
[![GoDoc](https://godoc.org/github.com/kubernetes/ingress?status.svg)](https://godoc.org/github.com/kubernetes/ingress)
## Description

10
controllers/README.md
View file

@ -1,11 +1,11 @@
# Ingress controllers
This directory contains ingress controllers.
=======
# Ingress Controllers
This directory contains Ingress controllers.
Configuring a webserver or loadbalancer is harder than it should be. Most webserver configuration files are very similar. There are some applications that have weird little quirks that tend to throw a wrench in things, but for the most part you can apply the same logic to them and achieve a desired result. The Ingress resource embodies this idea, and an Ingress controller is meant to handle all the quirks associated with a specific "class" of Ingress (be it a single instance of a loadbalancer, or a more complicated setup of frontends that provide GSLB, DDoS protection etc).
Configuring a webserver or loadbalancer is harder than it should be. Most webserver configuration files are very similar. There are some applications that have weird little quirks that tend to throw a wrench in things, but for the most part you can apply the same logic to them and achieve a desired result.
The Ingress resource embodies this idea, and an Ingress controller is meant to handle all the quirks associated with a specific "class" of Ingress (be it a single instance of a loadbalancer, or a more complicated setup of frontends that provide GSLB, DDoS protection, etc).
## What is an Ingress Controller?
An Ingress Controller is a daemon, deployed as a Kubernetes Pod, that watches the apiserver's `/ingresses` endpoint for updates to the [Ingress resource](https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/ingress.md). Its job is to satisfy requests for ingress.
An Ingress Controller is a daemon, deployed as a Kubernetes Pod, that watches the apiserver's `/ingresses` endpoint for updates to the [Ingress resource](https://kubernetes.io/docs/concepts/services-networking/ingress/). Its job is to satisfy requests for Ingresses.

9
controllers/gce/BETA_LIMITATIONS.md
View file

@ -1,6 +1,6 @@
# GLBC: Beta limitations
As of the Kubernetes 1.2 release, the GCE L7 Loadbalancer controller is still a *beta* product. We expect it to go GA in 1.3.
As of the Kubernetes 1.7 release, the GCE L7 Loadbalancer controller is still a *beta* product.
This is a list of beta limitations:
@ -13,13 +13,13 @@ This is a list of beta limitations:
* [Large clusters](#large-clusters): Ingress on GCE isn't supported on large (>1000 nodes), single-zone clusters.
* [Teardown](README.md#deletion): The recommended way to tear down a cluster with active Ingresses is to either delete each Ingress, or hit the `/delete-all-and-quit` endpoint on GLBC, before invoking a cluster teardown script (eg: kube-down.sh). You will have to manually cleanup GCE resources through the [cloud console](https://cloud.google.com/compute/docs/console#access) or [gcloud CLI](https://cloud.google.com/compute/docs/gcloud-compute/) if you simply tear down the cluster with active Ingresses.
* [Changing UIDs](#changing-the-cluster-uid): You can change the UID used as a suffix for all your GCE cloud resources, but this requires you to delete existing Ingresses first.
* [Cleaning up](#cleaning-up-cloud-resources): You can delete loadbalancers that older clusters might've leaked due to permature teardown through the GCE console.
* [Cleaning up](#cleaning-up-cloud-resources): You can delete loadbalancers that older clusters might've leaked due to premature teardown through the GCE console.
## Prerequisites
Before you can receive traffic through the GCE L7 Loadbalancer Controller you need:
* A Working Kubernetes cluster >= 1.1
* At least 1 Kubernetes [NodePort Service](../../../../docs/user-guide/services.md#type-nodeport) (this is the endpoint for your Ingress)
* At least 1 Kubernetes NodePort Service (this is the endpoint for your Ingress)
* A single instance of the L7 Loadbalancer Controller pod, if you're running Kubernetes < 1.3 (the GCP ingress controller runs on the master in later versions)
## Quota
@ -172,7 +172,6 @@ If you deleted a GKE/GCE cluster without first deleting the associated Ingresses
1. Navigate to the [cloud console](https://console.cloud.google.com/) and click on the "Networking" tab, then choose "LoadBalancing"
2. Find the loadbalancer you'd like to delete, it should have a name formatted as: k8s-um-ns-name--UUID
3. Delete it, check the boxes to also casade the deletion down to associated resources (eg: backend-services)
3. Delete it, check the boxes to also cascade the deletion down to associated resources (eg: backend-services)
4. Switch to the "Compute Engine" tab, then choose "Instance Groups"
5. Delete the Instance Group allocated for the leaked Ingress, it should have a name formatted as: k8s-ig-UUID

2
controllers/gce/Makefile
View file

@ -1,7 +1,7 @@
all: push
# 0.0 shouldn't clobber any released builds
TAG = 0.9.4
TAG = 0.9.6
PREFIX = gcr.io/google_containers/glbc
server:

20
controllers/gce/README.md
View file

@ -53,7 +53,7 @@ __Lines 8-9__: Each http rule contains the following information: A host (eg: fo
__Lines 10-12__: A `backend` is a service:port combination. It selects a group of pods capable of servicing traffic sent to the path specified in the parent rule. The `port` is the desired `spec.ports[*].port` from the Service Spec -- Note, though, that the L7 actually directs traffic to the corresponding `NodePort`.
__Global Prameters__: For the sake of simplicity the example Ingress has no global parameters. However, one can specify a default backend (see examples below) in the absence of which requests that don't match a path in the spec are sent to the default backend of glbc. Though glbc doesn't support HTTPS yet, security configs would also be global.
__Global Parameters__: For the sake of simplicity the example Ingress has no global parameters. However, one can specify a default backend (see examples below) in the absence of which requests that don't match a path in the spec are sent to the default backend of glbc.
## Load Balancer Management
@ -135,7 +135,7 @@ Go to your GCE console and confirm that the following resources have been create
* BackendServices (one for each Kubernetes nodePort service)
* An Instance Group (with ports corresponding to the BackendServices)
The HTTPLoadBalancing panel will also show you if your backends have responded to the health checks, wait till they do. This can take a few minutes. If you see `Health status will display here once configuration is complete.` the L7 is still bootstrapping. Wait till you have `Healthy instances: X`. Even though the GCE L7 is driven by our controller, which notices the Kubernetes healtchecks of a pod, we still need to wait on the first GCE L7 health check to complete. Once your backends are up and healthy:
The HTTPLoadBalancing panel will also show you if your backends have responded to the health checks, wait till they do. This can take a few minutes. If you see `Health status will display here once configuration is complete.` the L7 is still bootstrapping. Wait till you have `Healthy instances: X`. Even though the GCE L7 is driven by our controller, which notices the Kubernetes healthchecks of a pod, we still need to wait on the first GCE L7 health check to complete. Once your backends are up and healthy:
```shell
$ curl --resolve foo.bar.com:80:107.178.245.239 http://foo.bar.com/foo
@ -245,7 +245,7 @@ spec:
app: nginxtest
```
Running kubectl create against this manifest will given you a service with multiple endpoints:
Running kubectl create against this manifest will give you a service with multiple endpoints:
```shell
$ kubectl get svc nginxtest -o yaml | grep -i nodeport:
nodePort: 30404
@ -281,7 +281,7 @@ nginx-tester-pod-name
```
Note what just happened, the endpoint exposes /hostname, and the loadbalancer forwarded the entire matching url to the endpoint. This means if you had '/foo' in the Ingress and tried accessing /hostname, your endpoint would've received /foo/hostname and not known how to route it. Now update the Ingress to access static content via the /fs endpoint:
```
```yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
@ -300,7 +300,7 @@ As before, wait a while for the update to take effect, and try accessing `loadba
#### Deletion
Most production loadbalancers live as long as the nodes in the cluster and are torn down when the nodes are destroyed. That said, there are plenty of use cases for deleting an Ingress, deleting a loadbalancer controller, or just purging external loadbalancer resources alltogether. Deleting a loadbalancer controller pod will not affect the loadbalancers themselves, this way your backends won't suffer a loss of availability if the scheduler pre-empts your controller pod. Deleting a single loadbalancer is as easy as deleting an Ingress via kubectl:
Most production loadbalancers live as long as the nodes in the cluster and are torn down when the nodes are destroyed. That said, there are plenty of use cases for deleting an Ingress, deleting a loadbalancer controller, or just purging external loadbalancer resources altogether. Deleting a loadbalancer controller pod will not affect the loadbalancers themselves, this way your backends won't suffer a loss of availability if the scheduler pre-empts your controller pod. Deleting a single loadbalancer is as easy as deleting an Ingress via kubectl:
```shell
$ kubectl delete ing echomap
$ kubectl logs --follow glbc-6m6b6 l7-lb-controller
@ -313,7 +313,7 @@ I1007 00:26:02.043188 1 backends.go:134] Deleting backend k8-be-30301
I1007 00:26:05.591140 1 backends.go:134] Deleting backend k8-be-30284
I1007 00:26:09.159016 1 controller.go:232] Finished syncing default/echomap
```
Note that it takes ~30 seconds to purge cloud resources, the API calls to create and delete are a one time cost. GCE BackendServices are ref-counted and deleted by the controller as you delete Kubernetes Ingress'. This is not sufficient for cleanup, because you might have deleted the Ingress while glbc was down, in which case it would leak cloud resources. You can delete the glbc and purge cloud resources in 2 more ways:
Note that it takes ~30 seconds to purge cloud resources, the API calls to create and delete are a onetime cost. GCE BackendServices are ref-counted and deleted by the controller as you delete Kubernetes Ingress'. This is not sufficient for cleanup, because you might have deleted the Ingress while glbc was down, in which case it would leak cloud resources. You can delete the glbc and purge cloud resources in 2 more ways:
__The dev/test way__: If you want to delete everything in the cloud when the loadbalancer controller pod dies, start it with the --delete-all-on-quit flag. When a pod is killed it's first sent a SIGTERM, followed by a grace period (set to 10minutes for loadbalancer controllers), followed by a SIGKILL. The controller pod uses this time to delete cloud resources. Be careful with --delete-all-on-quit, because if you're running a production glbc and the scheduler re-schedules your pod for some reason, it will result in a loss of availability. You can do this because your rc.yaml has:
```yaml
@ -327,7 +327,7 @@ So simply delete the replication controller:
$ kubectl get rc glbc
CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGE
glbc default-http-backend gcr.io/google_containers/defaultbackend:1.0 k8s-app=glbc,version=v0.5 1 2m
l7-lb-controller gcr.io/google_containers/glbc:0.9.4
l7-lb-controller gcr.io/google_containers/glbc:0.9.6
$ kubectl delete rc glbc
replicationcontroller "glbc" deleted
@ -339,7 +339,7 @@ glbc-6m6b6 1/1 Terminating 0 13m
__The prod way__: If you didn't start the controller with `--delete-all-on-quit`, you can execute a GET on the `/delete-all-and-quit` endpoint. This endpoint is deliberately not exported.
```
```shell
$ kubectl exec -it glbc-6m6b6 -- wget -q -O- http://localhost:8081/delete-all-and-quit
..Hangs till quit is done..
@ -399,7 +399,7 @@ spec:
This creates 2 GCE forwarding rules that use a single static ip. Both `:80` and `:443` will direct traffic to your backend, which serves HTTP requests on the target port mentioned in the Service associated with the Ingress.
## Backend HTTPS
For encrypted communication between the load balancer and your Kubernetes service, you need to decorate the the service's port as expecting HTTPS. There's an alpha [Service annotation](examples/backside_https/app.yaml) for specifying the expected protocol per service port. Upon seeing the protocol as HTTPS, the ingress controller will assemble a GCP L7 load balancer with an HTTPS backend-service with a HTTPS health check.
For encrypted communication between the load balancer and your Kubernetes service, you need to decorate the service's port as expecting HTTPS. There's an alpha [Service annotation](examples/backside_https/app.yaml) for specifying the expected protocol per service port. Upon seeing the protocol as HTTPS, the ingress controller will assemble a GCP L7 load balancer with an HTTPS backend-service with a HTTPS health check.
The annotation value is a stringified JSON map of port-name to "HTTPS" or "HTTP". If you do not specify the port, "HTTP" is assumed.
```yaml
@ -698,7 +698,7 @@ The controller manages cloud resources through a notion of pools. Each pool is t
Periodically, each pool checks that it has a valid connection to the next hop in the above resource graph. So for example, the backend pool will check that each backend is connected to the instance group and that the node ports match, the instance group will check that all the Kubernetes nodes are a part of the instance group, and so on. Since Backends are a limited resource, they're shared (well, everything is limited by your quota, this applies doubly to backend services). This means you can setup N Ingress' exposing M services through different paths and the controller will only create M backends. When all the Ingress' are deleted, the backend pool GCs the backend.
## Wishlist:
## Wish list:
* More E2e, integration tests
* Better events

77
controllers/gce/backends/backends.go
View file

@ -26,10 +26,11 @@ import (
"github.com/golang/glog"
compute "google.golang.org/api/compute/v1"
v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/intstr"
"k8s.io/apimachinery/pkg/util/sets"
api_v1 "k8s.io/client-go/pkg/api/v1"
"k8s.io/ingress/controllers/gce/healthchecks"
"k8s.io/ingress/controllers/gce/instances"
@ -161,7 +162,7 @@ func (b *Backends) Init(pp probeProvider) {
// Get returns a single backend.
func (b *Backends) Get(port int64) (*compute.BackendService, error) {
be, err := b.cloud.GetBackendService(b.namer.BeName(port))
be, err := b.cloud.GetGlobalBackendService(b.namer.BeName(port))
if err != nil {
return nil, err
}
@ -171,13 +172,22 @@ func (b *Backends) Get(port int64) (*compute.BackendService, error) {
func (b *Backends) ensureHealthCheck(sp ServicePort) (string, error) {
hc := b.healthChecker.New(sp.Port, sp.Protocol)
if b.prober != nil {
existingLegacyHC, err := b.healthChecker.GetLegacy(sp.Port)
if err != nil && !utils.IsNotFoundError(err) {
return "", err
}
if existingLegacyHC != nil {
glog.V(4).Infof("Applying settings of existing health check to newer health check on port %+v", sp)
applyLegacyHCToHC(existingLegacyHC, hc)
} else if b.prober != nil {
probe, err := b.prober.GetProbe(sp)
if err != nil {
return "", err
}
if probe != nil {
glog.V(2).Infof("Applying httpGet settings of readinessProbe to health check on port %+v", sp)
glog.V(4).Infof("Applying httpGet settings of readinessProbe to health check on port %+v", sp)
applyProbeSettingsToHC(probe, hc)
}
}
@ -194,22 +204,29 @@ func (b *Backends) create(namedPort *compute.NamedPort, hcLink string, sp Servic
Port: namedPort.Port,
PortName: namedPort.Name,
}
if err := b.cloud.CreateBackendService(bs); err != nil {
if err := b.cloud.CreateGlobalBackendService(bs); err != nil {
return nil, err
}
return b.Get(namedPort.Port)
}
// Add will get or create a Backend for the given port.
func (b *Backends) Add(p ServicePort) error {
// Uses the given instance groups if non-nil, else creates instance groups.
func (b *Backends) Add(p ServicePort, igs []*compute.InstanceGroup) error {
// We must track the port even if creating the backend failed, because
// we might've created a health-check for it.
be := &compute.BackendService{}
defer func() { b.snapshotter.Add(portKey(p.Port), be) }()
igs, namedPort, err := b.nodePool.AddInstanceGroup(b.namer.IGName(), p.Port)
if err != nil {
return err
var err error
// Ideally callers should pass the instance groups to prevent recomputing them here.
// Igs can be nil in scenarios where we do not have instance groups such as
// while syncing default backend service.
if igs == nil {
igs, _, err = instances.EnsureInstanceGroupsAndPorts(b.nodePool, b.namer, p.Port)
if err != nil {
return err
}
}
// Ensure health check for backend service exists
@ -222,6 +239,7 @@ func (b *Backends) Add(p ServicePort) error {
pName := b.namer.BeName(p.Port)
be, _ = b.Get(p.Port)
if be == nil {
namedPort := utils.GetNamedPort(p.Port)
glog.V(2).Infof("Creating backend service for port %v named port %v", p.Port, namedPort)
be, err = b.create(namedPort, hcLink, p, pName)
if err != nil {
@ -239,7 +257,7 @@ func (b *Backends) Add(p ServicePort) error {
be.Protocol = string(p.Protocol)
be.HealthChecks = []string{hcLink}
be.Description = p.Description()
if err = b.cloud.UpdateBackendService(be); err != nil {
if err = b.cloud.UpdateGlobalBackendService(be); err != nil {
return err
}
}
@ -247,7 +265,7 @@ func (b *Backends) Add(p ServicePort) error {
// If previous health check was legacy type, we need to delete it.
if existingHCLink != hcLink && strings.Contains(existingHCLink, "/httpHealthChecks/") {
if err = b.healthChecker.DeleteLegacy(p.Port); err != nil {
return err
glog.Warning("Failed to delete legacy HttpHealthCheck %v; Will not try again, err: %v", pName, err)
}
}
@ -273,7 +291,7 @@ func (b *Backends) Delete(port int64) (err error) {
}
}()
// Try deleting health checks even if a backend is not found.
if err = b.cloud.DeleteBackendService(name); err != nil && !utils.IsHTTPErrorCode(err, http.StatusNotFound) {
if err = b.cloud.DeleteGlobalBackendService(name); err != nil && !utils.IsHTTPErrorCode(err, http.StatusNotFound) {
return err
}
@ -285,7 +303,7 @@ func (b *Backends) List() ([]interface{}, error) {
// TODO: for consistency with the rest of this sub-package this method
// should return a list of backend ports.
interList := []interface{}{}
be, err := b.cloud.ListBackendServices()
be, err := b.cloud.ListGlobalBackendServices()
if err != nil {
return interList, err
}
@ -352,7 +370,7 @@ func (b *Backends) edgeHop(be *compute.BackendService, igs []*compute.InstanceGr
newBackends := getBackendsForIGs(addIGs, bm)
be.Backends = append(originalBackends, newBackends...)
if err := b.cloud.UpdateBackendService(be); err != nil {
if err := b.cloud.UpdateGlobalBackendService(be); err != nil {
if utils.IsHTTPErrorCode(err, http.StatusBadRequest) {
glog.V(2).Infof("Updating backend service backends with balancing mode %v failed, will try another mode. err:%v", bm, err)
errs = append(errs, err.Error())
@ -371,12 +389,12 @@ func (b *Backends) edgeHop(be *compute.BackendService, igs []*compute.InstanceGr
}
// Sync syncs backend services corresponding to ports in the given list.
func (b *Backends) Sync(svcNodePorts []ServicePort) error {
func (b *Backends) Sync(svcNodePorts []ServicePort, igs []*compute.InstanceGroup) error {
glog.V(3).Infof("Sync: backends %v", svcNodePorts)
// create backends for new ports, perform an edge hop for existing ports
for _, port := range svcNodePorts {
if err := b.Add(port); err != nil {
if err := b.Add(port, igs); err != nil {
return err
}
}
@ -418,14 +436,14 @@ func (b *Backends) Shutdown() error {
// Status returns the status of the given backend by name.
func (b *Backends) Status(name string) string {
backend, err := b.cloud.GetBackendService(name)
backend, err := b.cloud.GetGlobalBackendService(name)
if err != nil || len(backend.Backends) == 0 {
return "Unknown"
}
// TODO: Look at more than one backend's status
// TODO: Include port, ip in the status, since it's in the health info.
hs, err := b.cloud.GetHealth(name, backend.Backends[0].Group)
hs, err := b.cloud.GetGlobalBackendServiceHealth(name, backend.Backends[0].Group)
if err != nil || len(hs.HealthStatus) == 0 || hs.HealthStatus[0] == nil {
return "Unknown"
}
@ -433,15 +451,34 @@ func (b *Backends) Status(name string) string {
return hs.HealthStatus[0].HealthState
}
func applyProbeSettingsToHC(p *api_v1.Probe, hc *healthchecks.HealthCheck) {
func applyLegacyHCToHC(existing *compute.HttpHealthCheck, hc *healthchecks.HealthCheck) {
hc.Description = existing.Description
hc.CheckIntervalSec = existing.CheckIntervalSec
hc.HealthyThreshold = existing.HealthyThreshold
hc.Host = existing.Host
hc.Port = existing.Port
hc.RequestPath = existing.RequestPath
hc.TimeoutSec = existing.TimeoutSec
hc.UnhealthyThreshold = existing.UnhealthyThreshold
}
func applyProbeSettingsToHC(p *v1.Probe, hc *healthchecks.HealthCheck) {
healthPath := p.Handler.HTTPGet.Path
// GCE requires a leading "/" for health check urls.
if !strings.HasPrefix(healthPath, "/") {
healthPath = "/" + healthPath
}
// Extract host from HTTP headers
host := p.Handler.HTTPGet.Host
for _, header := range p.Handler.HTTPGet.HTTPHeaders {
if header.Name == "Host" {
host = header.Value
break
}
}
hc.RequestPath = healthPath
hc.Host = p.Handler.HTTPGet.Host
hc.Host = host
hc.Description = "Kubernetes L7 health check generated with readiness probe settings."
hc.CheckIntervalSec = int64(p.PeriodSeconds) + int64(healthchecks.DefaultHealthCheckInterval.Seconds())
hc.TimeoutSec = int64(p.TimeoutSeconds)

119
controllers/gce/backends/backends_test.go
View file

@ -23,9 +23,9 @@ import (
compute "google.golang.org/api/compute/v1"
"google.golang.org/api/googleapi"
api_v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/util/intstr"
"k8s.io/apimachinery/pkg/util/sets"
api_v1 "k8s.io/client-go/pkg/api/v1"
"k8s.io/ingress/controllers/gce/healthchecks"
"k8s.io/ingress/controllers/gce/instances"
@ -50,21 +50,23 @@ var existingProbe = &api_v1.Probe{
},
}
func newBackendPool(f BackendServices, fakeIGs instances.InstanceGroups, syncWithCloud bool) *Backends {
func newTestJig(f BackendServices, fakeIGs instances.InstanceGroups, syncWithCloud bool) (*Backends, healthchecks.HealthCheckProvider) {
namer := &utils.Namer{}
nodePool := instances.NewNodePool(fakeIGs)
nodePool.Init(&instances.FakeZoneLister{Zones: []string{defaultZone}})
healthChecks := healthchecks.NewHealthChecker(healthchecks.NewFakeHealthCheckProvider(), "/", namer)
healthCheckProvider := healthchecks.NewFakeHealthCheckProvider()
healthChecks := healthchecks.NewHealthChecker(healthCheckProvider, "/", namer)
bp := NewBackendPool(f, healthChecks, nodePool, namer, []int64{}, syncWithCloud)
probes := map[ServicePort]*api_v1.Probe{{Port: 443, Protocol: utils.ProtocolHTTPS}: existingProbe}
bp.Init(NewFakeProbeProvider(probes))
return bp
return bp, healthCheckProvider
}
func TestBackendPoolAdd(t *testing.T) {
f := NewFakeBackendServices(noOpErrFunc)
fakeIGs := instances.NewFakeInstanceGroups(sets.NewString())
pool := newBackendPool(f, fakeIGs, false)
pool, _ := newTestJig(f, fakeIGs, false)
namer := utils.Namer{}
testCases := []ServicePort{
@ -78,14 +80,14 @@ func TestBackendPoolAdd(t *testing.T) {
// Add a backend for a port, then re-add the same port and
// make sure it corrects a broken link from the backend to
// the instance group.
err := pool.Add(nodePort)
err := pool.Add(nodePort, nil)
if err != nil {
t.Fatalf("Did not find expect error when adding a nodeport: %v, err: %v", nodePort, err)
}
beName := namer.BeName(nodePort.Port)
// Check that the new backend has the right port
be, err := f.GetBackendService(beName)
be, err := f.GetGlobalBackendService(beName)
if err != nil {
t.Fatalf("Did not find expected backend %v", beName)
}
@ -105,7 +107,6 @@ func TestBackendPoolAdd(t *testing.T) {
}
// Check the created healthcheck is the correct protocol
// pool.healthChecker.
hc, err := pool.healthChecker.Get(nodePort.Port)
if err != nil {
t.Fatalf("Unexpected err when querying fake healthchecker: %v", err)
@ -122,17 +123,55 @@ func TestBackendPoolAdd(t *testing.T) {
}
}
func TestHealthCheckMigration(t *testing.T) {
f := NewFakeBackendServices(noOpErrFunc)
fakeIGs := instances.NewFakeInstanceGroups(sets.NewString())
pool, hcp := newTestJig(f, fakeIGs, false)
namer := utils.Namer{}
p := ServicePort{Port: 7000, Protocol: utils.ProtocolHTTP}
// Create a legacy health check and insert it into the HC provider.
legacyHC := &compute.HttpHealthCheck{
Name: namer.BeName(p.Port),
RequestPath: "/my-healthz-path",
Host: "k8s.io",
Description: "My custom HC",
UnhealthyThreshold: 30,
CheckIntervalSec: 40,
}
hcp.CreateHttpHealthCheck(legacyHC)
// Add the service port to the backend pool
pool.Add(p, nil)
// Assert the proper health check was created
hc, _ := pool.healthChecker.Get(p.Port)
if hc == nil || hc.Protocol() != p.Protocol {
t.Fatalf("Expected %s health check, received %v: ", p.Protocol, hc)
}
// Assert the newer health check has the legacy health check settings
if hc.RequestPath != legacyHC.RequestPath ||
hc.Host != legacyHC.Host ||
hc.UnhealthyThreshold != legacyHC.UnhealthyThreshold ||
hc.CheckIntervalSec != legacyHC.CheckIntervalSec ||
hc.Description != legacyHC.Description {
t.Fatalf("Expected newer health check to have identical settings to legacy health check. Legacy: %+v, New: %+v", legacyHC, hc)
}
}
func TestBackendPoolUpdate(t *testing.T) {
f := NewFakeBackendServices(noOpErrFunc)
fakeIGs := instances.NewFakeInstanceGroups(sets.NewString())
pool := newBackendPool(f, fakeIGs, false)
pool, _ := newTestJig(f, fakeIGs, false)
namer := utils.Namer{}
p := ServicePort{Port: 3000, Protocol: utils.ProtocolHTTP}
pool.Add(p)
pool.Add(p, nil)
beName := namer.BeName(p.Port)
be, err := f.GetBackendService(beName)
be, err := f.GetGlobalBackendService(beName)
if err != nil {
t.Fatalf("Unexpected err: %v", err)
}
@ -149,9 +188,9 @@ func TestBackendPoolUpdate(t *testing.T) {
// Update service port to encrypted
p.Protocol = utils.ProtocolHTTPS
pool.Sync([]ServicePort{p})
pool.Sync([]ServicePort{p}, nil)
be, err = f.GetBackendService(beName)
be, err = f.GetGlobalBackendService(beName)
if err != nil {
t.Fatalf("Unexpected err retrieving backend service after update: %v", err)
}
@ -171,14 +210,14 @@ func TestBackendPoolUpdate(t *testing.T) {
func TestBackendPoolChaosMonkey(t *testing.T) {
f := NewFakeBackendServices(noOpErrFunc)
fakeIGs := instances.NewFakeInstanceGroups(sets.NewString())
pool := newBackendPool(f, fakeIGs, false)
pool, _ := newTestJig(f, fakeIGs, false)
namer := utils.Namer{}
nodePort := ServicePort{Port: 8080, Protocol: utils.ProtocolHTTP}
pool.Add(nodePort)
pool.Add(nodePort, nil)
beName := namer.BeName(nodePort.Port)
be, _ := f.GetBackendService(beName)
be, _ := f.GetGlobalBackendService(beName)
// Mess up the link between backend service and instance group.
// This simulates a user doing foolish things through the UI.
@ -186,15 +225,15 @@ func TestBackendPoolChaosMonkey(t *testing.T) {
{Group: "test edge hop"},
}
f.calls = []int{}
f.UpdateBackendService(be)
f.UpdateGlobalBackendService(be)
pool.Add(nodePort)
pool.Add(nodePort, nil)
for _, call := range f.calls {
if call == utils.Create {
t.Fatalf("Unexpected create for existing backend service")
}
}
gotBackend, err := f.GetBackendService(beName)
gotBackend, err := f.GetGlobalBackendService(beName)
if err != nil {
t.Fatalf("Failed to find a backend with name %v: %v", beName, err)
}
@ -220,10 +259,10 @@ func TestBackendPoolSync(t *testing.T) {
svcNodePorts := []ServicePort{{Port: 81, Protocol: utils.ProtocolHTTP}, {Port: 82, Protocol: utils.ProtocolHTTPS}, {Port: 83, Protocol: utils.ProtocolHTTP}}
f := NewFakeBackendServices(noOpErrFunc)
fakeIGs := instances.NewFakeInstanceGroups(sets.NewString())
pool := newBackendPool(f, fakeIGs, true)
pool.Add(ServicePort{Port: 81})
pool.Add(ServicePort{Port: 90})
if err := pool.Sync(svcNodePorts); err != nil {
pool, _ := newTestJig(f, fakeIGs, true)
pool.Add(ServicePort{Port: 81}, nil)
pool.Add(ServicePort{Port: 90}, nil)
if err := pool.Sync(svcNodePorts, nil); err != nil {
t.Errorf("Expected backend pool to sync, err: %v", err)
}
if err := pool.GC(svcNodePorts); err != nil {
@ -257,12 +296,12 @@ func TestBackendPoolSync(t *testing.T) {
// k8s-be-3001--uid - another cluster tagged with uid
unrelatedBackends := sets.NewString([]string{"foo", "k8s-be-foo", "k8s--bar--foo", "k8s-be-30001--uid"}...)
for _, name := range unrelatedBackends.List() {
f.CreateBackendService(&compute.BackendService{Name: name})
f.CreateGlobalBackendService(&compute.BackendService{Name: name})
}
namer := &utils.Namer{}
// This backend should get deleted again since it is managed by this cluster.
f.CreateBackendService(&compute.BackendService{Name: namer.BeName(deletedPorts[0].Port)})
f.CreateGlobalBackendService(&compute.BackendService{Name: namer.BeName(deletedPorts[0].Port)})
// TODO: Avoid casting.
// Repopulate the pool with a cloud list, which now includes the 82 port
@ -272,7 +311,7 @@ func TestBackendPoolSync(t *testing.T) {
pool.GC(svcNodePorts)
currBackends, _ := f.ListBackendServices()
currBackends, _ := f.ListGlobalBackendServices()
currSet := sets.NewString()
for _, b := range currBackends.Items {
currSet.Insert(b.Name)
@ -316,13 +355,13 @@ func TestBackendPoolDeleteLegacyHealthChecks(t *testing.T) {
}
// Create backend service with expected name and link to legacy health check
f.CreateBackendService(&compute.BackendService{
f.CreateGlobalBackendService(&compute.BackendService{
Name: beName,
HealthChecks: []string{hc.SelfLink},
})
// Have pool sync the above backend service
bp.Add(ServicePort{Port: 80, Protocol: utils.ProtocolHTTPS})
bp.Add(ServicePort{Port: 80, Protocol: utils.ProtocolHTTPS}, nil)
// Verify the legacy health check has been deleted
_, err = hcp.GetHttpHealthCheck(beName)
@ -345,13 +384,13 @@ func TestBackendPoolDeleteLegacyHealthChecks(t *testing.T) {
func TestBackendPoolShutdown(t *testing.T) {
f := NewFakeBackendServices(noOpErrFunc)
fakeIGs := instances.NewFakeInstanceGroups(sets.NewString())
pool := newBackendPool(f, fakeIGs, false)
pool, _ := newTestJig(f, fakeIGs, false)
namer := utils.Namer{}
// Add a backend-service and verify that it doesn't exist after Shutdown()
pool.Add(ServicePort{Port: 80})
pool.Add(ServicePort{Port: 80}, nil)
pool.Shutdown()
if _, err := f.GetBackendService(namer.BeName(80)); err == nil {
if _, err := f.GetGlobalBackendService(namer.BeName(80)); err == nil {
t.Fatalf("%v", err)
}
}
@ -359,13 +398,13 @@ func TestBackendPoolShutdown(t *testing.T) {
func TestBackendInstanceGroupClobbering(t *testing.T) {
f := NewFakeBackendServices(noOpErrFunc)
fakeIGs := instances.NewFakeInstanceGroups(sets.NewString())
pool := newBackendPool(f, fakeIGs, false)
pool, _ := newTestJig(f, fakeIGs, false)
namer := utils.Namer{}
// This will add the instance group k8s-ig to the instance pool
pool.Add(ServicePort{Port: 80})
pool.Add(ServicePort{Port: 80}, nil)
be, err := f.GetBackendService(namer.BeName(80))
be, err := f.GetGlobalBackendService(namer.BeName(80))
if err != nil {
t.Fatalf("%v", err)
}
@ -376,13 +415,13 @@ func TestBackendInstanceGroupClobbering(t *testing.T) {
{Group: "k8s-ig-foo"},
}
be.Backends = append(be.Backends, newGroups...)
if err = f.UpdateBackendService(be); err != nil {
if err = f.UpdateGlobalBackendService(be); err != nil {
t.Fatalf("Failed to update backend service %v", be.Name)
}
// Make sure repeated adds don't clobber the inserted instance group
pool.Add(ServicePort{Port: 80})
be, err = f.GetBackendService(namer.BeName(80))
pool.Add(ServicePort{Port: 80}, nil)
be, err = f.GetGlobalBackendService(namer.BeName(80))
if err != nil {
t.Fatalf("%v", err)
}
@ -405,7 +444,7 @@ func TestBackendCreateBalancingMode(t *testing.T) {
f := NewFakeBackendServices(noOpErrFunc)
fakeIGs := instances.NewFakeInstanceGroups(sets.NewString())
pool := newBackendPool(f, fakeIGs, false)
pool, _ := newTestJig(f, fakeIGs, false)
namer := utils.Namer{}
nodePort := ServicePort{Port: 8080}
modes := []BalancingMode{Rate, Utilization}
@ -423,8 +462,8 @@ func TestBackendCreateBalancingMode(t *testing.T) {
return nil
}
pool.Add(nodePort)
be, err := f.GetBackendService(namer.BeName(nodePort.Port))
pool.Add(nodePort, nil)
be, err := f.GetGlobalBackendService(namer.BeName(nodePort.Port))
if err != nil {
t.Fatalf("%v", err)
}

28
controllers/gce/backends/fakes.go
View file

@ -20,7 +20,7 @@ import (
"fmt"
compute "google.golang.org/api/compute/v1"
api_v1 "k8s.io/client-go/pkg/api/v1"
api_v1 "k8s.io/api/core/v1"
"k8s.io/client-go/tools/cache"
"k8s.io/ingress/controllers/gce/utils"
@ -44,8 +44,8 @@ type FakeBackendServices struct {
errFunc func(op int, be *compute.BackendService) error
}
// GetBackendService fakes getting a backend service from the cloud.
func (f *FakeBackendServices) GetBackendService(name string) (*compute.BackendService, error) {
// GetGlobalBackendService fakes getting a backend service from the cloud.
func (f *FakeBackendServices) GetGlobalBackendService(name string) (*compute.BackendService, error) {
f.calls = append(f.calls, utils.Get)
obj, exists, err := f.backendServices.GetByKey(name)
if !exists {
@ -62,8 +62,8 @@ func (f *FakeBackendServices) GetBackendService(name string) (*compute.BackendSe
return nil, fmt.Errorf("backend service %v not found", name)
}
// CreateBackendService fakes backend service creation.
func (f *FakeBackendServices) CreateBackendService(be *compute.BackendService) error {
// CreateGlobalBackendService fakes backend service creation.
func (f *FakeBackendServices) CreateGlobalBackendService(be *compute.BackendService) error {
if f.errFunc != nil {
if err := f.errFunc(utils.Create, be); err != nil {
return err
@ -74,8 +74,8 @@ func (f *FakeBackendServices) CreateBackendService(be *compute.BackendService) e
return f.backendServices.Update(be)
}
// DeleteBackendService fakes backend service deletion.
func (f *FakeBackendServices) DeleteBackendService(name string) error {
// DeleteGlobalBackendService fakes backend service deletion.
func (f *FakeBackendServices) DeleteGlobalBackendService(name string) error {
f.calls = append(f.calls, utils.Delete)
svc, exists, err := f.backendServices.GetByKey(name)
if !exists {
@ -87,8 +87,8 @@ func (f *FakeBackendServices) DeleteBackendService(name string) error {
return f.backendServices.Delete(svc)
}
// ListBackendServices fakes backend service listing.
func (f *FakeBackendServices) ListBackendServices() (*compute.BackendServiceList, error) {
// ListGlobalBackendServices fakes backend service listing.
func (f *FakeBackendServices) ListGlobalBackendServices() (*compute.BackendServiceList, error) {
var svcs []*compute.BackendService
for _, s := range f.backendServices.List() {
svc := s.(*compute.BackendService)
@ -97,8 +97,8 @@ func (f *FakeBackendServices) ListBackendServices() (*compute.BackendServiceList
return &compute.BackendServiceList{Items: svcs}, nil
}
// UpdateBackendService fakes updating a backend service.
func (f *FakeBackendServices) UpdateBackendService(be *compute.BackendService) error {
// UpdateGlobalBackendService fakes updating a backend service.
func (f *FakeBackendServices) UpdateGlobalBackendService(be *compute.BackendService) error {
if f.errFunc != nil {
if err := f.errFunc(utils.Update, be); err != nil {
return err
@ -108,9 +108,9 @@ func (f *FakeBackendServices) UpdateBackendService(be *compute.BackendService) e
return f.backendServices.Update(be)
}
// GetHealth fakes getting backend service health.
func (f *FakeBackendServices) GetHealth(name, instanceGroupLink string) (*compute.BackendServiceGroupHealth, error) {
be, err := f.GetBackendService(name)
// GetGlobalBackendServiceHealth fakes getting backend service health.
func (f *FakeBackendServices) GetGlobalBackendServiceHealth(name, instanceGroupLink string) (*compute.BackendServiceGroupHealth, error) {
be, err := f.GetGlobalBackendService(name)
if err != nil {
return nil, err
}

18
controllers/gce/backends/interfaces.go
View file

@ -18,7 +18,7 @@ package backends
import (
compute "google.golang.org/api/compute/v1"
api_v1 "k8s.io/client-go/pkg/api/v1"
api_v1 "k8s.io/api/core/v1"
)
// ProbeProvider retrieves a probe struct given a nodePort
@ -30,10 +30,10 @@ type probeProvider interface {
// as gce backendServices, and sync them through the BackendServices interface.
type BackendPool interface {
Init(p probeProvider)
Add(port ServicePort) error
Add(port ServicePort, igs []*compute.InstanceGroup) error
Get(port int64) (*compute.BackendService, error)
Delete(port int64) error
Sync(ports []ServicePort) error
Sync(ports []ServicePort, igs []*compute.InstanceGroup) error
GC(ports []ServicePort) error
Shutdown() error
Status(name string) string
@ -42,10 +42,10 @@ type BackendPool interface {
// BackendServices is an interface for managing gce backend services.
type BackendServices interface {
GetBackendService(name string) (*compute.BackendService, error)
UpdateBackendService(bg *compute.BackendService) error
CreateBackendService(bg *compute.BackendService) error
DeleteBackendService(name string) error
ListBackendServices() (*compute.BackendServiceList, error)
GetHealth(name, instanceGroupLink string) (*compute.BackendServiceGroupHealth, error)
GetGlobalBackendService(name string) (*compute.BackendService, error)
UpdateGlobalBackendService(bg *compute.BackendService) error
CreateGlobalBackendService(bg *compute.BackendService) error
DeleteGlobalBackendService(name string) error
ListGlobalBackendServices() (*compute.BackendServiceList, error)
GetGlobalBackendServiceHealth(name, instanceGroupLink string) (*compute.BackendServiceGroupHealth, error)
}

115
controllers/gce/controller/cluster_manager.go
View file

@ -17,14 +17,11 @@ limitations under the License.
package controller
import (
"io"
"net/http"
"os"
"time"
"github.com/golang/glog"
"k8s.io/kubernetes/pkg/cloudprovider"
compute "google.golang.org/api/compute/v1"
gce "k8s.io/kubernetes/pkg/cloudprovider/providers/gce"
"k8s.io/ingress/controllers/gce/backends"
@ -57,9 +54,6 @@ const (
// Names longer than this are truncated, because of GCE restrictions.
nameLenLimit = 62
// Sleep interval to retry cloud client creation.
cloudClientRetryInterval = 10 * time.Second
)
// ClusterManager manages cluster resource pools.
@ -115,41 +109,45 @@ func (c *ClusterManager) shutdown() error {
}
// Checkpoint performs a checkpoint with the cloud.
// - lbNames are the names of L7 loadbalancers we wish to exist. If they already
// - lbs are the single cluster L7 loadbalancers we wish to exist. If they already
// exist, they should not have any broken links between say, a UrlMap and
// TargetHttpProxy.
// - nodeNames are the names of nodes we wish to add to all loadbalancer
// instance groups.
// - nodePorts are the ports for which we require BackendServices. Each of
// these ports must also be opened on the corresponding Instance Group.
// - backendServicePorts are the ports for which we require BackendServices.
// - namedPorts are the ports which must be opened on instance groups.
// Returns the list of all instance groups corresponding to the given loadbalancers.
// If in performing the checkpoint the cluster manager runs out of quota, a
// googleapi 403 is returned.
func (c *ClusterManager) Checkpoint(lbs []*loadbalancers.L7RuntimeInfo, nodeNames []string, nodePorts []backends.ServicePort) error {
func (c *ClusterManager) Checkpoint(lbs []*loadbalancers.L7RuntimeInfo, nodeNames []string, backendServicePorts []backends.ServicePort, namedPorts []backends.ServicePort) ([]*compute.InstanceGroup, error) {
if len(namedPorts) != 0 {
// Add the default backend node port to the list of named ports for instance groups.
namedPorts = append(namedPorts, c.defaultBackendNodePort)
}
// Multiple ingress paths can point to the same service (and hence nodePort)
// but each nodePort can only have one set of cloud resources behind it. So
// don't waste time double validating GCE BackendServices.
portMap := map[int64]backends.ServicePort{}
for _, p := range nodePorts {
portMap[p.Port] = p
namedPorts = uniq(namedPorts)
backendServicePorts = uniq(backendServicePorts)
// Create Instance Groups.
igs, err := c.EnsureInstanceGroupsAndPorts(namedPorts)
if err != nil {
return igs, err
}
nodePorts = []backends.ServicePort{}
for _, sp := range portMap {
nodePorts = append(nodePorts, sp)
}
if err := c.backendPool.Sync(nodePorts); err != nil {
return err
if err := c.backendPool.Sync(backendServicePorts, igs); err != nil {
return igs, err
}
if err := c.instancePool.Sync(nodeNames); err != nil {
return err
return igs, err
}
if err := c.l7Pool.Sync(lbs); err != nil {
return err
return igs, err
}
// TODO: Manage default backend and its firewall rule in a centralized way.
// DefaultBackend is managed in l7 pool, which doesn't understand instances,
// which the firewall rule requires.
fwNodePorts := nodePorts
fwNodePorts := backendServicePorts
if len(lbs) != 0 {
// If there are no Ingresses, we shouldn't be allowing traffic to the
// default backend. Equally importantly if the cluster gets torn down
@ -162,10 +160,27 @@ func (c *ClusterManager) Checkpoint(lbs []*loadbalancers.L7RuntimeInfo, nodeName
np = append(np, p.Port)
}
if err := c.firewallPool.Sync(np, nodeNames); err != nil {
return err
return igs, err
}
return nil
return igs, nil
}
func (c *ClusterManager) EnsureInstanceGroupsAndPorts(servicePorts []backends.ServicePort) ([]*compute.InstanceGroup, error) {
var igs []*compute.InstanceGroup
var err error
for _, p := range servicePorts {
// EnsureInstanceGroupsAndPorts always returns all the instance groups, so we can return
// the output of any call, no need to append the return from all calls.
// TODO: Ideally, we want to call CreateInstaceGroups only the first time and
// then call AddNamedPort multiple times. Need to update the interface to
// achieve this.
igs, _, err = instances.EnsureInstanceGroupsAndPorts(c.instancePool, c.ClusterNamer, p.Port)
if err != nil {
return nil, err
}
}
return igs, nil
}
// GC garbage collects unused resources.
@ -209,65 +224,17 @@ func (c *ClusterManager) GC(lbNames []string, nodePorts []backends.ServicePort)
return nil
}
func getGCEClient(config io.Reader) *gce.GCECloud {
// Creating the cloud interface involves resolving the metadata server to get
// an oauth token. If this fails, the token provider assumes it's not on GCE.
// No errors are thrown. So we need to keep retrying till it works because
// we know we're on GCE.
for {
cloudInterface, err := cloudprovider.GetCloudProvider("gce", config)
if err == nil {
cloud := cloudInterface.(*gce.GCECloud)
// If this controller is scheduled on a node without compute/rw
// it won't be allowed to list backends. We can assume that the
// user has no need for Ingress in this case. If they grant
// permissions to the node they will have to restart the controller
// manually to re-create the client.
if _, err = cloud.ListBackendServices(); err == nil || utils.IsHTTPErrorCode(err, http.StatusForbidden) {
return cloud
}
glog.Warningf("Failed to list backend services, retrying: %v", err)
} else {
glog.Warningf("Failed to retrieve cloud interface, retrying: %v", err)
}
time.Sleep(cloudClientRetryInterval)
}
}
// NewClusterManager creates a cluster manager for shared resources.
// - namer: is the namer used to tag cluster wide shared resources.
// - defaultBackendNodePort: is the node port of glbc's default backend. This is
// the kubernetes Service that serves the 404 page if no urls match.
// - defaultHealthCheckPath: is the default path used for L7 health checks, eg: "/healthz".
func NewClusterManager(
configFilePath string,
cloud *gce.GCECloud,
namer *utils.Namer,
defaultBackendNodePort backends.ServicePort,
defaultHealthCheckPath string) (*ClusterManager, error) {
// TODO: Make this more resilient. Currently we create the cloud client
// and pass it through to all the pools. This makes unit testing easier.
// However if the cloud client suddenly fails, we should try to re-create it
// and continue.
var cloud *gce.GCECloud
if configFilePath != "" {
glog.Infof("Reading config from path %v", configFilePath)
config, err := os.Open(configFilePath)
if err != nil {
return nil, err
}
defer config.Close()
cloud = getGCEClient(config)
glog.Infof("Successfully loaded cloudprovider using config %q", configFilePath)
} else {
// While you might be tempted to refactor so we simply assing nil to the
// config and only invoke getGCEClient once, that will not do the right
// thing because a nil check against an interface isn't true in golang.
cloud = getGCEClient(nil)
glog.Infof("Created GCE client without a config file")
}
// Names are fundamental to the cluster, the uid allocator makes sure names don't collide.
cluster := ClusterManager{ClusterNamer: namer}

216
controllers/gce/controller/controller.go
View file

@ -18,26 +18,25 @@ package controller
import (
"fmt"
"net/http"
"reflect"
"sync"
"time"
"github.com/golang/glog"
apiv1 "k8s.io/api/core/v1"
extensions "k8s.io/api/extensions/v1beta1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/fields"
informerv1 "k8s.io/client-go/informers/core/v1"
informerv1beta1 "k8s.io/client-go/informers/extensions/v1beta1"
"k8s.io/client-go/kubernetes"
scheme "k8s.io/client-go/kubernetes/scheme"
unversionedcore "k8s.io/client-go/kubernetes/typed/core/v1"
listers "k8s.io/client-go/listers/core/v1"
api_v1 "k8s.io/client-go/pkg/api/v1"
extensions "k8s.io/client-go/pkg/apis/extensions/v1beta1"
"k8s.io/client-go/tools/cache"
"k8s.io/client-go/tools/record"
"k8s.io/ingress/controllers/gce/loadbalancers"
"k8s.io/ingress/controllers/gce/utils"
)
var (
@ -55,17 +54,45 @@ var (
storeSyncPollPeriod = 5 * time.Second
)
// ControllerContext holds
type ControllerContext struct {
IngressInformer cache.SharedIndexInformer
ServiceInformer cache.SharedIndexInformer
PodInformer cache.SharedIndexInformer
NodeInformer cache.SharedIndexInformer
// Stop is the stop channel shared among controllers
StopCh chan struct{}
}
func NewControllerContext(kubeClient kubernetes.Interface, namespace string, resyncPeriod time.Duration) *ControllerContext {
return &ControllerContext{
IngressInformer: informerv1beta1.NewIngressInformer(kubeClient, namespace, resyncPeriod, cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}),
ServiceInformer: informerv1.NewServiceInformer(kubeClient, namespace, resyncPeriod, cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}),
PodInformer: informerv1.NewPodInformer(kubeClient, namespace, resyncPeriod, cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}),
NodeInformer: informerv1.NewNodeInformer(kubeClient, resyncPeriod, cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}),
StopCh: make(chan struct{}),
}
}
func (ctx *ControllerContext) Start() {
go ctx.IngressInformer.Run(ctx.StopCh)
go ctx.ServiceInformer.Run(ctx.StopCh)
go ctx.PodInformer.Run(ctx.StopCh)
go ctx.NodeInformer.Run(ctx.StopCh)
}
// LoadBalancerController watches the kubernetes api and adds/removes services
// from the loadbalancer, via loadBalancerConfig.
type LoadBalancerController struct {
client kubernetes.Interface
ingController cache.Controller
nodeController cache.Controller
svcController cache.Controller
podController cache.Controller
ingLister StoreToIngressLister
nodeLister StoreToNodeLister
svcLister StoreToServiceLister
client kubernetes.Interface
ingressSynced cache.InformerSynced
serviceSynced cache.InformerSynced
podSynced cache.InformerSynced
nodeSynced cache.InformerSynced
ingLister StoreToIngressLister
nodeLister StoreToNodeLister
svcLister StoreToServiceLister
// Health checks are the readiness probes of containers on pods.
podLister StoreToPodLister
// TODO: Watch secrets
@ -92,7 +119,7 @@ type LoadBalancerController struct {
// - clusterManager: A ClusterManager capable of creating all cloud resources
// required for L7 loadbalancing.
// - resyncPeriod: Watchers relist from the Kubernetes API server this often.
func NewLoadBalancerController(kubeClient kubernetes.Interface, clusterManager *ClusterManager, resyncPeriod time.Duration, namespace string) (*LoadBalancerController, error) {
func NewLoadBalancerController(kubeClient kubernetes.Interface, ctx *ControllerContext, clusterManager *ClusterManager) (*LoadBalancerController, error) {
eventBroadcaster := record.NewBroadcaster()
eventBroadcaster.StartLogging(glog.Infof)
eventBroadcaster.StartRecordingToSink(&unversionedcore.EventSinkImpl{
@ -101,28 +128,37 @@ func NewLoadBalancerController(kubeClient kubernetes.Interface, clusterManager *
lbc := LoadBalancerController{
client: kubeClient,
CloudClusterManager: clusterManager,
stopCh: make(chan struct{}),
stopCh: ctx.StopCh,
recorder: eventBroadcaster.NewRecorder(scheme.Scheme,
api_v1.EventSource{Component: "loadbalancer-controller"}),
apiv1.EventSource{Component: "loadbalancer-controller"}),
}
lbc.nodeQueue = NewTaskQueue(lbc.syncNodes)
lbc.ingQueue = NewTaskQueue(lbc.sync)
lbc.hasSynced = lbc.storesSynced
// Ingress watch handlers
pathHandlers := cache.ResourceEventHandlerFuncs{
lbc.ingressSynced = ctx.IngressInformer.HasSynced
lbc.serviceSynced = ctx.ServiceInformer.HasSynced
lbc.podSynced = ctx.PodInformer.HasSynced
lbc.nodeSynced = ctx.NodeInformer.HasSynced
lbc.ingLister.Store = ctx.IngressInformer.GetStore()
lbc.svcLister.Indexer = ctx.ServiceInformer.GetIndexer()
lbc.podLister.Indexer = ctx.PodInformer.GetIndexer()
lbc.nodeLister.Indexer = ctx.NodeInformer.GetIndexer()
// ingress event handler
ctx.IngressInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: func(obj interface{}) {
addIng := obj.(*extensions.Ingress)
if !isGCEIngress(addIng) {
if !isGCEIngress(addIng) && !isGCEMultiClusterIngress(addIng) {
glog.Infof("Ignoring add for ingress %v based on annotation %v", addIng.Name, ingressClassKey)
return
}
lbc.recorder.Eventf(addIng, api_v1.EventTypeNormal, "ADD", fmt.Sprintf("%s/%s", addIng.Namespace, addIng.Name))
lbc.recorder.Eventf(addIng, apiv1.EventTypeNormal, "ADD", fmt.Sprintf("%s/%s", addIng.Namespace, addIng.Name))
lbc.ingQueue.enqueue(obj)
},
DeleteFunc: func(obj interface{}) {
delIng := obj.(*extensions.Ingress)
if !isGCEIngress(delIng) {
if !isGCEIngress(delIng) && !isGCEMultiClusterIngress(delIng) {
glog.Infof("Ignoring delete for ingress %v based on annotation %v", delIng.Name, ingressClassKey)
return
}
@ -131,7 +167,7 @@ func NewLoadBalancerController(kubeClient kubernetes.Interface, clusterManager *
},
UpdateFunc: func(old, cur interface{}) {
curIng := cur.(*extensions.Ingress)
if !isGCEIngress(curIng) {
if !isGCEIngress(curIng) && !isGCEMultiClusterIngress(curIng) {
return
}
if !reflect.DeepEqual(old, cur) {
@ -139,13 +175,10 @@ func NewLoadBalancerController(kubeClient kubernetes.Interface, clusterManager *
}
lbc.ingQueue.enqueue(cur)
},
}
lbc.ingLister.Store, lbc.ingController = cache.NewInformer(
cache.NewListWatchFromClient(lbc.client.Extensions().RESTClient(), "ingresses", namespace, fields.Everything()),
&extensions.Ingress{}, resyncPeriod, pathHandlers)
})
// Service watch handlers
svcHandlers := cache.ResourceEventHandlerFuncs{
// service event handler
ctx.ServiceInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: lbc.enqueueIngressForService,
UpdateFunc: func(old, cur interface{}) {
if !reflect.DeepEqual(old, cur) {
@ -153,32 +186,14 @@ func NewLoadBalancerController(kubeClient kubernetes.Interface, clusterManager *
}
},
// Ingress deletes matter, service deletes don't.
}
})
lbc.svcLister.Indexer, lbc.svcController = cache.NewIndexerInformer(
cache.NewListWatchFromClient(lbc.client.Core().RESTClient(), "services", namespace, fields.Everything()),
&api_v1.Service{},
resyncPeriod,
svcHandlers,
cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc},
)
lbc.podLister.Indexer, lbc.podController = cache.NewIndexerInformer(
cache.NewListWatchFromClient(lbc.client.Core().RESTClient(), "pods", namespace, fields.Everything()),
&api_v1.Pod{},
resyncPeriod,
cache.ResourceEventHandlerFuncs{},
cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc},
)
// Node watch handlers
lbc.nodeLister.Indexer, lbc.nodeController = cache.NewIndexerInformer(
cache.NewListWatchFromClient(lbc.client.Core().RESTClient(), "nodes", api_v1.NamespaceAll, fields.Everything()),
&api_v1.Node{},
resyncPeriod,
cache.ResourceEventHandlerFuncs{},
cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc},
)
// node event handler
ctx.NodeInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: lbc.nodeQueue.enqueue,
DeleteFunc: lbc.nodeQueue.enqueue,
// Nodes are updated every 10s and we don't care, so no update handler.
})
lbc.tr = &GCETranslator{&lbc}
lbc.tlsLoader = &apiServerTLSLoader{client: lbc.client}
@ -189,7 +204,7 @@ func NewLoadBalancerController(kubeClient kubernetes.Interface, clusterManager *
// enqueueIngressForService enqueues all the Ingress' for a Service.
func (lbc *LoadBalancerController) enqueueIngressForService(obj interface{}) {
svc := obj.(*api_v1.Service)
svc := obj.(*apiv1.Service)
ings, err := lbc.ingLister.GetServiceIngress(svc)
if err != nil {
glog.V(5).Infof("ignoring service %v: %v", svc.Name, err)
@ -206,10 +221,6 @@ func (lbc *LoadBalancerController) enqueueIngressForService(obj interface{}) {
// Run starts the loadbalancer controller.
func (lbc *LoadBalancerController) Run() {
glog.Infof("Starting loadbalancer controller")
go lbc.ingController.Run(lbc.stopCh)
go lbc.nodeController.Run(lbc.stopCh)
go lbc.svcController.Run(lbc.stopCh)
go lbc.podController.Run(lbc.stopCh)
go lbc.ingQueue.run(time.Second, lbc.stopCh)
go lbc.nodeQueue.run(time.Second, lbc.stopCh)
<-lbc.stopCh
@ -246,14 +257,14 @@ func (lbc *LoadBalancerController) storesSynced() bool {
return (
// wait for pods to sync so we don't allocate a default health check when
// an endpoint has a readiness probe.
lbc.podController.HasSynced() &&
lbc.podSynced() &&
// wait for services so we don't thrash on backend creation.
lbc.svcController.HasSynced() &&
lbc.serviceSynced() &&
// wait for nodes so we don't disconnect a backend from an instance
// group just because we don't realize there are nodes in that zone.
lbc.nodeController.HasSynced() &&
lbc.nodeSynced() &&
// Wait for ingresses as a safety measure. We don't really need this.
lbc.ingController.HasSynced())
lbc.ingressSynced())
}
// sync manages Ingress create/updates/deletes.
@ -264,13 +275,19 @@ func (lbc *LoadBalancerController) sync(key string) (err error) {
}
glog.V(3).Infof("Syncing %v", key)
ingresses, err := lbc.ingLister.List()
allIngresses, err := lbc.ingLister.ListAll()
if err != nil {
return err
}
nodePorts := lbc.tr.toNodePorts(&ingresses)
gceIngresses, err := lbc.ingLister.ListGCEIngresses()
if err != nil {
return err
}
allNodePorts := lbc.tr.toNodePorts(&allIngresses)
gceNodePorts := lbc.tr.toNodePorts(&gceIngresses)
lbNames := lbc.ingLister.Store.ListKeys()
lbs, err := lbc.ListRuntimeInfo()
lbs, err := lbc.toRuntimeInfo(gceIngresses)
if err != nil {
return err
}
@ -296,22 +313,19 @@ func (lbc *LoadBalancerController) sync(key string) (err error) {
var syncError error
defer func() {
if deferErr := lbc.CloudClusterManager.GC(lbNames, nodePorts); deferErr != nil {
if deferErr := lbc.CloudClusterManager.GC(lbNames, allNodePorts); deferErr != nil {
err = fmt.Errorf("error during sync %v, error during GC %v", syncError, deferErr)
}
glog.V(3).Infof("Finished syncing %v", key)
}()
// Record any errors during sync and throw a single error at the end. This
// allows us to free up associated cloud resources ASAP.
if err := lbc.CloudClusterManager.Checkpoint(lbs, nodeNames, nodePorts); err != nil {
igs, err := lbc.CloudClusterManager.Checkpoint(lbs, nodeNames, gceNodePorts, allNodePorts)
if err != nil {
// TODO: Implement proper backoff for the queue.
eventMsg := "GCE"
if utils.IsHTTPErrorCode(err, http.StatusForbidden) {
eventMsg += " :Quota"
}
if ingExists {
lbc.recorder.Eventf(obj.(*extensions.Ingress), api_v1.EventTypeWarning, eventMsg, err.Error())
lbc.recorder.Eventf(obj.(*extensions.Ingress), apiv1.EventTypeWarning, eventMsg, err.Error())
} else {
err = fmt.Errorf("%v, error: %v", eventMsg, err)
}
@ -321,6 +335,22 @@ func (lbc *LoadBalancerController) sync(key string) (err error) {
if !ingExists {
return syncError
}
ing := *obj.(*extensions.Ingress)
if isGCEMultiClusterIngress(&ing) {
// Add instance group names as annotation on the ingress.
if ing.Annotations == nil {
ing.Annotations = map[string]string{}
}
err = setInstanceGroupsAnnotation(ing.Annotations, igs)
if err != nil {
return err
}
if err := lbc.updateAnnotations(ing.Name, ing.Namespace, ing.Annotations); err != nil {
return err
}
return nil
}
// Update the UrlMap of the single loadbalancer that came through the watch.
l7, err := lbc.CloudClusterManager.l7Pool.Get(key)
if err != nil {
@ -328,14 +358,13 @@ func (lbc *LoadBalancerController) sync(key string) (err error) {
return syncError
}
ing := *obj.(*extensions.Ingress)
if urlMap, err := lbc.tr.toURLMap(&ing); err != nil {
syncError = fmt.Errorf("%v, convert to url map error %v", syncError, err)
} else if err := l7.UpdateUrlMap(urlMap); err != nil {
lbc.recorder.Eventf(&ing, api_v1.EventTypeWarning, "UrlMap", err.Error())
lbc.recorder.Eventf(&ing, apiv1.EventTypeWarning, "UrlMap", err.Error())
syncError = fmt.Errorf("%v, update url map error: %v", syncError, err)
} else if err := lbc.updateIngressStatus(l7, ing); err != nil {
lbc.recorder.Eventf(&ing, api_v1.EventTypeWarning, "Status", err.Error())
lbc.recorder.Eventf(&ing, apiv1.EventTypeWarning, "Status", err.Error())
syncError = fmt.Errorf("%v, update ingress error: %v", syncError, err)
}
return syncError
@ -353,8 +382,8 @@ func (lbc *LoadBalancerController) updateIngressStatus(l7 *loadbalancers.L7, ing
return err
}
currIng.Status = extensions.IngressStatus{
LoadBalancer: api_v1.LoadBalancerStatus{
Ingress: []api_v1.LoadBalancerIngress{
LoadBalancer: apiv1.LoadBalancerStatus{
Ingress: []apiv1.LoadBalancerIngress{
{IP: ip},
},
},
@ -368,17 +397,26 @@ func (lbc *LoadBalancerController) updateIngressStatus(l7 *loadbalancers.L7, ing
if _, err := ingClient.UpdateStatus(currIng); err != nil {
return err
}
lbc.recorder.Eventf(currIng, api_v1.EventTypeNormal, "CREATE", "ip: %v", ip)
lbc.recorder.Eventf(currIng, apiv1.EventTypeNormal, "CREATE", "ip: %v", ip)
}
}
annotations := loadbalancers.GetLBAnnotations(l7, currIng.Annotations, lbc.CloudClusterManager.backendPool)
if err := lbc.updateAnnotations(ing.Name, ing.Namespace, annotations); err != nil {
return err
}
return nil
}
func (lbc *LoadBalancerController) updateAnnotations(name, namespace string, annotations map[string]string) error {
// Update annotations through /update endpoint
currIng, err = ingClient.Get(ing.Name, metav1.GetOptions{})
ingClient := lbc.client.Extensions().Ingresses(namespace)
currIng, err := ingClient.Get(name, metav1.GetOptions{})
if err != nil {
return err
}
currIng.Annotations = loadbalancers.GetLBAnnotations(l7, currIng.Annotations, lbc.CloudClusterManager.backendPool)
if !reflect.DeepEqual(ing.Annotations, currIng.Annotations) {
glog.V(3).Infof("Updating annotations of %v/%v", ing.Namespace, ing.Name)
if !reflect.DeepEqual(currIng.Annotations, annotations) {
glog.V(3).Infof("Updating annotations of %v/%v", namespace, name)
currIng.Annotations = annotations
if _, err := ingClient.Update(currIng); err != nil {
return err
}
@ -386,12 +424,8 @@ func (lbc *LoadBalancerController) updateIngressStatus(l7 *loadbalancers.L7, ing
return nil
}
// ListRuntimeInfo lists L7RuntimeInfo as understood by the loadbalancer module.
func (lbc *LoadBalancerController) ListRuntimeInfo() (lbs []*loadbalancers.L7RuntimeInfo, err error) {
ingList, err := lbc.ingLister.List()
if err != nil {
return lbs, err
}
// toRuntimeInfo returns L7RuntimeInfo for the given ingresses.
func (lbc *LoadBalancerController) toRuntimeInfo(ingList extensions.IngressList) (lbs []*loadbalancers.L7RuntimeInfo, err error) {
for _, ing := range ingList.Items {
k, err := keyFunc(&ing)
if err != nil {
@ -436,11 +470,11 @@ func (lbc *LoadBalancerController) syncNodes(key string) error {
}
func getNodeReadyPredicate() listers.NodeConditionPredicate {
return func(node *api_v1.Node) bool {
return func(node *apiv1.Node) bool {
for ix := range node.Status.Conditions {
condition := &node.Status.Conditions[ix]
if condition.Type == api_v1.NodeReady {
return condition.Status == api_v1.ConditionTrue
if condition.Type == apiv1.NodeReady {
return condition.Status == apiv1.ConditionTrue
}
}
return false

11
controllers/gce/controller/controller_test.go
View file

@ -24,14 +24,14 @@ import (
compute "google.golang.org/api/compute/v1"
api_v1 "k8s.io/api/core/v1"
extensions "k8s.io/api/extensions/v1beta1"
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/intstr"
"k8s.io/apimachinery/pkg/util/sets"
"k8s.io/apimachinery/pkg/util/uuid"
"k8s.io/client-go/kubernetes/fake"
"k8s.io/client-go/pkg/api"
api_v1 "k8s.io/client-go/pkg/api/v1"
extensions "k8s.io/client-go/pkg/apis/extensions/v1beta1"
"k8s.io/kubernetes/pkg/api"
"k8s.io/ingress/controllers/gce/firewalls"
"k8s.io/ingress/controllers/gce/loadbalancers"
@ -53,7 +53,8 @@ func defaultBackendName(clusterName string) string {
// newLoadBalancerController create a loadbalancer controller.
func newLoadBalancerController(t *testing.T, cm *fakeClusterManager) *LoadBalancerController {
kubeClient := fake.NewSimpleClientset()
lb, err := NewLoadBalancerController(kubeClient, cm.ClusterManager, 1*time.Second, api_v1.NamespaceAll)
ctx := NewControllerContext(kubeClient, api_v1.NamespaceAll, 1*time.Second)
lb, err := NewLoadBalancerController(kubeClient, ctx, cm.ClusterManager)
if err != nil {
t.Fatalf("%v", err)
}
@ -428,7 +429,7 @@ func TestLbChangeStaticIP(t *testing.T) {
}
ing.Annotations = map[string]string{staticIPNameKey: "testip"}
cm.fakeLbs.ReserveGlobalStaticIP("testip", "1.2.3.4")
cm.fakeLbs.ReserveGlobalAddress(&compute.Address{Name: "testip", Address: "1.2.3.4"})
// Second sync reassigns 1.2.3.4 to existing forwarding rule (by recreating it)
lbc.sync(ingStoreKey)

2
controllers/gce/controller/fakes.go
View file

@ -65,7 +65,7 @@ func NewFakeClusterManager(clusterName, firewallName string) *fakeClusterManager
testDefaultBeNodePort,
namer,
)
frPool := firewalls.NewFirewallPool(firewalls.NewFakeFirewallsProvider(namer), namer)
frPool := firewalls.NewFirewallPool(firewalls.NewFakeFirewallsProvider(), namer)
cm := &ClusterManager{
ClusterNamer: namer,
instancePool: nodePool,

4
controllers/gce/controller/tls.go
View file

@ -21,10 +21,10 @@ import (
"github.com/golang/glog"
api_v1 "k8s.io/api/core/v1"
extensions "k8s.io/api/extensions/v1beta1"
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
api_v1 "k8s.io/client-go/pkg/api/v1"
extensions "k8s.io/client-go/pkg/apis/extensions/v1beta1"
"k8s.io/ingress/controllers/gce/loadbalancers"
)

119
controllers/gce/controller/utils.go
View file

@ -26,6 +26,9 @@ import (
"github.com/golang/glog"
compute "google.golang.org/api/compute/v1"
api_v1 "k8s.io/api/core/v1"
extensions "k8s.io/api/extensions/v1beta1"
"k8s.io/apimachinery/pkg/api/meta"
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
@ -34,8 +37,6 @@ import (
"k8s.io/apimachinery/pkg/util/sets"
"k8s.io/apimachinery/pkg/util/wait"
listers "k8s.io/client-go/listers/core/v1"
api_v1 "k8s.io/client-go/pkg/api/v1"
extensions "k8s.io/client-go/pkg/apis/extensions/v1beta1"
"k8s.io/client-go/tools/cache"
"k8s.io/client-go/util/workqueue"
@ -75,12 +76,19 @@ const (
// ingressClassKey picks a specific "class" for the Ingress. The controller
// only processes Ingresses with this annotation either unset, or set
// to either gceIngessClass or the empty string.
ingressClassKey = "kubernetes.io/ingress.class"
gceIngressClass = "gce"
ingressClassKey = "kubernetes.io/ingress.class"
gceIngressClass = "gce"
gceMultiIngressClass = "gce-multi-cluster"
// Label key to denote which GCE zone a Kubernetes node is in.
zoneKey = "failure-domain.beta.kubernetes.io/zone"
defaultZone = ""
// instanceGroupsAnnotationKey is the annotation key used by controller to
// specify the name and zone of instance groups created for the ingress.
// This is read only for users. Controller will overrite any user updates.
// This is only set for ingresses with ingressClass = "gce-multi-cluster"
instanceGroupsAnnotationKey = "ingress.gcp.kubernetes.io/instance-groups"
)
// ingAnnotations represents Ingress annotations.
@ -156,6 +164,13 @@ func isGCEIngress(ing *extensions.Ingress) bool {
return class == "" || class == gceIngressClass
}
// isGCEMultiClusterIngress returns true if the given Ingress has
// ingress.class annotation set to "gce-multi-cluster".
func isGCEMultiClusterIngress(ing *extensions.Ingress) bool {
class := ingAnnotations(ing.ObjectMeta.Annotations).ingressClass()
return class == gceMultiIngressClass
}
// errorNodePortNotFound is an implementation of error.
type errorNodePortNotFound struct {
backend extensions.IngressBackend
@ -285,8 +300,19 @@ func ListAll(store cache.Store, selector labels.Selector, appendFn cache.AppendF
return nil
}
// List lists all Ingress' in the store.
func (s *StoreToIngressLister) List() (ing extensions.IngressList, err error) {
// List lists all Ingress' in the store (both single and multi cluster ingresses).
func (s *StoreToIngressLister) ListAll() (ing extensions.IngressList, err error) {
for _, m := range s.Store.List() {
newIng := m.(*extensions.Ingress)
if isGCEIngress(newIng) || isGCEMultiClusterIngress(newIng) {
ing.Items = append(ing.Items, *newIng)
}
}
return ing, nil
}
// ListGCEIngresses lists all GCE Ingress' in the store.
func (s *StoreToIngressLister) ListGCEIngresses() (ing extensions.IngressList, err error) {
for _, m := range s.Store.List() {
newIng := m.(*extensions.Ingress)
if isGCEIngress(newIng) {
@ -471,32 +497,39 @@ PortLoop:
return p, nil
}
// toNodePorts converts a pathlist to a flat list of nodeports.
// toNodePorts is a helper method over ingressToNodePorts to process a list of ingresses.
func (t *GCETranslator) toNodePorts(ings *extensions.IngressList) []backends.ServicePort {
var knownPorts []backends.ServicePort
for _, ing := range ings.Items {
defaultBackend := ing.Spec.Backend
if defaultBackend != nil {
port, err := t.getServiceNodePort(*defaultBackend, ing.Namespace)
knownPorts = append(knownPorts, t.ingressToNodePorts(&ing)...)
}
return knownPorts
}
// ingressToNodePorts converts a pathlist to a flat list of nodeports for the given ingress.
func (t *GCETranslator) ingressToNodePorts(ing *extensions.Ingress) []backends.ServicePort {
var knownPorts []backends.ServicePort
defaultBackend := ing.Spec.Backend
if defaultBackend != nil {
port, err := t.getServiceNodePort(*defaultBackend, ing.Namespace)
if err != nil {
glog.Infof("%v", err)
} else {
knownPorts = append(knownPorts, port)
}
}
for _, rule := range ing.Spec.Rules {
if rule.HTTP == nil {
glog.Errorf("ignoring non http Ingress rule")
continue
}
for _, path := range rule.HTTP.Paths {
port, err := t.getServiceNodePort(path.Backend, ing.Namespace)
if err != nil {
glog.Infof("%v", err)
} else {
knownPorts = append(knownPorts, port)
}
}
for _, rule := range ing.Spec.Rules {
if rule.HTTP == nil {
glog.Errorf("ignoring non http Ingress rule")
continue
}
for _, path := range rule.HTTP.Paths {
port, err := t.getServiceNodePort(path.Backend, ing.Namespace)
if err != nil {
glog.Infof("%v", err)
continue
}
knownPorts = append(knownPorts, port)
}
knownPorts = append(knownPorts, port)
}
}
return knownPorts
@ -592,10 +625,11 @@ func (t *GCETranslator) getHTTPProbe(svc api_v1.Service, targetPort intstr.IntOr
// isSimpleHTTPProbe returns true if the given Probe is:
// - an HTTPGet probe, as opposed to a tcp or exec probe
// - has no special host or headers fields
// - has no special host or headers fields, except for possibly an HTTP Host header
func isSimpleHTTPProbe(probe *api_v1.Probe) bool {
return (probe != nil && probe.Handler.HTTPGet != nil && probe.Handler.HTTPGet.Host == "" &&
len(probe.Handler.HTTPGet.HTTPHeaders) == 0)
(len(probe.Handler.HTTPGet.HTTPHeaders) == 0 ||
(len(probe.Handler.HTTPGet.HTTPHeaders) == 1 && probe.Handler.HTTPGet.HTTPHeaders[0].Name == "Host")))
}
// GetProbe returns a probe that's used for the given nodeport
@ -639,3 +673,34 @@ func (o PodsByCreationTimestamp) Less(i, j int) bool {
}
return o[i].CreationTimestamp.Before(o[j].CreationTimestamp)
}
// setInstanceGroupsAnnotation sets the instance-groups annotation with names of the given instance groups.
func setInstanceGroupsAnnotation(existing map[string]string, igs []*compute.InstanceGroup) error {
type Value struct {
Name string
Zone string
}
var instanceGroups []Value
for _, ig := range igs {
instanceGroups = append(instanceGroups, Value{Name: ig.Name, Zone: ig.Zone})
}
jsonValue, err := json.Marshal(instanceGroups)
if err != nil {
return err
}
existing[instanceGroupsAnnotationKey] = string(jsonValue)
return nil
}
// uniq returns an array of unique service ports from the given array.
func uniq(nodePorts []backends.ServicePort) []backends.ServicePort {
portMap := map[int64]backends.ServicePort{}
for _, p := range nodePorts {
portMap[p.Port] = p
}
nodePorts = make([]backends.ServicePort, 0, len(portMap))
for _, sp := range portMap {
nodePorts = append(nodePorts, sp)
}
return nodePorts
}

53
controllers/gce/controller/util_test.go → controllers/gce/controller/utils_test.go
View file

@ -21,10 +21,12 @@ import (
"testing"
"time"
compute "google.golang.org/api/compute/v1"
api_v1 "k8s.io/api/core/v1"
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/intstr"
"k8s.io/apimachinery/pkg/util/sets"
api_v1 "k8s.io/client-go/pkg/api/v1"
"k8s.io/ingress/controllers/gce/backends"
"k8s.io/ingress/controllers/gce/utils"
)
@ -76,20 +78,19 @@ func TestInstancesAddedToZones(t *testing.T) {
lbc.CloudClusterManager.instancePool.Sync([]string{"n1", "n2", "n3"})
gotZonesToNode := cm.fakeIGs.GetInstancesByZone()
i := 0
if cm.fakeIGs.Ports[0] != testPort {
t.Errorf("Expected the same node port on all igs, got ports %+v", cm.fakeIGs.Ports)
}
for z, nodeNames := range zoneToNode {
if ig, err := cm.fakeIGs.GetInstanceGroup(testIG, z); err != nil {
t.Errorf("Failed to find ig %v in zone %v, found %+v: %v", testIG, z, ig, err)
}
if cm.fakeIGs.Ports[i] != testPort {
t.Errorf("Expected the same node port on all igs, got ports %+v", cm.fakeIGs.Ports)
}
expNodes := sets.NewString(nodeNames...)
gotNodes := sets.NewString(gotZonesToNode[z]...)
if !gotNodes.Equal(expNodes) {
t.Errorf("Nodes not added to zones, expected %+v got %+v", expNodes, gotNodes)
}
i++
}
}
@ -264,3 +265,43 @@ func addNodes(lbc *LoadBalancerController, zoneToNode map[string][]string) {
func getProbePath(p *api_v1.Probe) string {
return p.Handler.HTTPGet.Path
}
func TestAddInstanceGroupsAnnotation(t *testing.T) {
testCases := []struct {
Igs []*compute.InstanceGroup
ExpectedAnnotation string
}{
{
// Single zone.
[]*compute.InstanceGroup{&compute.InstanceGroup{
Name: "ig-name",
Zone: "https://www.googleapis.com/compute/v1/projects/my-project/zones/us-central1-b",
}},
`[{"Name":"ig-name","Zone":"https://www.googleapis.com/compute/v1/projects/my-project/zones/us-central1-b"}]`,
},
{
// Multiple zones.
[]*compute.InstanceGroup{
&compute.InstanceGroup{
Name: "ig-name-1",
Zone: "https://www.googleapis.com/compute/v1/projects/my-project/zones/us-central1-b",
},
&compute.InstanceGroup{
Name: "ig-name-2",
Zone: "https://www.googleapis.com/compute/v1/projects/my-project/zones/us-central1-a",
},
},
`[{"Name":"ig-name-1","Zone":"https://www.googleapis.com/compute/v1/projects/my-project/zones/us-central1-b"},{"Name":"ig-name-2","Zone":"https://www.googleapis.com/compute/v1/projects/my-project/zones/us-central1-a"}]`,
},
}
for _, c := range testCases {
annotations := map[string]string{}
err := setInstanceGroupsAnnotation(annotations, c.Igs)
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}
if annotations[instanceGroupsAnnotationKey] != c.ExpectedAnnotation {
t.Fatalf("Unexpected annotation value: %s, expected: %s", annotations[instanceGroupsAnnotationKey], c.ExpectedAnnotation)
}
}
}

34
controllers/gce/examples/health_checks/health_check_app.yaml
View file

@ -11,7 +11,7 @@ spec:
spec:
containers:
- name: echoheaders
image: gcr.io/google_containers/echoserver:1.5
image: gcr.io/google_containers/echoserver:1.8
ports:
- containerPort: 8080
readinessProbe:
@ -23,22 +23,22 @@ spec:
successThreshold: 1
failureThreshold: 10
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
---
apiVersion: v1

2
controllers/gce/examples/https/README.md
View file

@ -12,7 +12,7 @@ $ kubectl --namespace=kube-system get pod -l name=glbc
NAME
l7-lb-controller-v0.6.0-1770t ...
```
Also make sure you have a [firewall rule](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md#creating-the-fir-glbc-health-checks) for the node port of the Service.
Also make sure you have a [firewall rule](https://github.com/kubernetes/ingress/blob/master/controllers/gce/BETA_LIMITATIONS.md#creating-the-fir-glbc-health-checks) for the node port of the Service.
Create Ingress
```console

4
controllers/gce/examples/https/make_secret.go
View file

@ -26,13 +26,13 @@ import (
"io/ioutil"
"log"
api_v1 "k8s.io/api/core/v1"
registered "k8s.io/apimachinery/pkg/apimachinery/registered"
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/client-go/pkg/api"
api_v1 "k8s.io/client-go/pkg/api/v1"
// This installs the legacy v1 API
"k8s.io/kubernetes/pkg/api"
_ "k8s.io/kubernetes/pkg/api/install"
)

2
controllers/gce/examples/websocket/README.md
View file

@ -55,7 +55,7 @@ Wait for the loadbalancer to be created and functioning. When you receive a succ
Websocket example. Connect to /ws%
```
The binary we deployed does not have any html/javascript to demonstrate thwe websocket, so we'll use websocket.org's client.
The binary we deployed does not have any html/javascript to demonstrate the websocket, so we'll use websocket.org's client.
Visit http://www.websocket.org/echo.html. It's important to use `HTTP` instead of `HTTPS` since we assembled an `HTTP` load balancer. Browsers may prevent `HTTP` websocket connections as a security feature.
Set the `Location` to

70
controllers/gce/firewalls/fakes.go
View file

@ -18,86 +18,66 @@ package firewalls
import (
"fmt"
"strconv"
compute "google.golang.org/api/compute/v1"
netset "k8s.io/kubernetes/pkg/util/net/sets"
"k8s.io/ingress/controllers/gce/utils"
)
type fakeFirewallsProvider struct {
fw map[string]*compute.Firewall
namer *utils.Namer
fw map[string]*compute.Firewall
networkUrl string
}
// NewFakeFirewallsProvider creates a fake for firewall rules.
func NewFakeFirewallsProvider(namer *utils.Namer) *fakeFirewallsProvider {
func NewFakeFirewallsProvider() *fakeFirewallsProvider {
return &fakeFirewallsProvider{
fw: make(map[string]*compute.Firewall),
namer: namer,
fw: make(map[string]*compute.Firewall),
}
}
func (f *fakeFirewallsProvider) GetFirewall(prefixedName string) (*compute.Firewall, error) {
rule, exists := f.fw[prefixedName]
func (ff *fakeFirewallsProvider) GetFirewall(name string) (*compute.Firewall, error) {
rule, exists := ff.fw[name]
if exists {
return rule, nil
}
return nil, utils.FakeGoogleAPINotFoundErr()
}
func (f *fakeFirewallsProvider) CreateFirewall(name, msgTag string, srcRange netset.IPNet, ports []int64, hosts []string) error {
prefixedName := f.namer.FrName(name)
strPorts := []string{}
for _, p := range ports {
strPorts = append(strPorts, strconv.FormatInt(p, 10))
}
if _, exists := f.fw[prefixedName]; exists {
return fmt.Errorf("firewall rule %v already exists", prefixedName)
}
f.fw[prefixedName] = &compute.Firewall{
// To accurately mimic the cloudprovider we need to add the k8s-fw
// prefix to the given rule name.
Name: prefixedName,
SourceRanges: srcRange.StringSlice(),
Allowed: []*compute.FirewallAllowed{{Ports: strPorts}},
TargetTags: hosts, // WARNING: This is actually not correct, but good enough for testing this package
func (ff *fakeFirewallsProvider) CreateFirewall(f *compute.Firewall) error {
if _, exists := ff.fw[f.Name]; exists {
return fmt.Errorf("firewall rule %v already exists", f.Name)
}
ff.fw[f.Name] = f
return nil
}
func (f *fakeFirewallsProvider) DeleteFirewall(name string) error {
func (ff *fakeFirewallsProvider) DeleteFirewall(name string) error {
// We need the full name for the same reason as CreateFirewall.
prefixedName := f.namer.FrName(name)
_, exists := f.fw[prefixedName]
_, exists := ff.fw[name]
if !exists {
return utils.FakeGoogleAPINotFoundErr()
}
delete(f.fw, prefixedName)
delete(ff.fw, name)
return nil
}
func (f *fakeFirewallsProvider) UpdateFirewall(name, msgTag string, srcRange netset.IPNet, ports []int64, hosts []string) error {
strPorts := []string{}
for _, p := range ports {
strPorts = append(strPorts, strconv.FormatInt(p, 10))
}
func (ff *fakeFirewallsProvider) UpdateFirewall(f *compute.Firewall) error {
// We need the full name for the same reason as CreateFirewall.
prefixedName := f.namer.FrName(name)
_, exists := f.fw[prefixedName]
_, exists := ff.fw[f.Name]
if !exists {
return fmt.Errorf("update failed for rule %v, srcRange %v ports %v, rule not found", prefixedName, srcRange, ports)
return fmt.Errorf("update failed for rule %v, srcRange %v ports %+v, rule not found", f.Name, f.SourceRanges, f.Allowed)
}
f.fw[prefixedName] = &compute.Firewall{
Name: name,
SourceRanges: srcRange.StringSlice(),
Allowed: []*compute.FirewallAllowed{{Ports: strPorts}},
TargetTags: hosts, // WARNING: This is actually not correct, but good enough for testing this package
}
ff.fw[f.Name] = f
return nil
}
func (ff *fakeFirewallsProvider) NetworkURL() string {
return ff.networkUrl
}
func (ff *fakeFirewallsProvider) GetNodeTags(nodeNames []string) ([]string, error) {
return nodeNames, nil
}

52
controllers/gce/firewalls/firewalls.go
View file

@ -35,18 +35,18 @@ var l7SrcRanges = []string{"130.211.0.0/22", "35.191.0.0/16"}
type FirewallRules struct {
cloud Firewall
namer *utils.Namer
srcRanges netset.IPNet
srcRanges []string
}
// NewFirewallPool creates a new firewall rule manager.
// cloud: the cloud object implementing Firewall.
// namer: cluster namer.
func NewFirewallPool(cloud Firewall, namer *utils.Namer) SingleFirewallPool {
srcNetSet, err := netset.ParseIPNets(l7SrcRanges...)
_, err := netset.ParseIPNets(l7SrcRanges...)
if err != nil {
glog.Fatalf("Could not parse L7 src ranges %v for firewall rule: %v", l7SrcRanges, err)
}
return &FirewallRules{cloud: cloud, namer: namer, srcRanges: srcNetSet}
return &FirewallRules{cloud: cloud, namer: namer, srcRanges: l7SrcRanges}
}
// Sync sync firewall rules with the cloud.
@ -60,9 +60,15 @@ func (fr *FirewallRules) Sync(nodePorts []int64, nodeNames []string) error {
// instead of the whole name.
name := fr.namer.FrName(suffix)
rule, _ := fr.cloud.GetFirewall(name)
firewall, err := fr.createFirewallObject(name, "GCE L7 firewall rule", nodePorts, nodeNames)
if err != nil {
return err
}
if rule == nil {
glog.Infof("Creating global l7 firewall rule %v", name)
return fr.cloud.CreateFirewall(suffix, "GCE L7 firewall rule", fr.srcRanges, nodePorts, nodeNames)
return fr.cloud.CreateFirewall(firewall)
}
requiredPorts := sets.NewString()
@ -85,17 +91,17 @@ func (fr *FirewallRules) Sync(nodePorts []int64, nodeNames []string) error {
glog.V(4).Info("Firewall does not need update of ports or source ranges")
return nil
}
glog.V(3).Infof("Firewall %v already exists, updating nodeports %v", name, nodePorts)
return fr.cloud.UpdateFirewall(suffix, "GCE L7 firewall", fr.srcRanges, nodePorts, nodeNames)
return fr.cloud.UpdateFirewall(firewall)
}
// Shutdown shuts down this firewall rules manager.
func (fr *FirewallRules) Shutdown() error {
glog.Infof("Deleting firewall with suffix %v", fr.namer.FrSuffix())
err := fr.cloud.DeleteFirewall(fr.namer.FrSuffix())
name := fr.namer.FrName(fr.namer.FrSuffix())
glog.Infof("Deleting firewall %v", name)
err := fr.cloud.DeleteFirewall(name)
if err != nil && utils.IsHTTPErrorCode(err, 404) {
glog.Infof("Firewall with suffix %v didn't exist at Shutdown", fr.namer.FrSuffix())
glog.Infof("Firewall with name %v didn't exist at Shutdown", name)
return nil
}
return err
@ -107,3 +113,31 @@ func (fr *FirewallRules) Shutdown() error {
func (fr *FirewallRules) GetFirewall(name string) (*compute.Firewall, error) {
return fr.cloud.GetFirewall(name)
}
func (fr *FirewallRules) createFirewallObject(firewallName, description string, nodePorts []int64, nodeNames []string) (*compute.Firewall, error) {
ports := make([]string, len(nodePorts))
for ix := range nodePorts {
ports[ix] = strconv.Itoa(int(nodePorts[ix]))
}
// If the node tags to be used for this cluster have been predefined in the
// provider config, just use them. Otherwise, invoke computeHostTags method to get the tags.
targetTags, err := fr.cloud.GetNodeTags(nodeNames)
if err != nil {
return nil, err
}
return &compute.Firewall{
Name: firewallName,
Description: description,
SourceRanges: fr.srcRanges,
Network: fr.cloud.NetworkURL(),
Allowed: []*compute.FirewallAllowed{
{
IPProtocol: "tcp",
Ports: ports,
},
},
TargetTags: targetTags,
}, nil
}

15
controllers/gce/firewalls/firewalls_test.go
View file

@ -22,14 +22,11 @@ import (
"k8s.io/apimachinery/pkg/util/sets"
"k8s.io/ingress/controllers/gce/utils"
netset "k8s.io/kubernetes/pkg/util/net/sets"
)
const allCIDR = "0.0.0.0/0"
func TestSyncFirewallPool(t *testing.T) {
namer := utils.NewNamer("ABC", "XYZ")
fwp := NewFakeFirewallsProvider(namer)
fwp := NewFakeFirewallsProvider()
fp := NewFirewallPool(fwp, namer)
ruleName := namer.FrName(namer.FrSuffix())
@ -50,12 +47,16 @@ func TestSyncFirewallPool(t *testing.T) {
}
verifyFirewallRule(fwp, ruleName, nodePorts, nodes, l7SrcRanges, t)
srcRanges, _ := netset.ParseIPNets(allCIDR)
err = fwp.UpdateFirewall(namer.FrSuffix(), "", srcRanges, nodePorts, nodes)
firewall, err := fp.(*FirewallRules).createFirewallObject(namer.FrName(namer.FrSuffix()), "", nodePorts, nodes)
if err != nil {
t.Errorf("unexpected err when creating firewall object, err: %v", err)
}
err = fwp.UpdateFirewall(firewall)
if err != nil {
t.Errorf("failed to update firewall rule, err: %v", err)
}
verifyFirewallRule(fwp, ruleName, nodePorts, nodes, []string{allCIDR}, t)
verifyFirewallRule(fwp, ruleName, nodePorts, nodes, l7SrcRanges, t)
// Run Sync and expect l7 src ranges to be returned
err = fp.Sync(nodePorts, nodes)

7
controllers/gce/firewalls/interfaces.go
View file

@ -18,7 +18,6 @@ package firewalls
import (
compute "google.golang.org/api/compute/v1"
netset "k8s.io/kubernetes/pkg/util/net/sets"
)
// SingleFirewallPool syncs the firewall rule for L7 traffic.
@ -32,8 +31,10 @@ type SingleFirewallPool interface {
// This interface is a little different from the rest because it dovetails into
// the same firewall methods used by the TCPLoadBalancer.
type Firewall interface {
CreateFirewall(name, msgTag string, srcRange netset.IPNet, ports []int64, hosts []string) error
CreateFirewall(f *compute.Firewall) error
GetFirewall(name string) (*compute.Firewall, error)
DeleteFirewall(name string) error
UpdateFirewall(name, msgTag string, srcRange netset.IPNet, ports []int64, hosts []string) error
UpdateFirewall(f *compute.Firewall) error
GetNodeTags(nodeNames []string) ([]string, error)
NetworkURL() string
}

6
controllers/gce/healthchecks/healthchecks.go
View file

@ -125,6 +125,12 @@ func (h *HealthChecks) Get(port int64) (*HealthCheck, error) {
return NewHealthCheck(hc), err
}
// GetLegacy deletes legacy HTTP health checks
func (h *HealthChecks) GetLegacy(port int64) (*compute.HttpHealthCheck, error) {
name := h.namer.BeName(port)
return h.cloud.GetHttpHealthCheck(name)
}
// DeleteLegacy deletes legacy HTTP health checks
func (h *HealthChecks) DeleteLegacy(port int64) error {
name := h.namer.BeName(port)

1
controllers/gce/healthchecks/interfaces.go
View file

@ -41,5 +41,6 @@ type HealthChecker interface {
Sync(hc *HealthCheck) (string, error)
Delete(port int64) error
Get(port int64) (*HealthCheck, error)
GetLegacy(port int64) (*compute.HttpHealthCheck, error)
DeleteLegacy(port int64) error
}

34
controllers/gce/ingress-app.yaml
View file

@ -49,26 +49,26 @@ spec:
spec:
containers:
- name: echoheaders
image: gcr.io/google_containers/echoserver:1.5
image: gcr.io/google_containers/echoserver:1.8
ports:
- containerPort: 8080
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
---
# This is the Ingress resource that creates a HTTP Loadbalancer configured

68
controllers/gce/instances/fakes.go
View file

@ -18,6 +18,7 @@ package instances
import (
"fmt"
"strings"
compute "google.golang.org/api/compute/v1"
"k8s.io/apimachinery/pkg/util/sets"
@ -75,15 +76,16 @@ func (f *FakeInstanceGroups) GetInstanceGroup(name, zone string) (*compute.Insta
return ig, nil
}
}
// TODO: Return googleapi 404 error
return nil, fmt.Errorf("instance group %v not found", name)
return nil, utils.FakeGoogleAPINotFoundErr()
}
// CreateInstanceGroup fakes instance group creation.
func (f *FakeInstanceGroups) CreateInstanceGroup(name, zone string) (*compute.InstanceGroup, error) {
newGroup := &compute.InstanceGroup{Name: name, SelfLink: name, Zone: zone}
f.instanceGroups = append(f.instanceGroups, newGroup)
return newGroup, nil
func (f *FakeInstanceGroups) CreateInstanceGroup(ig *compute.InstanceGroup, zone string) error {
ig.SelfLink = ig.Name
ig.Zone = zone
f.instanceGroups = append(f.instanceGroups, ig)
return nil
}
// DeleteInstanceGroup fakes instance group deletion.
@ -110,7 +112,8 @@ func (f *FakeInstanceGroups) ListInstancesInInstanceGroup(name, zone string, sta
}
// AddInstancesToInstanceGroup fakes adding instances to an instance group.
func (f *FakeInstanceGroups) AddInstancesToInstanceGroup(name, zone string, instanceNames []string) error {
func (f *FakeInstanceGroups) AddInstancesToInstanceGroup(name, zone string, instanceRefs []*compute.InstanceReference) error {
instanceNames := toInstanceNames(instanceRefs)
f.calls = append(f.calls, utils.AddInstances)
f.instances.Insert(instanceNames...)
if _, ok := f.zonesToInstances[zone]; !ok {
@ -126,7 +129,8 @@ func (f *FakeInstanceGroups) GetInstancesByZone() map[string][]string {
}
// RemoveInstancesFromInstanceGroup fakes removing instances from an instance group.
func (f *FakeInstanceGroups) RemoveInstancesFromInstanceGroup(name, zone string, instanceNames []string) error {
func (f *FakeInstanceGroups) RemoveInstancesFromInstanceGroup(name, zone string, instanceRefs []*compute.InstanceReference) error {
instanceNames := toInstanceNames(instanceRefs)
f.calls = append(f.calls, utils.RemoveInstances)
f.instances.Delete(instanceNames...)
l, ok := f.zonesToInstances[zone]
@ -145,10 +149,23 @@ func (f *FakeInstanceGroups) RemoveInstancesFromInstanceGroup(name, zone string,
return nil
}
// AddPortToInstanceGroup fakes adding ports to an Instance Group.
func (f *FakeInstanceGroups) AddPortToInstanceGroup(ig *compute.InstanceGroup, port int64) (*compute.NamedPort, error) {
f.Ports = append(f.Ports, port)
return &compute.NamedPort{Name: f.namer.BeName(port), Port: port}, nil
func (f *FakeInstanceGroups) SetNamedPortsOfInstanceGroup(igName, zone string, namedPorts []*compute.NamedPort) error {
found := false
for _, ig := range f.instanceGroups {
if ig.Name == igName && ig.Zone == zone {
found = true
break
}
}
if !found {
return fmt.Errorf("Failed to find instance group %q in zone %q", igName, zone)
}
f.Ports = f.Ports[:0]
for _, port := range namedPorts {
f.Ports = append(f.Ports, port.Port)
}
return nil
}
// getInstanceList returns an instance list based on the given names.
@ -157,9 +174,7 @@ func getInstanceList(nodeNames sets.String) *compute.InstanceGroupsListInstances
instanceNames := nodeNames.List()
computeInstances := []*compute.InstanceWithNamedPorts{}
for _, name := range instanceNames {
instanceLink := fmt.Sprintf(
"https://www.googleapis.com/compute/v1/projects/%s/zones/%s/instances/%s",
"project", "zone", name)
instanceLink := getInstanceUrl(name)
computeInstances = append(
computeInstances, &compute.InstanceWithNamedPorts{
Instance: instanceLink})
@ -168,3 +183,26 @@ func getInstanceList(nodeNames sets.String) *compute.InstanceGroupsListInstances
Items: computeInstances,
}
}
func (f *FakeInstanceGroups) ToInstanceReferences(zone string, instanceNames []string) (refs []*compute.InstanceReference) {
for _, ins := range instanceNames {
instanceLink := getInstanceUrl(ins)
refs = append(refs, &compute.InstanceReference{Instance: instanceLink})
}
return refs
}
func getInstanceUrl(instanceName string) string {
return fmt.Sprintf("https://www.googleapis.com/compute/v1/projects/%s/zones/%s/instances/%s",
"project", "zone", instanceName)
}
func toInstanceNames(instanceRefs []*compute.InstanceReference) []string {
instanceNames := make([]string, len(instanceRefs))
for ix := range instanceRefs {
url := instanceRefs[ix].Instance
parts := strings.Split(url, "/")
instanceNames[ix] = parts[len(parts)-1]
}
return instanceNames
}

60
controllers/gce/instances/instances.go
View file

@ -63,29 +63,55 @@ func (i *Instances) Init(zl zoneLister) {
// all of which have the exact same named port.
func (i *Instances) AddInstanceGroup(name string, port int64) ([]*compute.InstanceGroup, *compute.NamedPort, error) {
igs := []*compute.InstanceGroup{}
namedPort := &compute.NamedPort{}
namedPort := utils.GetNamedPort(port)
zones, err := i.ListZones()
if err != nil {
return igs, namedPort, err
}
defer i.snapshotter.Add(name, struct{}{})
for _, zone := range zones {
ig, _ := i.Get(name, zone)
var err error
ig, err := i.Get(name, zone)
if err != nil && !utils.IsHTTPErrorCode(err, http.StatusNotFound) {
glog.Errorf("Failed to get instance group %v/%v, err: %v", zone, name, err)
return nil, nil, err
}
if ig == nil {
glog.Infof("Creating instance group %v in zone %v", name, zone)
ig, err = i.cloud.CreateInstanceGroup(name, zone)
if err = i.cloud.CreateInstanceGroup(&compute.InstanceGroup{Name: name}, zone); err != nil {
// Error may come back with StatusConflict meaning the instance group was created by another controller
// possibly the Service Controller for internal load balancers.
if utils.IsHTTPErrorCode(err, http.StatusConflict) {
glog.Warningf("Failed to create instance group %v/%v due to conflict status, but continuing sync. err: %v", zone, name, err)
} else {
glog.Errorf("Failed to create instance group %v/%v, err: %v", zone, name, err)
return nil, nil, err
}
}
ig, err = i.cloud.GetInstanceGroup(name, zone)
if err != nil {
glog.Errorf("Failed to get instance group %v/%v after ensuring existence, err: %v", zone, name, err)
return nil, nil, err
}
} else {
glog.V(3).Infof("Instance group %v already exists in zone %v, adding port %d to it", name, zone, port)
glog.V(3).Infof("Instance group %v already exists in zone %v", name, zone)
}
defer i.snapshotter.Add(name, struct{}{})
namedPort, err = i.cloud.AddPortToInstanceGroup(ig, port)
if err != nil {
return nil, nil, err
found := false
for _, np := range ig.NamedPorts {
if np.Port == port {
glog.V(3).Infof("Instance group %v already has named port %+v", ig.Name, np)
found = true
break
}
}
if !found {
glog.V(3).Infof("Instance group %v/%v does not have port %+v, adding it now.", zone, name, namedPort)
if err := i.cloud.SetNamedPortsOfInstanceGroup(ig.Name, zone, append(ig.NamedPorts, namedPort)); err != nil {
return nil, nil, err
}
}
igs = append(igs, ig)
}
@ -103,11 +129,15 @@ func (i *Instances) DeleteInstanceGroup(name string) error {
}
for _, zone := range zones {
if err := i.cloud.DeleteInstanceGroup(name, zone); err != nil {
if !utils.IsHTTPErrorCode(err, http.StatusNotFound) {
if utils.IsNotFoundError(err) {
glog.V(3).Infof("Instance group %v in zone %v did not exist", name, zone)
} else if utils.IsInUsedByError(err) {
glog.V(3).Infof("Could not delete instance group %v in zone %v because it's still in use. Ignoring: %v", name, zone, err)
} else {
errs = append(errs, err)
}
} else {
glog.Infof("Deleted instance group %v in zone %v", name, zone)
glog.V(3).Infof("Deleted instance group %v in zone %v", name, zone)
}
}
if len(errs) == 0 {
@ -173,7 +203,7 @@ func (i *Instances) Add(groupName string, names []string) error {
errs := []error{}
for zone, nodeNames := range i.splitNodesByZone(names) {
glog.V(1).Infof("Adding nodes %v to %v in zone %v", nodeNames, groupName, zone)
if err := i.cloud.AddInstancesToInstanceGroup(groupName, zone, nodeNames); err != nil {
if err := i.cloud.AddInstancesToInstanceGroup(groupName, zone, i.cloud.ToInstanceReferences(zone, nodeNames)); err != nil {
errs = append(errs, err)
}
}
@ -187,8 +217,8 @@ func (i *Instances) Add(groupName string, names []string) error {
func (i *Instances) Remove(groupName string, names []string) error {
errs := []error{}
for zone, nodeNames := range i.splitNodesByZone(names) {
glog.V(1).Infof("Adding nodes %v to %v in zone %v", nodeNames, groupName, zone)
if err := i.cloud.RemoveInstancesFromInstanceGroup(groupName, zone, nodeNames); err != nil {
glog.V(1).Infof("Removing nodes %v from %v in zone %v", nodeNames, groupName, zone)
if err := i.cloud.RemoveInstancesFromInstanceGroup(groupName, zone, i.cloud.ToInstanceReferences(zone, nodeNames)); err != nil {
errs = append(errs, err)
}
}
@ -238,7 +268,7 @@ func (i *Instances) Sync(nodes []string) (err error) {
}
if len(addNodes) != 0 {
glog.V(4).Infof("Adding nodes to IG: %v", removeNodes)
glog.V(4).Infof("Adding nodes to IG: %v", addNodes)
if err = i.Add(igName, addNodes); err != nil {
return err
}

9
controllers/gce/instances/interfaces.go
View file

@ -45,12 +45,13 @@ type NodePool interface {
// InstanceGroups is an interface for managing gce instances groups, and the instances therein.
type InstanceGroups interface {
GetInstanceGroup(name, zone string) (*compute.InstanceGroup, error)
CreateInstanceGroup(name, zone string) (*compute.InstanceGroup, error)
CreateInstanceGroup(ig *compute.InstanceGroup, zone string) error
DeleteInstanceGroup(name, zone string) error
// TODO: Refactor for modulatiry.
ListInstancesInInstanceGroup(name, zone string, state string) (*compute.InstanceGroupsListInstances, error)
AddInstancesToInstanceGroup(name, zone string, instanceNames []string) error
RemoveInstancesFromInstanceGroup(name, zone string, instanceName []string) error
AddPortToInstanceGroup(ig *compute.InstanceGroup, port int64) (*compute.NamedPort, error)
AddInstancesToInstanceGroup(name, zone string, instanceRefs []*compute.InstanceReference) error
RemoveInstancesFromInstanceGroup(name, zone string, instanceRefs []*compute.InstanceReference) error
ToInstanceReferences(zone string, instanceNames []string) (refs []*compute.InstanceReference)
SetNamedPortsOfInstanceGroup(igName, zone string, namedPorts []*compute.NamedPort) error
}

13
controllers/gce/instances/utils.go Normal file
View file

@ -0,0 +1,13 @@
package instances
import (
compute "google.golang.org/api/compute/v1"
"k8s.io/ingress/controllers/gce/utils"
)
// Helper method to create instance groups.
// This method exists to ensure that we are using the same logic at all places.
func EnsureInstanceGroupsAndPorts(nodePool NodePool, namer *utils.Namer, port int64) ([]*compute.InstanceGroup, *compute.NamedPort, error) {
return nodePool.AddInstanceGroup(namer.IGName(), port)
}

84
controllers/gce/loadbalancers/fakes.go
View file

@ -114,28 +114,21 @@ func (f *FakeLoadBalancers) GetGlobalForwardingRule(name string) (*compute.Forwa
}
// CreateGlobalForwardingRule fakes forwarding rule creation.
func (f *FakeLoadBalancers) CreateGlobalForwardingRule(proxyLink, ip, name, portRange string) (*compute.ForwardingRule, error) {
func (f *FakeLoadBalancers) CreateGlobalForwardingRule(rule *compute.ForwardingRule) error {
f.calls = append(f.calls, "CreateGlobalForwardingRule")
if ip == "" {
ip = fmt.Sprintf(testIPManager.ip())
}
rule := &compute.ForwardingRule{
Name: name,
IPAddress: ip,
Target: proxyLink,
PortRange: portRange,
IPProtocol: "TCP",
SelfLink: name,
if rule.IPAddress == "" {
rule.IPAddress = fmt.Sprintf(testIPManager.ip())
}
rule.SelfLink = rule.Name
f.Fw = append(f.Fw, rule)
return rule, nil
return nil
}
// SetProxyForGlobalForwardingRule fakes setting a global forwarding rule.
func (f *FakeLoadBalancers) SetProxyForGlobalForwardingRule(fw *compute.ForwardingRule, proxyLink string) error {
func (f *FakeLoadBalancers) SetProxyForGlobalForwardingRule(forwardingRuleName, proxyLink string) error {
f.calls = append(f.calls, "SetProxyForGlobalForwardingRule")
for i := range f.Fw {
if f.Fw[i].Name == fw.Name {
if f.Fw[i].Name == forwardingRuleName {
f.Fw[i].Target = proxyLink
}
}
@ -181,27 +174,23 @@ func (f *FakeLoadBalancers) GetUrlMap(name string) (*compute.UrlMap, error) {
}
// CreateUrlMap fakes url-map creation.
func (f *FakeLoadBalancers) CreateUrlMap(backend *compute.BackendService, name string) (*compute.UrlMap, error) {
func (f *FakeLoadBalancers) CreateUrlMap(urlMap *compute.UrlMap) error {
f.calls = append(f.calls, "CreateUrlMap")
urlMap := &compute.UrlMap{
Name: name,
DefaultService: backend.SelfLink,
SelfLink: f.umName(),
}
urlMap.SelfLink = f.umName()
f.Um = append(f.Um, urlMap)
return urlMap, nil
return nil
}
// UpdateUrlMap fakes updating url-maps.
func (f *FakeLoadBalancers) UpdateUrlMap(urlMap *compute.UrlMap) (*compute.UrlMap, error) {
func (f *FakeLoadBalancers) UpdateUrlMap(urlMap *compute.UrlMap) error {
f.calls = append(f.calls, "UpdateUrlMap")
for i := range f.Um {
if f.Um[i].Name == urlMap.Name {
f.Um[i] = urlMap
return urlMap, nil
return nil
}
}
return nil, nil
return fmt.Errorf("url map %v not found", urlMap.Name)
}
// DeleteUrlMap fakes url-map deletion.
@ -231,15 +220,11 @@ func (f *FakeLoadBalancers) GetTargetHttpProxy(name string) (*compute.TargetHttp
}
// CreateTargetHttpProxy fakes creating a target http proxy.
func (f *FakeLoadBalancers) CreateTargetHttpProxy(urlMap *compute.UrlMap, name string) (*compute.TargetHttpProxy, error) {
func (f *FakeLoadBalancers) CreateTargetHttpProxy(proxy *compute.TargetHttpProxy) error {
f.calls = append(f.calls, "CreateTargetHttpProxy")
proxy := &compute.TargetHttpProxy{
Name: name,
UrlMap: urlMap.SelfLink,
SelfLink: name,
}
proxy.SelfLink = proxy.Name
f.Tp = append(f.Tp, proxy)
return proxy, nil
return nil
}
// DeleteTargetHttpProxy fakes deleting a target http proxy.
@ -280,16 +265,11 @@ func (f *FakeLoadBalancers) GetTargetHttpsProxy(name string) (*compute.TargetHtt
}
// CreateTargetHttpsProxy fakes creating a target http proxy.
func (f *FakeLoadBalancers) CreateTargetHttpsProxy(urlMap *compute.UrlMap, cert *compute.SslCertificate, name string) (*compute.TargetHttpsProxy, error) {
func (f *FakeLoadBalancers) CreateTargetHttpsProxy(proxy *compute.TargetHttpsProxy) error {
f.calls = append(f.calls, "CreateTargetHttpsProxy")
proxy := &compute.TargetHttpsProxy{
Name: name,
UrlMap: urlMap.SelfLink,
SslCertificates: []string{cert.SelfLink},
SelfLink: name,
}
proxy.SelfLink = proxy.Name
f.Tps = append(f.Tps, proxy)
return proxy, nil
return nil
}
// DeleteTargetHttpsProxy fakes deleting a target http proxy.
@ -397,20 +377,16 @@ func (f *FakeLoadBalancers) CheckURLMap(t *testing.T, l7 *L7, expectedMap map[st
// Static IP fakes
// ReserveGlobalStaticIP fakes out static IP reservation.
func (f *FakeLoadBalancers) ReserveGlobalStaticIP(name, IPAddress string) (*compute.Address, error) {
f.calls = append(f.calls, "ReserveGlobalStaticIP")
ip := &compute.Address{
Name: name,
Address: IPAddress,
}
f.IP = append(f.IP, ip)
return ip, nil
// ReserveGlobalAddress fakes out static IP reservation.
func (f *FakeLoadBalancers) ReserveGlobalAddress(addr *compute.Address) error {
f.calls = append(f.calls, "ReserveGlobalAddress")
f.IP = append(f.IP, addr)
return nil
}
// GetGlobalStaticIP fakes out static IP retrieval.
func (f *FakeLoadBalancers) GetGlobalStaticIP(name string) (*compute.Address, error) {
f.calls = append(f.calls, "GetGlobalStaticIP")
// GetGlobalAddress fakes out static IP retrieval.
func (f *FakeLoadBalancers) GetGlobalAddress(name string) (*compute.Address, error) {
f.calls = append(f.calls, "GetGlobalAddress")
for i := range f.IP {
if f.IP[i].Name == name {
return f.IP[i], nil
@ -419,9 +395,9 @@ func (f *FakeLoadBalancers) GetGlobalStaticIP(name string) (*compute.Address, er
return nil, fmt.Errorf("static IP %v not found", name)
}
// DeleteGlobalStaticIP fakes out static IP deletion.
func (f *FakeLoadBalancers) DeleteGlobalStaticIP(name string) error {
f.calls = append(f.calls, "DeleteGlobalStaticIP")
// DeleteGlobalAddress fakes out static IP deletion.
func (f *FakeLoadBalancers) DeleteGlobalAddress(name string) error {
f.calls = append(f.calls, "DeleteGlobalAddress")
ip := []*compute.Address{}
for i := range f.IP {
if f.IP[i].Name != name {

19
controllers/gce/loadbalancers/interfaces.go
View file

@ -28,25 +28,25 @@ import (
type LoadBalancers interface {
// Forwarding Rules
GetGlobalForwardingRule(name string) (*compute.ForwardingRule, error)
CreateGlobalForwardingRule(proxyLink, ip, name, portRange string) (*compute.ForwardingRule, error)
CreateGlobalForwardingRule(rule *compute.ForwardingRule) error
DeleteGlobalForwardingRule(name string) error
SetProxyForGlobalForwardingRule(fw *compute.ForwardingRule, proxy string) error
SetProxyForGlobalForwardingRule(fw, proxy string) error
// UrlMaps
GetUrlMap(name string) (*compute.UrlMap, error)
CreateUrlMap(backend *compute.BackendService, name string) (*compute.UrlMap, error)
UpdateUrlMap(urlMap *compute.UrlMap) (*compute.UrlMap, error)
CreateUrlMap(urlMap *compute.UrlMap) error
UpdateUrlMap(urlMap *compute.UrlMap) error
DeleteUrlMap(name string) error
// TargetProxies
GetTargetHttpProxy(name string) (*compute.TargetHttpProxy, error)
CreateTargetHttpProxy(urlMap *compute.UrlMap, name string) (*compute.TargetHttpProxy, error)
CreateTargetHttpProxy(proxy *compute.TargetHttpProxy) error
DeleteTargetHttpProxy(name string) error
SetUrlMapForTargetHttpProxy(proxy *compute.TargetHttpProxy, urlMap *compute.UrlMap) error
// TargetHttpsProxies
GetTargetHttpsProxy(name string) (*compute.TargetHttpsProxy, error)
CreateTargetHttpsProxy(urlMap *compute.UrlMap, SSLCerts *compute.SslCertificate, name string) (*compute.TargetHttpsProxy, error)
CreateTargetHttpsProxy(proxy *compute.TargetHttpsProxy) error
DeleteTargetHttpsProxy(name string) error
SetUrlMapForTargetHttpsProxy(proxy *compute.TargetHttpsProxy, urlMap *compute.UrlMap) error
SetSslCertificateForTargetHttpsProxy(proxy *compute.TargetHttpsProxy, SSLCerts *compute.SslCertificate) error
@ -57,9 +57,10 @@ type LoadBalancers interface {
DeleteSslCertificate(name string) error
// Static IP
ReserveGlobalStaticIP(name, IPAddress string) (*compute.Address, error)
GetGlobalStaticIP(name string) (*compute.Address, error)
DeleteGlobalStaticIP(name string) error
ReserveGlobalAddress(addr *compute.Address) error
GetGlobalAddress(name string) (*compute.Address, error)
DeleteGlobalAddress(name string) error
}
// LoadBalancerPool is an interface to manage the cloud resources associated

76
controllers/gce/loadbalancers/loadbalancers.go
View file

@ -163,13 +163,13 @@ func (l *L7s) Delete(name string) error {
// Sync loadbalancers with the given runtime info from the controller.
func (l *L7s) Sync(lbs []*L7RuntimeInfo) error {
glog.V(3).Infof("Creating loadbalancers %+v", lbs)
glog.V(3).Infof("Syncing loadbalancers %v", lbs)
if len(lbs) != 0 {
// Lazily create a default backend so we don't tax users who don't care
// about Ingress by consuming 1 of their 3 GCE BackendServices. This
// BackendService is GC'd when there are no more Ingresses.
if err := l.defaultBackendPool.Add(l.defaultBackendNodePort); err != nil {
if err := l.defaultBackendPool.Add(l.defaultBackendNodePort, nil); err != nil {
return err
}
defaultBackend, err := l.defaultBackendPool.Get(l.defaultBackendNodePort.Port)
@ -257,6 +257,11 @@ type L7RuntimeInfo struct {
StaticIPName string
}
// String returns the load balancer name
func (l *L7RuntimeInfo) String() string {
return l.Name
}
// L7 represents a single L7 loadbalancer.
type L7 struct {
Name string
@ -304,7 +309,14 @@ func (l *L7) checkUrlMap(backend *compute.BackendService) (err error) {
}
glog.Infof("Creating url map %v for backend %v", urlMapName, l.glbcDefaultBackend.Name)
urlMap, err = l.cloud.CreateUrlMap(l.glbcDefaultBackend, urlMapName)
newUrlMap := &compute.UrlMap{
Name: urlMapName,
DefaultService: l.glbcDefaultBackend.SelfLink,
}
if err = l.cloud.CreateUrlMap(newUrlMap); err != nil {
return err
}
urlMap, err = l.cloud.GetUrlMap(urlMapName)
if err != nil {
return err
}
@ -320,7 +332,14 @@ func (l *L7) checkProxy() (err error) {
proxy, _ := l.cloud.GetTargetHttpProxy(proxyName)
if proxy == nil {
glog.Infof("Creating new http proxy for urlmap %v", l.um.Name)
proxy, err = l.cloud.CreateTargetHttpProxy(l.um, proxyName)
newProxy := &compute.TargetHttpProxy{
Name: proxyName,
UrlMap: l.um.SelfLink,
}
if err = l.cloud.CreateTargetHttpProxy(newProxy); err != nil {
return err
}
proxy, err = l.cloud.GetTargetHttpProxy(proxyName)
if err != nil {
return err
}
@ -488,10 +507,20 @@ func (l *L7) checkHttpsProxy() (err error) {
proxy, _ := l.cloud.GetTargetHttpsProxy(proxyName)
if proxy == nil {
glog.Infof("Creating new https proxy for urlmap %v", l.um.Name)
proxy, err = l.cloud.CreateTargetHttpsProxy(l.um, l.sslCert, proxyName)
newProxy := &compute.TargetHttpsProxy{
Name: proxyName,
UrlMap: l.um.SelfLink,
SslCertificates: []string{l.sslCert.SelfLink},
}
if err = l.cloud.CreateTargetHttpsProxy(newProxy); err != nil {
return err
}
proxy, err = l.cloud.GetTargetHttpsProxy(proxyName)
if err != nil {
return err
}
l.tps = proxy
return nil
}
@ -528,7 +557,17 @@ func (l *L7) checkForwardingRule(name, proxyLink, ip, portRange string) (fw *com
if fw == nil {
parts := strings.Split(proxyLink, "/")
glog.Infof("Creating forwarding rule for proxy %v and ip %v:%v", parts[len(parts)-1:], ip, portRange)
fw, err = l.cloud.CreateGlobalForwardingRule(proxyLink, ip, name, portRange)
rule := &compute.ForwardingRule{
Name: name,
IPAddress: ip,
Target: proxyLink,
PortRange: portRange,
IPProtocol: "TCP",
}
if err = l.cloud.CreateGlobalForwardingRule(rule); err != nil {
return nil, err
}
fw, err = l.cloud.GetGlobalForwardingRule(name)
if err != nil {
return nil, err
}
@ -539,7 +578,7 @@ func (l *L7) checkForwardingRule(name, proxyLink, ip, portRange string) (fw *com
} else {
glog.Infof("Forwarding rule %v has the wrong proxy, setting %v overwriting %v",
fw.Name, fw.Target, proxyLink)
if err := l.cloud.SetProxyForGlobalForwardingRule(fw, proxyLink); err != nil {
if err := l.cloud.SetProxyForGlobalForwardingRule(fw.Name, proxyLink); err != nil {
return nil, err
}
}
@ -571,7 +610,7 @@ func (l *L7) getEffectiveIP() (string, bool) {
if l.runtimeInfo.StaticIPName != "" {
// Existing static IPs allocated to forwarding rules will get orphaned
// till the Ingress is torn down.
if ip, err := l.cloud.GetGlobalStaticIP(l.runtimeInfo.StaticIPName); err != nil || ip == nil {
if ip, err := l.cloud.GetGlobalAddress(l.runtimeInfo.StaticIPName); err != nil || ip == nil {
glog.Warningf("The given static IP name %v doesn't translate to an existing global static IP, ignoring it and allocating a new IP: %v",
l.runtimeInfo.StaticIPName, err)
} else {
@ -624,10 +663,10 @@ func (l *L7) checkStaticIP() (err error) {
return nil
}
staticIPName := l.namer.Truncate(fmt.Sprintf("%v-%v", forwardingRulePrefix, l.Name))
ip, _ := l.cloud.GetGlobalStaticIP(staticIPName)
ip, _ := l.cloud.GetGlobalAddress(staticIPName)
if ip == nil {
glog.Infof("Creating static ip %v", staticIPName)
ip, err = l.cloud.ReserveGlobalStaticIP(staticIPName, l.fw.IPAddress)
err = l.cloud.ReserveGlobalAddress(&compute.Address{Name: staticIPName, Address: l.fw.IPAddress})
if err != nil {
if utils.IsHTTPErrorCode(err, http.StatusConflict) ||
utils.IsHTTPErrorCode(err, http.StatusBadRequest) {
@ -637,6 +676,10 @@ func (l *L7) checkStaticIP() (err error) {
}
return err
}
ip, err = l.cloud.GetGlobalAddress(staticIPName)
if err != nil {
return err
}
}
l.ip = ip
return nil
@ -757,7 +800,6 @@ func (l *L7) UpdateUrlMap(ingressRules utils.GCEURLMap) error {
if l.um == nil {
return fmt.Errorf("cannot add url without an urlmap")
}
glog.V(3).Infof("Updating urlmap for l7 %v", l.Name)
// All UrlMaps must have a default backend. If the Ingress has a default
// backend, it applies to all host rules as well as to the urlmap itself.
@ -807,11 +849,17 @@ func (l *L7) UpdateUrlMap(ingressRules utils.GCEURLMap) error {
glog.Infof("UrlMap for l7 %v is unchanged", l.Name)
return nil
}
glog.Infof("Updating url map: %+v", ingressRules)
um, err := l.cloud.UpdateUrlMap(l.um)
glog.V(3).Infof("Updating URLMap: %q", l.Name)
if err := l.cloud.UpdateUrlMap(l.um); err != nil {
return err
}
um, err := l.cloud.GetUrlMap(l.um.Name)
if err != nil {
return err
}
l.um = um
return nil
}
@ -898,7 +946,7 @@ func (l *L7) Cleanup() error {
}
if l.ip != nil {
glog.V(2).Infof("Deleting static IP %v(%v)", l.ip.Name, l.ip.Address)
if err := utils.IgnoreHTTPNotFound(l.cloud.DeleteGlobalStaticIP(l.ip.Name)); err != nil {
if err := utils.IgnoreHTTPNotFound(l.cloud.DeleteGlobalAddress(l.ip.Name)); err != nil {
return err
}
l.ip = nil

2
controllers/gce/loadbalancers/loadbalancers_test.go
View file

@ -289,7 +289,7 @@ func TestCreateBothLoadBalancers(t *testing.T) {
if err != nil || fw.Target != tp.SelfLink {
t.Fatalf("%v", err)
}
ip, err := f.GetGlobalStaticIP(f.fwName(false))
ip, err := f.GetGlobalAddress(f.fwName(false))
if err != nil || ip.Address != fw.IPAddress || ip.Address != fws.IPAddress {
t.Fatalf("%v", err)
}

102
controllers/gce/main.go
View file

@ -17,8 +17,11 @@ limitations under the License.
package main
import (
"bytes"
go_flag "flag"
"fmt"
"io"
"io/ioutil"
"net/http"
"os"
"os/signal"
@ -30,14 +33,13 @@ import (
"github.com/prometheus/client_golang/prometheus/promhttp"
flag "github.com/spf13/pflag"
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/intstr"
"k8s.io/apimachinery/pkg/util/wait"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/pkg/api"
api_v1 "k8s.io/client-go/pkg/api/v1"
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/clientcmd"
clientcmdapi "k8s.io/client-go/tools/clientcmd/api"
@ -47,6 +49,8 @@ import (
"k8s.io/ingress/controllers/gce/loadbalancers"
"k8s.io/ingress/controllers/gce/storage"
"k8s.io/ingress/controllers/gce/utils"
"k8s.io/kubernetes/pkg/cloudprovider"
"k8s.io/kubernetes/pkg/cloudprovider/providers/gce"
)
// Entrypoint of GLBC. Example invocation:
@ -69,10 +73,13 @@ const (
alphaNumericChar = "0"
// Current docker image version. Only used in debug logging.
imageVersion = "glbc:0.9.4"
imageVersion = "glbc:0.9.6"
// Key used to persist UIDs to configmaps.
uidConfigMapName = "ingress-uid"
// Sleep interval to retry cloud client creation.
cloudClientRetryInterval = 10 * time.Second
)
var (
@ -122,7 +129,7 @@ var (
`Path used to health-check a backend service. All Services must serve
a 200 page on this path. Currently this is only configurable globally.`)
watchNamespace = flags.String("watch-namespace", api.NamespaceAll,
watchNamespace = flags.String("watch-namespace", v1.NamespaceAll,
`Namespace to watch for Ingress/Services/Endpoints.`)
verbose = flags.Bool("verbose", false,
@ -242,13 +249,36 @@ func main() {
SvcPort: intstr.FromInt(int(port)),
}
var cloud *gce.GCECloud
if *inCluster || *useRealCloud {
// Create cluster manager
namer, err := newNamer(kubeClient, *clusterName, controller.DefaultFirewallName)
if err != nil {
glog.Fatalf("%v", err)
}
clusterManager, err = controller.NewClusterManager(*configFilePath, namer, defaultBackendNodePort, *healthCheckPath)
// TODO: Make this more resilient. Currently we create the cloud client
// and pass it through to all the pools. This makes unit testing easier.
// However if the cloud client suddenly fails, we should try to re-create it
// and continue.
if *configFilePath != "" {
glog.Infof("Reading config from path %v", configFilePath)
config, err := os.Open(*configFilePath)
if err != nil {
glog.Fatalf("%v", err)
}
defer config.Close()
cloud = getGCEClient(config)
glog.Infof("Successfully loaded cloudprovider using config %q", configFilePath)
} else {
// While you might be tempted to refactor so we simply assing nil to the
// config and only invoke getGCEClient once, that will not do the right
// thing because a nil check against an interface isn't true in golang.
cloud = getGCEClient(nil)
glog.Infof("Created GCE client without a config file")
}
clusterManager, err = controller.NewClusterManager(cloud, namer, defaultBackendNodePort, *healthCheckPath)
if err != nil {
glog.Fatalf("%v", err)
}
@ -257,11 +287,14 @@ func main() {
clusterManager = controller.NewFakeClusterManager(*clusterName, controller.DefaultFirewallName).ClusterManager
}
ctx := controller.NewControllerContext(kubeClient, *watchNamespace, *resyncPeriod)
// Start loadbalancer controller
lbc, err := controller.NewLoadBalancerController(kubeClient, clusterManager, *resyncPeriod, *watchNamespace)
lbc, err := controller.NewLoadBalancerController(kubeClient, ctx, clusterManager)
if err != nil {
glog.Fatalf("%v", err)
}
if clusterManager.ClusterNamer.GetClusterName() != "" {
glog.V(3).Infof("Cluster name %+v", clusterManager.ClusterNamer.GetClusterName())
}
@ -269,6 +302,7 @@ func main() {
go registerHandlers(lbc)
go handleSigterm(lbc, *deleteAllOnQuit)
ctx.Start()
lbc.Run()
for {
glog.Infof("Handled quit, awaiting pod deletion.")
@ -287,7 +321,7 @@ func newNamer(kubeClient kubernetes.Interface, clusterName string, fwName string
}
namer := utils.NewNamer(name, fw_name)
uidVault := storage.NewConfigMapVault(kubeClient, api.NamespaceSystem, uidConfigMapName)
uidVault := storage.NewConfigMapVault(kubeClient, metav1.NamespaceSystem, uidConfigMapName)
// Start a goroutine to poll the cluster UID config map
// We don't watch because we know exactly which configmap we want and this
@ -359,7 +393,7 @@ func useDefaultOrLookupVault(cfgVault *storage.ConfigMapVault, cm_key, default_n
// Use getFlagOrLookupVault to obtain a stored or overridden value for the firewall name.
// else, use the cluster UID as a backup (this retains backwards compatibility).
func getFirewallName(kubeClient kubernetes.Interface, name, cluster_uid string) (string, error) {
cfgVault := storage.NewConfigMapVault(kubeClient, api.NamespaceSystem, uidConfigMapName)
cfgVault := storage.NewConfigMapVault(kubeClient, metav1.NamespaceSystem, uidConfigMapName)
if fw_name, err := useDefaultOrLookupVault(cfgVault, storage.ProviderDataKey, name); err != nil {
return "", err
} else if fw_name != "" {
@ -377,7 +411,7 @@ func getFirewallName(kubeClient kubernetes.Interface, name, cluster_uid string)
// - remember that "" is the cluster uid
// else, allocate a new uid
func getClusterUID(kubeClient kubernetes.Interface, name string) (string, error) {
cfgVault := storage.NewConfigMapVault(kubeClient, api.NamespaceSystem, uidConfigMapName)
cfgVault := storage.NewConfigMapVault(kubeClient, metav1.NamespaceSystem, uidConfigMapName)
if name, err := useDefaultOrLookupVault(cfgVault, storage.UidDataKey, name); err != nil {
return "", err
} else if name != "" {
@ -385,7 +419,7 @@ func getClusterUID(kubeClient kubernetes.Interface, name string) (string, error)
}
// Check if the cluster has an Ingress with ip
ings, err := kubeClient.Extensions().Ingresses(api.NamespaceAll).List(meta_v1.ListOptions{
ings, err := kubeClient.Extensions().Ingresses(metav1.NamespaceAll).List(metav1.ListOptions{
LabelSelector: labels.Everything().String(),
})
if err != nil {
@ -419,10 +453,10 @@ func getClusterUID(kubeClient kubernetes.Interface, name string) (string, error)
// getNodePort waits for the Service, and returns it's first node port.
func getNodePort(client kubernetes.Interface, ns, name string) (port, nodePort int32, err error) {
var svc *api_v1.Service
var svc *v1.Service
glog.V(3).Infof("Waiting for %v/%v", ns, name)
wait.Poll(1*time.Second, 5*time.Minute, func() (bool, error) {
svc, err = client.Core().Services(ns).Get(name, meta_v1.GetOptions{})
svc, err = client.Core().Services(ns).Get(name, metav1.GetOptions{})
if err != nil {
return false, nil
}
@ -438,3 +472,45 @@ func getNodePort(client kubernetes.Interface, ns, name string) (port, nodePort i
})
return
}
func getGCEClient(config io.Reader) *gce.GCECloud {
getConfigReader := func() io.Reader { return nil }
if config != nil {
allConfig, err := ioutil.ReadAll(config)
if err != nil {
glog.Fatalf("Error while reading entire config: %v", err)
}
glog.V(2).Infof("Using cloudprovider config file:\n%v ", string(allConfig))
getConfigReader = func() io.Reader {
return bytes.NewReader(allConfig)
}
} else {
glog.V(2).Infoln("No cloudprovider config file provided. Continuing with default values.")
}
// Creating the cloud interface involves resolving the metadata server to get
// an oauth token. If this fails, the token provider assumes it's not on GCE.
// No errors are thrown. So we need to keep retrying till it works because
// we know we're on GCE.
for {
cloudInterface, err := cloudprovider.GetCloudProvider("gce", getConfigReader())
if err == nil {
cloud := cloudInterface.(*gce.GCECloud)
// If this controller is scheduled on a node without compute/rw
// it won't be allowed to list backends. We can assume that the
// user has no need for Ingress in this case. If they grant
// permissions to the node they will have to restart the controller
// manually to re-create the client.
if _, err = cloud.ListGlobalBackendServices(); err == nil || utils.IsHTTPErrorCode(err, http.StatusForbidden) {
return cloud
}
glog.Warningf("Failed to list backend services, retrying: %v", err)
} else {
glog.Warningf("Failed to retrieve cloud interface, retrying: %v", err)
}
time.Sleep(cloudClientRetryInterval)
}
}

8
controllers/gce/rc.yaml
View file

@ -24,18 +24,18 @@ metadata:
name: l7-lb-controller
labels:
k8s-app: glbc
version: v0.9.4
version: v0.9.6
spec:
# There should never be more than 1 controller alive simultaneously.
replicas: 1
selector:
k8s-app: glbc
version: v0.9.4
version: v0.9.6
template:
metadata:
labels:
k8s-app: glbc
version: v0.9.4
version: v0.9.6
name: glbc
spec:
terminationGracePeriodSeconds: 600
@ -61,7 +61,7 @@ spec:
requests:
cpu: 10m
memory: 20Mi
- image: gcr.io/google_containers/glbc:0.9.4
- image: gcr.io/google_containers/glbc:0.9.6
livenessProbe:
httpGet:
path: /healthz

2
controllers/gce/storage/configmaps.go
View file

@ -23,10 +23,10 @@ import (
"github.com/golang/glog"
api_v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
api_v1 "k8s.io/client-go/pkg/api/v1"
"k8s.io/client-go/tools/cache"
)

2
controllers/gce/storage/configmaps_test.go
View file

@ -19,7 +19,7 @@ package storage
import (
"testing"
"k8s.io/client-go/pkg/api"
api "k8s.io/apimachinery/pkg/apis/meta/v1"
)
func TestConfigMapUID(t *testing.T) {

22
controllers/gce/utils/utils.go
View file

@ -191,7 +191,6 @@ func (n *Namer) ParseName(name string) *NameComponents {
// NameBelongsToCluster checks if a given name is tagged with this cluster's UID.
func (n *Namer) NameBelongsToCluster(name string) bool {
if !strings.HasPrefix(name, "k8s-") {
glog.V(4).Infof("%v not part of cluster", name)
return false
}
parts := strings.Split(name, clusterNameDelimiter)
@ -203,7 +202,6 @@ func (n *Namer) NameBelongsToCluster(name string) bool {
return false
}
if len(parts) > 2 {
glog.Warningf("Too many parts to name %v, ignoring", name)
return false
}
return parts[1] == clusterName
@ -332,6 +330,20 @@ func IgnoreHTTPNotFound(err error) error {
return err
}
// IsInUsedByError returns true if the resource is being used by another GCP resource
func IsInUsedByError(err error) bool {
apiErr, ok := err.(*googleapi.Error)
if !ok || apiErr.Code != http.StatusBadRequest {
return false
}
return strings.Contains(apiErr.Message, "being used by")
}
// IsNotFoundError returns true if the resource does not exist
func IsNotFoundError(err error) bool {
return IsHTTPErrorCode(err, http.StatusNotFound)
}
// CompareLinks returns true if the 2 self links are equal.
func CompareLinks(l1, l2 string) bool {
// TODO: These can be partial links
@ -341,3 +353,9 @@ func CompareLinks(l1, l2 string) bool {
// FakeIngressRuleValueMap is a convenience type used by multiple submodules
// that share the same testing methods.
type FakeIngressRuleValueMap map[string]string
// GetNamedPort creates the NamedPort API object for the given port.
func GetNamedPort(port int64) *compute.NamedPort {
// TODO: move port naming to namer
return &compute.NamedPort{Name: fmt.Sprintf("port%v", port), Port: port}
}

2
controllers/nginx/.dockerignore Normal file
View file

@ -0,0 +1,2 @@
core

389
controllers/nginx/Changelog.md
View file

@ -1,5 +1,286 @@
Changelog
Changelog
### 0.9-beta.13
**Image:** `gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.13`
*New Features:*
- NGINX 1.3.5
- New flag to disable node listing
- Custom X-Forwarder-Header (CloudFlare uses `CF-Connecting-IP` as header)
- Custom error page in Client Certificate Authentication
*Changes:*
- [X] [#1272](https://github.com/kubernetes/ingress/pull/1272) Delete useless statement
- [X] [#1277](https://github.com/kubernetes/ingress/pull/1277) Add indent for nginx.conf
- [X] [#1278](https://github.com/kubernetes/ingress/pull/1278) Add proxy-pass-params annotation and Backend field
- [X] [#1282](https://github.com/kubernetes/ingress/pull/1282) Fix nginx stats
- [X] [#1288](https://github.com/kubernetes/ingress/pull/1288) Allow PATCH in enable-cors
- [X] [#1290](https://github.com/kubernetes/ingress/pull/1290) Add flag to disabling node listing
- [X] [#1293](https://github.com/kubernetes/ingress/pull/1293) Adds support for error page in Client Certificate Authentication
- [X] [#1308](https://github.com/kubernetes/ingress/pull/1308) A trivial typo in config
- [X] [#1310](https://github.com/kubernetes/ingress/pull/1310) Refactoring nginx configuration configmap
- [X] [#1311](https://github.com/kubernetes/ingress/pull/1311) Enable nginx async writes
- [X] [#1312](https://github.com/kubernetes/ingress/pull/1312) Allow custom forwarded for header
- [X] [#1313](https://github.com/kubernetes/ingress/pull/1313) Fix eol in nginx template
- [X] [#1315](https://github.com/kubernetes/ingress/pull/1315) Fix nginx custom error pages
*Documentation:*
- [X] [#1270](https://github.com/kubernetes/ingress/pull/1270) add missing yamls in controllers/nginx
- [X] [#1276](https://github.com/kubernetes/ingress/pull/1276) Link rbac sample from deployment docs
- [X] [#1291](https://github.com/kubernetes/ingress/pull/1291) fix link to conformance suite
- [X] [#1295](https://github.com/kubernetes/ingress/pull/1295) fix README of nginx-ingress-controller
- [X] [#1299](https://github.com/kubernetes/ingress/pull/1299) fix two doc issues in nginx/README
- [X] [#1306](https://github.com/kubernetes/ingress/pull/1306) Fix kubeconfig example for nginx deployment
### 0.9-beta.12
**Image:** `gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.12`
*Breaking changes:*
- SSL passthrough is disabled by default. To enable the feature use `--enable-ssl-passthrough`
*New Features:*
- Support for arm64
- New flags to customize listen ports
- Per minute rate limiting
- Rate limit whitelist
- Configuration of nginx worker timeout (to avoid zombie nginx workers processes)
- Redirects from non-www to www
- Custom default backend (per Ingress)
- Graceful shutdown for NGINX
*Changes:*
- [X] [#977](https://github.com/kubernetes/ingress/pull/977) Add sort-backends command line option
- [X] [#981](https://github.com/kubernetes/ingress/pull/981) Add annotation to allow use of service ClusterIP for NGINX upstream.
- [X] [#991](https://github.com/kubernetes/ingress/pull/991) Remove secret sync loop
- [X] [#992](https://github.com/kubernetes/ingress/pull/992) Check errors generating pem files
- [X] [#993](https://github.com/kubernetes/ingress/pull/993) Fix the sed command to work on macOS
- [X] [#1013](https://github.com/kubernetes/ingress/pull/1013) The fields of vtsDate are unified in the form of plural
- [X] [#1025](https://github.com/kubernetes/ingress/pull/1025) Fix file watch
- [X] [#1027](https://github.com/kubernetes/ingress/pull/1027) Lint code
- [X] [#1031](https://github.com/kubernetes/ingress/pull/1031) Change missing secret name log level to V(3)
- [X] [#1032](https://github.com/kubernetes/ingress/pull/1032) Alternative syncSecret approach #1030
- [X] [#1042](https://github.com/kubernetes/ingress/pull/1042) Add function to allow custom values in Ingress status
- [X] [#1043](https://github.com/kubernetes/ingress/pull/1043) Return reference to object providing Endpoint
- [X] [#1046](https://github.com/kubernetes/ingress/pull/1046) Add field FileSHA in BasicDigest struct
- [X] [#1058](https://github.com/kubernetes/ingress/pull/1058) add per minute rate limiting
- [X] [#1060](https://github.com/kubernetes/ingress/pull/1060) Update fsnotify dependency to fix arm64 issue
- [X] [#1065](https://github.com/kubernetes/ingress/pull/1065) Add more descriptive steps in Dev Documentation
- [X] [#1073](https://github.com/kubernetes/ingress/pull/1073) Release nginx-slim 0.22
- [X] [#1074](https://github.com/kubernetes/ingress/pull/1074) Remove lua and use fastcgi to render errors
- [X] [#1075](https://github.com/kubernetes/ingress/pull/1075) (feat/ #374) support proxy timeout
- [X] [#1076](https://github.com/kubernetes/ingress/pull/1076) Add more ssl test cases
- [X] [#1078](https://github.com/kubernetes/ingress/pull/1078) fix the same udp port and tcp port, update nginx.conf error
- [X] [#1080](https://github.com/kubernetes/ingress/pull/1080) Disable platform s390x
- [X] [#1081](https://github.com/kubernetes/ingress/pull/1081) Spit Static check and Coverage in diff Stages of Travis CI
- [X] [#1082](https://github.com/kubernetes/ingress/pull/1082) Fix build tasks
- [X] [#1087](https://github.com/kubernetes/ingress/pull/1087) Release nginx-slim 0.23
- [X] [#1088](https://github.com/kubernetes/ingress/pull/1088) Configure nginx worker timeout
- [X] [#1089](https://github.com/kubernetes/ingress/pull/1089) Update nginx to 1.13.4
- [X] [#1098](https://github.com/kubernetes/ingress/pull/1098) Exposing the event recorder to allow other controllers to create events
- [X] [#1102](https://github.com/kubernetes/ingress/pull/1102) Fix lose SSL Passthrough
- [X] [#1104](https://github.com/kubernetes/ingress/pull/1104) Simplify verification of hostname in ssl certificates
- [X] [#1109](https://github.com/kubernetes/ingress/pull/1109) Cleanup remote address in nginx template
- [X] [#1110](https://github.com/kubernetes/ingress/pull/1110) Fix Endpoint comparison
- [X] [#1118](https://github.com/kubernetes/ingress/pull/1118) feat(#733)Support nginx bandwidth control
- [X] [#1124](https://github.com/kubernetes/ingress/pull/1124) check fields len in dns.go
- [X] [#1130](https://github.com/kubernetes/ingress/pull/1130) Update nginx.go
- [X] [#1134](https://github.com/kubernetes/ingress/pull/1134) replace deprecated interface with versioned ones
- [X] [#1136](https://github.com/kubernetes/ingress/pull/1136) Fix status update - changed in #1074
- [X] [#1138](https://github.com/kubernetes/ingress/pull/1138) update nginx.go: preformance improve
- [X] [#1139](https://github.com/kubernetes/ingress/pull/1139) Fix Todo:convert sequence to table
- [X] [#1162](https://github.com/kubernetes/ingress/pull/1162) Optimize CI build time
- [X] [#1164](https://github.com/kubernetes/ingress/pull/1164) Use variable request_uri as redirect after auth
- [X] [#1179](https://github.com/kubernetes/ingress/pull/1179) Fix sticky upstream not used when enable rewrite
- [X] [#1184](https://github.com/kubernetes/ingress/pull/1184) Add support for temporal and permanent redirects
- [X] [#1185](https://github.com/kubernetes/ingress/pull/1185) Add more info about Server-Alias usage
- [X] [#1186](https://github.com/kubernetes/ingress/pull/1186) Add annotation for client-body-buffer-size per location
- [X] [#1190](https://github.com/kubernetes/ingress/pull/1190) Add flag to disable SSL passthrough
- [X] [#1193](https://github.com/kubernetes/ingress/pull/1193) fix broken link
- [X] [#1198](https://github.com/kubernetes/ingress/pull/1198) Add option for specific scheme for base url
- [X] [#1202](https://github.com/kubernetes/ingress/pull/1202) formatIP issue
- [X] [#1203](https://github.com/kubernetes/ingress/pull/1203) NGINX not reloading correctly
- [X] [#1204](https://github.com/kubernetes/ingress/pull/1204) Fix template error
- [X] [#1205](https://github.com/kubernetes/ingress/pull/1205) Add initial sync of secrets
- [X] [#1206](https://github.com/kubernetes/ingress/pull/1206) Update ssl-passthrough docs
- [X] [#1207](https://github.com/kubernetes/ingress/pull/1207) delete broken link
- [X] [#1208](https://github.com/kubernetes/ingress/pull/1208) fix some typo
- [X] [#1210](https://github.com/kubernetes/ingress/pull/1210) add rate limit whitelist
- [X] [#1215](https://github.com/kubernetes/ingress/pull/1215) Replace base64 encoding with random uuid
- [X] [#1218](https://github.com/kubernetes/ingress/pull/1218) Trivial fixes in core/pkg/net
- [X] [#1219](https://github.com/kubernetes/ingress/pull/1219) keep zones unique per ingress resource
- [X] [#1221](https://github.com/kubernetes/ingress/pull/1221) Move certificate authentication from location to server
- [X] [#1223](https://github.com/kubernetes/ingress/pull/1223) Add doc for non-www to www annotation
- [X] [#1224](https://github.com/kubernetes/ingress/pull/1224) refactor rate limit whitelist
- [X] [#1226](https://github.com/kubernetes/ingress/pull/1226) Remove useless variable in nginx.tmpl
- [X] [#1227](https://github.com/kubernetes/ingress/pull/1227) Update annotations doc with base-url-scheme
- [X] [#1233](https://github.com/kubernetes/ingress/pull/1233) Fix ClientBodyBufferSize annotation
- [X] [#1234](https://github.com/kubernetes/ingress/pull/1234) Lint code
- [X] [#1235](https://github.com/kubernetes/ingress/pull/1235) Fix Equal comparison
- [X] [#1236](https://github.com/kubernetes/ingress/pull/1236) Add Validation for Client Body Buffer Size
- [X] [#1238](https://github.com/kubernetes/ingress/pull/1238) Add support for 'client_body_timeout' and 'client_header_timeout'
- [X] [#1239](https://github.com/kubernetes/ingress/pull/1239) Add flags to customize listen ports and detect port collisions
- [X] [#1243](https://github.com/kubernetes/ingress/pull/1243) Add support for access-log-path and error-log-path
- [X] [#1244](https://github.com/kubernetes/ingress/pull/1244) Add custom default backend annotation
- [X] [#1246](https://github.com/kubernetes/ingress/pull/1246) Add additional headers when custom default backend is used
- [X] [#1247](https://github.com/kubernetes/ingress/pull/1247) Make Ingress annotations available in template
- [X] [#1248](https://github.com/kubernetes/ingress/pull/1248) Improve nginx controller performance
- [X] [#1254](https://github.com/kubernetes/ingress/pull/1254) fix Type transform panic
- [X] [#1257](https://github.com/kubernetes/ingress/pull/1257) Graceful shutdown for Nginx
- [X] [#1261](https://github.com/kubernetes/ingress/pull/1261) Add support for 'worker-shutdown-timeout'
*Documentation:*
- [X] [#976](https://github.com/kubernetes/ingress/pull/976) Update annotations doc
- [X] [#979](https://github.com/kubernetes/ingress/pull/979) Missing auth example
- [X] [#980](https://github.com/kubernetes/ingress/pull/980) Add nginx basic auth example
- [X] [#1001](https://github.com/kubernetes/ingress/pull/1001) examples/nginx/rbac: Give access to own namespace
- [X] [#1005](https://github.com/kubernetes/ingress/pull/1005) Update configuration.md
- [X] [#1018](https://github.com/kubernetes/ingress/pull/1018) add docs for `proxy-set-headers` and `add-headers`
- [X] [#1038](https://github.com/kubernetes/ingress/pull/1038) typo / spelling in README.md
- [X] [#1039](https://github.com/kubernetes/ingress/pull/1039) typo in examples/tcp/nginx/README.md
- [X] [#1049](https://github.com/kubernetes/ingress/pull/1049) Fix config name in the example.
- [X] [#1054](https://github.com/kubernetes/ingress/pull/1054) Fix link to UDP example
- [X] [#1084](https://github.com/kubernetes/ingress/pull/1084) (issue #310)Fix some broken link
- [X] [#1103](https://github.com/kubernetes/ingress/pull/1103) Add GoDoc Widget
- [X] [#1105](https://github.com/kubernetes/ingress/pull/1105) Make Readme file more readable
- [X] [#1106](https://github.com/kubernetes/ingress/pull/1106) Update annotations.md
- [X] [#1107](https://github.com/kubernetes/ingress/pull/1107) Fix Broken Link
- [X] [#1119](https://github.com/kubernetes/ingress/pull/1119) fix typos in controllers/nginx/README.md
- [X] [#1122](https://github.com/kubernetes/ingress/pull/1122) Fix broken link
- [X] [#1131](https://github.com/kubernetes/ingress/pull/1131) Add short help doc in configuration for nginx limit rate
- [X] [#1143](https://github.com/kubernetes/ingress/pull/1143) Minor Typo Fix
- [X] [#1144](https://github.com/kubernetes/ingress/pull/1144) Minor Typo fix
- [X] [#1145](https://github.com/kubernetes/ingress/pull/1145) Minor Typo fix
- [X] [#1146](https://github.com/kubernetes/ingress/pull/1146) Fix Minor Typo in Readme
- [X] [#1147](https://github.com/kubernetes/ingress/pull/1147) Minor Typo Fix
- [X] [#1148](https://github.com/kubernetes/ingress/pull/1148) Minor Typo Fix in Getting-Started.md
- [X] [#1149](https://github.com/kubernetes/ingress/pull/1149) Fix Minor Typo in TLS authentication
- [X] [#1150](https://github.com/kubernetes/ingress/pull/1150) Fix Minor Typo in Customize the HAProxy configuration
- [X] [#1151](https://github.com/kubernetes/ingress/pull/1151) Fix Minor Typo in customization custom-template
- [X] [#1152](https://github.com/kubernetes/ingress/pull/1152) Fix minor typo in HAProxy Multi TLS certificate termination
- [X] [#1153](https://github.com/kubernetes/ingress/pull/1153) Fix minor typo in Multi TLS certificate termination
- [X] [#1154](https://github.com/kubernetes/ingress/pull/1154) Fix minor typo in Role Based Access Control
- [X] [#1155](https://github.com/kubernetes/ingress/pull/1155) Fix minor typo in TCP loadbalancing
- [X] [#1156](https://github.com/kubernetes/ingress/pull/1156) Fix minor typo in UDP loadbalancing
- [X] [#1157](https://github.com/kubernetes/ingress/pull/1157) Fix minor typos in Prerequisites
- [X] [#1158](https://github.com/kubernetes/ingress/pull/1158) Fix minor typo in Ingress examples
- [X] [#1159](https://github.com/kubernetes/ingress/pull/1159) Fix minor typos in Ingress admin guide
- [X] [#1160](https://github.com/kubernetes/ingress/pull/1160) Fix a broken href and typo in Ingress FAQ
- [X] [#1165](https://github.com/kubernetes/ingress/pull/1165) Update CONTRIBUTING.md
- [X] [#1168](https://github.com/kubernetes/ingress/pull/1168) finx link to running-locally.md
- [X] [#1170](https://github.com/kubernetes/ingress/pull/1170) Update dead link in nginx/HTTPS section
- [X] [#1172](https://github.com/kubernetes/ingress/pull/1172) Update README.md
- [X] [#1173](https://github.com/kubernetes/ingress/pull/1173) Update admin.md
- [X] [#1174](https://github.com/kubernetes/ingress/pull/1174) fix several titles
- [X] [#1177](https://github.com/kubernetes/ingress/pull/1177) fix typos
- [X] [#1188](https://github.com/kubernetes/ingress/pull/1188) Fix minor typo
- [X] [#1189](https://github.com/kubernetes/ingress/pull/1189) Fix sign in URL redirect parameter
- [X] [#1192](https://github.com/kubernetes/ingress/pull/1192) Update README.md
- [X] [#1195](https://github.com/kubernetes/ingress/pull/1195) Update troubleshooting.md
- [X] [#1196](https://github.com/kubernetes/ingress/pull/1196) Update README.md
- [X] [#1209](https://github.com/kubernetes/ingress/pull/1209) Update README.md
- [X] [#1085](https://github.com/kubernetes/ingress/pull/1085) Fix ConfigMap's namespace in custom configuration example for nginx
- [X] [#1142](https://github.com/kubernetes/ingress/pull/1142) Fix typo in multiple docs
- [X] [#1228](https://github.com/kubernetes/ingress/pull/1228) Update release doc in getting-started.md
- [X] [#1230](https://github.com/kubernetes/ingress/pull/1230) Update godep guide link
### 0.9-beta.11
**Image:** `gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.11`
Fixes NGINX [CVE-2017-7529](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-7529)
*Changes:*
- [X] [#659](https://github.com/kubernetes/ingress/pull/659) [nginx] TCP configmap should allow listen proxy_protocol per service
- [X] [#730](https://github.com/kubernetes/ingress/pull/730) Add support for add_headers
- [X] [#808](https://github.com/kubernetes/ingress/pull/808) HTTP->HTTPS redirect does not work with use-proxy-protocol: "true"
- [X] [#921](https://github.com/kubernetes/ingress/pull/921) Make proxy-real-ip-cidr a comma separated list
- [X] [#930](https://github.com/kubernetes/ingress/pull/930) Add support for proxy protocol in TCP services
- [X] [#933](https://github.com/kubernetes/ingress/pull/933) Lint code
- [X] [#937](https://github.com/kubernetes/ingress/pull/937) Fix lint code errors
- [X] [#940](https://github.com/kubernetes/ingress/pull/940) Sets parameters for a shared memory zone of limit_conn_zone
- [X] [#949](https://github.com/kubernetes/ingress/pull/949) fix nginx version to 1.13.3 to fix integer overflow
- [X] [#956](https://github.com/kubernetes/ingress/pull/956) Simplify handling of ssl certificates
- [X] [#958](https://github.com/kubernetes/ingress/pull/958) Release ubuntu-slim:0.13
- [X] [#959](https://github.com/kubernetes/ingress/pull/959) Release nginx-slim 0.21
- [X] [#960](https://github.com/kubernetes/ingress/pull/960) Update nginx in ingress controller
- [X] [#964](https://github.com/kubernetes/ingress/pull/964) Support for proxy_headers_hash_bucket_size and proxy_headers_hash_max_size
- [X] [#966](https://github.com/kubernetes/ingress/pull/966) Fix error checking for pod name & NS
- [X] [#967](https://github.com/kubernetes/ingress/pull/967) Fix runningAddresses typo
- [X] [#968](https://github.com/kubernetes/ingress/pull/968) Fix missing hyphen in yaml for nginx RBAC example
- [X] [#973](https://github.com/kubernetes/ingress/pull/973) check number of servers in configuration comparator
### 0.9-beta.10
**Image:** `gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.10`
Fix release 0.9-beta.9
### 0.9-beta.9
**Image:** `gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.9`
*New Features:*
- Add support for arm and ppc64le
*Changes:*
- [X] [#548](https://github.com/kubernetes/ingress/pull/548) nginx: support multidomain certificates
- [X] [#620](https://github.com/kubernetes/ingress/pull/620) [nginx] Listening ports are not configurable, so ingress can't be run multiple times per node when using CNI
- [X] [#648](https://github.com/kubernetes/ingress/pull/648) publish-service argument isn't honored when ELB is internal only facing.
- [X] [#833](https://github.com/kubernetes/ingress/pull/833) WIP: Avoid reloads implementing Equals in structs
- [X] [#838](https://github.com/kubernetes/ingress/pull/838) Feature request: Add ingress annotation to enable upstream "keepalive" option
- [X] [#844](https://github.com/kubernetes/ingress/pull/844) ingress annotations affinity is not working
- [X] [#862](https://github.com/kubernetes/ingress/pull/862) Avoid reloads implementing Equaler interface
- [X] [#864](https://github.com/kubernetes/ingress/pull/864) Remove dead code
- [X] [#868](https://github.com/kubernetes/ingress/pull/868) Lint nginx code
- [X] [#871](https://github.com/kubernetes/ingress/pull/871) Add feature to allow sticky sessions per location
- [X] [#873](https://github.com/kubernetes/ingress/pull/873) Update README.md
- [X] [#876](https://github.com/kubernetes/ingress/pull/876) Add information about nginx controller flags
- [X] [#878](https://github.com/kubernetes/ingress/pull/878) Update go to 1.8.3
- [X] [#881](https://github.com/kubernetes/ingress/pull/881) Option to not remove loadBalancer status record?
- [X] [#882](https://github.com/kubernetes/ingress/pull/882) Add flag to skip the update of Ingress status on shutdown
- [X] [#885](https://github.com/kubernetes/ingress/pull/885) Don't use $proxy_protocol var which may be undefined.
- [X] [#886](https://github.com/kubernetes/ingress/pull/886) Add support for SubjectAltName in SSL certificates
- [X] [#888](https://github.com/kubernetes/ingress/pull/888) Update nginx-slim to 0.19
- [X] [#889](https://github.com/kubernetes/ingress/pull/889) Add PHOST to backend
- [X] [#890](https://github.com/kubernetes/ingress/pull/890) Improve variable configuration for source IP address
- [X] [#892](https://github.com/kubernetes/ingress/pull/892) Add upstream keepalive connections cache
- [X] [#897](https://github.com/kubernetes/ingress/pull/897) Update outdated ingress resource link
- [X] [#898](https://github.com/kubernetes/ingress/pull/898) add error check right when reload nginx fail
- [X] [#899](https://github.com/kubernetes/ingress/pull/899) Fix nginx error check
- [X] [#900](https://github.com/kubernetes/ingress/pull/900) After #862 changes in the configmap do not trigger a reload
- [X] [#901](https://github.com/kubernetes/ingress/pull/901) [doc] Update NGinX status port to 18080
- [X] [#902](https://github.com/kubernetes/ingress/pull/902) Always reload after a change in the configuration
- [X] [#904](https://github.com/kubernetes/ingress/pull/904) Fix nginx sticky sessions
- [X] [#906](https://github.com/kubernetes/ingress/pull/906) Fix race condition with closed channels
- [X] [#907](https://github.com/kubernetes/ingress/pull/907) nginx/proxy: allow specifying next upstream behaviour
- [X] [#910](https://github.com/kubernetes/ingress/pull/910) Feature request: use `X-Forwarded-Host` from the reverse proxy before
- [X] [#911](https://github.com/kubernetes/ingress/pull/911) Improve X-Forwarded-Host support
- [X] [#915](https://github.com/kubernetes/ingress/pull/915) Release nginx-slim 0.20
- [X] [#916](https://github.com/kubernetes/ingress/pull/916) Add arm and ppc64le support
- [X] [#919](https://github.com/kubernetes/ingress/pull/919) Apply the 'ssl-redirect' annotation per-location
- [X] [#922](https://github.com/kubernetes/ingress/pull/922) Add example of TLS termination using a classic ELB
### 0.9-beta.8
**Image:** `gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.8`
@ -29,7 +310,7 @@ Changelog
- [X] [#829](https://github.com/kubernetes/ingress/pull/829) feat(template): wrap IPv6 addresses in []
- [X] [#786](https://github.com/kubernetes/ingress/pull/786) Update echoserver image version in examples
- [X] [#825](https://github.com/kubernetes/ingress/pull/825) Create or delete ingress based on class annotation
- [X] [#790](https://github.com/kubernetes/ingress/pull/790) #789 removing duplicate X-Real-IP header
- [X] [#790](https://github.com/kubernetes/ingress/pull/790) #789 removing duplicate X-Real-IP header
- [X] [#792](https://github.com/kubernetes/ingress/pull/792) Avoid checking if the controllers are synced
- [X] [#798](https://github.com/kubernetes/ingress/pull/798) nginx: RBAC for leader election
- [X] [#799](https://github.com/kubernetes/ingress/pull/799) could not build variables_hash
@ -52,7 +333,7 @@ Changelog
*Changes:*
- [X] [#777](https://github.com/kubernetes/ingress/pull/777) Update sniff parser to fix index out of bound error
- [X] [#777](https://github.com/kubernetes/ingress/pull/777) Update sniff parser to fix index out of bound error
### 0.9-beta.6
@ -109,8 +390,8 @@ Changelog
- [X] [#663](https://github.com/kubernetes/ingress/pull/663) Remove helper required in go < 1.8
- [X] [#662](https://github.com/kubernetes/ingress/pull/662) Add debug information about ingress class
- [X] [#661](https://github.com/kubernetes/ingress/pull/661) Avoid running nginx if the configuration file is empty
- [X] [#660](https://github.com/kubernetes/ingress/pull/660) Rollback queue refactoring
- [X] [#661](https://github.com/kubernetes/ingress/pull/661) Avoid running nginx if the configuration file is empty
- [X] [#660](https://github.com/kubernetes/ingress/pull/660) Rollback queue refactoring
- [X] [#654](https://github.com/kubernetes/ingress/pull/654) Update go version to 1.8
@ -140,13 +421,13 @@ Changelog
- [X] [#595](https://github.com/kubernetes/ingress/pull/595) Remove Host header from auth_request proxy configuration
- [X] [#588](https://github.com/kubernetes/ingress/pull/588) Read resolv.conf file just once
- [X] [#586](https://github.com/kubernetes/ingress/pull/586) Updated instructions to create an ingress controller build
- [X] [#583](https://github.com/kubernetes/ingress/pull/583) fixed lua_package_path in nginx.tmpl
- [X] [#583](https://github.com/kubernetes/ingress/pull/583) fixed lua_package_path in nginx.tmpl
- [X] [#580](https://github.com/kubernetes/ingress/pull/580) Updated faq for running multiple ingress controller
- [X] [#579](https://github.com/kubernetes/ingress/pull/579) Detect if the ingress controller is running with multiple replicas
- [X] [#578](https://github.com/kubernetes/ingress/pull/578) Set different listeners per protocol version
- [X] [#577](https://github.com/kubernetes/ingress/pull/577) Avoid zombie child processes
- [X] [#576](https://github.com/kubernetes/ingress/pull/576) Replace secret workqueue
- [X] [#568](https://github.com/kubernetes/ingress/pull/568) Revert merge annotations to the implicit root context
- [X] [#568](https://github.com/kubernetes/ingress/pull/568) Revert merge annotations to the implicit root context
- [X] [#563](https://github.com/kubernetes/ingress/pull/563) Add option to disable hsts preload
- [X] [#560](https://github.com/kubernetes/ingress/pull/560) Fix intermittent misconfiguration of backend.secure and SessionAffinity
- [X] [#556](https://github.com/kubernetes/ingress/pull/556) Update nginx version and remove dumb-init
@ -170,7 +451,7 @@ Changelog
- [X] [#512](https://github.com/kubernetes/ingress/pull/512) Fix typos regarding the ssl-passthrough annotation documentation
- [X] [#505](https://github.com/kubernetes/ingress/pull/505) add unit test cases for core/pkg/ingress/controller/annotations
- [X] [#503](https://github.com/kubernetes/ingress/pull/503) Add example for nginx in aws
- [X] [#502](https://github.com/kubernetes/ingress/pull/502) Add information about SSL Passthrough annotation
- [X] [#502](https://github.com/kubernetes/ingress/pull/502) Add information about SSL Passthrough annotation
- [X] [#500](https://github.com/kubernetes/ingress/pull/500) Improve TLS secret configuration
- [X] [#498](https://github.com/kubernetes/ingress/pull/498) Proper enqueue a secret on the secret queue
- [X] [#493](https://github.com/kubernetes/ingress/pull/493) Update nginx and vts module
@ -179,7 +460,7 @@ Changelog
- [X] [#485](https://github.com/kubernetes/ingress/pull/485) Fix typo nginx configMap vts metrics customization
- [X] [#481](https://github.com/kubernetes/ingress/pull/481) Remove unnecessary quote in nginx log format
- [X] [#471](https://github.com/kubernetes/ingress/pull/471) prometheus scrape annotations
- [X] [#460](https://github.com/kubernetes/ingress/pull/460) add example of 'run multiple haproxy ingress controllers as a deployment'
- [X] [#460](https://github.com/kubernetes/ingress/pull/460) add example of 'run multiple haproxy ingress controllers as a deployment'
- [X] [#459](https://github.com/kubernetes/ingress/pull/459) Add information about SSL certificates in the default log level
- [X] [#456](https://github.com/kubernetes/ingress/pull/456) Avoid upstreams with multiple servers with the same port
- [X] [#454](https://github.com/kubernetes/ingress/pull/454) Pass request port to real server
@ -211,55 +492,55 @@ Changelog
*Changes:*
- [X] [#433](https://github.com/kubernetes/ingress/pull/433) close over the ingress variable or the last assignment will be used
- [X] [#424](https://github.com/kubernetes/ingress/pull/424) Manually sync secrets from certificate authentication annotations
- [X] [#423](https://github.com/kubernetes/ingress/pull/423) Scrap json metrics from nginx vts module when enabled
- [X] [#418](https://github.com/kubernetes/ingress/pull/418) Only update Ingress status for the configured class
- [X] [#415](https://github.com/kubernetes/ingress/pull/415) Improve external authentication docs
- [X] [#410](https://github.com/kubernetes/ingress/pull/410) Add support for "signin url"
- [X] [#409](https://github.com/kubernetes/ingress/pull/409) Allow custom http2 header sizes
- [X] [#408](https://github.com/kubernetes/ingress/pull/408) Review docs
- [X] [#406](https://github.com/kubernetes/ingress/pull/406) Add debug info and fix spelling
- [X] [#402](https://github.com/kubernetes/ingress/pull/402) allow specifying custom dh param
- [X] [#433](https://github.com/kubernetes/ingress/pull/433) close over the ingress variable or the last assignment will be used
- [X] [#424](https://github.com/kubernetes/ingress/pull/424) Manually sync secrets from certificate authentication annotations
- [X] [#423](https://github.com/kubernetes/ingress/pull/423) Scrap json metrics from nginx vts module when enabled
- [X] [#418](https://github.com/kubernetes/ingress/pull/418) Only update Ingress status for the configured class
- [X] [#415](https://github.com/kubernetes/ingress/pull/415) Improve external authentication docs
- [X] [#410](https://github.com/kubernetes/ingress/pull/410) Add support for "signin url"
- [X] [#409](https://github.com/kubernetes/ingress/pull/409) Allow custom http2 header sizes
- [X] [#408](https://github.com/kubernetes/ingress/pull/408) Review docs
- [X] [#406](https://github.com/kubernetes/ingress/pull/406) Add debug info and fix spelling
- [X] [#402](https://github.com/kubernetes/ingress/pull/402) allow specifying custom dh param
- [X] [#397](https://github.com/kubernetes/ingress/pull/397) Fix external auth
- [X] [#394](https://github.com/kubernetes/ingress/pull/394) Update README.md
- [X] [#394](https://github.com/kubernetes/ingress/pull/394) Update README.md
- [X] [#392](https://github.com/kubernetes/ingress/pull/392) Fix http2 header size
- [X] [#391](https://github.com/kubernetes/ingress/pull/391) remove tmp nginx-diff files
- [X] [#390](https://github.com/kubernetes/ingress/pull/390) Fix RateLimit comment
- [X] [#385](https://github.com/kubernetes/ingress/pull/385) add Copyright
- [X] [#382](https://github.com/kubernetes/ingress/pull/382) Ingress Fake Certificate generation
- [X] [#380](https://github.com/kubernetes/ingress/pull/380) Fix custom log format
- [X] [#373](https://github.com/kubernetes/ingress/pull/373) Cleanup
- [X] [#371](https://github.com/kubernetes/ingress/pull/371) add configuration to disable listening on ipv6
- [X] [#370](https://github.com/kubernetes/ingress/pull/270) Add documentation for ingress.kubernetes.io/force-ssl-redirect
- [X] [#369](https://github.com/kubernetes/ingress/pull/369) Minor text fix for "ApiServer"
- [X] [#391](https://github.com/kubernetes/ingress/pull/391) remove tmp nginx-diff files
- [X] [#390](https://github.com/kubernetes/ingress/pull/390) Fix RateLimit comment
- [X] [#385](https://github.com/kubernetes/ingress/pull/385) add Copyright
- [X] [#382](https://github.com/kubernetes/ingress/pull/382) Ingress Fake Certificate generation
- [X] [#380](https://github.com/kubernetes/ingress/pull/380) Fix custom log format
- [X] [#373](https://github.com/kubernetes/ingress/pull/373) Cleanup
- [X] [#371](https://github.com/kubernetes/ingress/pull/371) add configuration to disable listening on ipv6
- [X] [#370](https://github.com/kubernetes/ingress/pull/270) Add documentation for ingress.kubernetes.io/force-ssl-redirect
- [X] [#369](https://github.com/kubernetes/ingress/pull/369) Minor text fix for "ApiServer"
- [X] [#367](https://github.com/kubernetes/ingress/pull/367) BuildLogFormatUpstream was always using the default log-format
- [X] [#366](https://github.com/kubernetes/ingress/pull/366) add_judgment
- [X] [#365](https://github.com/kubernetes/ingress/pull/365) add ForceSSLRedirect ingress annotation
- [X] [#364](https://github.com/kubernetes/ingress/pull/364) Fix error caused by increasing proxy_buffer_size (#363)
- [X] [#362](https://github.com/kubernetes/ingress/pull/362) Fix ingress class
- [X] [#360](https://github.com/kubernetes/ingress/pull/360) add example of 'run multiple nginx ingress controllers as a deployment'
- [X] [#358](https://github.com/kubernetes/ingress/pull/358) Checks if the TLS secret contains a valid keypair structure
- [X] [#356](https://github.com/kubernetes/ingress/pull/356) Disable listen only on ipv6 and fix proxy_protocol
- [X] [#354](https://github.com/kubernetes/ingress/pull/354) add judgment
- [X] [#352](https://github.com/kubernetes/ingress/pull/352) Add ability to customize upstream and stream log format
- [X] [#351](https://github.com/kubernetes/ingress/pull/351) Enable custom election id for status sync.
- [X] [#347](https://github.com/kubernetes/ingress/pull/347) Fix client source IP address
- [X] [#366](https://github.com/kubernetes/ingress/pull/366) add_judgment
- [X] [#365](https://github.com/kubernetes/ingress/pull/365) add ForceSSLRedirect ingress annotation
- [X] [#364](https://github.com/kubernetes/ingress/pull/364) Fix error caused by increasing proxy_buffer_size (#363)
- [X] [#362](https://github.com/kubernetes/ingress/pull/362) Fix ingress class
- [X] [#360](https://github.com/kubernetes/ingress/pull/360) add example of 'run multiple nginx ingress controllers as a deployment'
- [X] [#358](https://github.com/kubernetes/ingress/pull/358) Checks if the TLS secret contains a valid keypair structure
- [X] [#356](https://github.com/kubernetes/ingress/pull/356) Disable listen only on ipv6 and fix proxy_protocol
- [X] [#354](https://github.com/kubernetes/ingress/pull/354) add judgment
- [X] [#352](https://github.com/kubernetes/ingress/pull/352) Add ability to customize upstream and stream log format
- [X] [#351](https://github.com/kubernetes/ingress/pull/351) Enable custom election id for status sync.
- [X] [#347](https://github.com/kubernetes/ingress/pull/347) Fix client source IP address
- [X] [#345](https://github.com/kubernetes/ingress/pull/345) Fix lint error
- [X] [#344](https://github.com/kubernetes/ingress/pull/344) Refactoring of TCP and UDP services
- [X] [#343](https://github.com/kubernetes/ingress/pull/343) Fix node lister when --watch-namespace is used
- [X] [#341](https://github.com/kubernetes/ingress/pull/341) Do not run coverage check in the default target.
- [X] [#340](https://github.com/kubernetes/ingress/pull/340) Add support for specify proxy cookie path/domain
- [X] [#337](https://github.com/kubernetes/ingress/pull/337) Fix for formatting error introduced in #304
- [X] [#335](https://github.com/kubernetes/ingress/pull/335) Fix for vet complaints:
- [X] [#332](https://github.com/kubernetes/ingress/pull/332) Add annotation to customize nginx configuration
- [X] [#331](https://github.com/kubernetes/ingress/pull/331) Correct spelling mistake
- [X] [#328](https://github.com/kubernetes/ingress/pull/328) fix misspell "affinity" in main.go
- [X] [#326](https://github.com/kubernetes/ingress/pull/326) add nginx daemonset example
- [X] [#311](https://github.com/kubernetes/ingress/pull/311) Sort stream service ports to avoid extra reloads
- [X] [#344](https://github.com/kubernetes/ingress/pull/344) Refactoring of TCP and UDP services
- [X] [#343](https://github.com/kubernetes/ingress/pull/343) Fix node lister when --watch-namespace is used
- [X] [#341](https://github.com/kubernetes/ingress/pull/341) Do not run coverage check in the default target.
- [X] [#340](https://github.com/kubernetes/ingress/pull/340) Add support for specify proxy cookie path/domain
- [X] [#337](https://github.com/kubernetes/ingress/pull/337) Fix for formatting error introduced in #304
- [X] [#335](https://github.com/kubernetes/ingress/pull/335) Fix for vet complaints:
- [X] [#332](https://github.com/kubernetes/ingress/pull/332) Add annotation to customize nginx configuration
- [X] [#331](https://github.com/kubernetes/ingress/pull/331) Correct spelling mistake
- [X] [#328](https://github.com/kubernetes/ingress/pull/328) fix misspell "affinity" in main.go
- [X] [#326](https://github.com/kubernetes/ingress/pull/326) add nginx daemonset example
- [X] [#311](https://github.com/kubernetes/ingress/pull/311) Sort stream service ports to avoid extra reloads
- [X] [#307](https://github.com/kubernetes/ingress/pull/307) Add docs for body-size annotation
- [X] [#306](https://github.com/kubernetes/ingress/pull/306) modify nginx readme
- [X] [#304](https://github.com/kubernetes/ingress/pull/304) change 'buildSSPassthrouthUpstreams' to 'buildSSLPassthroughUpstreams'
- [X] [#306](https://github.com/kubernetes/ingress/pull/306) modify nginx readme
- [X] [#304](https://github.com/kubernetes/ingress/pull/304) change 'buildSSPassthrouthUpstreams' to 'buildSSLPassthroughUpstreams'
### 0.9-beta.2
@ -300,9 +581,9 @@ Changelog
- [X] [#227](https://github.com/kubernetes/ingress/pull/227) proxy_protocol on ssl_passthrough listener
- [X] [#223](https://github.com/kubernetes/ingress/pull/223) Fix panic if a tempfile cannot be created
- [X] [#220](https://github.com/kubernetes/ingress/pull/220) Fixes for minikube usage instructions.
- [X] [#219](https://github.com/kubernetes/ingress/pull/219) Fix typo, add a couple of links.
- [X] [#219](https://github.com/kubernetes/ingress/pull/219) Fix typo, add a couple of links.
- [X] [#218](https://github.com/kubernetes/ingress/pull/218) Improve links from CONTRIBUTING.
- [X] [#217](https://github.com/kubernetes/ingress/pull/217) Fix an e2e link.
- [X] [#217](https://github.com/kubernetes/ingress/pull/217) Fix an e2e link.
- [X] [#212](https://github.com/kubernetes/ingress/pull/212) Simplify code to obtain TCP or UDP services
- [X] [#208](https://github.com/kubernetes/ingress/pull/208) Fix nil HTTP field
- [X] [#198](https://github.com/kubernetes/ingress/pull/198) Add an example for static-ip and deployment

109
controllers/nginx/Makefile
View file

@ -3,10 +3,16 @@ all: push
BUILDTAGS=
# Use the 0.0 tag for testing, it shouldn't clobber any release builds
RELEASE?=0.9.0-beta.8
PREFIX?=gcr.io/google_containers/nginx-ingress-controller
TAG?=0.9.0-beta.13
REGISTRY?=gcr.io/google_containers
GOOS?=linux
DOCKER?=gcloud docker --
SED_I?=sed -i
GOHOSTOS ?= $(shell go env GOHOSTOS)
ifeq ($(GOHOSTOS),darwin)
SED_I=sed -i ''
endif
REPO_INFO=$(shell git config --get remote.origin.url)
@ -16,16 +22,93 @@ endif
PKG=k8s.io/ingress/controllers/nginx
ARCH ?= $(shell go env GOARCH)
GOARCH = ${ARCH}
DUMB_ARCH = ${ARCH}
ALL_ARCH = amd64 arm arm64 ppc64le
QEMUVERSION=v2.9.1
IMGNAME = nginx-ingress-controller
IMAGE = $(REGISTRY)/$(IMGNAME)
MULTI_ARCH_IMG = $(IMAGE)-$(ARCH)
# Set default base image dynamically for each arch
BASEIMAGE?=gcr.io/google_containers/nginx-slim-$(ARCH):0.24
ifeq ($(ARCH),arm)
QEMUARCH=arm
GOARCH=arm
DUMB_ARCH=armhf
endif
ifeq ($(ARCH),arm64)
QEMUARCH=aarch64
endif
ifeq ($(ARCH),ppc64le)
QEMUARCH=ppc64le
GOARCH=ppc64le
DUMB_ARCH=ppc64el
endif
#ifeq ($(ARCH),s390x)
# QEMUARCH=s390x
#endif
TEMP_DIR := $(shell mktemp -d)
DOCKERFILE := $(TEMP_DIR)/rootfs/Dockerfile
all: all-container
sub-container-%:
$(MAKE) ARCH=$* build container
sub-push-%:
$(MAKE) ARCH=$* push
all-container: $(addprefix sub-container-,$(ALL_ARCH))
all-push: $(addprefix sub-push-,$(ALL_ARCH))
container: .container-$(ARCH)
.container-$(ARCH):
cp -r ./* $(TEMP_DIR)
$(SED_I) 's|BASEIMAGE|$(BASEIMAGE)|g' $(DOCKERFILE)
$(SED_I) "s|QEMUARCH|$(QEMUARCH)|g" $(DOCKERFILE)
$(SED_I) "s|DUMB_ARCH|$(DUMB_ARCH)|g" $(DOCKERFILE)
ifeq ($(ARCH),amd64)
# When building "normally" for amd64, remove the whole line, it has no part in the amd64 image
$(SED_I) "/CROSS_BUILD_/d" $(DOCKERFILE)
else
# When cross-building, only the placeholder "CROSS_BUILD_" should be removed
# Register /usr/bin/qemu-ARCH-static as the handler for ARM binaries in the kernel
$(DOCKER) run --rm --privileged multiarch/qemu-user-static:register --reset
curl -sSL https://github.com/multiarch/qemu-user-static/releases/download/$(QEMUVERSION)/x86_64_qemu-$(QEMUARCH)-static.tar.gz | tar -xz -C $(TEMP_DIR)/rootfs
$(SED_I) "s/CROSS_BUILD_//g" $(DOCKERFILE)
endif
$(DOCKER) build -t $(MULTI_ARCH_IMG):$(TAG) $(TEMP_DIR)/rootfs
ifeq ($(ARCH), amd64)
# This is for to maintain the backward compatibility
$(DOCKER) tag $(MULTI_ARCH_IMG):$(TAG) $(IMAGE):$(TAG)
endif
push: .push-$(ARCH)
.push-$(ARCH):
$(DOCKER) push $(MULTI_ARCH_IMG):$(TAG)
ifeq ($(ARCH), amd64)
$(DOCKER) push $(IMAGE):$(TAG)
endif
clean:
$(DOCKER) rmi -f $(MULTI_ARCH_IMG):$(TAG) || true
build: clean
CGO_ENABLED=0 GOOS=${GOOS} go build -a -installsuffix cgo \
-ldflags "-s -w -X ${PKG}/pkg/version.RELEASE=${RELEASE} -X ${PKG}/pkg/version.COMMIT=${COMMIT} -X ${PKG}/pkg/version.REPO=${REPO_INFO}" \
-o rootfs/nginx-ingress-controller ${PKG}/pkg/cmd/controller
container: build
$(DOCKER) build --pull -t $(PREFIX):$(RELEASE) rootfs
push: container
$(DOCKER) push $(PREFIX):$(RELEASE)
CGO_ENABLED=0 GOOS=${GOOS} GOARCH=${GOARCH} go build -a -installsuffix cgo \
-ldflags "-s -w -X ${PKG}/pkg/version.RELEASE=${TAG} -X ${PKG}/pkg/version.COMMIT=${COMMIT} -X ${PKG}/pkg/version.REPO=${REPO_INFO}" \
-o ${TEMP_DIR}/rootfs/nginx-ingress-controller ${PKG}/pkg/cmd/controller
fmt:
@echo "+ $@"
@ -49,5 +132,5 @@ vet:
@echo "+ $@"
@go vet $(shell go list ${PKG}/... | grep -v vendor)
clean:
rm -f rootfs/nginx-ingress-controller
release: all-container all-push
echo "done"

85
controllers/nginx/README.md
View file

@ -5,6 +5,7 @@ This is an nginx Ingress controller that uses [ConfigMap](https://github.com/kub
## Contents
* [Conventions](#conventions)
* [Requirements](#requirements)
* [Command line arguments](#command-line-arguments)
* [Dry running](#try-running-the-ingress-controller)
* [Deployment](#deployment)
* [HTTP](#http)
@ -41,6 +42,61 @@ Anytime we reference a tls secret, we mean (x509, pem encoded, RSA 2048, etc). Y
- Default backend [404-server](https://github.com/kubernetes/contrib/tree/master/404-server)
## Command line arguments
```
Usage of :
--alsologtostderr log to standard error as well as files
--apiserver-host string The address of the Kubernetes Apiserver to connect to in the format of protocol://address:port, e.g., http://localhost:8080. If not specified, the assumption is that the binary runs inside a Kubernetes cluster and local discovery is attempted.
--configmap string Name of the ConfigMap that contains the custom configuration to use
--default-backend-service string Service used to serve a 404 page for the default backend. Takes the form
namespace/name. The controller uses the first node port of this Service for
the default backend.
--default-server-port int Default port to use for exposing the default server (catch all) (default 8181)
--default-ssl-certificate string Name of the secret
that contains a SSL certificate to be used as default for a HTTPS catch-all server
--disable-node-list Disable querying nodes. If --force-namespace-isolation is true, this should also be set.
--election-id string Election id to use for status update. (default "ingress-controller-leader")
--enable-ssl-passthrough Enable SSL passthrough feature. Default is disabled
--force-namespace-isolation Force namespace isolation. This flag is required to avoid the reference of secrets or
configmaps located in a different namespace than the specified in the flag --watch-namespace.
--health-check-path string Defines
the URL to be used as health check inside in the default server in NGINX. (default "/healthz")
--healthz-port int port for healthz endpoint. (default 10254)
--http-port int Indicates the port to use for HTTP traffic (default 80)
--https-port int Indicates the port to use for HTTPS traffic (default 443)
--ingress-class string Name of the ingress class to route through this controller.
--kubeconfig string Path to kubeconfig file with authorization and master location information.
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--logtostderr log to standard error instead of files
--profiling Enable profiling via web interface host:port/debug/pprof/ (default true)
--publish-service string Service fronting the ingress controllers. Takes the form
namespace/name. The controller will set the endpoint records on the
ingress objects to reflect those on the service.
--sort-backends Defines if backends and it's endpoints should be sorted
--ssl-passtrough-proxy-port int Default port to use internally for SSL when SSL Passthgough is enabled (default 442)
--status-port int Indicates the TCP port to use for exposing the nginx status page (default 18080)
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
--sync-period duration Relist and confirm cloud resources this often. Default is 10 minutes (default 10m0s)
--tcp-services-configmap string Name of the ConfigMap that contains the definition of the TCP services to expose.
The key in the map indicates the external port to be used. The value is the name of the
service with the format namespace/serviceName and the port of the service could be a
number of the name of the port.
The ports 80 and 443 are not allowed as external ports. This ports are reserved for the backend
--udp-services-configmap string Name of the ConfigMap that contains the definition of the UDP services to expose.
The key in the map indicates the external port to be used. The value is the name of the
service with the format namespace/serviceName and the port of the service could be a
number of the name of the port.
--update-status Indicates if the
ingress controller should update the Ingress status IP/hostname. Default is true (default true)
--update-status-on-shutdown Indicates if the
ingress controller should update the Ingress status IP/hostname when the controller
is being stopped. Default is true (default true)
-v, --v Level log level for V logs
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
--watch-namespace string Namespace to watch for Ingress. Default is to watch all namespaces
```
## Try running the Ingress controller
Before deploying the controller to production you might want to run it outside the cluster and observe it.
@ -53,23 +109,20 @@ $ ./rootfs/nginx-ingress-controller --running-in-cluster=false --default-backend
## Deployment
First create a default backend:
First create a default backend and it's corresponding service:
```
$ kubectl create -f examples/deployment/nginx/default-backend.yaml
$ kubectl expose rc default-http-backend --port=80 --target-port=8080 --name=default-http-backend
$ kubectl create -f examples/default-backend.yaml
```
Follow the [example-deployment](../../examples/deployment/nginx/README.md) steps to deploy nginx-ingress-controller in Kubernetes cluster (you may prefer other type of workloads, like Daemonset, in production environment).
Loadbalancers are created via a ReplicationController or Daemonset:
```
$ kubectl create -f examples/default/rc-default.yaml
```
## HTTP
First we need to deploy some application to publish. To keep this simple we will use the [echoheaders app](https://github.com/kubernetes/contrib/blob/master/ingress/echoheaders/echo-app.yaml) that just returns information about the http request as output
```
kubectl run echoheaders --image=gcr.io/google_containers/echoserver:1.5 --replicas=1 --port=8080
kubectl run echoheaders --image=gcr.io/google_containers/echoserver:1.8 --replicas=1 --port=8080
```
Now we expose the same application in two different services (so we can create different Ingress rules)
@ -138,9 +191,9 @@ spec:
serviceName: s1
servicePort: 80
```
Please follow [test.sh](https://github.com/bprashanth/Ingress/blob/master/examples/sni/nginx/test.sh) as a guide on how to generate secrets containing SSL certificates. The name of the secret can be different than the name of the certificate.
Please follow [PREREQUISITES](../../examples/PREREQUISITES.md) as a guide on how to generate secrets containing SSL certificates. The name of the secret can be different than the name of the certificate.
Check the [example](examples/tls/README.md)
Check the [example](../../examples/tls-termination/nginx)
### Default SSL Certificate
@ -264,8 +317,8 @@ To disable this behavior use `hsts=false` in the NGINX config map.
### Automated Certificate Management with Kube-Lego
[Kube-Lego] automatically requests missing certificates or expired from
[Let's Encrypt] by monitoring ingress resources and its referenced secrets. To
[Kube-Lego] automatically requests missing or expired certificates from
[Let's Encrypt] by monitoring ingress resources and their referenced secrets. To
enable this for an ingress resource you have to add an annotation:
```
@ -281,8 +334,8 @@ version to fully support Kube-Lego is nginx Ingress controller 0.8.
## Exposing TCP services
Ingress does not support TCP services (yet). For this reason this Ingress controller uses the flag `--tcp-services-configmap` to point to an existing config map where the key is the external port to use and the value is `<namespace/service name>:<service port>`
It is possible to use a number or the name of the port.
Ingress does not support TCP services (yet). For this reason this Ingress controller uses the flag `--tcp-services-configmap` to point to an existing config map where the key is the external port to use and the value is `<namespace/service name>:<service port>:[PROXY]`
It is possible to use a number or the name of the port. The last field is optional. Adding `PROXY` in the last field we can enable Proxy Protocol in a TCP service.
The next example shows how to expose the service `example-go` running in the namespace `default` in the port `8080` using the port `9000`
```
@ -315,7 +368,7 @@ data:
```
Please check the [udp services](examples/udp/README.md) example
Please check the [udp services](../../examples/udp/nginx/README.md) example
## Proxy Protocol
@ -338,7 +391,7 @@ Using this two headers is possible to use a custom backend service like [this on
The ngx_http_stub_status_module module provides access to basic status information. This is the default module active in the url `/nginx_status`.
This controller provides an alternative to this module using [nginx-module-vts](https://github.com/vozlt/nginx-module-vts) third party module.
To use this module just provide a config map with the key `enable-vts-status=true`. The URL is exposed in the port 8080.
To use this module just provide a config map with the key `enable-vts-status=true`. The URL is exposed in the port 18080.
Please check the example `example/rc-default.yaml`
![nginx-module-vts screenshot](https://cloud.githubusercontent.com/assets/3648408/10876811/77a67b70-8183-11e5-9924-6a6d0c5dc73a.png "screenshot with filter")
@ -399,7 +452,7 @@ Description:
### Local cluster
Using [`hack/local-up-cluster.sh`](https://github.com/kubernetes/kubernetes/blob/master/hack/local-up-cluster.sh) is possible to start a local kubernetes cluster consisting of a master and a single node. Please read [running-locally.md](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/running-locally.md) for more details.
Using [`hack/local-up-cluster.sh`](https://github.com/kubernetes/kubernetes/blob/master/hack/local-up-cluster.sh) is possible to start a local kubernetes cluster consisting of a master and a single node. Please read [running-locally.md](https://github.com/kubernetes/community/blob/master/contributors/devel/running-locally.md) for more details.
Use of `hostNetwork: true` in the ingress controller is required to falls back at localhost:8080 for the apiserver if every other client creation check fails (eg: service account not present, kubeconfig doesn't exist, no master env vars...)

206
controllers/nginx/configuration.md
View file

@ -1,4 +1,5 @@
## Contents
* [Customizing NGINX](#customizing-nginx)
* [Custom NGINX configuration](#custom-nginx-configuration)
* [Custom NGINX template](#custom-nginx-template)
@ -7,6 +8,7 @@
* [Authentication](#authentication)
* [Rewrite](#rewrite)
* [Rate limiting](#rate-limiting)
* [SSL Passthrough](#ssl-passthrough)
* [Secure backends](#secure-backends)
* [Server-side HTTPS enforcement through redirect](#server-side-https-enforcement-through-redirect)
* [Whitelist source range](#whitelist-source-range)
@ -17,7 +19,6 @@
* [Retries in non-idempotent methods](#retries-in-non-idempotent-methods)
* [Custom max body size](#custom-max-body-size)
### Customizing NGINX
There are 3 ways to customize NGINX:
@ -26,11 +27,9 @@ There are 3 ways to customize NGINX:
2. [annotations](#annotations): use this if you want a specific configuration for the site defined in the Ingress rule.
3. custom template: when more specific settings are required, like [open_file_cache](http://nginx.org/en/docs/http/ngx_http_core_module.html#open_file_cache), custom [log_format](http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format), adjust [listen](http://nginx.org/en/docs/http/ngx_http_core_module.html#listen) options as `rcvbuf` or when is not possible to change an through the ConfigMap.
#### Custom NGINX configuration
It is possible to customize the defaults in NGINX using a ConfigMap.
Please check the [custom configuration](../../examples/customization/custom-configuration/nginx/README.md) example.
#### Annotations
@ -48,24 +47,29 @@ The following annotations are supported:
|[ingress.kubernetes.io/auth-url](#external-authentication)|string|
|[ingress.kubernetes.io/auth-tls-secret](#certificate-authentication)|string|
|[ingress.kubernetes.io/auth-tls-verify-depth](#certificate-authentication)|number|
|[ingress.kubernetes.io/auth-tls-error-page](#certificate-authentication)|string|
|[ingress.kubernetes.io/base-url-scheme](#rewrite)|string|
|[ingress.kubernetes.io/client-body-buffer-size](#client-body-buffer-size)|string|
|[ingress.kubernetes.io/configuration-snippet](#configuration-snippet)|string|
|[ingress.kubernetes.io/default-backend](#default-backend)|string|
|[ingress.kubernetes.io/enable-cors](#enable-cors)|true or false|
|[ingress.kubernetes.io/force-ssl-redirect](#server-side-https-enforcement-through-redirect)|true or false|
|[ingress.kubernetes.io/from-to-www-redirect](#redirect-from-to-www)|true or false|
|[ingress.kubernetes.io/limit-connections](#rate-limiting)|number|
|[ingress.kubernetes.io/limit-rps](#rate-limiting)|number|
|[ingress.kubernetes.io/ssl-passthrough](#ssl-passthrough)|true or false|
|[ingress.kubernetes.io/proxy-body-size](#custom-max-body-size)|string|
|[ingress.kubernetes.io/rewrite-target](#rewrite)|URI|
|[ingress.kubernetes.io/secure-backends](#secure-backends)|true or false|
|[ingress.kubernetes.io/server-alias](#server-alias)|string|
|[ingress.kubernetes.io/service-upstream](#service-upstream)|true or false|
|[ingress.kubernetes.io/session-cookie-name](#cookie-affinity)|string|
|[ingress.kubernetes.io/session-cookie-hash](#cookie-affinity)|string|
|[ingress.kubernetes.io/ssl-redirect](#server-side-https-enforcement-through-redirect)|true or false|
|[ingress.kubernetes.io/ssl-passthrough](#ssl-passthrough)|true or false|
|[ingress.kubernetes.io/upstream-max-fails](#custom-nginx-upstream-checks)|number|
|[ingress.kubernetes.io/upstream-fail-timeout](#custom-nginx-upstream-checks)|number|
|[ingress.kubernetes.io/whitelist-source-range](#whitelist-source-range)|CIDR|
#### Custom NGINX template
The NGINX template is located in the file `/etc/nginx/template/nginx.tmpl`. Mounting a volume is possible to use a custom version.
@ -82,11 +86,10 @@ In addition to the built-in functions provided by the Go package the following f
- hasSuffix: [strings.HasSuffix](https://golang.org/pkg/strings/#HasSuffix)
- toUpper: [strings.ToUpper](https://golang.org/pkg/strings/#ToUpper)
- toLower: [strings.ToLower](https://golang.org/pkg/strings/#ToLower)
- buildLocation: helper to build the NGINX Location section in each server
- buildLocation: helps to build the NGINX Location section in each server
- buildProxyPass: builds the reverse proxy configuration
- buildRateLimitZones: helper to build all the required rate limit zones
- buildRateLimit: helper to build a limit zone inside a location if contains a rate limit annotation
- buildRateLimitZones: helps to build all the required rate limit zones
- buildRateLimit: helps to build a limit zone inside a location if contains a rate limit annotation
### Custom NGINX upstream checks
@ -106,13 +109,11 @@ In NGINX, backend server pools are called "[upstreams](http://nginx.org/en/docs/
Please check the [custom upstream check](../../examples/customization/custom-upstream-check/README.md) example.
### Authentication
Is possible to add authentication adding additional annotations in the Ingress rule. The source of the authentication is a secret that contains usernames and passwords inside the the key `auth`.
Is possible to add authentication adding additional annotations in the Ingress rule. The source of the authentication is a secret that contains usernames and passwords inside the key `auth`.
The annotations are:
```
ingress.kubernetes.io/auth-type: [basic|digest]
```
@ -130,14 +131,13 @@ The secret must be created in the same namespace as the Ingress rule.
ingress.kubernetes.io/auth-realm: "realm string"
```
Please check the [auth](/examples/auth/nginx/README.md) example.
Please check the [auth](/examples/auth/basic/nginx/README.md) example.
### Certificate Authentication
It's possible to enable Certificate based authentication using additional annotations in Ingress Rule.
The annotations are:
```
ingress.kubernetes.io/auth-tls-secret: secretName
```
@ -150,22 +150,60 @@ ingress.kubernetes.io/auth-tls-verify-depth
The validation depth between the provided client certificate and the Certification Authority chain.
```
ingress.kubernetes.io/auth-tls-error-page
```
The URL/Page that user should be redirected in case of a Certificate Authentication Error
Please check the [tls-auth](/examples/auth/client-certs/nginx/README.md) example.
### Configuration snippet
Using this annotion you can add additional configuration to the NGINX location. For example:
Using this annotation you can add additional configuration to the NGINX location. For example:
```
ingress.kubernetes.io/configuration-snippet: |
more_set_headers "Request-Id: $request_id";
```
### Default Backend
The ingress controller requires a default backend. This service is handle the response when the service in the Ingress rule does not have endpoints.
This is a global configuration for the ingress controller. In some cases could be required to return a custom content or format. In this scenario we can use the annotation `ingress.kubernetes.io/default-backend: <svc name>` to specify a custom default backend.
### Enable CORS
To enable Cross-Origin Resource Sharing (CORS) in an Ingress rule add the annotation `ingress.kubernetes.io/enable-cors: "true"`. This will add a section in the server location enabling this functionality.
For more information please check https://enable-cors.org/server_nginx.html
### Server Alias
To add Server Aliases to an Ingress rule add the annotation `ingress.kubernetes.io/server-alias: "<alias>"`.
This will create a server with the same configuration, but a different server_name as the provided host.
*Note:* A server-alias name cannot conflict with the hostname of an existing server. If it does the server-alias
annotation will be ignored. If a server-alias is created and later a new server with the same hostname is created
the new server configuration will take place over the alias configuration.
For more information please see http://nginx.org/en/docs/http/ngx_http_core_module.html#server_name
### Client Body Buffer Size
Sets buffer size for reading client request body per location. In case the request body is larger than the buffer,
the whole body or only its part is written to a temporary file. By default, buffer size is equal to two memory pages.
This is 8K on x86, other 32-bit platforms, and x86-64. It is usually 16K on other 64-bit platforms. This annotation is
applied to each location provided in the ingress rule.
*Note:* The annotation value must be given in a valid format otherwise the
For example to set the client-body-buffer-size the following can be done:
* `ingress.kubernetes.io/client-body-buffer-size: "1000"` # 1000 bytes
* `ingress.kubernetes.io/client-body-buffer-size: 1k` # 1 kilobyte
* `ingress.kubernetes.io/client-body-buffer-size: 1K` # 1 kilobyte
* `ingress.kubernetes.io/client-body-buffer-size: 1m` # 1 megabyte
* `ingress.kubernetes.io/client-body-buffer-size: 1M` # 1 megabyte
For more information please see http://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size
### External Authentication
To use an existing service that provides authentication the Ingress rule can be annotated with `ingress.kubernetes.io/auth-url` to indicate the URL where the HTTP request should be sent.
@ -177,7 +215,6 @@ ingress.kubernetes.io/auth-url: "URL to the authentication service"
Please check the [external-auth](/examples/auth/external-auth/nginx/README.md) example.
### Rewrite
In some scenarios the exposed URL in the backend service differs from the specified path in the Ingress rule. Without a rewrite any request will return 404.
@ -185,42 +222,72 @@ Set the annotation `ingress.kubernetes.io/rewrite-target` to the path expected b
If the application contains relative links it is possible to add an additional annotation `ingress.kubernetes.io/add-base-url` that will prepend a [`base` tag](https://developer.mozilla.org/en/docs/Web/HTML/Element/base) in the header of the returned HTML from the backend.
If the scheme of [`base` tag](https://developer.mozilla.org/en/docs/Web/HTML/Element/base) need to be specific, set the annotation `ingress.kubernetes.io/base-url-scheme` to the scheme such as `http` and `https`.
If the Application Root is exposed in a different path and needs to be redirected, set the annotation `ingress.kubernetes.io/app-root` to redirect requests for `/`.
Please check the [rewrite](/examples/rewrite/nginx/README.md) example.
### Rate limiting
The annotations `ingress.kubernetes.io/limit-connections` and `ingress.kubernetes.io/limit-rps` define a limit on the connections that can be opened by a single client IP address. This can be used to mitigate [DDoS Attacks](https://www.nginx.com/blog/mitigating-ddos-attacks-with-nginx-and-nginx-plus).
The annotations `ingress.kubernetes.io/limit-connections`, `ingress.kubernetes.io/limit-rps`, and `ingress.kubernetes.io/limit-rpm` define a limit on the connections that can be opened by a single client IP address. This can be used to mitigate [DDoS Attacks](https://www.nginx.com/blog/mitigating-ddos-attacks-with-nginx-and-nginx-plus).
`ingress.kubernetes.io/limit-connections`: number of concurrent connections allowed from a single IP address.
`ingress.kubernetes.io/limit-rps`: number of connections that may be accepted from a given IP each second.
If you specify both annotations in a single Ingress rule, `limit-rps` takes precedence.
`ingress.kubernetes.io/limit-rpm`: number of connections that may be accepted from a given IP each minute.
You can specify the client IP source ranges to be excluded from rate-limiting through the `ingress.kubernetes.io/limit-whitelist` annotation. The value is a comma separated list of CIDRs.
If you specify multiple annotations in a single Ingress rule, `limit-rpm`, and then `limit-rps` takes precedence.
The annotation `ingress.kubernetes.io/limit-rate`, `ingress.kubernetes.io/limit-rate-after` define a limit the rate of response transmission to a client. The rate is specified in bytes per second. The zero value disables rate limiting. The limit is set per a request, and so if a client simultaneously opens two connections, the overall rate will be twice as much as the specified limit.
`ingress.kubernetes.io/limit-rate-after`: sets the initial amount after which the further transmission of a response to a client will be rate limited.
`ingress.kubernetes.io/limit-rate`: rate of request that accepted from a client each second.
To configure this setting globally for all Ingress rules, the `limit-rate-after` and `limit-rate` value may be set in the NGINX ConfigMap. if you set the value in ingress annotation will cover global setting.
### SSL Passthrough
The annotation `ingress.kubernetes.io/ssl-passthrough` allows to configure TLS termination in the pod and not in NGINX.
This is possible thanks to the [ngx_stream_ssl_preread_module](https://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html) that enables the extraction of the server name information requested through SNI from the ClientHello message at the preread phase.
**Important:** using the annotation `ingress.kubernetes.io/ssl-passthrough` invalidates all the other available annotations. This is because SSL Passthrough works in L4 (TCP).
**Important:**
- Using the annotation `ingress.kubernetes.io/ssl-passthrough` invalidates all the other available annotations. This is because SSL Passthrough works in L4 (TCP).
- The use of this annotation requires the flag `--enable-ssl-passthrough` (By default it is disabled)
### Secure backends
By default NGINX uses `http` to reach the services. Adding the annotation `ingress.kubernetes.io/secure-backends: "true"` in the Ingress rule changes the protocol to `https`.
### Service Upstream
By default the NGINX ingress controller uses a list of all endpoints (Pod IP/port) in the NGINX upstream configuration. This annotation disables that behavior and instead uses a single upstream in NGINX, the service's Cluster IP and port. This can be desirable for things like zero-downtime deployments as it reduces the need to reload NGINX configuration when Pods come up and down. See issue [#257](https://github.com/kubernetes/ingress/issues/257).
#### Known Issues
If the `service-upstream` annotation is specified the following things should be taken into consideration:
* Sticky Sessions will not work as only round-robin load balancing is supported.
* The `proxy_next_upstream` directive will not have any effect meaning on error the request will not be dispatched to another upstream.
### Server-side HTTPS enforcement through redirect
By default the controller redirects (301) to `HTTPS` if TLS is enabled for that ingress. If you want to disable that behaviour globally, you can use `ssl-redirect: "false"` in the NGINX config map.
By default the controller redirects (301) to `HTTPS` if TLS is enabled for that ingress. If you want to disable that behavior globally, you can use `ssl-redirect: "false"` in the NGINX config map.
To configure this feature for specific ingress resources, you can use the `ingress.kubernetes.io/ssl-redirect: "false"` annotation in the particular resource.
When using SSL offloading outside of cluster (e.g. AWS ELB) it may be usefull to enforce a redirect to `HTTPS` even when there is not TLS cert available. This can be achieved by using the `ingress.kubernetes.io/force-ssl-redirect: "true"` annotation in the particular resource.
When using SSL offloading outside of cluster (e.g. AWS ELB) it may be useful to enforce a redirect to `HTTPS` even when there is not TLS cert available. This can be achieved by using the `ingress.kubernetes.io/force-ssl-redirect: "true"` annotation in the particular resource.
### Redirect from to www
In some scenarios is required to redirect from `www.domain.com` to `domain.com` or viceversa.
To enable this feature use the annotation `ingress.kubernetes.io/from-to-www-redirect: "true"`
**Important:**
If at some point a new Ingress is created with a host equal to one of the options (like `domain.com`) the annotation will be omitted.
### Whitelist source range
@ -233,67 +300,53 @@ To configure this setting globally for all Ingress rules, the `whitelist-source-
Please check the [whitelist](/examples/affinity/cookie/nginx/README.md) example.
### Session Affinity
The annotation `ingress.kubernetes.io/affinity` enables and sets the affinity type in all Upstreams of an Ingress. This way, a request will always be directed to the same upstream server.
The only affinity type available for NGINX is `cookie`.
#### Cookie affinity
If you use the ``cookie`` type you can also specify the name of the cookie that will be used to route the requests with the annotation `ingress.kubernetes.io/session-cookie-name`. The default is to create a cookie named 'route'.
In case of NGINX the annotation `ingress.kubernetes.io/session-cookie-hash` defines which algorithm will be used to 'hash' the used upstream. Default value is `md5` and possible values are `md5`, `sha1` and `index`.
The `index` option is not hashed, an in-memory index is used instead, it's quicker and the overhead is shorter Warning: the matching against upstream servers list is inconsistent. So, at reload, if upstreams servers has changed, index values are not guaranted to correspond to the same server as before! USE IT WITH CAUTION and only if you need to!
The `index` option is not hashed, an in-memory index is used instead, it's quicker and the overhead is shorter Warning: the matching against upstream servers list is inconsistent. So, at reload, if upstreams servers has changed, index values are not guaranteed to correspond to the same server as before! USE IT WITH CAUTION and only if you need to!
In NGINX this feature is implemented by the third party module [nginx-sticky-module-ng](https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng). The workflow used to define which upstream server will be used is explained [here](https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng/raw/08a395c66e425540982c00482f55034e1fee67b6/docs/sticky.pdf)
### **Allowed parameters in configuration ConfigMap**
**proxy-body-size:** Sets the maximum allowed size of the client request body. See NGINX [client_max_body_size](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size).
**custom-http-errors:** Enables which HTTP codes should be passed for processing with the [error_page directive](http://nginx.org/en/docs/http/ngx_http_core_module.html#error_page).
Setting at least one code also enables [proxy_intercept_errors](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_intercept_errors) which are required to process error_page.
Example usage: `custom-http-errors: 404,415`
**disable-access-log:** Disables the Access Log from the entire Ingress Controller. This is 'false' by default.
**access-log-path:** Access log path. Goes to '/var/log/nginx/access.log' by default. http://nginx.org/en/docs/http/ngx_http_log_module.html#access_log
**error-log-path:** Error log path. Goes to '/var/log/nginx/error.log' by default. http://nginx.org/en/docs/ngx_core_module.html#error_log
**disable-ipv6:** Disable listening on IPV6. This is 'false' by default.
**enable-dynamic-tls-records:** Enables dynamically sized TLS records to improve time-to-first-byte. Enabled by default. See [CloudFlare's blog](https://blog.cloudflare.com/optimizing-tls-over-tcp-to-reduce-latency) for more information.
**enable-underscores-in-headers:** Enables underscores in header names. This is disabled by default.
**enable-underscores-in-headers:** Enables underscores in header names. This is disabled by default.
**enable-vts-status:** Allows the replacement of the default status page with a third party module named [nginx-module-vts](https://github.com/vozlt/nginx-module-vts).
**error-log-level:** Configures the logging level of errors. Log levels above are listed in the order of increasing severity.
http://nginx.org/en/docs/ngx_core_module.html#error_log
**gzip-types:** Sets the MIME types in addition to "text/html" to compress. The special value "\*" matches any MIME type.
Responses with the "text/html" type are always compressed if `use-gzip` is enabled.
**hsts:** Enables or disables the header HSTS in servers running SSL.
HTTP Strict Transport Security (often abbreviated as HSTS) is a security feature (HTTP header) that tell browsers that it should only be communicated with using HTTPS, instead of using HTTP. It provides protection against protocol downgrade attacks and cookie theft.
https://developer.mozilla.org/en-US/docs/Web/Security/HTTP_strict_transport_security
https://blog.qualys.com/securitylabs/2016/03/28/the-importance-of-a-proper-http-strict-transport-security-implementation-on-your-web-server
**hsts-include-subdomains:** Enables or disables the use of HSTS in all the subdomains of the servername.
**hsts-include-subdomains:** Enables or disables the use of HSTS in all the subdomains of the server-name.
**hsts-max-age:** Sets the time, in seconds, that the browser should remember that this site is only to be accessed using HTTPS.
@ -306,7 +359,7 @@ The zero value disables keep-alive client connections.
http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout
**load-balance:** Sets the algorithm to use for load balancing. The value can either be round_robin to
use the default round robin load balancer, least_conn to use the least connected method, or
use the default round robin loadbalancer, least_conn to use the least connected method, or
ip_hash to use a hash of the server for routing. The default is least_conn.
http://nginx.org/en/docs/http/load_balancing.html.
@ -321,59 +374,56 @@ log-format-upstream: '{ "time": "$time_iso8601", "remote_addr": "$proxy_protocol
$status, "vhost": "$host", "request_proto": "$server_protocol", "path": "$uri",
"request_query": "$args", "request_length": $request_length, "duration": $request_time,
"method": "$request_method", "http_referrer": "$http_referer", "http_user_agent":
"$http_user_agent" }'
"$http_user_agent" }'
```
**log-format-stream:** Sets the nginx [stream format](https://nginx.org/en/docs/stream/ngx_stream_log_module.html#log_format)
.
**log-format-stream:** Sets the nginx [stream format](https://nginx.org/en/docs/stream/ngx_stream_log_module.html#log_format).
**max-worker-connections:** Sets the maximum number of simultaneous connections that can be opened by each [worker process](http://nginx.org/en/docs/ngx_core_module.html#worker_connections).
**proxy-buffer-size:** Sets the size of the buffer used for [reading the first part of the response](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffer_size) received from the proxied server. This part usually contains a small response header.
**proxy-connect-timeout:** Sets the timeout for [establishing a connection with a proxied server](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout). It should be noted that this timeout cannot usually exceed 75 seconds.
**proxy-cookie-domain:** Sets a text that [should be changed in the domain attribute](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cookie_domain) of the “Set-Cookie” header fields of a proxied server response.
**proxy-cookie-path:** Sets a text that [should be changed in the path attribute](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cookie_path) of the “Set-Cookie” header fields of a proxied server response.
**proxy-read-timeout:** Sets the timeout in seconds for [reading a response from the proxied server](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout). The timeout is set only between two successive read operations, not for the transmission of the whole response.
**proxy-send-timeout:** Sets the timeout in seconds for [transmitting a request to the proxied server](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout). The timeout is set only between two successive write operations, not for the transmission of the whole request.
**proxy-next-upstream:** Specifies in [which cases](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream) a request should be passed to the next server.
**proxy-request-buffering:** Enables or disables [buffering of a client request body](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering).
**retry-non-idempotent:** Since 1.9.13 NGINX will not retry non-idempotent requests (POST, LOCK, PATCH) in case of an error in the upstream server.
The previous behavior can be restored using the value "true".
**server-name-hash-bucket-size:** Sets the size of the bucket for the server names hash tables.
http://nginx.org/en/docs/hash.html
http://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_bucket_size
**server-name-hash-max-size:** Sets the maximum size of the [server names hash tables](http://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_max_size) used in server names, map directive’s values, MIME types, names of request header strings, etc.
http://nginx.org/en/docs/hash.html
**proxy-headers-hash-bucket-size:** Sets the size of the bucket for the proxy headers hash tables.
http://nginx.org/en/docs/hash.html
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_headers_hash_bucket_size
**proxy-headers-hash-max-size:** Sets the maximum size of the proxy headers hash tables.
http://nginx.org/en/docs/hash.html
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_headers_hash_max_size
**server-tokens:** Send NGINX Server header in responses and display NGINX version in error pages. Enabled by default.
**map-hash-bucket-size:** Sets the bucket size for the [map variables hash tables](http://nginx.org/en/docs/http/ngx_http_map_module.html#map_hash_bucket_size). The details of setting up hash tables are provided in a separate [document](http://nginx.org/en/docs/hash.html).
**ssl-buffer-size:** Sets the size of the [SSL buffer](http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_buffer_size) used for sending data.
The default of 4k helps NGINX to improve TLS Time To First Byte (TTTFB).
https://www.igvita.com/2013/12/16/optimizing-nginx-tls-time-to-first-byte/
**ssl-ciphers:** Sets the [ciphers](http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ciphers) list to enable. The ciphers are specified in the format understood by the OpenSSL library.
The default cipher list is:
@ -384,13 +434,11 @@ The recommendation above prioritizes algorithms that provide perfect [forward se
Please check the [Mozilla SSL Configuration Generator](https://mozilla.github.io/server-side-tls/ssl-config-generator/).
**ssl-dh-param:** Sets the name of the secret that contains Diffie-Hellman key to help with "Perfect Forward Secrecy".
https://www.openssl.org/docs/manmaster/apps/dhparam.html
https://wiki.mozilla.org/Security/Server_Side_TLS#DHE_handshake_and_dhparam
http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_dhparam
**ssl-protocols:** Sets the [SSL protocols](http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_protocols) to use.
The default is: `TLSv1 TLSv1.1 TLSv1.2`.
@ -402,44 +450,41 @@ If you don't need to support these clients please remove `TLSv1` to improve secu
Please check the result of the configuration using `https://ssllabs.com/ssltest/analyze.html` or `https://testssl.sh`.
**ssl-redirect:** Sets the global value of redirects (301) to HTTPS if the server has a TLS certificate (defined in an Ingress rule)
Default is "true".
**ssl-session-cache:** Enables or disables the use of shared [SSL cache](http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_session_cache) among worker processes.
**ssl-session-cache-size:** Sets the size of the [SSL shared session cache](http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_session_cache) between all worker processes.
**ssl-session-tickets:** Enables or disables session resumption through [TLS session tickets](http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_session_tickets).
**ssl-session-timeout:** Sets the time during which a client may [reuse the session](http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_session_timeout) parameters stored in a cache.
**upstream-max-fails:** Sets the number of unsuccessful attempts to communicate with the [server](http://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream) that should happen in the duration set by the `fail_timeout` parameter to consider the server unavailable.
**upstream-fail-timeout:** Sets the time during which the specified number of unsuccessful attempts to communicate with the [server](http://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream) should happen to consider the server unavailable.
**use-gzip:** Enables or disables compression of HTTP responses using the ["gzip" module](http://nginx.org/en/docs/http/ngx_http_gzip_module.html)
The default mime type list to compress is: `application/atom+xml application/javascript aplication/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component`.
**use-http2:** Enables or disables [HTTP/2](http://nginx.org/en/docs/http/ngx_http_v2_module.html) support in secure connections.
**use-proxy-protocol:** Enables or disables the [PROXY protocol](https://www.nginx.com/resources/admin-guide/proxy-protocol/) to receive client connection (real IP address) information passed through proxy servers and load balancers such as HAProxy and Amazon Elastic Load Balancer (ELB).
**whitelist-source-range:** Sets the default whitelisted IPs for each `server` block. This can be overwritten by an annotation on an Ingress rule. See [ngx_http_access_module](http://nginx.org/en/docs/http/ngx_http_access_module.html).
**worker-processes:** Sets the number of [worker processes](http://nginx.org/en/docs/ngx_core_module.html#worker_processes). The default of "auto" means number of available CPU cores.
**worker-shutdown-timeout:** Sets a timeout for Nginx to [wait for worker to gracefully shutdown](http://nginx.org/en/docs/ngx_core_module.html#worker_shutdown_timeout). The default is "10s".
**limit-conn-zone-variable:** Sets parameters for a shared memory zone that will keep states for various keys of [limit_conn_zone](http://nginx.org/en/docs/http/ngx_http_limit_conn_module.html#limit_conn_zone). The default of "$binary_remote_addr" variable’s size is always 4 bytes for IPv4 addresses or 16 bytes for IPv6 addresses.
**proxy-set-headers:** Sets custom headers from a configmap before sending traffic to backends. See [example](https://github.com/kubernetes/ingress/tree/master/examples/customization/custom-headers/nginx)
**add-headers:** Sets custom headers from a configmap before sending traffic to the client. See `proxy-set-headers` [example](https://github.com/kubernetes/ingress/tree/master/examples/customization/custom-headers/nginx)
**bind-address:** Sets the addresses on which the server will accept requests instead of *. It should be noted that these addresses must exist in the runtime environment or the controller will crash loop.
### Default configuration options
@ -460,13 +505,14 @@ The following table shows the options, the default value and a description.
|hsts-max-age|"15724800"|
|hsts-preload|"false"|
|ignore-invalid-headers|"true"|
|keep-alive|"75"|
|keep-alive|"75"|
|log-format-stream|[$time_local] $protocol $status $bytes_sent $bytes_received $session_time|
|log-format-upstream|[$the_x_forwarded_for] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status|
|log-format-upstream|[$the_real_ip] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status|
|map-hash-bucket-size|"64"|
|max-worker-connections|"16384"|
|proxy-body-size|same as body-size|
|proxy-buffer-size|"4k"|
|proxy-request-buffering|"on"|
|proxy-connect-timeout|"5"|
|proxy-cookie-domain|"off"|
|proxy-cookie-path|"off"|
@ -487,12 +533,14 @@ The following table shows the options, the default value and a description.
|ssl-session-timeout|10m|
|use-gzip|"true"|
|use-http2|"true"|
|upstream-keepalive-connections|"0" (disabled)|
|variables-hash-bucket-size|64|
|variables-hash-max-size|2048|
|vts-status-zone-size|10m|
|whitelist-source-range|permit all|
|worker-processes|number of CPUs|
|limit-conn-zone-variable|$binary_remote_addr|
|bind-address||
### Websockets
@ -501,23 +549,19 @@ Support for websockets is provided by NGINX out of the box. No special configura
The only requirement to avoid the close of connections is the increase of the values of `proxy-read-timeout` and `proxy-send-timeout`. The default value of this settings is `60 seconds`.
A more adequate value to support websockets is a value higher than one hour (`3600`).
### Optimizing TLS Time To First Byte (TTTFB)
NGINX provides the configuration option [ssl_buffer_size](http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_buffer_size) to allow the optimization of the TLS record size. This improves the [Time To First Byte](https://www.igvita.com/2013/12/16/optimizing-nginx-tls-time-to-first-byte/) (TTTFB). The default value in the Ingress controller is `4k` (NGINX default is `16k`).
### Retries in non-idempotent methods
Since 1.9.13 NGINX will not retry non-idempotent requests (POST, LOCK, PATCH) in case of an error.
The previous behavior can be restored using `retry-non-idempotent=true` in the configuration ConfigMap.
### Custom max body size
For NGINX, 413 error will be returned to the client when the size in a request exceeds the maximum allowed size of the client request body. This size can be configured by the parameter [`client_max_body_size`](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size).
To configure this setting globally for all Ingress rules, the `proxy-body-size` value may be set in the NGINX ConfigMap.
To use custom values in an Ingress rule define these annotation:
```

51
controllers/nginx/examples/default-backend.yaml Normal file
View file

@ -0,0 +1,51 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: default-http-backend
labels:
k8s-app: default-http-backend
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
k8s-app: default-http-backend
spec:
terminationGracePeriodSeconds: 60
containers:
- name: default-http-backend
# Any image is permissable as long as:
# 1. It serves a 404 page at /
# 2. It serves 200 on a /healthz endpoint
image: gcr.io/google_containers/defaultbackend:1.0
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
ports:
- containerPort: 8080
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
namespace: kube-system
labels:
k8s-app: default-http-backend
spec:
ports:
- port: 80
targetPort: 8080
selector:
k8s-app: default-http-backend

46
controllers/nginx/examples/echo-header.yaml Normal file
View file

@ -0,0 +1,46 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoheaders
spec:
replicas: 1
template:
metadata:
labels:
app: echoheaders
spec:
containers:
- name: echoheaders
image: gcr.io/google_containers/echoserver:1.8
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: echoheaders-x
labels:
app: echoheaders-x
spec:
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
selector:
app: echoheaders
---
apiVersion: v1
kind: Service
metadata:
name: echoheaders-y
labels:
app: echoheaders-y
spec:
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
selector:
app: echoheaders

26
controllers/nginx/examples/ingress.yaml Normal file
View file

@ -0,0 +1,26 @@
# This is the Ingress resource that creates a HTTP Loadbalancer configured
# according to the Ingress rules.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: echomap
spec:
rules:
- host: foo.bar.com
http:
paths:
- path: /foo
backend:
serviceName: echoheaders-x
servicePort: 80
- host: bar.baz.com
http:
paths:
- path: /bar
backend:
serviceName: echoheaders-y
servicePort: 80
- path: /foo
backend:
serviceName: echoheaders-x
servicePort: 80

12
controllers/nginx/pkg/cmd/controller/main.go
View file

@ -23,17 +23,15 @@ import (
"time"
"github.com/golang/glog"
"k8s.io/ingress/core/pkg/ingress/controller"
)
func main() {
// start a new nginx controller
ngx := newNGINXController()
// create a custom Ingress controller using NGINX as backend
ic := controller.NewIngressController(ngx)
go handleSigterm(ic)
go handleSigterm(ngx)
// start the controller
ic.Start()
ngx.Start()
// wait
glog.Infof("shutting down Ingress controller...")
for {
@ -42,14 +40,14 @@ func main() {
}
}
func handleSigterm(ic *controller.GenericController) {
func handleSigterm(ngx *NGINXController) {
signalChan := make(chan os.Signal, 1)
signal.Notify(signalChan, syscall.SIGTERM)
<-signalChan
glog.Infof("Received SIGTERM, shutting down")
exitCode := 0
if err := ic.Stop(); err != nil {
if err := ngx.Stop(); err != nil {
glog.Infof("Error during shutdown %v", err)
exitCode = 1
}

14
controllers/nginx/pkg/cmd/controller/metrics.go
View file

@ -24,7 +24,7 @@ import (
)
const (
ngxStatusPath = "/internal_nginx_status"
ngxStatusPath = "/nginx_status"
ngxVtsPath = "/nginx_status/format/json"
)
@ -45,6 +45,8 @@ type statsCollector struct {
namespace string
watchClass string
port int
}
func (s *statsCollector) stop(sm statusModule) {
@ -52,29 +54,28 @@ func (s *statsCollector) stop(sm statusModule) {
case defaultStatusModule:
s.basic.Stop()
prometheus.Unregister(s.basic)
break
case vtsStatusModule:
s.vts.Stop()
prometheus.Unregister(s.vts)
break
}
}
func (s *statsCollector) start(sm statusModule) {
switch sm {
case defaultStatusModule:
s.basic = collector.NewNginxStatus(s.namespace, s.watchClass, ngxHealthPort, ngxStatusPath)
s.basic = collector.NewNginxStatus(s.namespace, s.watchClass, s.port, ngxStatusPath)
prometheus.Register(s.basic)
break
case vtsStatusModule:
s.vts = collector.NewNGINXVTSCollector(s.namespace, s.watchClass, ngxHealthPort, ngxVtsPath)
s.vts = collector.NewNGINXVTSCollector(s.namespace, s.watchClass, s.port, ngxVtsPath)
prometheus.Register(s.vts)
break
}
}
func newStatsCollector(ns, class, binary string) *statsCollector {
func newStatsCollector(ns, class, binary string, port int) *statsCollector {
glog.Infof("starting new nginx stats collector for Ingress controller running in namespace %v (class %v)", ns, class)
glog.Infof("collector extracting information from port %v", port)
pc, err := collector.NewNamedProcess(true, collector.BinaryNameMatcher{
Name: "nginx",
Binary: binary,
@ -91,5 +92,6 @@ func newStatsCollector(ns, class, binary string) *statsCollector {
namespace: ns,
watchClass: class,
process: pc,
port: port,
}
}

364
controllers/nginx/pkg/cmd/controller/nginx.go
View file

@ -31,15 +31,19 @@ import (
"time"
"github.com/golang/glog"
"github.com/mitchellh/go-ps"
"github.com/spf13/pflag"
proxyproto "github.com/armon/go-proxyproto"
api_v1 "k8s.io/client-go/pkg/api/v1"
api "k8s.io/api/core/v1"
api_v1 "k8s.io/api/core/v1"
extensions "k8s.io/api/extensions/v1beta1"
"k8s.io/ingress/controllers/nginx/pkg/config"
ngx_template "k8s.io/ingress/controllers/nginx/pkg/template"
"k8s.io/ingress/controllers/nginx/pkg/version"
"k8s.io/ingress/core/pkg/ingress"
"k8s.io/ingress/core/pkg/ingress/controller"
"k8s.io/ingress/core/pkg/ingress/defaults"
"k8s.io/ingress/core/pkg/net/dns"
"k8s.io/ingress/core/pkg/net/ssl"
@ -48,11 +52,12 @@ import (
type statusModule string
const (
ngxHealthPort = 18080
ngxHealthPath = "/healthz"
defaultStatusModule statusModule = "default"
vtsStatusModule statusModule = "vts"
defUpstreamName = "upstream-default-backend"
)
var (
@ -65,7 +70,7 @@ var (
// newNGINXController creates a new NGINX Ingress controller.
// If the environment variable NGINX_BINARY exists it will be used
// as source for nginx commands
func newNGINXController() ingress.Controller {
func newNGINXController() *NGINXController {
ngx := os.Getenv("NGINX_BINARY")
if ngx == "" {
ngx = binary
@ -77,51 +82,14 @@ func newNGINXController() ingress.Controller {
}
n := &NGINXController{
binary: ngx,
configmap: &api_v1.ConfigMap{},
isIPV6Enabled: isIPv6Enabled(),
resolver: h,
proxy: &proxy{
Default: &server{
Hostname: "localhost",
IP: "127.0.0.1",
Port: 442,
ProxyProtocol: true,
},
},
binary: ngx,
configmap: &api_v1.ConfigMap{},
isIPV6Enabled: isIPv6Enabled(),
resolver: h,
ports: &config.ListenPorts{},
backendDefaults: config.NewDefault().Backend,
}
listener, err := net.Listen("tcp", ":443")
if err != nil {
glog.Fatalf("%v", err)
}
proxyList := &proxyproto.Listener{Listener: listener}
// start goroutine that accepts tcp connections in port 443
go func() {
for {
var conn net.Conn
var err error
if n.isProxyProtocolEnabled {
// we need to wrap the listener in order to decode
// proxy protocol before handling the connection
conn, err = proxyList.Accept()
} else {
conn, err = listener.Accept()
}
if err != nil {
glog.Warningf("unexpected error accepting tcp connection: %v", err)
continue
}
glog.V(3).Infof("remote address %s to local %s", conn.RemoteAddr(), conn.LocalAddr())
go n.proxy.Handle(conn)
}
}()
var onChange func()
onChange = func() {
template, err := ngx_template.NewTemplate(tmplPath, onChange)
@ -147,14 +115,13 @@ Error loading new template : %v
n.t = ngxTpl
go n.Start()
return ingress.Controller(n)
return n
}
// NGINXController ...
type NGINXController struct {
t *ngx_template.Template
controller *controller.GenericController
t *ngx_template.Template
configmap *api_v1.ConfigMap
@ -165,9 +132,6 @@ type NGINXController struct {
cmdArgs []string
watchClass string
namespace string
stats *statsCollector
statusModule statusModule
@ -177,15 +141,35 @@ type NGINXController struct {
// returns true if proxy protocol es enabled
isProxyProtocolEnabled bool
isSSLPassthroughEnabled bool
isShuttingDown bool
proxy *proxy
ports *config.ListenPorts
backendDefaults defaults.Backend
}
// Start start a new NGINX master process running in foreground.
func (n *NGINXController) Start() {
glog.Info("starting NGINX process...")
n.isShuttingDown = false
n.controller = controller.NewIngressController(n)
go n.controller.Start()
done := make(chan error, 1)
cmd := exec.Command(n.binary, "-c", cfgPath)
// put nginx in another process group to prevent it
// to receive signals meant for the controller
cmd.SysProcAttr = &syscall.SysProcAttr{
Setpgid: true,
Pgid: 0,
}
glog.Info("starting NGINX process...")
n.start(cmd, done)
// if the nginx master process dies the workers continue to process requests,
@ -195,6 +179,11 @@ func (n *NGINXController) Start() {
// To avoid this issue we restart nginx in case of errors.
for {
err := <-done
if n.isShuttingDown {
break
}
if exitError, ok := err.(*exec.ExitError); ok {
waitStatus := exitError.Sys().(syscall.WaitStatus)
glog.Warningf(`
@ -214,11 +203,34 @@ NGINX master process died (%v): %v
conn.Close()
time.Sleep(1 * time.Second)
}
// start a new nginx master process
// restart a new nginx master process if the controller
// is not being stopped
n.start(cmd, done)
}
}
// Stop gracefully stops the NGINX master process.
func (n *NGINXController) Stop() error {
n.isShuttingDown = true
n.controller.Stop()
// Send stop signal to Nginx
glog.Info("stopping NGINX process...")
cmd := exec.Command(n.binary, "-c", cfgPath, "-s", "quit")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err := cmd.Run()
if err != nil {
return err
}
// Wait for the Nginx process disappear
waitForNginxShutdown()
glog.Info("NGINX process has stopped")
return nil
}
func (n *NGINXController) start(cmd *exec.Cmd, done chan error) {
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
@ -237,17 +249,16 @@ func (n *NGINXController) start(cmd *exec.Cmd, done chan error) {
// BackendDefaults returns the nginx defaults
func (n NGINXController) BackendDefaults() defaults.Backend {
if n.configmap == nil {
d := config.NewDefault()
return d.Backend
}
return ngx_template.ReadConfig(n.configmap.Data).Backend
return n.backendDefaults
}
// printDiff returns the difference between the running configuration
// and the new one
func (n NGINXController) printDiff(data []byte) {
if !glog.V(2) {
return
}
in, err := os.Open(cfgPath)
if err != nil {
return
@ -276,10 +287,9 @@ func (n NGINXController) printDiff(data []byte) {
return
}
if glog.V(2) {
glog.Infof("NGINX configuration diff\n")
glog.Infof("%v", string(diffOutput))
}
glog.Infof("NGINX configuration diff\n")
glog.Infof("%v", string(diffOutput))
os.Remove(tmpfile.Name())
}
}
@ -294,13 +304,42 @@ func (n NGINXController) Info() *ingress.BackendInfo {
}
}
// DefaultEndpoint returns the default endpoint to be use as default server that returns 404.
func (n NGINXController) DefaultEndpoint() ingress.Endpoint {
return ingress.Endpoint{
Address: "127.0.0.1",
Port: fmt.Sprintf("%v", n.ports.Default),
Target: &api.ObjectReference{},
}
}
// ConfigureFlags allow to configure more flags before the parsing of
// command line arguments
func (n *NGINXController) ConfigureFlags(flags *pflag.FlagSet) {
flags.BoolVar(&n.isSSLPassthroughEnabled, "enable-ssl-passthrough", false, `Enable SSL passthrough feature. Default is disabled`)
flags.IntVar(&n.ports.HTTP, "http-port", 80, `Indicates the port to use for HTTP traffic`)
flags.IntVar(&n.ports.HTTPS, "https-port", 443, `Indicates the port to use for HTTPS traffic`)
flags.IntVar(&n.ports.Status, "status-port", 18080, `Indicates the TCP port to use for exposing the nginx status page`)
flags.IntVar(&n.ports.SSLProxy, "ssl-passtrough-proxy-port", 442, `Default port to use internally for SSL when SSL Passthgough is enabled`)
flags.IntVar(&n.ports.Default, "default-server-port", 8181, `Default port to use for exposing the default server (catch all)`)
}
// OverrideFlags customize NGINX controller flags
func (n *NGINXController) OverrideFlags(flags *pflag.FlagSet) {
// we check port collisions
if !isPortAvailable(n.ports.HTTP) {
glog.Fatalf("Port %v is already in use. Please check the flag --http-port", n.ports.HTTP)
}
if !isPortAvailable(n.ports.HTTPS) {
glog.Fatalf("Port %v is already in use. Please check the flag --https-port", n.ports.HTTPS)
}
if !isPortAvailable(n.ports.Status) {
glog.Fatalf("Port %v is already in use. Please check the flag --status-port", n.ports.Status)
}
if !isPortAvailable(n.ports.Default) {
glog.Fatalf("Port %v is already in use. Please check the flag --default-server-port", n.ports.Default)
}
ic, _ := flags.GetString("ingress-class")
wc, _ := flags.GetString("watch-namespace")
@ -313,7 +352,58 @@ func (n *NGINXController) OverrideFlags(flags *pflag.FlagSet) {
}
flags.Set("ingress-class", ic)
n.stats = newStatsCollector(wc, ic, n.binary)
h, _ := flags.GetInt("healthz-port")
n.ports.Health = h
n.stats = newStatsCollector(wc, ic, n.binary, n.ports.Status)
if n.isSSLPassthroughEnabled {
if !isPortAvailable(n.ports.SSLProxy) {
glog.Fatalf("Port %v is already in use. Please check the flag --ssl-passtrough-proxy-port", n.ports.SSLProxy)
}
glog.Info("starting TLS proxy for SSL passthrough")
n.proxy = &proxy{
Default: &server{
Hostname: "localhost",
IP: "127.0.0.1",
Port: n.ports.SSLProxy,
ProxyProtocol: true,
},
}
listener, err := net.Listen("tcp", fmt.Sprintf(":%v", n.ports.HTTPS))
if err != nil {
glog.Fatalf("%v", err)
}
proxyList := &proxyproto.Listener{Listener: listener}
// start goroutine that accepts tcp connections in port 443
go func() {
for {
var conn net.Conn
var err error
if n.isProxyProtocolEnabled {
// we need to wrap the listener in order to decode
// proxy protocol before handling the connection
conn, err = proxyList.Accept()
} else {
conn, err = listener.Accept()
}
if err != nil {
glog.Warningf("unexpected error accepting tcp connection: %v", err)
continue
}
glog.V(3).Infof("remote address %s to local %s", conn.RemoteAddr(), conn.LocalAddr())
go n.proxy.Handle(conn)
}
}()
}
}
// DefaultIngressClass just return the default ingress class
@ -355,20 +445,22 @@ Error: %v
// SetConfig sets the configured configmap
func (n *NGINXController) SetConfig(cmap *api_v1.ConfigMap) {
n.configmap = cmap
n.isProxyProtocolEnabled = false
if cmap == nil {
return
m := map[string]string{}
if cmap != nil {
m = cmap.Data
}
val, ok := cmap.Data["use-proxy-protocol"]
val, ok := m["use-proxy-protocol"]
if ok {
b, err := strconv.ParseBool(val)
if err == nil {
n.isProxyProtocolEnabled = b
return
}
}
n.backendDefaults = ngx_template.ReadConfig(m).Backend
}
// SetListers sets the configured store listers in the generic ingress controller
@ -376,7 +468,12 @@ func (n *NGINXController) SetListers(lister ingress.StoreLister) {
n.storeLister = lister
}
// OnUpdate is called by syncQueue in https://github.com/aledbf/ingress-controller/blob/master/pkg/ingress/controller/controller.go#L82
// UpdateIngressStatus custom Ingress status update
func (n *NGINXController) UpdateIngressStatus(*extensions.Ingress) []api_v1.LoadBalancerIngress {
return nil
}
// OnUpdate is called by syncQueue in https://github.com/kubernetes/ingress/blob/master/core/pkg/ingress/controller/controller.go#L426
// periodically to keep the configuration in sync.
//
// convert configmap to custom configuration object (different in each implementation)
@ -385,15 +482,6 @@ func (n *NGINXController) SetListers(lister ingress.StoreLister) {
// returning nill implies the backend will be reloaded.
// if an error is returned means requeue the update
func (n *NGINXController) OnUpdate(ingressCfg ingress.Configuration) error {
var longestName int
var serverNameBytes int
for _, srv := range ingressCfg.Servers {
if longestName < len(srv.Hostname) {
longestName = len(srv.Hostname)
}
serverNameBytes += len(srv.Hostname)
}
cfg := ngx_template.ReadConfig(n.configmap.Data)
cfg.Resolver = n.resolver
@ -430,7 +518,9 @@ func (n *NGINXController) OnUpdate(ingressCfg ingress.Configuration) error {
})
}
n.proxy.ServerList = servers
if n.isSSLPassthroughEnabled {
n.proxy.ServerList = servers
}
// we need to check if the status module configuration changed
if cfg.EnableVtsStatus {
@ -439,14 +529,44 @@ func (n *NGINXController) OnUpdate(ingressCfg ingress.Configuration) error {
n.setupMonitor(defaultStatusModule)
}
// NGINX cannot resize the has tables used to store server names.
// NGINX cannot resize the hash tables used to store server names.
// For this reason we check if the defined size defined is correct
// for the FQDN defined in the ingress rules adjusting the value
// if is required.
// https://trac.nginx.org/nginx/ticket/352
// https://trac.nginx.org/nginx/ticket/631
nameHashBucketSize := nginxHashBucketSize(longestName)
var longestName int
var serverNameBytes int
redirectServers := make(map[string]string)
for _, srv := range ingressCfg.Servers {
if longestName < len(srv.Hostname) {
longestName = len(srv.Hostname)
}
serverNameBytes += len(srv.Hostname)
if srv.RedirectFromToWWW {
var n string
if strings.HasPrefix(srv.Hostname, "www.") {
n = strings.TrimLeft(srv.Hostname, "www.")
} else {
n = fmt.Sprintf("www.%v", srv.Hostname)
}
glog.V(3).Infof("creating redirect from %v to %v", srv.Hostname, n)
if _, ok := redirectServers[n]; !ok {
found := false
for _, esrv := range ingressCfg.Servers {
if esrv.Hostname == n {
found = true
break
}
}
if !found {
redirectServers[n] = srv.Hostname
}
}
}
}
if cfg.ServerNameHashBucketSize == 0 {
nameHashBucketSize := nginxHashBucketSize(longestName)
glog.V(3).Infof("adjusting ServerNameHashBucketSize variable to %v", nameHashBucketSize)
cfg.ServerNameHashBucketSize = nameHashBucketSize
}
@ -482,6 +602,18 @@ func (n *NGINXController) OnUpdate(ingressCfg ingress.Configuration) error {
}
}
addHeaders := map[string]string{}
if cfg.AddHeaders != "" {
cmap, exists, err := n.storeLister.ConfigMap.GetByKey(cfg.AddHeaders)
if err != nil {
glog.Warningf("unexpected error reading configmap %v: %v", cfg.AddHeaders, err)
}
if exists {
addHeaders = cmap.(*api_v1.ConfigMap).Data
}
}
sslDHParam := ""
if cfg.SSLDHParam != "" {
secretName := cfg.SSLDHParam
@ -508,20 +640,26 @@ func (n *NGINXController) OnUpdate(ingressCfg ingress.Configuration) error {
cfg.SSLDHParam = sslDHParam
content, err := n.t.Write(config.TemplateConfig{
ProxySetHeaders: setHeaders,
MaxOpenFiles: maxOpenFiles,
BacklogSize: sysctlSomaxconn(),
Backends: ingressCfg.Backends,
PassthroughBackends: ingressCfg.PassthroughBackends,
Servers: ingressCfg.Servers,
TCPBackends: ingressCfg.TCPEndpoints,
UDPBackends: ingressCfg.UDPEndpoints,
HealthzURI: ngxHealthPath,
CustomErrors: len(cfg.CustomHTTPErrors) > 0,
Cfg: cfg,
IsIPV6Enabled: n.isIPV6Enabled && !cfg.DisableIpv6,
})
tc := config.TemplateConfig{
ProxySetHeaders: setHeaders,
AddHeaders: addHeaders,
MaxOpenFiles: maxOpenFiles,
BacklogSize: sysctlSomaxconn(),
Backends: ingressCfg.Backends,
PassthroughBackends: ingressCfg.PassthroughBackends,
Servers: ingressCfg.Servers,
TCPBackends: ingressCfg.TCPEndpoints,
UDPBackends: ingressCfg.UDPEndpoints,
HealthzURI: ngxHealthPath,
CustomErrors: len(cfg.CustomHTTPErrors) > 0,
Cfg: cfg,
IsIPV6Enabled: n.isIPV6Enabled && !cfg.DisableIpv6,
RedirectServers: redirectServers,
IsSSLPassthroughEnabled: n.isSSLPassthroughEnabled,
ListenPorts: n.ports,
}
content, err := n.t.Write(tc)
if err != nil {
return err
@ -539,9 +677,9 @@ func (n *NGINXController) OnUpdate(ingressCfg ingress.Configuration) error {
return err
}
o, e := exec.Command(n.binary, "-s", "reload", "-c", cfgPath).CombinedOutput()
o, err := exec.Command(n.binary, "-s", "reload", "-c", cfgPath).CombinedOutput()
if err != nil {
return fmt.Errorf("%v\n%v", e, string(o))
return fmt.Errorf("%v\n%v", err, string(o))
}
return nil
@ -564,7 +702,7 @@ func (n NGINXController) Name() string {
// Check returns if the nginx healthz endpoint is returning ok (status code 200)
func (n NGINXController) Check(_ *http.Request) error {
res, err := http.Get(fmt.Sprintf("http://localhost:%v%v", ngxHealthPort, ngxHealthPath))
res, err := http.Get(fmt.Sprintf("http://localhost:%v%v", n.ports.Status, ngxHealthPath))
if err != nil {
return err
}
@ -593,3 +731,27 @@ func isIPv6Enabled() bool {
cmd := exec.Command("test", "-f", "/proc/net/if_inet6")
return cmd.Run() == nil
}
// isNginxRunning returns true if a process with the name 'nginx' is found
func isNginxProcessPresent() bool {
processes, _ := ps.Processes()
for _, p := range processes {
if p.Executable() == "nginx" {
return true
}
}
return false
}
func waitForNginxShutdown() {
timer := time.NewTicker(time.Second * 1)
defer timer.Stop()
for {
select {
case <-timer.C:
if !isNginxProcessPresent() {
return
}
}
}
}

11
controllers/nginx/pkg/cmd/controller/utils.go
View file

@ -17,7 +17,9 @@ limitations under the License.
package main
import (
"fmt"
"io/ioutil"
"net"
"os"
"os/exec"
"syscall"
@ -74,3 +76,12 @@ func diff(b1, b2 []byte) ([]byte, error) {
out, _ := exec.Command("diff", "-u", f1.Name(), f2.Name()).CombinedOutput()
return out, nil
}
func isPortAvailable(p int) bool {
ln, err := net.Listen("tcp", fmt.Sprintf(":%v", p))
if err != nil {
return false
}
ln.Close()
return true
}

198
controllers/nginx/pkg/config/config.go
View file

@ -43,12 +43,9 @@ const (
// max-age is the time, in seconds, that the browser should remember that this site is only to be accessed using HTTPS.
hstsMaxAge = "15724800"
// If UseProxyProtocol is enabled defIPCIDR defines the default the IP/network address of your external load balancer
defIPCIDR = "0.0.0.0/0"
gzipTypes = "application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component"
logFormatUpstream = `%v - [$the_x_forwarded_for] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status`
logFormatUpstream = `%v - [$the_real_ip] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status`
logFormatStream = `[$time_local] $protocol $status $bytes_sent $bytes_received $session_time`
@ -76,18 +73,35 @@ const (
// Default setting for load balancer algorithm
defaultLoadBalancerAlgorithm = "least_conn"
// Parameters for a shared memory zone that will keep states for various keys.
// http://nginx.org/en/docs/http/ngx_http_limit_conn_module.html#limit_conn_zone
defaultLimitConnZoneVariable = "$binary_remote_addr"
)
// Configuration represents the content of nginx.conf file
type Configuration struct {
defaults.Backend `json:",squash"`
// Sets the name of the configmap that contains the headers to pass to the client
AddHeaders string `json:"add-headers,omitempty"`
// AllowBackendServerHeader enables the return of the header Server from the backend
// instead of the generic nginx string.
// http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header
// By default this is disabled
AllowBackendServerHeader bool `json:"allow-backend-server-header"`
// AccessLogPath sets the path of the access logs if enabled
// http://nginx.org/en/docs/http/ngx_http_log_module.html#access_log
// By default access logs go to /var/log/nginx/access.log
AccessLogPath string `json:"access-log-path,omitempty"`
// ErrorLogPath sets the path of the error logs
// http://nginx.org/en/docs/ngx_core_module.html#error_log
// By default error logs go to /var/log/nginx/error.log
ErrorLogPath string `json:"error-log-path,omitempty"`
// EnableDynamicTLSRecords enables dynamic TLS record sizes
// https://blog.cloudflare.com/optimizing-tls-over-tcp-to-reduce-latency
// By default this is enabled
@ -98,10 +112,18 @@ type Configuration struct {
// http://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_buffer_size
ClientHeaderBufferSize string `json:"client-header-buffer-size"`
// Defines a timeout for reading client request header, in seconds
// http://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_timeout
ClientHeaderTimeout int `json:"client-header-timeout,omitempty"`
// Sets buffer size for reading client request body
// http://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size
ClientBodyBufferSize string `json:"client-body-buffer-size,omitempty"`
// Defines a timeout for reading client request body, in seconds
// http://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_timeout
ClientBodyTimeout int `json:"client-body-timeout,omitempty"`
// DisableAccessLog disables the Access Log globally from NGINX ingress controller
//http://nginx.org/en/docs/http/ngx_http_log_module.html
DisableAccessLog bool `json:"disable-access-log,omitempty"`
@ -177,7 +199,7 @@ type Configuration struct {
// Enable json escaping
// http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format
LogFormatEscapeJson bool `json:"log-format-escape-json,omitempty"`
LogFormatEscapeJSON bool `json:"log-format-escape-json,omitempty"`
// Customize upstream log_format
// http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format
@ -198,7 +220,7 @@ type Configuration struct {
// If UseProxyProtocol is enabled ProxyRealIPCIDR defines the default the IP/network address
// of your external load balancer
ProxyRealIPCIDR string `json:"proxy-real-ip-cidr,omitempty"`
ProxyRealIPCIDR []string `json:"proxy-real-ip-cidr,omitempty"`
// Sets the name of the configmap that contains the headers to pass to the backend
ProxySetHeaders string `json:"proxy-set-headers,omitempty"`
@ -214,6 +236,16 @@ type Configuration struct {
// http://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_bucket_size
ServerNameHashBucketSize int `json:"server-name-hash-bucket-size,omitempty"`
// Size of the bucket for the proxy headers hash tables
// http://nginx.org/en/docs/hash.html
// https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_headers_hash_max_size
ProxyHeadersHashMaxSize int `json:"proxy-headers-hash-max-size,omitempty"`
// Maximum size of the bucket for the proxy headers hash tables
// http://nginx.org/en/docs/hash.html
// https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_headers_hash_bucket_size
ProxyHeadersHashBucketSize int `json:"proxy-headers-hash-bucket-size,omitempty"`
// Enables or disables emitting nginx version in error messages and in the “Server” response header field.
// http://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens
// Default: true
@ -283,6 +315,10 @@ type Configuration struct {
// http://nginx.org/en/docs/ngx_core_module.html#worker_processes
WorkerProcesses string `json:"worker-processes,omitempty"`
// Defines a timeout for a graceful shutdown of worker processes
// http://nginx.org/en/docs/ngx_core_module.html#worker_shutdown_timeout
WorkerShutdownTimeout string `json:"worker-shutdown-timeout,omitempty"`
// Defines the load balancing algorithm to use. The deault is round-robin
LoadBalanceAlgorithm string `json:"load-balance,omitempty"`
@ -293,51 +329,90 @@ type Configuration struct {
// Sets the maximum size of the variables hash table.
// http://nginx.org/en/docs/http/ngx_http_map_module.html#variables_hash_max_size
VariablesHashMaxSize int `json:"variables-hash-max-size,omitempty"`
// Activates the cache for connections to upstream servers.
// The connections parameter sets the maximum number of idle keepalive connections to
// upstream servers that are preserved in the cache of each worker process. When this
// number is exceeded, the least recently used connections are closed.
// http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive
// Default: 32
UpstreamKeepaliveConnections int `json:"upstream-keepalive-connections,omitempty"`
// Sets the maximum size of the variables hash table.
// http://nginx.org/en/docs/http/ngx_http_map_module.html#variables_hash_max_size
LimitConnZoneVariable string `json:"limit-conn-zone-variable,omitempty"`
// Sets the timeout between two successive read or write operations on client or proxied server connections.
// If no data is transmitted within this time, the connection is closed.
// http://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_timeout
ProxyStreamTimeout string `json:"proxy-stream-timeout,omitempty"`
// Sets the ipv4 addresses on which the server will accept requests.
BindAddressIpv4 []string `json:"bind-address-ipv4,omitempty"`
// Sets the ipv6 addresses on which the server will accept requests.
BindAddressIpv6 []string `json:"bind-address-ipv6,omitempty"`
// Sets the header field for identifying the originating IP address of a client
// Default is X-Forwarded-For
ForwardedForHeader string `json:"forwarded-for-header,omitempty"`
}
// NewDefault returns the default nginx configuration
func NewDefault() Configuration {
defIPCIDR := make([]string, 0)
defIPCIDR = append(defIPCIDR, "0.0.0.0/0")
defBindAddress := make([]string, 0)
cfg := Configuration{
AllowBackendServerHeader: false,
AccessLogPath: "/var/log/nginx/access.log",
ErrorLogPath: "/var/log/nginx/error.log",
ClientHeaderBufferSize: "1k",
ClientHeaderTimeout: 60,
ClientBodyBufferSize: "8k",
ClientBodyTimeout: 60,
EnableDynamicTLSRecords: true,
EnableUnderscoresInHeaders: false,
ErrorLogLevel: errorLevel,
ForwardedForHeader: "X-Forwarded-For",
HTTP2MaxFieldSize: "4k",
HTTP2MaxHeaderSize: "16k",
HSTS: true,
HSTSIncludeSubdomains: true,
HSTSMaxAge: hstsMaxAge,
HSTSPreload: false,
IgnoreInvalidHeaders: true,
GzipTypes: gzipTypes,
KeepAlive: 75,
KeepAliveRequests: 100,
LargeClientHeaderBuffers: "4 8k",
LogFormatEscapeJson: false,
LogFormatStream: logFormatStream,
LogFormatUpstream: logFormatUpstream,
MaxWorkerConnections: 16384,
MapHashBucketSize: 64,
ProxyRealIPCIDR: defIPCIDR,
ServerNameHashMaxSize: 1024,
ShowServerTokens: true,
SSLBufferSize: sslBufferSize,
SSLCiphers: sslCiphers,
SSLECDHCurve: "secp384r1",
SSLProtocols: sslProtocols,
SSLSessionCache: true,
SSLSessionCacheSize: sslSessionCacheSize,
SSLSessionTickets: true,
SSLSessionTimeout: sslSessionTimeout,
UseGzip: true,
WorkerProcesses: strconv.Itoa(runtime.NumCPU()),
LoadBalanceAlgorithm: defaultLoadBalancerAlgorithm,
VtsStatusZoneSize: "10m",
VariablesHashBucketSize: 64,
VariablesHashMaxSize: 2048,
UseHTTP2: true,
HSTSIncludeSubdomains: true,
HSTSMaxAge: hstsMaxAge,
HSTSPreload: false,
IgnoreInvalidHeaders: true,
GzipTypes: gzipTypes,
KeepAlive: 75,
KeepAliveRequests: 100,
LargeClientHeaderBuffers: "4 8k",
LogFormatEscapeJSON: false,
LogFormatStream: logFormatStream,
LogFormatUpstream: logFormatUpstream,
MaxWorkerConnections: 16384,
MapHashBucketSize: 64,
ProxyRealIPCIDR: defIPCIDR,
ServerNameHashMaxSize: 1024,
ProxyHeadersHashMaxSize: 512,
ProxyHeadersHashBucketSize: 64,
ShowServerTokens: true,
SSLBufferSize: sslBufferSize,
SSLCiphers: sslCiphers,
SSLECDHCurve: "secp384r1",
SSLProtocols: sslProtocols,
SSLSessionCache: true,
SSLSessionCacheSize: sslSessionCacheSize,
SSLSessionTickets: true,
SSLSessionTimeout: sslSessionTimeout,
UseGzip: true,
WorkerProcesses: strconv.Itoa(runtime.NumCPU()),
WorkerShutdownTimeout: "10s",
LoadBalanceAlgorithm: defaultLoadBalancerAlgorithm,
VtsStatusZoneSize: "10m",
VariablesHashBucketSize: 64,
VariablesHashMaxSize: 2048,
UseHTTP2: true,
ProxyStreamTimeout: "600s",
Backend: defaults.Backend{
ProxyBodySize: bodySize,
ProxyConnectTimeout: 5,
@ -346,11 +421,19 @@ func NewDefault() Configuration {
ProxyBufferSize: "4k",
ProxyCookieDomain: "off",
ProxyCookiePath: "off",
ProxyNextUpstream: "error timeout invalid_header http_502 http_503 http_504",
ProxyRequestBuffering: "on",
SSLRedirect: true,
CustomHTTPErrors: []int{},
WhitelistSourceRange: []string{},
SkipAccessLogURLs: []string{},
LimitRate: 0,
LimitRateAfter: 0,
},
UpstreamKeepaliveConnections: 32,
LimitConnZoneVariable: defaultLimitConnZoneVariable,
BindAddressIpv4: defBindAddress,
BindAddressIpv6: defBindAddress,
}
if glog.V(5) {
@ -365,7 +448,7 @@ func NewDefault() Configuration {
// is enabled.
func (cfg Configuration) BuildLogFormatUpstream() string {
if cfg.LogFormatUpstream == logFormatUpstream {
return fmt.Sprintf(cfg.LogFormatUpstream, "$the_x_forwarded_for")
return fmt.Sprintf(cfg.LogFormatUpstream, "$the_real_ip")
}
return cfg.LogFormatUpstream
@ -373,16 +456,31 @@ func (cfg Configuration) BuildLogFormatUpstream() string {
// TemplateConfig contains the nginx configuration to render the file nginx.conf
type TemplateConfig struct {
ProxySetHeaders map[string]string
MaxOpenFiles int
BacklogSize int
Backends []*ingress.Backend
PassthroughBackends []*ingress.SSLPassthroughBackend
Servers []*ingress.Server
TCPBackends []ingress.L4Service
UDPBackends []ingress.L4Service
HealthzURI string
CustomErrors bool
Cfg Configuration
IsIPV6Enabled bool
ProxySetHeaders map[string]string
AddHeaders map[string]string
MaxOpenFiles int
BacklogSize int
Backends []*ingress.Backend
PassthroughBackends []*ingress.SSLPassthroughBackend
Servers []*ingress.Server
TCPBackends []ingress.L4Service
UDPBackends []ingress.L4Service
HealthzURI string
CustomErrors bool
Cfg Configuration
IsIPV6Enabled bool
IsSSLPassthroughEnabled bool
RedirectServers map[string]string
ListenPorts *ListenPorts
}
// ListenPorts describe the ports required to run the
// NGINX Ingress controller
type ListenPorts struct {
HTTP int
HTTPS int
Status int
Health int
Default int
SSLProxy int
}

4
controllers/nginx/pkg/config/config_test.go
View file

@ -28,8 +28,8 @@ func TestBuildLogFormatUpstream(t *testing.T) {
curLogFormat string
expected string
}{
{true, logFormatUpstream, fmt.Sprintf(logFormatUpstream, "$the_x_forwarded_for")},
{false, logFormatUpstream, fmt.Sprintf(logFormatUpstream, "$the_x_forwarded_for")},
{true, logFormatUpstream, fmt.Sprintf(logFormatUpstream, "$the_real_ip")},
{false, logFormatUpstream, fmt.Sprintf(logFormatUpstream, "$the_real_ip")},
{true, "my-log-format", "my-log-format"},
{false, "john-log-format", "john-log-format"},
}

8
controllers/nginx/pkg/metric/collector/status.go
View file

@ -143,8 +143,8 @@ func (bit BoolToFloat64) UnmarshalJSON(data []byte) error {
return nil
}
func getNginxStatus(ngxHealthPort int, ngxStatusPath string) (*basicStatus, error) {
url := fmt.Sprintf("http://localhost:%v%v", ngxHealthPort, ngxStatusPath)
func getNginxStatus(port int, path string) (*basicStatus, error) {
url := fmt.Sprintf("http://localhost:%v%v", port, path)
glog.V(3).Infof("start scrapping url: %v", url)
data, err := httpBody(url)
@ -174,8 +174,8 @@ func httpBody(url string) ([]byte, error) {
return data, nil
}
func getNginxVtsMetrics(ngxHealthPort int, ngxVtsPath string) (*vts, error) {
url := fmt.Sprintf("http://localhost:%v%v", ngxHealthPort, ngxVtsPath)
func getNginxVtsMetrics(port int, path string) (*vts, error) {
url := fmt.Sprintf("http://localhost:%v%v", port, path)
glog.V(3).Infof("start scrapping url: %v", url)
data, err := httpBody(url)

50
controllers/nginx/pkg/metric/collector/vts.go
View file

@ -28,8 +28,8 @@ const ns = "nginx"
type (
vtsCollector struct {
scrapeChan chan scrapeRequest
ngxHealthPort int
ngxVtsPath string
port int
path string
data *vtsData
watchNamespace string
ingressClass string
@ -39,10 +39,10 @@ type (
bytes *prometheus.Desc
cache *prometheus.Desc
connections *prometheus.Desc
response *prometheus.Desc
request *prometheus.Desc
responses *prometheus.Desc
requests *prometheus.Desc
filterZoneBytes *prometheus.Desc
filterZoneResponse *prometheus.Desc
filterZoneResponses *prometheus.Desc
filterZoneCache *prometheus.Desc
upstreamBackup *prometheus.Desc
upstreamBytes *prometheus.Desc
@ -50,19 +50,19 @@ type (
upstreamFailTimeout *prometheus.Desc
upstreamMaxFails *prometheus.Desc
upstreamResponses *prometheus.Desc
upstreamRequest *prometheus.Desc
upstreamRequests *prometheus.Desc
upstreamResponseMsec *prometheus.Desc
upstreamWeight *prometheus.Desc
}
)
// NewNGINXVTSCollector returns a new prometheus collector for the VTS module
func NewNGINXVTSCollector(watchNamespace, ingressClass string, ngxHealthPort int, ngxVtsPath string) Stopable {
func NewNGINXVTSCollector(watchNamespace, ingressClass string, port int, path string) Stopable {
p := vtsCollector{
scrapeChan: make(chan scrapeRequest),
ngxHealthPort: ngxHealthPort,
ngxVtsPath: ngxVtsPath,
port: port,
path: path,
watchNamespace: watchNamespace,
ingressClass: ingressClass,
}
@ -83,12 +83,12 @@ func NewNGINXVTSCollector(watchNamespace, ingressClass string, ngxHealthPort int
"Nginx connections count",
[]string{"ingress_class", "namespace", "type"}, nil),
response: prometheus.NewDesc(
responses: prometheus.NewDesc(
prometheus.BuildFQName(ns, "", "responses_total"),
"The number of responses with status codes 1xx, 2xx, 3xx, 4xx, and 5xx.",
[]string{"ingress_class", "namespace", "server_zone", "status_code"}, nil),
request: prometheus.NewDesc(
requests: prometheus.NewDesc(
prometheus.BuildFQName(ns, "", "requests_total"),
"The total number of requested client connections.",
[]string{"ingress_class", "namespace", "server_zone"}, nil),
@ -98,7 +98,7 @@ func NewNGINXVTSCollector(watchNamespace, ingressClass string, ngxHealthPort int
"Nginx bytes count",
[]string{"ingress_class", "namespace", "server_zone", "country", "direction"}, nil),
filterZoneResponse: prometheus.NewDesc(
filterZoneResponses: prometheus.NewDesc(
prometheus.BuildFQName(ns, "", "filterzone_responses_total"),
"The number of responses with status codes 1xx, 2xx, 3xx, 4xx, and 5xx.",
[]string{"ingress_class", "namespace", "server_zone", "country", "status_code"}, nil),
@ -138,7 +138,7 @@ func NewNGINXVTSCollector(watchNamespace, ingressClass string, ngxHealthPort int
"The number of upstream responses with status codes 1xx, 2xx, 3xx, 4xx, and 5xx.",
[]string{"ingress_class", "namespace", "upstream", "server", "status_code"}, nil),
upstreamRequest: prometheus.NewDesc(
upstreamRequests: prometheus.NewDesc(
prometheus.BuildFQName(ns, "", "upstream_requests_total"),
"The total number of client connections forwarded to this server.",
[]string{"ingress_class", "namespace", "upstream", "server"}, nil),
@ -164,20 +164,20 @@ func (p vtsCollector) Describe(ch chan<- *prometheus.Desc) {
ch <- p.data.bytes
ch <- p.data.cache
ch <- p.data.connections
ch <- p.data.request
ch <- p.data.response
ch <- p.data.requests
ch <- p.data.responses
ch <- p.data.upstreamBackup
ch <- p.data.upstreamBytes
ch <- p.data.upstreamDown
ch <- p.data.upstreamFailTimeout
ch <- p.data.upstreamMaxFails
ch <- p.data.upstreamRequest
ch <- p.data.upstreamRequests
ch <- p.data.upstreamResponseMsec
ch <- p.data.upstreamResponses
ch <- p.data.upstreamWeight
ch <- p.data.filterZoneBytes
ch <- p.data.filterZoneCache
ch <- p.data.filterZoneResponse
ch <- p.data.filterZoneResponses
}
// Collect implements prometheus.Collector.
@ -201,7 +201,7 @@ func (p vtsCollector) Stop() {
// scrapeVts scrape nginx vts metrics
func (p vtsCollector) scrapeVts(ch chan<- prometheus.Metric) {
nginxMetrics, err := getNginxVtsMetrics(p.ngxHealthPort, p.ngxVtsPath)
nginxMetrics, err := getNginxVtsMetrics(p.port, p.path)
if err != nil {
glog.Warningf("unexpected error obtaining nginx status info: %v", err)
return
@ -213,7 +213,7 @@ func (p vtsCollector) scrapeVts(ch chan<- prometheus.Metric) {
for pos, value := range zones {
reflectMetrics(&zones[pos].Responses, p.data.upstreamResponses, ch, p.ingressClass, p.watchNamespace, name, value.Server)
ch <- prometheus.MustNewConstMetric(p.data.upstreamRequest,
ch <- prometheus.MustNewConstMetric(p.data.upstreamRequests,
prometheus.CounterValue, zones[pos].RequestCounter, p.ingressClass, p.watchNamespace, name, value.Server)
ch <- prometheus.MustNewConstMetric(p.data.upstreamDown,
prometheus.CounterValue, float64(zones[pos].Down), p.ingressClass, p.watchNamespace, name, value.Server)
@ -235,10 +235,10 @@ func (p vtsCollector) scrapeVts(ch chan<- prometheus.Metric) {
}
for name, zone := range nginxMetrics.ServerZones {
reflectMetrics(&zone.Responses, p.data.response, ch, p.ingressClass, p.watchNamespace, name)
reflectMetrics(&zone.Responses, p.data.responses, ch, p.ingressClass, p.watchNamespace, name)
reflectMetrics(&zone.Cache, p.data.cache, ch, p.ingressClass, p.watchNamespace, name)
ch <- prometheus.MustNewConstMetric(p.data.request,
ch <- prometheus.MustNewConstMetric(p.data.requests,
prometheus.CounterValue, zone.RequestCounter, p.ingressClass, p.watchNamespace, name)
ch <- prometheus.MustNewConstMetric(p.data.bytes,
prometheus.CounterValue, zone.InBytes, p.ingressClass, p.watchNamespace, name, "in")
@ -248,13 +248,13 @@ func (p vtsCollector) scrapeVts(ch chan<- prometheus.Metric) {
for serverZone, countries := range nginxMetrics.FilterZones {
for country, zone := range countries {
reflectMetrics(&zone.Responses, p.data.filterZoneResponse, ch, p.ingressClass, p.watchNamespace, serverZone, country)
reflectMetrics(&zone.Responses, p.data.filterZoneResponses, ch, p.ingressClass, p.watchNamespace, serverZone, country)
reflectMetrics(&zone.Cache, p.data.filterZoneCache, ch, p.ingressClass, p.watchNamespace, serverZone, country)
ch <- prometheus.MustNewConstMetric(p.data.filterZoneBytes,
prometheus.CounterValue, float64(zone.InBytes), p.ingressClass, p.watchNamespace, serverZone, country, "in")
prometheus.CounterValue, zone.InBytes, p.ingressClass, p.watchNamespace, serverZone, country, "in")
ch <- prometheus.MustNewConstMetric(p.data.filterZoneBytes,
prometheus.CounterValue, float64(zone.OutBytes), p.ingressClass, p.watchNamespace, serverZone, country, "out")
prometheus.CounterValue, zone.OutBytes, p.ingressClass, p.watchNamespace, serverZone, country, "out")
}
}
}
@ -266,7 +266,7 @@ func reflectMetrics(value interface{}, desc *prometheus.Desc, ch chan<- promethe
tag := val.Type().Field(i).Tag
l := append(labels, tag.Get("json"))
ch <- prometheus.MustNewConstMetric(desc,
prometheus.CounterValue, float64(val.Field(i).Interface().(float64)),
prometheus.CounterValue, val.Field(i).Interface().(float64),
l...)
}
}

32
controllers/nginx/pkg/template/configmap.go
View file

@ -17,6 +17,8 @@ limitations under the License.
package template
import (
"fmt"
"net"
"strconv"
"strings"
@ -24,12 +26,15 @@ import (
"github.com/mitchellh/mapstructure"
"k8s.io/ingress/controllers/nginx/pkg/config"
ing_net "k8s.io/ingress/core/pkg/net"
)
const (
customHTTPErrors = "custom-http-errors"
skipAccessLogUrls = "skip-access-log-urls"
whitelistSourceRange = "whitelist-source-range"
proxyRealIPCIDR = "proxy-real-ip-cidr"
bindAddress = "bind-address"
)
// ReadConfig obtains the configuration defined by the user merged with the defaults.
@ -45,6 +50,9 @@ func ReadConfig(src map[string]string) config.Configuration {
errors := make([]int, 0)
skipUrls := make([]string, 0)
whitelist := make([]string, 0)
proxylist := make([]string, 0)
bindAddressIpv4List := make([]string, 0)
bindAddressIpv6List := make([]string, 0)
if val, ok := conf[customHTTPErrors]; ok {
delete(conf, customHTTPErrors)
@ -65,11 +73,35 @@ func ReadConfig(src map[string]string) config.Configuration {
delete(conf, whitelistSourceRange)
whitelist = append(whitelist, strings.Split(val, ",")...)
}
if val, ok := conf[proxyRealIPCIDR]; ok {
delete(conf, proxyRealIPCIDR)
proxylist = append(proxylist, strings.Split(val, ",")...)
} else {
proxylist = append(proxylist, "0.0.0.0/0")
}
if val, ok := conf[bindAddress]; ok {
delete(conf, bindAddress)
for _, i := range strings.Split(val, ",") {
ns := net.ParseIP(i)
if ns != nil {
if ing_net.IsIPV6(ns) {
bindAddressIpv6List = append(bindAddressIpv6List, fmt.Sprintf("[%v]", ns))
} else {
bindAddressIpv4List = append(bindAddressIpv4List, fmt.Sprintf("%v", ns))
}
} else {
glog.Warningf("%v is not a valid textual representation of an IP address", i)
}
}
}
to := config.NewDefault()
to.CustomHTTPErrors = filterErrors(errors)
to.SkipAccessLogURLs = skipUrls
to.WhitelistSourceRange = whitelist
to.ProxyRealIPCIDR = proxylist
to.BindAddressIpv4 = bindAddressIpv4List
to.BindAddressIpv6 = bindAddressIpv6List
config := &mapstructure.DecoderConfig{
Metadata: nil,

11
controllers/nginx/pkg/template/configmap_test.go
View file

@ -39,19 +39,30 @@ func TestMergeConfigMapToStruct(t *testing.T) {
"skip-access-log-urls": "/log,/demo,/test",
"use-proxy-protocol": "true",
"disable-access-log": "true",
"access-log-path": "/var/log/test/access.log",
"error-log-path": "/var/log/test/error.log",
"use-gzip": "true",
"enable-dynamic-tls-records": "false",
"gzip-types": "text/html",
"proxy-real-ip-cidr": "1.1.1.1/8,2.2.2.2/24",
"bind-address": "1.1.1.1,2.2.2.2,3.3.3,2001:db8:a0b:12f0::1,3731:54:65fe:2::a7,33:33:33::33::33",
"worker-shutdown-timeout": "99s",
}
def := config.NewDefault()
def.CustomHTTPErrors = []int{300, 400}
def.DisableAccessLog = true
def.AccessLogPath = "/var/log/test/access.log"
def.ErrorLogPath = "/var/log/test/error.log"
def.SkipAccessLogURLs = []string{"/log", "/demo", "/test"}
def.ProxyReadTimeout = 1
def.ProxySendTimeout = 2
def.EnableDynamicTLSRecords = false
def.UseProxyProtocol = true
def.GzipTypes = "text/html"
def.ProxyRealIPCIDR = []string{"1.1.1.1/8", "2.2.2.2/24"}
def.BindAddressIpv4 = []string{"1.1.1.1", "2.2.2.2"}
def.BindAddressIpv6 = []string{"[2001:db8:a0b:12f0::1]", "[3731:54:65fe:2::a7]"}
def.WorkerShutdownTimeout = "99s"
to := ReadConfig(conf)
if diff := pretty.Compare(to, def); diff != "" {

313
controllers/nginx/pkg/template/template.go
View file

@ -22,19 +22,21 @@ import (
"encoding/json"
"fmt"
"net"
"net/url"
"os"
"os/exec"
"regexp"
"strconv"
"strings"
text_template "text/template"
"k8s.io/apimachinery/pkg/util/sets"
"github.com/golang/glog"
"github.com/pborman/uuid"
extensions "k8s.io/api/extensions/v1beta1"
"k8s.io/apimachinery/pkg/util/sets"
"k8s.io/ingress/controllers/nginx/pkg/config"
"k8s.io/ingress/core/pkg/ingress"
"k8s.io/ingress/core/pkg/ingress/annotations/ratelimit"
ing_net "k8s.io/ingress/core/pkg/net"
"k8s.io/ingress/core/pkg/watch"
)
@ -133,9 +135,11 @@ var (
"buildAuthLocation": buildAuthLocation,
"buildAuthResponseHeaders": buildAuthResponseHeaders,
"buildProxyPass": buildProxyPass,
"filterRateLimits": filterRateLimits,
"buildRateLimitZones": buildRateLimitZones,
"buildRateLimit": buildRateLimit,
"buildResolvers": buildResolvers,
"buildUpstreamName": buildUpstreamName,
"isLocationAllowed": isLocationAllowed,
"buildLogFormatUpstream": buildLogFormatUpstream,
"buildDenyVariable": buildDenyVariable,
@ -146,10 +150,18 @@ var (
"toUpper": strings.ToUpper,
"toLower": strings.ToLower,
"formatIP": formatIP,
"buildNextUpstream": buildNextUpstream,
"getIngressInformation": getIngressInformation,
"serverConfig": func(all config.TemplateConfig, server *ingress.Server) interface{} {
return struct{ First, Second interface{} }{all, server}
},
"buildAuthSignURL": buildAuthSignURL,
"isValidClientBodyBufferSize": isValidClientBodyBufferSize,
"buildForwardedFor": buildForwardedFor,
}
)
// fomatIP will wrap IPv6 addresses in [] and return IPv4 addresses
// formatIP will wrap IPv6 addresses in [] and return IPv4 addresses
// without modification. If the input cannot be parsed as an IP address
// it is returned without modification.
func formatIP(input string) string {
@ -165,7 +177,7 @@ func formatIP(input string) string {
// buildResolvers returns the resolvers reading the /etc/resolv.conf file
func buildResolvers(a interface{}) string {
// NGINX need IPV6 addresses to be surrounded by brakets
// NGINX need IPV6 addresses to be surrounded by brackets
nss := a.([]net.IP)
if len(nss) == 0 {
return ""
@ -193,7 +205,7 @@ func buildLocation(input interface{}) string {
}
path := location.Path
if len(location.Redirect.Target) > 0 && location.Redirect.Target != path {
if len(location.Rewrite.Target) > 0 && location.Rewrite.Target != path {
if path == slash {
return fmt.Sprintf("~* %s", path)
}
@ -209,6 +221,7 @@ func buildLocation(input interface{}) string {
return path
}
// TODO: Needs Unit Tests
func buildAuthLocation(input interface{}) string {
location, ok := input.(*ingress.Location)
if !ok {
@ -248,7 +261,7 @@ func buildAuthResponseHeaders(input interface{}) []string {
func buildLogFormatUpstream(input interface{}) string {
cfg, ok := input.(config.Configuration)
if !ok {
glog.Errorf("error an ingress.buildLogFormatUpstream type but %T was returned", input)
glog.Errorf("error an ingress.buildLogFormatUpstream type but %T was returned", input)
}
return cfg.BuildLogFormatUpstream()
@ -258,7 +271,7 @@ func buildLogFormatUpstream(input interface{}) string {
// (specified through the ingress.kubernetes.io/rewrite-to annotation)
// If the annotation ingress.kubernetes.io/add-base-url:"true" is specified it will
// add a base tag in the head of the response from the service
func buildProxyPass(b interface{}, loc interface{}) string {
func buildProxyPass(host string, b interface{}, loc interface{}) string {
backends := b.([]*ingress.Backend)
location, ok := loc.(*ingress.Location)
if !ok {
@ -268,59 +281,93 @@ func buildProxyPass(b interface{}, loc interface{}) string {
path := location.Path
proto := "http"
upstreamName := location.Backend
for _, backend := range backends {
if backend.Name == location.Backend {
if backend.Secure || backend.SSLPassthrough {
proto = "https"
}
if isSticky(host, location, backend.SessionAffinity.CookieSessionAffinity.Locations) {
upstreamName = fmt.Sprintf("sticky-%v", upstreamName)
}
break
}
}
// defProxyPass returns the default proxy_pass, just the name of the upstream
defProxyPass := fmt.Sprintf("proxy_pass %s://%s;", proto, location.Backend)
defProxyPass := fmt.Sprintf("proxy_pass %s://%s;", proto, upstreamName)
// if the path in the ingress rule is equals to the target: no special rewrite
if path == location.Redirect.Target {
if path == location.Rewrite.Target {
return defProxyPass
}
if path != slash && !strings.HasSuffix(path, slash) {
if !strings.HasSuffix(path, slash) {
path = fmt.Sprintf("%s/", path)
}
if len(location.Redirect.Target) > 0 {
if len(location.Rewrite.Target) > 0 {
abu := ""
if location.Redirect.AddBaseURL {
if location.Rewrite.AddBaseURL {
// path has a slash suffix, so that it can be connected with baseuri directly
bPath := fmt.Sprintf("%s%s", path, "$baseuri")
abu = fmt.Sprintf(`subs_filter '<head(.*)>' '<head$1><base href="$scheme://$http_host%v">' r;
subs_filter '<HEAD(.*)>' '<HEAD$1><base href="$scheme://$http_host%v">' r;
`, bPath, bPath)
if len(location.Rewrite.BaseURLScheme) > 0 {
abu = fmt.Sprintf(`subs_filter '<head(.*)>' '<head$1><base href="%v://$http_host%v">' r;
subs_filter '<HEAD(.*)>' '<HEAD$1><base href="%v://$http_host%v">' r;
`, location.Rewrite.BaseURLScheme, bPath, location.Rewrite.BaseURLScheme, bPath)
} else {
abu = fmt.Sprintf(`subs_filter '<head(.*)>' '<head$1><base href="$scheme://$http_host%v">' r;
subs_filter '<HEAD(.*)>' '<HEAD$1><base href="$scheme://$http_host%v">' r;
`, bPath, bPath)
}
}
if location.Redirect.Target == slash {
if location.Rewrite.Target == slash {
// special case redirect to /
// ie /something to /
return fmt.Sprintf(`
rewrite %s(.*) /$1 break;
rewrite %s / break;
proxy_pass %s://%s;
%v`, path, location.Path, proto, location.Backend, abu)
rewrite %s(.*) /$1 break;
rewrite %s / break;
proxy_pass %s://%s;
%v`, path, location.Path, proto, upstreamName, abu)
}
return fmt.Sprintf(`
rewrite %s(.*) %s/$1 break;
proxy_pass %s://%s;
%v`, path, location.Redirect.Target, proto, location.Backend, abu)
rewrite %s(.*) %s/$1 break;
proxy_pass %s://%s;
%v`, path, location.Rewrite.Target, proto, upstreamName, abu)
}
// default proxy_pass
return defProxyPass
}
// TODO: Needs Unit Tests
func filterRateLimits(input interface{}) []ratelimit.RateLimit {
ratelimits := []ratelimit.RateLimit{}
found := sets.String{}
servers, ok := input.([]*ingress.Server)
if !ok {
return ratelimits
}
for _, server := range servers {
for _, loc := range server.Locations {
if loc.RateLimit.ID != "" && !found.Has(loc.RateLimit.ID) {
found.Insert(loc.RateLimit.ID)
ratelimits = append(ratelimits, loc.RateLimit)
}
}
}
return ratelimits
}
// TODO: Needs Unit Tests
// buildRateLimitZones produces an array of limit_conn_zone in order to allow
// rate limiting of request. Each Ingress rule could have up to two zones, one
// for connection limit by IP address and other for limiting request per second
// rate limiting of request. Each Ingress rule could have up to three zones, one
// for connection limit by IP address, one for limiting requests per minute, and
// one for limiting requests per second.
func buildRateLimitZones(input interface{}) []string {
zones := sets.String{}
@ -331,9 +378,9 @@ func buildRateLimitZones(input interface{}) []string {
for _, server := range servers {
for _, loc := range server.Locations {
if loc.RateLimit.Connections.Limit > 0 {
zone := fmt.Sprintf("limit_conn_zone $binary_remote_addr zone=%v:%vm;",
zone := fmt.Sprintf("limit_conn_zone $limit_%s zone=%v:%vm;",
loc.RateLimit.ID,
loc.RateLimit.Connections.Name,
loc.RateLimit.Connections.SharedSize)
if !zones.Has(zone) {
@ -341,8 +388,20 @@ func buildRateLimitZones(input interface{}) []string {
}
}
if loc.RateLimit.RPM.Limit > 0 {
zone := fmt.Sprintf("limit_req_zone $limit_%s zone=%v:%vm rate=%vr/m;",
loc.RateLimit.ID,
loc.RateLimit.RPM.Name,
loc.RateLimit.RPM.SharedSize,
loc.RateLimit.RPM.Limit)
if !zones.Has(zone) {
zones.Insert(zone)
}
}
if loc.RateLimit.RPS.Limit > 0 {
zone := fmt.Sprintf("limit_req_zone $binary_remote_addr zone=%v:%vm rate=%vr/s;",
zone := fmt.Sprintf("limit_req_zone $limit_%s zone=%v:%vm rate=%vr/s;",
loc.RateLimit.ID,
loc.RateLimit.RPS.Name,
loc.RateLimit.RPS.SharedSize,
loc.RateLimit.RPS.Limit)
@ -357,7 +416,7 @@ func buildRateLimitZones(input interface{}) []string {
}
// buildRateLimit produces an array of limit_req to be used inside the Path of
// Ingress rules. The order: connections by IP first and RPS next.
// Ingress rules. The order: connections by IP first, then RPS, and RPM last.
func buildRateLimit(input interface{}) []string {
limits := []string{}
@ -378,6 +437,24 @@ func buildRateLimit(input interface{}) []string {
limits = append(limits, limit)
}
if loc.RateLimit.RPM.Limit > 0 {
limit := fmt.Sprintf("limit_req zone=%v burst=%v nodelay;",
loc.RateLimit.RPM.Name, loc.RateLimit.RPM.Burst)
limits = append(limits, limit)
}
if loc.RateLimit.LimitRateAfter > 0 {
limit := fmt.Sprintf("limit_rate_after %vk;",
loc.RateLimit.LimitRateAfter)
limits = append(limits, limit)
}
if loc.RateLimit.LimitRate > 0 {
limit := fmt.Sprintf("limit_rate %vk;",
loc.RateLimit.LimitRate)
limits = append(limits, limit)
}
return limits
}
@ -392,7 +469,6 @@ func isLocationAllowed(input interface{}) bool {
}
var (
nonAlpha = regexp.MustCompile("[^a-zA-Z0-9]+")
denyPathSlugMap = map[string]string{}
)
@ -405,8 +481,179 @@ func buildDenyVariable(a interface{}) string {
l := a.(string)
if _, ok := denyPathSlugMap[l]; !ok {
denyPathSlugMap[l] = uuid.New()
denyPathSlugMap[l] = buildRandomUUID()
}
return fmt.Sprintf("$deny_%v", denyPathSlugMap[l])
}
// TODO: Needs Unit Tests
func buildUpstreamName(host string, b interface{}, loc interface{}) string {
backends := b.([]*ingress.Backend)
location, ok := loc.(*ingress.Location)
if !ok {
return ""
}
upstreamName := location.Backend
for _, backend := range backends {
if backend.Name == location.Backend {
if backend.SessionAffinity.AffinityType == "cookie" &&
isSticky(host, location, backend.SessionAffinity.CookieSessionAffinity.Locations) {
upstreamName = fmt.Sprintf("sticky-%v", upstreamName)
}
break
}
}
return upstreamName
}
// TODO: Needs Unit Tests
func isSticky(host string, loc *ingress.Location, stickyLocations map[string][]string) bool {
if _, ok := stickyLocations[host]; ok {
for _, sl := range stickyLocations[host] {
if sl == loc.Path {
return true
}
}
}
return false
}
func buildNextUpstream(input interface{}) string {
nextUpstream, ok := input.(string)
if !ok {
glog.Errorf("expected an string type but %T was returned", input)
}
parts := strings.Split(nextUpstream, " ")
nextUpstreamCodes := make([]string, 0, len(parts))
for _, v := range parts {
if v != "" && v != "non_idempotent" {
nextUpstreamCodes = append(nextUpstreamCodes, v)
}
}
return strings.Join(nextUpstreamCodes, " ")
}
func buildAuthSignURL(input interface{}) string {
s, ok := input.(string)
if !ok {
glog.Errorf("expected an string type but %T was returned", input)
}
u, _ := url.Parse(s)
q := u.Query()
if len(q) == 0 {
return fmt.Sprintf("%v?rd=$request_uri", s)
}
return fmt.Sprintf("%v&rd=$request_uri", s)
}
// buildRandomUUID return a random string to be used in the template
func buildRandomUUID() string {
s := uuid.New()
return strings.Replace(s, "-", "", -1)
}
func isValidClientBodyBufferSize(input interface{}) bool {
s, ok := input.(string)
if !ok {
glog.Errorf("expected an string type but %T was returned", input)
return false
}
if s == "" {
return false
}
_, err := strconv.Atoi(s)
if err != nil {
sLowercase := strings.ToLower(s)
kCheck := strings.TrimSuffix(sLowercase, "k")
_, err := strconv.Atoi(kCheck)
if err == nil {
return true
}
mCheck := strings.TrimSuffix(sLowercase, "m")
_, err = strconv.Atoi(mCheck)
if err == nil {
return true
}
glog.Errorf("client-body-buffer-size '%v' was provided in an incorrect format, hence it will not be set.", s)
return false
}
return true
}
type ingressInformation struct {
Namespace string
Rule string
Service string
Annotations map[string]string
}
func getIngressInformation(i, p interface{}) *ingressInformation {
ing, ok := i.(*extensions.Ingress)
if !ok {
glog.V(3).Infof("expected an Ingress type but %T was returned", i)
return &ingressInformation{}
}
path, ok := p.(string)
if !ok {
glog.V(3).Infof("expected a string type but %T was returned", p)
return &ingressInformation{}
}
if ing == nil {
return &ingressInformation{}
}
info := &ingressInformation{
Namespace: ing.GetNamespace(),
Rule: ing.GetName(),
Annotations: ing.Annotations,
}
if ing.Spec.Backend != nil {
info.Service = ing.Spec.Backend.ServiceName
}
for _, rule := range ing.Spec.Rules {
if rule.HTTP == nil {
continue
}
for _, rPath := range rule.HTTP.Paths {
if path == rPath.Path {
info.Service = rPath.Backend.ServiceName
return info
}
}
}
return info
}
func buildForwardedFor(input interface{}) string {
s, ok := input.(string)
if !ok {
glog.Errorf("expected an string type but %T was returned", input)
}
ffh := strings.Replace(s, "-", "_", -1)
ffh = strings.ToLower(ffh)
return fmt.Sprintf("$http_%v", ffh)
}

245
controllers/nginx/pkg/template/template_test.go
View file

@ -18,73 +18,81 @@ package template
import (
"encoding/json"
"io/ioutil"
"os"
"path"
"reflect"
"strings"
"testing"
"io/ioutil"
"k8s.io/ingress/controllers/nginx/pkg/config"
"k8s.io/ingress/core/pkg/ingress"
"k8s.io/ingress/core/pkg/ingress/annotations/authreq"
"k8s.io/ingress/core/pkg/ingress/annotations/rewrite"
"net"
)
var (
// TODO: add tests for secure endpoints
tmplFuncTestcases = map[string]struct {
Path string
Target string
Location string
ProxyPass string
AddBaseURL bool
Path string
Target string
Location string
ProxyPass string
AddBaseURL bool
BaseURLScheme string
}{
"invalid redirect / to /": {"/", "/", "/", "proxy_pass http://upstream-name;", false},
"invalid redirect / to /": {"/", "/", "/", "proxy_pass http://upstream-name;", false, ""},
"redirect / to /jenkins": {"/", "/jenkins", "~* /",
`
rewrite /(.*) /jenkins/$1 break;
proxy_pass http://upstream-name;
`, false},
rewrite /(.*) /jenkins/$1 break;
proxy_pass http://upstream-name;
`, false, ""},
"redirect /something to /": {"/something", "/", `~* ^/something\/?(?<baseuri>.*)`, `
rewrite /something/(.*) /$1 break;
rewrite /something / break;
proxy_pass http://upstream-name;
`, false},
rewrite /something/(.*) /$1 break;
rewrite /something / break;
proxy_pass http://upstream-name;
`, false, ""},
"redirect /end-with-slash/ to /not-root": {"/end-with-slash/", "/not-root", "~* ^/end-with-slash/(?<baseuri>.*)", `
rewrite /end-with-slash/(.*) /not-root/$1 break;
proxy_pass http://upstream-name;
`, false},
rewrite /end-with-slash/(.*) /not-root/$1 break;
proxy_pass http://upstream-name;
`, false, ""},
"redirect /something-complex to /not-root": {"/something-complex", "/not-root", `~* ^/something-complex\/?(?<baseuri>.*)`, `
rewrite /something-complex/(.*) /not-root/$1 break;
proxy_pass http://upstream-name;
`, false},
rewrite /something-complex/(.*) /not-root/$1 break;
proxy_pass http://upstream-name;
`, false, ""},
"redirect / to /jenkins and rewrite": {"/", "/jenkins", "~* /", `
rewrite /(.*) /jenkins/$1 break;
proxy_pass http://upstream-name;
subs_filter '<head(.*)>' '<head$1><base href="$scheme://$http_host/$baseuri">' r;
subs_filter '<HEAD(.*)>' '<HEAD$1><base href="$scheme://$http_host/$baseuri">' r;
`, true},
rewrite /(.*) /jenkins/$1 break;
proxy_pass http://upstream-name;
subs_filter '<head(.*)>' '<head$1><base href="$scheme://$http_host/$baseuri">' r;
subs_filter '<HEAD(.*)>' '<HEAD$1><base href="$scheme://$http_host/$baseuri">' r;
`, true, ""},
"redirect /something to / and rewrite": {"/something", "/", `~* ^/something\/?(?<baseuri>.*)`, `
rewrite /something/(.*) /$1 break;
rewrite /something / break;
proxy_pass http://upstream-name;
subs_filter '<head(.*)>' '<head$1><base href="$scheme://$http_host/something/$baseuri">' r;
subs_filter '<HEAD(.*)>' '<HEAD$1><base href="$scheme://$http_host/something/$baseuri">' r;
`, true},
rewrite /something/(.*) /$1 break;
rewrite /something / break;
proxy_pass http://upstream-name;
subs_filter '<head(.*)>' '<head$1><base href="$scheme://$http_host/something/$baseuri">' r;
subs_filter '<HEAD(.*)>' '<HEAD$1><base href="$scheme://$http_host/something/$baseuri">' r;
`, true, ""},
"redirect /end-with-slash/ to /not-root and rewrite": {"/end-with-slash/", "/not-root", `~* ^/end-with-slash/(?<baseuri>.*)`, `
rewrite /end-with-slash/(.*) /not-root/$1 break;
proxy_pass http://upstream-name;
subs_filter '<head(.*)>' '<head$1><base href="$scheme://$http_host/end-with-slash/$baseuri">' r;
subs_filter '<HEAD(.*)>' '<HEAD$1><base href="$scheme://$http_host/end-with-slash/$baseuri">' r;
`, true},
rewrite /end-with-slash/(.*) /not-root/$1 break;
proxy_pass http://upstream-name;
subs_filter '<head(.*)>' '<head$1><base href="$scheme://$http_host/end-with-slash/$baseuri">' r;
subs_filter '<HEAD(.*)>' '<HEAD$1><base href="$scheme://$http_host/end-with-slash/$baseuri">' r;
`, true, ""},
"redirect /something-complex to /not-root and rewrite": {"/something-complex", "/not-root", `~* ^/something-complex\/?(?<baseuri>.*)`, `
rewrite /something-complex/(.*) /not-root/$1 break;
proxy_pass http://upstream-name;
subs_filter '<head(.*)>' '<head$1><base href="$scheme://$http_host/something-complex/$baseuri">' r;
subs_filter '<HEAD(.*)>' '<HEAD$1><base href="$scheme://$http_host/something-complex/$baseuri">' r;
`, true},
rewrite /something-complex/(.*) /not-root/$1 break;
proxy_pass http://upstream-name;
subs_filter '<head(.*)>' '<head$1><base href="$scheme://$http_host/something-complex/$baseuri">' r;
subs_filter '<HEAD(.*)>' '<HEAD$1><base href="$scheme://$http_host/something-complex/$baseuri">' r;
`, true, ""},
"redirect /something to / and rewrite with specific scheme": {"/something", "/", `~* ^/something\/?(?<baseuri>.*)`, `
rewrite /something/(.*) /$1 break;
rewrite /something / break;
proxy_pass http://upstream-name;
subs_filter '<head(.*)>' '<head$1><base href="http://$http_host/something/$baseuri">' r;
subs_filter '<HEAD(.*)>' '<HEAD$1><base href="http://$http_host/something/$baseuri">' r;
`, true, "http"},
}
)
@ -110,8 +118,8 @@ func TestFormatIP(t *testing.T) {
func TestBuildLocation(t *testing.T) {
for k, tc := range tmplFuncTestcases {
loc := &ingress.Location{
Path: tc.Path,
Redirect: rewrite.Redirect{Target: tc.Target, AddBaseURL: tc.AddBaseURL},
Path: tc.Path,
Rewrite: rewrite.Redirect{Target: tc.Target, AddBaseURL: tc.AddBaseURL},
}
newLoc := buildLocation(loc)
@ -124,12 +132,12 @@ func TestBuildLocation(t *testing.T) {
func TestBuildProxyPass(t *testing.T) {
for k, tc := range tmplFuncTestcases {
loc := &ingress.Location{
Path: tc.Path,
Redirect: rewrite.Redirect{Target: tc.Target, AddBaseURL: tc.AddBaseURL},
Backend: "upstream-name",
Path: tc.Path,
Rewrite: rewrite.Redirect{Target: tc.Target, AddBaseURL: tc.AddBaseURL, BaseURLScheme: tc.BaseURLScheme},
Backend: "upstream-name",
}
pp := buildProxyPass([]*ingress.Backend{}, loc)
pp := buildProxyPass("", []*ingress.Backend{}, loc)
if !strings.EqualFold(tc.ProxyPass, pp) {
t.Errorf("%s: expected \n'%v'\nbut returned \n'%v'", k, tc.ProxyPass, pp)
}
@ -168,7 +176,9 @@ func TestTemplateWithData(t *testing.T) {
if err := json.Unmarshal(data, &dat); err != nil {
t.Errorf("unexpected error unmarshalling json: %v", err)
}
if dat.ListenPorts == nil {
dat.ListenPorts = &config.ListenPorts{}
}
tf, err := os.Open(path.Join(pwd, "../../rootfs/etc/nginx/template/nginx.tmpl"))
if err != nil {
t.Errorf("unexpected error reading json file: %v", err)
@ -225,3 +235,140 @@ func TestBuildDenyVariable(t *testing.T) {
t.Errorf("Expected '%v' but returned '%v'", a, b)
}
}
func TestBuildClientBodyBufferSize(t *testing.T) {
a := isValidClientBodyBufferSize("1000")
if a != true {
t.Errorf("Expected '%v' but returned '%v'", true, a)
}
b := isValidClientBodyBufferSize("1000k")
if b != true {
t.Errorf("Expected '%v' but returned '%v'", true, b)
}
c := isValidClientBodyBufferSize("1000m")
if c != true {
t.Errorf("Expected '%v' but returned '%v'", true, c)
}
d := isValidClientBodyBufferSize("1000km")
if d != false {
t.Errorf("Expected '%v' but returned '%v'", false, d)
}
e := isValidClientBodyBufferSize("1000mk")
if e != false {
t.Errorf("Expected '%v' but returned '%v'", false, e)
}
f := isValidClientBodyBufferSize("1000kk")
if f != false {
t.Errorf("Expected '%v' but returned '%v'", false, f)
}
g := isValidClientBodyBufferSize("1000mm")
if g != false {
t.Errorf("Expected '%v' but returned '%v'", false, g)
}
h := isValidClientBodyBufferSize(nil)
if h != false {
t.Errorf("Expected '%v' but returned '%v'", false, h)
}
i := isValidClientBodyBufferSize("")
if i != false {
t.Errorf("Expected '%v' but returned '%v'", false, i)
}
}
func TestIsLocationAllowed(t *testing.T) {
loc := ingress.Location{
Denied: nil,
}
isAllowed := isLocationAllowed(&loc)
if !isAllowed {
t.Errorf("Expected '%v' but returned '%v'", true, isAllowed)
}
}
func TestBuildForwardedFor(t *testing.T) {
inputStr := "X-Forwarded-For"
outputStr := buildForwardedFor(inputStr)
validStr := "$http_x_forwarded_for"
if outputStr != validStr {
t.Errorf("Expected '%v' but returned '%v'", validStr, outputStr)
}
}
func TestBuildResolvers(t *testing.T) {
ipOne := net.ParseIP("192.0.0.1")
ipTwo := net.ParseIP("2001:db8:1234:0000:0000:0000:0000:0000")
ipList := []net.IP{ipOne, ipTwo}
validResolver := "resolver 192.0.0.1 [2001:db8:1234::] valid=30s;"
resolver := buildResolvers(ipList)
if resolver != validResolver {
t.Errorf("Expected '%v' but returned '%v'", validResolver, resolver)
}
}
func TestBuildAuthSignURL(t *testing.T) {
urlOne := "http://google.com"
validUrlOne := "http://google.com?rd=$request_uri"
urlTwo := "http://google.com?cat"
validUrlTwo := "http://google.com?cat&rd=$request_uri"
authSignURLOne := buildAuthSignURL(urlOne)
if authSignURLOne != validUrlOne {
t.Errorf("Expected '%v' but returned '%v'", validUrlOne, authSignURLOne)
}
authSignURLTwo := buildAuthSignURL(urlTwo)
if authSignURLTwo != validUrlTwo {
t.Errorf("Expected '%v' but returned '%v'", validUrlTwo, authSignURLTwo)
}
}
func TestBuildNextUpstream(t *testing.T) {
nextUpstream := "timeout http_500 http_502 non_idempotent"
validNextUpstream := "timeout http_500 http_502"
buildNextUpstream := buildNextUpstream(nextUpstream)
if buildNextUpstream != validNextUpstream {
t.Errorf("Expected '%v' but returned '%v'", validNextUpstream, buildNextUpstream)
}
}
func TestBuildRateLimit(t *testing.T) {
loc := ingress.Location{}
loc.RateLimit.Connections.Name = "con"
loc.RateLimit.Connections.Limit = 1
loc.RateLimit.RPS.Name = "rps"
loc.RateLimit.RPS.Limit = 1
loc.RateLimit.RPS.Burst = 1
loc.RateLimit.RPM.Name = "rpm"
loc.RateLimit.RPM.Limit = 2
loc.RateLimit.RPM.Burst = 2
loc.RateLimit.LimitRateAfter = 1
loc.RateLimit.LimitRate = 1
validLimits := []string{
"limit_conn con 1;",
"limit_req zone=rps burst=1 nodelay;",
"limit_req zone=rpm burst=2 nodelay;",
"limit_rate_after 1k;",
"limit_rate 1k;",
}
limits := buildRateLimit(loc)
for i, limit := range limits {
if limit != validLimits[i] {
t.Errorf("Expected '%v' but returned '%v'", validLimits, limits)
}
}
}

11
controllers/nginx/rootfs/Dockerfile
View file

@ -12,17 +12,20 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM gcr.io/google_containers/nginx-slim-amd64:0.18
FROM BASEIMAGE
CROSS_BUILD_COPY qemu-QEMUARCH-static /usr/bin/
RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y \
diffutils \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
RUN curl -sSL -o /sbin/tini https://github.com/krallin/tini/releases/download/v0.14.0/tini-amd64 && \
chmod +x /sbin/tini
RUN curl -sSL -o /tmp/dumb-init.deb http://ftp.us.debian.org/debian/pool/main/d/dumb-init/dumb-init_1.2.0-1_DUMB_ARCH.deb && \
dpkg -i /tmp/dumb-init.deb && \
rm /tmp/dumb-init.deb
ENTRYPOINT ["/sbin/tini", "--"]
ENTRYPOINT ["/usr/bin/dumb-init"]
COPY . /

67
controllers/nginx/rootfs/etc/nginx/lua/error_page.lua
View file

@ -1,67 +0,0 @@
http = require "resty.http"
def_backend = "upstream-default-backend"
local concat = table.concat
local upstream = require "ngx.upstream"
local get_servers = upstream.get_servers
local get_upstreams = upstream.get_upstreams
local random = math.random
local us = get_upstreams()
function openURL(original_headers, status)
local httpc = http.new()
original_headers["X-Code"] = status or "404"
original_headers["X-Format"] = original_headers["Accept"] or "text/html"
local random_backend = get_destination()
local res, err = httpc:request_uri(random_backend, {
path = "/",
method = "GET",
headers = original_headers,
})
if not res then
ngx.log(ngx.ERR, err)
ngx.exit(500)
end
for k,v in pairs(res.headers) do
ngx.header[k] = v
end
ngx.status = tonumber(status)
ngx.say(res.body)
end
function get_destination()
for _, u in ipairs(us) do
if u == def_backend then
local srvs, err = get_servers(u)
local us_table = {}
if not srvs then
return "http://127.0.0.1:8181"
else
for _, srv in ipairs(srvs) do
us_table[srv["name"]] = srv["weight"]
end
end
local destination = random_weight(us_table)
return "http://"..destination
end
end
end
function random_weight(tbl)
local total = 0
for k, v in pairs(tbl) do
total = total + v
end
local offset = random(0, total - 1)
for k1, v1 in pairs(tbl) do
if offset < v1 then
return k1
end
offset = offset - v1
end
end

78
controllers/nginx/rootfs/etc/nginx/lua/trie.lua
View file

@ -1,78 +0,0 @@
-- Simple trie for URLs
local _M = {}
local mt = {
__index = _M
}
-- http://lua-users.org/wiki/SplitJoin
local strfind, tinsert, strsub = string.find, table.insert, string.sub
function _M.strsplit(delimiter, text)
local list = {}
local pos = 1
while 1 do
local first, last = strfind(text, delimiter, pos)
if first then -- found?
tinsert(list, strsub(text, pos, first-1))
pos = last+1
else
tinsert(list, strsub(text, pos))
break
end
end
return list
end
local strsplit = _M.strsplit
function _M.new()
local t = { }
return setmetatable(t, mt)
end
function _M.add(t, key, val)
local parts = {}
-- hack for just /
if key == "/" then
parts = { "" }
else
parts = strsplit("/", key)
end
local l = t
for i = 1, #parts do
local p = parts[i]
if not l[p] then
l[p] = {}
end
l = l[p]
end
l.__value = val
end
function _M.get(t, key)
local parts = strsplit("/", key)
local l = t
-- this may be nil
local val = t.__value
for i = 1, #parts do
local p = parts[i]
if l[p] then
l = l[p]
local v = l.__value
if v then
val = v
end
else
break
end
end
-- may be nil
return val
end
return _M

2
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/.gitignore vendored
View file

@ -1,2 +0,0 @@
t/servroot/
t/error.log

23
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/LICENSE vendored
View file

@ -1,23 +0,0 @@
Copyright (c) 2013, James Hurst
All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this
list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

20
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/Makefile vendored
View file

@ -1,20 +0,0 @@
OPENRESTY_PREFIX=/usr/local/openresty
PREFIX ?= /usr/local
LUA_INCLUDE_DIR ?= $(PREFIX)/include
LUA_LIB_DIR ?= $(PREFIX)/lib/lua/$(LUA_VERSION)
INSTALL ?= install
TEST_FILE ?= t
.PHONY: all test install
all: ;
install: all
$(INSTALL) -d $(DESTDIR)/$(LUA_LIB_DIR)/resty
$(INSTALL) lib/resty/*.lua $(DESTDIR)/$(LUA_LIB_DIR)/resty/
test: all
PATH=$(OPENRESTY_PREFIX)/nginx/sbin:$$PATH TEST_NGINX_NO_SHUFFLE=1 prove -I../test-nginx/lib -r $(TEST_FILE)
util/lua-releng

424
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/README.md vendored
View file

@ -1,424 +0,0 @@
# lua-resty-http
Lua HTTP client cosocket driver for [OpenResty](http://openresty.org/) / [ngx_lua](https://github.com/openresty/lua-nginx-module).
# Status
Production ready.
# Features
* HTTP 1.0 and 1.1
* SSL
* Streaming interface to the response body, for predictable memory usage
* Alternative simple interface for singleshot requests without manual connection step
* Chunked and non-chunked transfer encodings
* Keepalive
* Pipelining
* Trailers
# API
* [new](#name)
* [connect](#connect)
* [set_timeout](#set_timeout)
* [ssl_handshake](#ssl_handshake)
* [set_keepalive](#set_keepalive)
* [get_reused_times](#get_reused_times)
* [close](#close)
* [request](#request)
* [request_uri](#request_uri)
* [request_pipeline](#request_pipeline)
* [Response](#response)
* [body_reader](#resbody_reader)
* [read_body](#resread_body)
* [read_trailes](#resread_trailers)
* [Proxy](#proxy)
* [proxy_request](#proxy_request)
* [proxy_response](#proxy_response)
* [Utility](#utility)
* [parse_uri](#parse_uri)
* [get_client_body_reader](#get_client_body_reader)
## Synopsis
```` lua
lua_package_path "/path/to/lua-resty-http/lib/?.lua;;";
server {
location /simpleinterface {
resolver 8.8.8.8; # use Google's open DNS server for an example
content_by_lua '
-- For simple singleshot requests, use the URI interface.
local http = require "resty.http"
local httpc = http.new()
local res, err = httpc:request_uri("http://example.com/helloworld", {
method = "POST",
body = "a=1&b=2",
headers = {
["Content-Type"] = "application/x-www-form-urlencoded",
}
})
if not res then
ngx.say("failed to request: ", err)
return
end
-- In this simple form, there is no manual connection step, so the body is read
-- all in one go, including any trailers, and the connection closed or keptalive
-- for you.
ngx.status = res.status
for k,v in pairs(res.headers) do
--
end
ngx.say(res.body)
';
}
location /genericinterface {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
-- The generic form gives us more control. We must connect manually.
httpc:set_timeout(500)
httpc:connect("127.0.0.1", 80)
-- And request using a path, rather than a full URI.
local res, err = httpc:request{
path = "/helloworld",
headers = {
["Host"] = "example.com",
},
}
if not res then
ngx.say("failed to request: ", err)
return
end
-- Now we can use the body_reader iterator, to stream the body according to our desired chunk size.
local reader = res.body_reader
repeat
local chunk, err = reader(8192)
if err then
ngx.log(ngx.ERR, err)
break
end
if chunk then
-- process
end
until not chunk
local ok, err = httpc:set_keepalive()
if not ok then
ngx.say("failed to set keepalive: ", err)
return
end
';
}
}
````
# Connection
## new
`syntax: httpc = http.new()`
Creates the http object. In case of failures, returns `nil` and a string describing the error.
## connect
`syntax: ok, err = httpc:connect(host, port, options_table?)`
`syntax: ok, err = httpc:connect("unix:/path/to/unix.sock", options_table?)`
Attempts to connect to the web server.
Before actually resolving the host name and connecting to the remote backend, this method will always look up the connection pool for matched idle connections created by previous calls of this method.
An optional Lua table can be specified as the last argument to this method to specify various connect options:
* `pool`
: Specifies a custom name for the connection pool being used. If omitted, then the connection pool name will be generated from the string template `<host>:<port>` or `<unix-socket-path>`.
## set_timeout
`syntax: httpc:set_timeout(time)`
Sets the timeout (in ms) protection for subsequent operations, including the `connect` method.
## ssl_handshake
`syntax: session, err = httpc:ssl_handshake(session, host, verify)`
Performs an SSL handshake on the TCP connection, only availble in ngx_lua > v0.9.11
See docs for [ngx.socket.tcp](https://github.com/openresty/lua-nginx-module#ngxsockettcp) for details.
## set_keepalive
`syntax: ok, err = httpc:set_keepalive(max_idle_timeout, pool_size)`
Attempts to puts the current connection into the ngx_lua cosocket connection pool.
You can specify the max idle timeout (in ms) when the connection is in the pool and the maximal size of the pool every nginx worker process.
Only call this method in the place you would have called the `close` method instead. Calling this method will immediately turn the current http object into the `closed` state. Any subsequent operations other than `connect()` on the current objet will return the `closed` error.
Note that calling this instead of `close` is "safe" in that it will conditionally close depending on the type of request. Specifically, a `1.0` request without `Connection: Keep-Alive` will be closed, as will a `1.1` request with `Connection: Close`.
In case of success, returns `1`. In case of errors, returns `nil, err`. In the case where the conneciton is conditionally closed as described above, returns `2` and the error string `connection must be closed`.
## get_reused_times
`syntax: times, err = httpc:get_reused_times()`
This method returns the (successfully) reused times for the current connection. In case of error, it returns `nil` and a string describing the error.
If the current connection does not come from the built-in connection pool, then this method always returns `0`, that is, the connection has never been reused (yet). If the connection comes from the connection pool, then the return value is always non-zero. So this method can also be used to determine if the current connection comes from the pool.
## close
`syntax: ok, err = http:close()`
Closes the current connection and returns the status.
In case of success, returns `1`. In case of errors, returns `nil` with a string describing the error.
# Requesting
## request
`syntax: res, err = httpc:request(params)`
Returns a `res` table or `nil` and an error message.
The `params` table accepts the following fields:
* `version` The HTTP version number, currently supporting 1.0 or 1.1.
* `method` The HTTP method string.
* `path` The path string.
* `headers` A table of request headers.
* `body` The request body as a string, or an iterator function (see [get_client_body_reader](#get_client_body_reader)).
* `ssl_verify` Verify SSL cert matches hostname
When the request is successful, `res` will contain the following fields:
* `status` The status code.
* `reason` The status reason phrase.
* `headers` A table of headers. Multiple headers with the same field name will be presented as a table of values.
* `has_body` A boolean flag indicating if there is a body to be read.
* `body_reader` An iterator function for reading the body in a streaming fashion.
* `read_body` A method to read the entire body into a string.
* `read_trailers` A method to merge any trailers underneath the headers, after reading the body.
## request_uri
`syntax: res, err = httpc:request_uri(uri, params)`
The simple interface. Options supplied in the `params` table are the same as in the generic interface, and will override components found in the uri itself.
In this mode, there is no need to connect manually first. The connection is made on your behalf, suiting cases where you simply need to grab a URI without too much hassle.
Additionally there is no ability to stream the response body in this mode. If the request is successful, `res` will contain the following fields:
* `status` The status code.
* `headers` A table of headers.
* `body` The response body as a string.
## request_pipeline
`syntax: responses, err = httpc:request_pipeline(params)`
This method works as per the [request](#request) method above, but `params` is instead a table of param tables. Each request is sent in order, and `responses` is returned as a table of response handles. For example:
```lua
local responses = httpc:request_pipeline{
{
path = "/b",
},
{
path = "/c",
},
{
path = "/d",
}
}
for i,r in ipairs(responses) do
if r.status then
ngx.say(r.status)
ngx.say(r:read_body())
end
end
```
Due to the nature of pipelining, no responses are actually read until you attempt to use the response fields (status / headers etc). And since the responses are read off in order, you must read the entire body (and any trailers if you have them), before attempting to read the next response.
Note this doesn't preclude the use of the streaming response body reader. Responses can still be streamed, so long as the entire body is streamed before attempting to access the next response.
Be sure to test at least one field (such as status) before trying to use the others, in case a socket read error has occurred.
# Response
## res.body_reader
The `body_reader` iterator can be used to stream the response body in chunk sizes of your choosing, as follows:
````lua
local reader = res.body_reader
repeat
local chunk, err = reader(8192)
if err then
ngx.log(ngx.ERR, err)
break
end
if chunk then
-- process
end
until not chunk
````
If the reader is called with no arguments, the behaviour depends on the type of connection. If the response is encoded as chunked, then the iterator will return the chunks as they arrive. If not, it will simply return the entire body.
Note that the size provided is actually a **maximum** size. So in the chunked transfer case, you may get chunks smaller than the size you ask, as a remainder of the actual HTTP chunks.
## res:read_body
`syntax: body, err = res:read_body()`
Reads the entire body into a local string.
## res:read_trailers
`syntax: res:read_trailers()`
This merges any trailers underneath the `res.headers` table itself. Must be called after reading the body.
# Proxy
There are two convenience methods for when one simply wishes to proxy the current request to the connected upstream, and safely send it downstream to the client, as a reverse proxy. A complete example:
```lua
local http = require "resty.http"
local httpc = http.new()
httpc:set_timeout(500)
local ok, err = httpc:connect(HOST, PORT)
if not ok then
ngx.log(ngx.ERR, err)
return
end
httpc:set_timeout(2000)
httpc:proxy_response(httpc:proxy_request())
httpc:set_keepalive()
```
## proxy_request
`syntax: local res, err = httpc:proxy_request(request_body_chunk_size?)`
Performs a request using the current client request arguments, effectively proxying to the connected upstream. The request body will be read in a streaming fashion, according to `request_body_chunk_size` (see [documentation on the client body reader](#get_client_body_reader) below).
## proxy_response
`syntax: httpc:proxy_response(res, chunksize?)`
Sets the current response based on the given `res`. Ensures that hop-by-hop headers are not sent downstream, and will read the response according to `chunksize` (see [documentation on the body reader](#resbody_reader) above).
# Utility
## parse_uri
`syntax: local scheme, host, port, path = unpack(httpc:parse_uri(uri))`
This is a convenience function allowing one to more easily use the generic interface, when the input data is a URI.
## get_client_body_reader
`syntax: reader, err = httpc:get_client_body_reader(chunksize?, sock?)`
Returns an iterator function which can be used to read the downstream client request body in a streaming fashion. You may also specify an optional default chunksize (default is `65536`), or an already established socket in
place of the client request.
Example:
```lua
local req_reader = httpc:get_client_body_reader()
repeat
local chunk, err = req_reader(8192)
if err then
ngx.log(ngx.ERR, err)
break
end
if chunk then
-- process
end
until not chunk
```
This iterator can also be used as the value for the body field in request params, allowing one to stream the request body into a proxied upstream request.
```lua
local client_body_reader, err = httpc:get_client_body_reader()
local res, err = httpc:request{
path = "/helloworld",
body = client_body_reader,
}
```
If `sock` is specified,
# Author
James Hurst <james@pintsized.co.uk>
Originally started life based on https://github.com/bakins/lua-resty-http-simple. Cosocket docs and implementation borrowed from the other lua-resty-* cosocket modules.
# Licence
This module is licensed under the 2-clause BSD license.
Copyright (c) 2013-2016, James Hurst <james@pintsized.co.uk>
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

850
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/lib/resty/http.lua vendored
View file

@ -1,850 +0,0 @@
local http_headers = require "resty.http_headers"
local ngx_socket_tcp = ngx.socket.tcp
local ngx_req = ngx.req
local ngx_req_socket = ngx_req.socket
local ngx_req_get_headers = ngx_req.get_headers
local ngx_req_get_method = ngx_req.get_method
local str_gmatch = string.gmatch
local str_lower = string.lower
local str_upper = string.upper
local str_find = string.find
local str_sub = string.sub
local str_gsub = string.gsub
local tbl_concat = table.concat
local tbl_insert = table.insert
local ngx_encode_args = ngx.encode_args
local ngx_re_match = ngx.re.match
local ngx_re_gsub = ngx.re.gsub
local ngx_log = ngx.log
local ngx_DEBUG = ngx.DEBUG
local ngx_ERR = ngx.ERR
local ngx_NOTICE = ngx.NOTICE
local ngx_var = ngx.var
local co_yield = coroutine.yield
local co_create = coroutine.create
local co_status = coroutine.status
local co_resume = coroutine.resume
-- http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.5.1
local HOP_BY_HOP_HEADERS = {
["connection"] = true,
["keep-alive"] = true,
["proxy-authenticate"] = true,
["proxy-authorization"] = true,
["te"] = true,
["trailers"] = true,
["transfer-encoding"] = true,
["upgrade"] = true,
["content-length"] = true, -- Not strictly hop-by-hop, but Nginx will deal
-- with this (may send chunked for example).
}
-- Reimplemented coroutine.wrap, returning "nil, err" if the coroutine cannot
-- be resumed. This protects user code from inifite loops when doing things like
-- repeat
-- local chunk, err = res.body_reader()
-- if chunk then -- <-- This could be a string msg in the core wrap function.
-- ...
-- end
-- until not chunk
local co_wrap = function(func)
local co = co_create(func)
if not co then
return nil, "could not create coroutine"
else
return function(...)
if co_status(co) == "suspended" then
return select(2, co_resume(co, ...))
else
return nil, "can't resume a " .. co_status(co) .. " coroutine"
end
end
end
end
local _M = {
_VERSION = '0.09',
}
_M._USER_AGENT = "lua-resty-http/" .. _M._VERSION .. " (Lua) ngx_lua/" .. ngx.config.ngx_lua_version
local mt = { __index = _M }
local HTTP = {
[1.0] = " HTTP/1.0\r\n",
[1.1] = " HTTP/1.1\r\n",
}
local DEFAULT_PARAMS = {
method = "GET",
path = "/",
version = 1.1,
}
function _M.new(self)
local sock, err = ngx_socket_tcp()
if not sock then
return nil, err
end
return setmetatable({ sock = sock, keepalive = true }, mt)
end
function _M.set_timeout(self, timeout)
local sock = self.sock
if not sock then
return nil, "not initialized"
end
return sock:settimeout(timeout)
end
function _M.ssl_handshake(self, ...)
local sock = self.sock
if not sock then
return nil, "not initialized"
end
self.ssl = true
return sock:sslhandshake(...)
end
function _M.connect(self, ...)
local sock = self.sock
if not sock then
return nil, "not initialized"
end
self.host = select(1, ...)
self.port = select(2, ...)
-- If port is not a number, this is likely a unix domain socket connection.
if type(self.port) ~= "number" then
self.port = nil
end
self.keepalive = true
return sock:connect(...)
end
function _M.set_keepalive(self, ...)
local sock = self.sock
if not sock then
return nil, "not initialized"
end
if self.keepalive == true then
return sock:setkeepalive(...)
else
-- The server said we must close the connection, so we cannot setkeepalive.
-- If close() succeeds we return 2 instead of 1, to differentiate between
-- a normal setkeepalive() failure and an intentional close().
local res, err = sock:close()
if res then
return 2, "connection must be closed"
else
return res, err
end
end
end
function _M.get_reused_times(self)
local sock = self.sock
if not sock then
return nil, "not initialized"
end
return sock:getreusedtimes()
end
function _M.close(self)
local sock = self.sock
if not sock then
return nil, "not initialized"
end
return sock:close()
end
local function _should_receive_body(method, code)
if method == "HEAD" then return nil end
if code == 204 or code == 304 then return nil end
if code >= 100 and code < 200 then return nil end
return true
end
function _M.parse_uri(self, uri)
local m, err = ngx_re_match(uri, [[^(http[s]?)://([^:/]+)(?::(\d+))?(.*)]],
"jo")
if not m then
if err then
return nil, "failed to match the uri: " .. uri .. ", " .. err
end
return nil, "bad uri: " .. uri
else
if m[3] then
m[3] = tonumber(m[3])
else
if m[1] == "https" then
m[3] = 443
else
m[3] = 80
end
end
if not m[4] or "" == m[4] then m[4] = "/" end
return m, nil
end
end
local function _format_request(params)
local version = params.version
local headers = params.headers or {}
local query = params.query or ""
if query then
if type(query) == "table" then
query = "?" .. ngx_encode_args(query)
end
end
-- Initialize request
local req = {
str_upper(params.method),
" ",
params.path,
query,
HTTP[version],
-- Pre-allocate slots for minimum headers and carriage return.
true,
true,
true,
}
local c = 6 -- req table index it's faster to do this inline vs table.insert
-- Append headers
for key, values in pairs(headers) do
if type(values) ~= "table" then
values = {values}
end
key = tostring(key)
for _, value in pairs(values) do
req[c] = key .. ": " .. tostring(value) .. "\r\n"
c = c + 1
end
end
-- Close headers
req[c] = "\r\n"
return tbl_concat(req)
end
local function _receive_status(sock)
local line, err = sock:receive("*l")
if not line then
return nil, nil, nil, err
end
return tonumber(str_sub(line, 10, 12)), tonumber(str_sub(line, 6, 8)), str_sub(line, 14)
end
local function _receive_headers(sock)
local headers = http_headers.new()
repeat
local line, err = sock:receive("*l")
if not line then
return nil, err
end
for key, val in str_gmatch(line, "([^:%s]+):%s*(.+)") do
if headers[key] then
if type(headers[key]) ~= "table" then
headers[key] = { headers[key] }
end
tbl_insert(headers[key], tostring(val))
else
headers[key] = tostring(val)
end
end
until str_find(line, "^%s*$")
return headers, nil
end
local function _chunked_body_reader(sock, default_chunk_size)
return co_wrap(function(max_chunk_size)
local max_chunk_size = max_chunk_size or default_chunk_size
local remaining = 0
local length
repeat
-- If we still have data on this chunk
if max_chunk_size and remaining > 0 then
if remaining > max_chunk_size then
-- Consume up to max_chunk_size
length = max_chunk_size
remaining = remaining - max_chunk_size
else
-- Consume all remaining
length = remaining
remaining = 0
end
else -- This is a fresh chunk
-- Receive the chunk size
local str, err = sock:receive("*l")
if not str then
co_yield(nil, err)
end
length = tonumber(str, 16)
if not length then
co_yield(nil, "unable to read chunksize")
end
if max_chunk_size and length > max_chunk_size then
-- Consume up to max_chunk_size
remaining = length - max_chunk_size
length = max_chunk_size
end
end
if length > 0 then
local str, err = sock:receive(length)
if not str then
co_yield(nil, err)
end
max_chunk_size = co_yield(str) or default_chunk_size
-- If we're finished with this chunk, read the carriage return.
if remaining == 0 then
sock:receive(2) -- read \r\n
end
else
-- Read the last (zero length) chunk's carriage return
sock:receive(2) -- read \r\n
end
until length == 0
end)
end
local function _body_reader(sock, content_length, default_chunk_size)
return co_wrap(function(max_chunk_size)
local max_chunk_size = max_chunk_size or default_chunk_size
if not content_length and max_chunk_size then
-- We have no length, but wish to stream.
-- HTTP 1.0 with no length will close connection, so read chunks to the end.
repeat
local str, err, partial = sock:receive(max_chunk_size)
if not str and err == "closed" then
max_chunk_size = tonumber(co_yield(partial, err) or default_chunk_size)
end
max_chunk_size = tonumber(co_yield(str) or default_chunk_size)
if max_chunk_size and max_chunk_size < 0 then max_chunk_size = nil end
if not max_chunk_size then
ngx_log(ngx_ERR, "Buffer size not specified, bailing")
break
end
until not str
elseif not content_length then
-- We have no length but don't wish to stream.
-- HTTP 1.0 with no length will close connection, so read to the end.
co_yield(sock:receive("*a"))
elseif not max_chunk_size then
-- We have a length and potentially keep-alive, but want everything.
co_yield(sock:receive(content_length))
else
-- We have a length and potentially a keep-alive, and wish to stream
-- the response.
local received = 0
repeat
local length = max_chunk_size
if received + length > content_length then
length = content_length - received
end
if length > 0 then
local str, err = sock:receive(length)
if not str then
max_chunk_size = tonumber(co_yield(nil, err) or default_chunk_size)
end
received = received + length
max_chunk_size = tonumber(co_yield(str) or default_chunk_size)
if max_chunk_size and max_chunk_size < 0 then max_chunk_size = nil end
if not max_chunk_size then
ngx_log(ngx_ERR, "Buffer size not specified, bailing")
break
end
end
until length == 0
end
end)
end
local function _no_body_reader()
return nil
end
local function _read_body(res)
local reader = res.body_reader
if not reader then
-- Most likely HEAD or 304 etc.
return nil, "no body to be read"
end
local chunks = {}
local c = 1
local chunk, err
repeat
chunk, err = reader()
if err then
return nil, err, tbl_concat(chunks) -- Return any data so far.
end
if chunk then
chunks[c] = chunk
c = c + 1
end
until not chunk
return tbl_concat(chunks)
end
local function _trailer_reader(sock)
return co_wrap(function()
co_yield(_receive_headers(sock))
end)
end
local function _read_trailers(res)
local reader = res.trailer_reader
if not reader then
return nil, "no trailers"
end
local trailers = reader()
setmetatable(res.headers, { __index = trailers })
end
local function _send_body(sock, body)
if type(body) == 'function' then
repeat
local chunk, err, partial = body()
if chunk then
local ok,err = sock:send(chunk)
if not ok then
return nil, err
end
elseif err ~= nil then
return nil, err, partial
end
until chunk == nil
elseif body ~= nil then
local bytes, err = sock:send(body)
if not bytes then
return nil, err
end
end
return true, nil
end
local function _handle_continue(sock, body)
local status, version, reason, err = _receive_status(sock)
if not status then
return nil, nil, err
end
-- Only send body if we receive a 100 Continue
if status == 100 then
local ok, err = sock:receive("*l") -- Read carriage return
if not ok then
return nil, nil, err
end
_send_body(sock, body)
end
return status, version, err
end
function _M.send_request(self, params)
-- Apply defaults
setmetatable(params, { __index = DEFAULT_PARAMS })
local sock = self.sock
local body = params.body
local headers = http_headers.new()
local params_headers = params.headers
if params_headers then
-- We assign one by one so that the metatable can handle case insensitivity
-- for us. You can blame the spec for this inefficiency.
for k,v in pairs(params_headers) do
headers[k] = v
end
end
-- Ensure minimal headers are set
if type(body) == 'string' and not headers["Content-Length"] then
headers["Content-Length"] = #body
end
if not headers["Host"] then
if (str_sub(self.host, 1, 5) == "unix:") then
return nil, "Unable to generate a useful Host header for a unix domain socket. Please provide one."
end
-- If we have a port (i.e. not connected to a unix domain socket), and this
-- port is non-standard, append it to the Host heaer.
if self.port then
if self.ssl and self.port ~= 443 then
headers["Host"] = self.host .. ":" .. self.port
elseif not self.ssl and self.port ~= 80 then
headers["Host"] = self.host .. ":" .. self.port
else
headers["Host"] = self.host
end
else
headers["Host"] = self.host
end
end
if not headers["User-Agent"] then
headers["User-Agent"] = _M._USER_AGENT
end
if params.version == 1.0 and not headers["Connection"] then
headers["Connection"] = "Keep-Alive"
end
params.headers = headers
-- Format and send request
local req = _format_request(params)
ngx_log(ngx_DEBUG, "\n", req)
local bytes, err = sock:send(req)
if not bytes then
return nil, err
end
-- Send the request body, unless we expect: continue, in which case
-- we handle this as part of reading the response.
if headers["Expect"] ~= "100-continue" then
local ok, err, partial = _send_body(sock, body)
if not ok then
return nil, err, partial
end
end
return true
end
function _M.read_response(self, params)
local sock = self.sock
local status, version, reason, err
-- If we expect: continue, we need to handle this, sending the body if allowed.
-- If we don't get 100 back, then status is the actual status.
if params.headers["Expect"] == "100-continue" then
local _status, _version, _err = _handle_continue(sock, params.body)
if not _status then
return nil, _err
elseif _status ~= 100 then
status, version, err = _status, _version, _err
end
end
-- Just read the status as normal.
if not status then
status, version, reason, err = _receive_status(sock)
if not status then
return nil, err
end
end
local res_headers, err = _receive_headers(sock)
if not res_headers then
return nil, err
end
-- keepalive is true by default. Determine if this is correct or not.
local ok, connection = pcall(str_lower, res_headers["Connection"])
if ok then
if (version == 1.1 and connection == "close") or
(version == 1.0 and connection ~= "keep-alive") then
self.keepalive = false
end
else
-- no connection header
if version == 1.0 then
self.keepalive = false
end
end
local body_reader = _no_body_reader
local trailer_reader, err = nil, nil
local has_body = false
-- Receive the body_reader
if _should_receive_body(params.method, status) then
local ok, encoding = pcall(str_lower, res_headers["Transfer-Encoding"])
if ok and version == 1.1 and encoding == "chunked" then
body_reader, err = _chunked_body_reader(sock)
has_body = true
else
local ok, length = pcall(tonumber, res_headers["Content-Length"])
if ok then
body_reader, err = _body_reader(sock, length)
has_body = true
end
end
end
if res_headers["Trailer"] then
trailer_reader, err = _trailer_reader(sock)
end
if err then
return nil, err
else
return {
status = status,
reason = reason,
headers = res_headers,
has_body = has_body,
body_reader = body_reader,
read_body = _read_body,
trailer_reader = trailer_reader,
read_trailers = _read_trailers,
}
end
end
function _M.request(self, params)
local res, err = self:send_request(params)
if not res then
return res, err
else
return self:read_response(params)
end
end
function _M.request_pipeline(self, requests)
for i, params in ipairs(requests) do
if params.headers and params.headers["Expect"] == "100-continue" then
return nil, "Cannot pipeline request specifying Expect: 100-continue"
end
local res, err = self:send_request(params)
if not res then
return res, err
end
end
local responses = {}
for i, params in ipairs(requests) do
responses[i] = setmetatable({
params = params,
response_read = false,
}, {
-- Read each actual response lazily, at the point the user tries
-- to access any of the fields.
__index = function(t, k)
local res, err
if t.response_read == false then
res, err = _M.read_response(self, t.params)
t.response_read = true
if not res then
ngx_log(ngx_ERR, err)
else
for rk, rv in pairs(res) do
t[rk] = rv
end
end
end
return rawget(t, k)
end,
})
end
return responses
end
function _M.request_uri(self, uri, params)
if not params then params = {} end
local parsed_uri, err = self:parse_uri(uri)
if not parsed_uri then
return nil, err
end
local scheme, host, port, path = unpack(parsed_uri)
if not params.path then params.path = path end
local c, err = self:connect(host, port)
if not c then
return nil, err
end
if scheme == "https" then
local verify = true
if params.ssl_verify == false then
verify = false
end
local ok, err = self:ssl_handshake(nil, host, verify)
if not ok then
return nil, err
end
end
local res, err = self:request(params)
if not res then
return nil, err
end
local body, err = res:read_body()
if not body then
return nil, err
end
res.body = body
local ok, err = self:set_keepalive()
if not ok then
ngx_log(ngx_ERR, err)
end
return res, nil
end
function _M.get_client_body_reader(self, chunksize, sock)
local chunksize = chunksize or 65536
if not sock then
local ok, err
ok, sock, err = pcall(ngx_req_socket)
if not ok then
return nil, sock -- pcall err
end
if not sock then
if err == "no body" then
return nil
else
return nil, err
end
end
end
local headers = ngx_req_get_headers()
local length = headers.content_length
local encoding = headers.transfer_encoding
if length then
return _body_reader(sock, tonumber(length), chunksize)
elseif encoding and str_lower(encoding) == 'chunked' then
-- Not yet supported by ngx_lua but should just work...
return _chunked_body_reader(sock, chunksize)
else
return nil
end
end
function _M.proxy_request(self, chunksize)
return self:request{
method = ngx_req_get_method(),
path = ngx_re_gsub(ngx_var.uri, "\\s", "%20", "jo") .. ngx_var.is_args .. (ngx_var.query_string or ""),
body = self:get_client_body_reader(chunksize),
headers = ngx_req_get_headers(),
}
end
function _M.proxy_response(self, response, chunksize)
if not response then
ngx_log(ngx_ERR, "no response provided")
return
end
ngx.status = response.status
-- Filter out hop-by-hop headeres
for k,v in pairs(response.headers) do
if not HOP_BY_HOP_HEADERS[str_lower(k)] then
ngx.header[k] = v
end
end
local reader = response.body_reader
repeat
local chunk, err = reader(chunksize)
if err then
ngx_log(ngx_ERR, err)
break
end
if chunk then
local res, err = ngx.print(chunk)
if not res then
ngx_log(ngx_ERR, err)
break
end
end
until not chunk
end
return _M

62
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/lib/resty/http_headers.lua vendored
View file

@ -1,62 +0,0 @@
local rawget, rawset, setmetatable =
rawget, rawset, setmetatable
local str_gsub = string.gsub
local str_lower = string.lower
local _M = {
_VERSION = '0.01',
}
-- Returns an empty headers table with internalised case normalisation.
-- Supports the same cases as in ngx_lua:
--
-- headers.content_length
-- headers["content-length"]
-- headers["Content-Length"]
function _M.new(self)
local mt = {
normalised = {},
}
mt.__index = function(t, k)
local k_hyphened = str_gsub(k, "_", "-")
local matched = rawget(t, k)
if matched then
return matched
else
local k_normalised = str_lower(k_hyphened)
return rawget(t, mt.normalised[k_normalised])
end
end
-- First check the normalised table. If there's no match (first time) add an entry for
-- our current case in the normalised table. This is to preserve the human (prettier) case
-- instead of outputting lowercased header names.
--
-- If there's a match, we're being updated, just with a different case for the key. We use
-- the normalised table to give us the original key, and perorm a rawset().
mt.__newindex = function(t, k, v)
-- we support underscore syntax, so always hyphenate.
local k_hyphened = str_gsub(k, "_", "-")
-- lowercase hyphenated is "normalised"
local k_normalised = str_lower(k_hyphened)
if not mt.normalised[k_normalised] then
mt.normalised[k_normalised] = k_hyphened
rawset(t, k_hyphened, v)
else
rawset(t, mt.normalised[k_normalised], v)
end
end
return setmetatable({}, mt)
end
return _M

22
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/lua-resty-http-0.09-0.rockspec vendored
View file

@ -1,22 +0,0 @@
package = "lua-resty-http"
version = "0.09-0"
source = {
url = "git://github.com/pintsized/lua-resty-http",
tag = "v0.09"
}
description = {
summary = "Lua HTTP client cosocket driver for OpenResty / ngx_lua.",
homepage = "https://github.com/pintsized/lua-resty-http",
license = "2-clause BSD",
maintainer = "James Hurst <james@pintsized.co.uk>"
}
dependencies = {
"lua >= 5.1"
}
build = {
type = "builtin",
modules = {
["resty.http"] = "lib/resty/http.lua",
["resty.http_headers"] = "lib/resty/http_headers.lua"
}
}

233
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/t/01-basic.t vendored
View file

@ -1,233 +0,0 @@
# vim:set ft= ts=4 sw=4 et:
use Test::Nginx::Socket;
use Cwd qw(cwd);
plan tests => repeat_each() * (blocks() * 4) + 1;
my $pwd = cwd();
our $HttpConfig = qq{
lua_package_path "$pwd/lib/?.lua;;";
error_log logs/error.log debug;
};
$ENV{TEST_NGINX_RESOLVER} = '8.8.8.8';
no_long_string();
#no_diff();
run_tests();
__DATA__
=== TEST 1: Simple default get.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b"
}
ngx.status = res.status
ngx.print(res:read_body())
httpc:close()
';
}
location = /b {
echo "OK";
}
--- request
GET /a
--- response_body
OK
--- no_error_log
[error]
[warn]
=== TEST 2: HTTP 1.0
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
version = 1.0,
path = "/b"
}
ngx.status = res.status
ngx.print(res:read_body())
httpc:close()
';
}
location = /b {
echo "OK";
}
--- request
GET /a
--- response_body
OK
--- no_error_log
[error]
[warn]
=== TEST 3: Status code and reason phrase
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b"
}
ngx.status = res.status
ngx.say(res.reason)
ngx.print(res:read_body())
httpc:close()
';
}
location = /b {
content_by_lua '
ngx.status = 404
ngx.say("OK")
';
}
--- request
GET /a
--- response_body
Not Found
OK
--- error_code: 404
--- no_error_log
[error]
[warn]
=== TEST 4: Response headers
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b"
}
ngx.status = res.status
ngx.say(res.headers["X-Test"])
httpc:close()
';
}
location = /b {
content_by_lua '
ngx.header["X-Test"] = "x-value"
ngx.say("OK")
';
}
--- request
GET /a
--- response_body
x-value
--- no_error_log
[error]
[warn]
=== TEST 5: Query
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
query = {
a = 1,
b = 2,
},
path = "/b"
}
ngx.status = res.status
for k,v in pairs(res.headers) do
ngx.header[k] = v
end
ngx.print(res:read_body())
httpc:close()
';
}
location = /b {
content_by_lua '
for k,v in pairs(ngx.req.get_uri_args()) do
ngx.header["X-Header-" .. string.upper(k)] = v
end
';
}
--- request
GET /a
--- response_headers
X-Header-A: 1
X-Header-B: 2
--- no_error_log
[error]
[warn]
=== TEST 7: HEAD has no body.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
method = "HEAD",
path = "/b"
}
local body = res:read_body()
if body then
ngx.print(body)
end
httpc:close()
';
}
location = /b {
echo "OK";
}
--- request
GET /a
--- response_body
--- no_error_log
[error]
[warn]

158
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/t/02-chunked.t vendored
View file

@ -1,158 +0,0 @@
# vim:set ft= ts=4 sw=4 et:
use Test::Nginx::Socket;
use Cwd qw(cwd);
plan tests => repeat_each() * (blocks() * 4);
my $pwd = cwd();
our $HttpConfig = qq{
lua_package_path "$pwd/lib/?.lua;;";
};
$ENV{TEST_NGINX_RESOLVER} = '8.8.8.8';
no_long_string();
#no_diff();
run_tests();
__DATA__
=== TEST 1: Non chunked.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b"
}
local body = res:read_body()
ngx.say(#body)
httpc:close()
';
}
location = /b {
chunked_transfer_encoding off;
content_by_lua '
local len = 32768
local t = {}
for i=1,len do
t[i] = 0
end
ngx.print(table.concat(t))
';
}
--- request
GET /a
--- response_body
32768
--- no_error_log
[error]
[warn]
=== TEST 2: Chunked. The number of chunks received when no max size is given proves the response was in fact chunked.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b"
}
local chunks = {}
local c = 1
repeat
local chunk, err = res.body_reader()
if chunk then
chunks[c] = chunk
c = c + 1
end
until not chunk
local body = table.concat(chunks)
ngx.say(#body)
ngx.say(#chunks)
httpc:close()
';
}
location = /b {
content_by_lua '
local len = 32768
local t = {}
for i=1,len do
t[i] = 0
end
ngx.print(table.concat(t))
local len = 32768
local t = {}
for i=1,len do
t[i] = 0
end
ngx.print(table.concat(t))
';
}
--- request
GET /a
--- response_body
65536
2
--- no_error_log
[error]
[warn]
=== TEST 3: Chunked using read_body method.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b"
}
local body = res:read_body()
ngx.say(#body)
httpc:close()
';
}
location = /b {
content_by_lua '
local len = 32768
local t = {}
for i=1,len do
t[i] = 0
end
ngx.print(table.concat(t))
local len = 32768
local t = {}
for i=1,len do
t[i] = 0
end
ngx.print(table.concat(t))
';
}
--- request
GET /a
--- response_body
65536
--- no_error_log
[error]
[warn]

185
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/t/03-requestbody.t vendored
View file

@ -1,185 +0,0 @@
# vim:set ft= ts=4 sw=4 et:
use Test::Nginx::Socket;
use Cwd qw(cwd);
plan tests => repeat_each() * (blocks() * 4);
my $pwd = cwd();
our $HttpConfig = qq{
lua_package_path "$pwd/lib/?.lua;;";
};
$ENV{TEST_NGINX_RESOLVER} = '8.8.8.8';
no_long_string();
#no_diff();
run_tests();
__DATA__
=== TEST 1: POST form-urlencoded
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
body = "a=1&b=2&c=3",
path = "/b",
headers = {
["Content-Type"] = "application/x-www-form-urlencoded",
}
}
ngx.say(res:read_body())
httpc:close()
';
}
location = /b {
content_by_lua '
ngx.req.read_body()
local args = ngx.req.get_post_args()
ngx.say("a: ", args.a)
ngx.say("b: ", args.b)
ngx.print("c: ", args.c)
';
}
--- request
GET /a
--- response_body
a: 1
b: 2
c: 3
--- no_error_log
[error]
[warn]
=== TEST 2: POST form-urlencoded 1.0
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
method = "POST",
body = "a=1&b=2&c=3",
path = "/b",
headers = {
["Content-Type"] = "application/x-www-form-urlencoded",
},
version = 1.0,
}
ngx.say(res:read_body())
httpc:close()
';
}
location = /b {
content_by_lua '
ngx.req.read_body()
local args = ngx.req.get_post_args()
ngx.say(ngx.req.get_method())
ngx.say("a: ", args.a)
ngx.say("b: ", args.b)
ngx.print("c: ", args.c)
';
}
--- request
GET /a
--- response_body
POST
a: 1
b: 2
c: 3
--- no_error_log
[error]
[warn]
=== TEST 3: 100 Continue does not end requset
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
body = "a=1&b=2&c=3",
path = "/b",
headers = {
["Expect"] = "100-continue",
["Content-Type"] = "application/x-www-form-urlencoded",
}
}
ngx.say(res.status)
ngx.say(res:read_body())
httpc:close()
';
}
location = /b {
content_by_lua '
ngx.req.read_body()
local args = ngx.req.get_post_args()
ngx.say("a: ", args.a)
ngx.say("b: ", args.b)
ngx.print("c: ", args.c)
';
}
--- request
GET /a
--- response_body
200
a: 1
b: 2
c: 3
--- no_error_log
[error]
[warn]
=== TEST 4: Return non-100 status to user
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b",
headers = {
["Expect"] = "100-continue",
["Content-Type"] = "application/x-www-form-urlencoded",
}
}
if not res then
ngx.say(err)
end
ngx.say(res.status)
ngx.say(res:read_body())
httpc:close()
';
}
location = /b {
return 417 "Expectation Failed";
}
--- request
GET /a
--- response_body
417
Expectation Failed
--- no_error_log
[error]
[warn]

151
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/t/04-trailers.t vendored
View file

@ -1,151 +0,0 @@
# vim:set ft= ts=4 sw=4 et:
use Test::Nginx::Socket;
use Cwd qw(cwd);
plan tests => repeat_each() * (blocks() * 4);
my $pwd = cwd();
our $HttpConfig = qq{
lua_package_path "$pwd/lib/?.lua;;";
};
$ENV{TEST_NGINX_RESOLVER} = '8.8.8.8';
no_long_string();
#no_diff();
run_tests();
__DATA__
=== TEST 1: Trailers. Check Content-MD5 generated after the body is sent matches up.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b",
headers = {
["TE"] = "trailers",
}
}
local body = res:read_body()
local hash = ngx.md5(body)
res:read_trailers()
if res.headers["Content-MD5"] == hash then
ngx.say("OK")
else
ngx.say(res.headers["Content-MD5"])
end
';
}
location = /b {
content_by_lua '
-- We use the raw socket to compose a response, since OpenResty
-- doesnt support trailers natively.
ngx.req.read_body()
local sock, err = ngx.req.socket(true)
if not sock then
ngx.say(err)
end
local res = {}
table.insert(res, "HTTP/1.1 200 OK")
table.insert(res, "Date: " .. ngx.http_time(ngx.time()))
table.insert(res, "Transfer-Encoding: chunked")
table.insert(res, "Trailer: Content-MD5")
table.insert(res, "")
local body = "Hello, World"
table.insert(res, string.format("%x", #body))
table.insert(res, body)
table.insert(res, "0")
table.insert(res, "")
table.insert(res, "Content-MD5: " .. ngx.md5(body))
table.insert(res, "")
table.insert(res, "")
sock:send(table.concat(res, "\\r\\n"))
';
}
--- request
GET /a
--- response_body
OK
--- no_error_log
[error]
[warn]
=== TEST 2: Advertised trailer does not exist, handled gracefully.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b",
headers = {
["TE"] = "trailers",
}
}
local body = res:read_body()
local hash = ngx.md5(body)
res:read_trailers()
ngx.say("OK")
httpc:close()
';
}
location = /b {
content_by_lua '
-- We use the raw socket to compose a response, since OpenResty
-- doesnt support trailers natively.
ngx.req.read_body()
local sock, err = ngx.req.socket(true)
if not sock then
ngx.say(err)
end
local res = {}
table.insert(res, "HTTP/1.1 200 OK")
table.insert(res, "Date: " .. ngx.http_time(ngx.time()))
table.insert(res, "Transfer-Encoding: chunked")
table.insert(res, "Trailer: Content-MD5")
table.insert(res, "")
local body = "Hello, World"
table.insert(res, string.format("%x", #body))
table.insert(res, body)
table.insert(res, "0")
table.insert(res, "")
table.insert(res, "")
sock:send(table.concat(res, "\\r\\n"))
';
}
--- request
GET /a
--- response_body
OK
--- no_error_log
[error]
[warn]

566
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/t/05-stream.t vendored
View file

@ -1,566 +0,0 @@
# vim:set ft= ts=4 sw=4 et:
use Test::Nginx::Socket;
use Cwd qw(cwd);
plan tests => repeat_each() * (blocks() * 4) - 1;
my $pwd = cwd();
our $HttpConfig = qq{
lua_package_path "$pwd/lib/?.lua;;";
};
$ENV{TEST_NGINX_RESOLVER} = '8.8.8.8';
no_long_string();
#no_diff();
run_tests();
__DATA__
=== TEST 1: Chunked streaming body reader returns the right content length.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b",
}
local chunks = {}
repeat
local chunk = res.body_reader()
if chunk then
table.insert(chunks, chunk)
end
until not chunk
local body = table.concat(chunks)
ngx.say(#body)
ngx.say(res.headers["Transfer-Encoding"])
httpc:close()
';
}
location = /b {
content_by_lua '
local len = 32768
local t = {}
for i=1,len do
t[i] = 0
end
ngx.print(table.concat(t))
';
}
--- request
GET /a
--- response_body
32768
chunked
--- no_error_log
[error]
[warn]
=== TEST 2: Non-Chunked streaming body reader returns the right content length.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b",
}
local chunks = {}
repeat
local chunk = res.body_reader()
if chunk then
table.insert(chunks, chunk)
end
until not chunk
local body = table.concat(chunks)
ngx.say(#body)
ngx.say(res.headers["Transfer-Encoding"])
ngx.say(#chunks)
httpc:close()
';
}
location = /b {
chunked_transfer_encoding off;
content_by_lua '
local len = 32768
local t = {}
for i=1,len do
t[i] = 0
end
ngx.print(table.concat(t))
';
}
--- request
GET /a
--- response_body
32768
nil
1
--- no_error_log
[error]
[warn]
=== TEST 2b: Non-Chunked streaming body reader, buffer size becomes nil
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b",
}
local chunks = {}
local buffer_size = 16384
repeat
local chunk = res.body_reader(buffer_size)
if chunk then
table.insert(chunks, chunk)
end
buffer_size = nil
until not chunk
local body = table.concat(chunks)
ngx.say(res.headers["Transfer-Encoding"])
httpc:close()
';
}
location = /b {
chunked_transfer_encoding off;
content_by_lua '
local len = 32768
local t = {}
for i=1,len do
t[i] = 0
end
ngx.print(table.concat(t))
';
}
--- request
GET /a
--- response_body
nil
--- error_log
Buffer size not specified, bailing
=== TEST 3: HTTP 1.0 body reader with no max size returns the right content length.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b",
version = 1.0,
}
local chunks = {}
repeat
local chunk = res.body_reader()
if chunk then
table.insert(chunks, chunk)
end
until not chunk
local body = table.concat(chunks)
ngx.say(#body)
ngx.say(res.headers["Transfer-Encoding"])
ngx.say(#chunks)
httpc:close()
';
}
location = /b {
chunked_transfer_encoding off;
content_by_lua '
local len = 32768
local t = {}
for i=1,len do
t[i] = 0
end
ngx.print(table.concat(t))
';
}
--- request
GET /a
--- response_body
32768
nil
1
--- no_error_log
[error]
[warn]
=== TEST 4: HTTP 1.0 body reader with max chunk size returns the right content length.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b",
version = 1.0,
}
local chunks = {}
local size = 8192
repeat
local chunk = res.body_reader(size)
if chunk then
table.insert(chunks, chunk)
end
size = size + size
until not chunk
local body = table.concat(chunks)
ngx.say(#body)
ngx.say(res.headers["Transfer-Encoding"])
ngx.say(#chunks)
httpc:close()
';
}
location = /b {
chunked_transfer_encoding off;
content_by_lua '
local len = 32769
local t = {}
for i=1,len do
t[i] = 0
end
ngx.print(table.concat(t))
';
}
--- request
GET /a
--- response_body
32769
nil
3
--- no_error_log
[error]
[warn]
=== TEST 4b: HTTP 1.0 body reader with no content length, stream works as expected.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b",
version = 1.0,
}
local chunks = {}
local size = 8192
repeat
local chunk = res.body_reader(size)
if chunk then
table.insert(chunks, chunk)
end
size = size + size
until not chunk
local body = table.concat(chunks)
ngx.say(#body)
ngx.say(#chunks)
httpc:close()
';
}
location = /b {
content_by_lua '
ngx.req.read_body()
local sock, err = ngx.req.socket(true)
if not sock then
ngx.say(err)
end
local res = {}
table.insert(res, "HTTP/1.0 200 OK")
table.insert(res, "Date: " .. ngx.http_time(ngx.time()))
table.insert(res, "")
local len = 32769
local t = {}
for i=1,len do
t[i] = 0
end
table.insert(res, table.concat(t))
sock:send(table.concat(res, "\\r\\n"))
';
}
--- request
GET /a
--- response_body
32769
3
--- no_error_log
[error]
[warn]
=== TEST 5: Chunked streaming body reader with max chunk size returns the right content length.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b",
}
local chunks = {}
local size = 8192
repeat
local chunk = res.body_reader(size)
if chunk then
table.insert(chunks, chunk)
end
size = size + size
until not chunk
local body = table.concat(chunks)
ngx.say(#body)
ngx.say(res.headers["Transfer-Encoding"])
ngx.say(#chunks)
httpc:close()
';
}
location = /b {
content_by_lua '
local len = 32768
local t = {}
for i=1,len do
t[i] = 0
end
ngx.print(table.concat(t))
';
}
--- request
GET /a
--- response_body
32768
chunked
3
--- no_error_log
[error]
[warn]
=== TEST 6: Request reader correctly reads body
--- http_config eval: $::HttpConfig
--- config
location = /a {
lua_need_request_body off;
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
local reader, err = httpc:get_client_body_reader(8192)
repeat
local chunk, err = reader()
if chunk then
ngx.print(chunk)
end
until chunk == nil
';
}
--- request
POST /a
foobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbaz
--- response_body: foobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbaz
--- no_error_log
[error]
[warn]
=== TEST 7: Request reader correctly reads body in chunks
--- http_config eval: $::HttpConfig
--- config
location = /a {
lua_need_request_body off;
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
local reader, err = httpc:get_client_body_reader(64)
local chunks = 0
repeat
chunks = chunks +1
local chunk, err = reader()
if chunk then
ngx.print(chunk)
end
until chunk == nil
ngx.say("\\n"..chunks)
';
}
--- request
POST /a
foobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbaz
--- response_body
foobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbaz
3
--- no_error_log
[error]
[warn]
=== TEST 8: Request reader passes into client
--- http_config eval: $::HttpConfig
--- config
location = /a {
lua_need_request_body off;
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local reader, err = httpc:get_client_body_reader(64)
local res, err = httpc:request{
method = POST,
path = "/b",
body = reader,
headers = ngx.req.get_headers(100, true),
}
local body = res:read_body()
ngx.say(body)
httpc:close()
';
}
location = /b {
content_by_lua '
ngx.req.read_body()
local body, err = ngx.req.get_body_data()
ngx.print(body)
';
}
--- request
POST /a
foobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbaz
--- response_body
foobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbaz
--- no_error_log
[error]
[warn]
=== TEST 9: Body reader is a function returning nil when no body is present.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b",
method = "HEAD",
}
repeat
local chunk = res.body_reader()
until not chunk
httpc:close()
';
}
location = /b {
content_by_lua '
ngx.exit(200)
';
}
--- request
GET /a
--- no_error_log
[error]
[warn]
=== TEST 10: Issue a notice (but do not error) if trying to read the request body in a subrequest
--- http_config eval: $::HttpConfig
--- config
location = /a {
echo_location /b;
}
location = /b {
lua_need_request_body off;
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
local reader, err = httpc:get_client_body_reader(8192)
if not reader then
ngx.log(ngx.NOTICE, err)
return
end
repeat
local chunk, err = reader()
if chunk then
ngx.print(chunk)
end
until chunk == nil
';
}
--- request
POST /a
foobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbazfoobarbaz
--- response_body:
--- no_error_log
[error]
[warn]
--- error_log
attempt to read the request body in a subrequest

145
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/t/06-simpleinterface.t vendored
View file

@ -1,145 +0,0 @@
# vim:set ft= ts=4 sw=4 et:
use Test::Nginx::Socket;
use Cwd qw(cwd);
plan tests => repeat_each() * (blocks() * 4) + 6;
my $pwd = cwd();
our $HttpConfig = qq{
lua_package_path "$pwd/lib/?.lua;;";
error_log logs/error.log debug;
};
$ENV{TEST_NGINX_RESOLVER} = '8.8.8.8';
no_long_string();
#no_diff();
run_tests();
__DATA__
=== TEST 1: Simple URI interface
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
local res, err = httpc:request_uri("http://127.0.0.1:"..ngx.var.server_port.."/b?a=1&b=2")
if not res then
ngx.log(ngx.ERR, err)
end
ngx.status = res.status
ngx.header["X-Header-A"] = res.headers["X-Header-A"]
ngx.header["X-Header-B"] = res.headers["X-Header-B"]
ngx.print(res.body)
';
}
location = /b {
content_by_lua '
for k,v in pairs(ngx.req.get_uri_args()) do
ngx.header["X-Header-" .. string.upper(k)] = v
end
ngx.say("OK")
';
}
--- request
GET /a
--- response_headers
X-Header-A: 1
X-Header-B: 2
--- response_body
OK
--- no_error_log
[error]
[warn]
=== TEST 2: Simple URI interface HTTP 1.0
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
local res, err = httpc:request_uri(
"http://127.0.0.1:"..ngx.var.server_port.."/b?a=1&b=2", {
}
)
ngx.status = res.status
ngx.header["X-Header-A"] = res.headers["X-Header-A"]
ngx.header["X-Header-B"] = res.headers["X-Header-B"]
ngx.print(res.body)
';
}
location = /b {
content_by_lua '
for k,v in pairs(ngx.req.get_uri_args()) do
ngx.header["X-Header-" .. string.upper(k)] = v
end
ngx.say("OK")
';
}
--- request
GET /a
--- response_headers
X-Header-A: 1
X-Header-B: 2
--- response_body
OK
--- no_error_log
[error]
[warn]
=== TEST 3 Simple URI interface, params override
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
local res, err = httpc:request_uri(
"http://127.0.0.1:"..ngx.var.server_port.."/b?a=1&b=2", {
path = "/c",
query = {
a = 2,
b = 3,
},
}
)
ngx.status = res.status
ngx.header["X-Header-A"] = res.headers["X-Header-A"]
ngx.header["X-Header-B"] = res.headers["X-Header-B"]
ngx.print(res.body)
';
}
location = /c {
content_by_lua '
for k,v in pairs(ngx.req.get_uri_args()) do
ngx.header["X-Header-" .. string.upper(k)] = v
end
ngx.say("OK")
';
}
--- request
GET /a
--- response_headers
X-Header-A: 2
X-Header-B: 3
--- response_body
OK
--- no_error_log
[error]
[warn]

240
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/t/07-keepalive.t vendored
View file

@ -1,240 +0,0 @@
# vim:set ft= ts=4 sw=4 et:
use Test::Nginx::Socket;
use Cwd qw(cwd);
plan tests => repeat_each() * (blocks() * 4);
my $pwd = cwd();
our $HttpConfig = qq{
lua_package_path "$pwd/lib/?.lua;;";
error_log logs/error.log debug;
};
$ENV{TEST_NGINX_RESOLVER} = '8.8.8.8';
no_long_string();
#no_diff();
run_tests();
__DATA__
=== TEST 1 Simple interface, Connection: Keep-alive. Test the connection is reused.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
local res, err = httpc:request_uri(
"http://127.0.0.1:"..ngx.var.server_port.."/b", {
}
)
ngx.say(res.headers["Connection"])
httpc:connect("127.0.0.1", ngx.var.server_port)
ngx.say(httpc:get_reused_times())
';
}
location = /b {
content_by_lua '
ngx.say("OK")
';
}
--- request
GET /a
--- response_body
keep-alive
1
--- no_error_log
[error]
[warn]
=== TEST 2 Simple interface, Connection: close, test we don't try to keepalive, but also that subsequent connections can keepalive.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
local res, err = httpc:request_uri(
"http://127.0.0.1:"..ngx.var.server_port.."/b", {
version = 1.0,
headers = {
["Connection"] = "close",
},
}
)
httpc:connect("127.0.0.1", ngx.var.server_port)
ngx.say(httpc:get_reused_times())
httpc:set_keepalive()
httpc:connect("127.0.0.1", ngx.var.server_port)
ngx.say(httpc:get_reused_times())
';
}
location = /b {
content_by_lua '
ngx.say("OK")
';
}
--- request
GET /a
--- response_body
0
1
--- no_error_log
[error]
[warn]
=== TEST 3 Generic interface, Connection: Keep-alive. Test the connection is reused.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b"
}
local body = res:read_body()
ngx.say(res.headers["Connection"])
ngx.say(httpc:set_keepalive())
httpc:connect("127.0.0.1", ngx.var.server_port)
ngx.say(httpc:get_reused_times())
';
}
location = /b {
content_by_lua '
ngx.say("OK")
';
}
--- request
GET /a
--- response_body
keep-alive
1
1
--- no_error_log
[error]
[warn]
=== TEST 4 Generic interface, Connection: Close. Test we don't try to keepalive, but also that subsequent connections can keepalive.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
version = 1.0,
headers = {
["Connection"] = "Close",
},
path = "/b"
}
local body = res:read_body()
ngx.say(res.headers["Connection"])
local r, e = httpc:set_keepalive()
ngx.say(r)
ngx.say(e)
httpc:connect("127.0.0.1", ngx.var.server_port)
ngx.say(httpc:get_reused_times())
httpc:set_keepalive()
httpc:connect("127.0.0.1", ngx.var.server_port)
ngx.say(httpc:get_reused_times())
';
}
location = /b {
content_by_lua '
ngx.say("OK")
';
}
--- request
GET /a
--- response_body
close
2
connection must be closed
0
1
--- no_error_log
[error]
[warn]
=== TEST 5: Generic interface, HTTP 1.0, no connection header. Test we don't try to keepalive, but also that subsequent connections can keepalive.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", 12345)
local res, err = httpc:request{
version = 1.0,
path = "/b"
}
local body = res:read_body()
ngx.print(body)
ngx.say(res.headers["Connection"])
local r, e = httpc:set_keepalive()
ngx.say(r)
ngx.say(e)
httpc:connect("127.0.0.1", ngx.var.server_port)
ngx.say(httpc:get_reused_times())
httpc:set_keepalive()
httpc:connect("127.0.0.1", ngx.var.server_port)
ngx.say(httpc:get_reused_times())
';
}
location = /b {
content_by_lua '
ngx.say("OK")
';
}
--- request
GET /a
--- tcp_listen: 12345
--- tcp_reply
HTTP/1.0 200 OK
Date: Fri, 08 Aug 2016 08:12:31 GMT
Server: OpenResty
OK
--- response_body
OK
nil
2
connection must be closed
0
1
--- no_error_log
[error]
[warn]

143
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/t/08-pipeline.t vendored
View file

@ -1,143 +0,0 @@
# vim:set ft= ts=4 sw=4 et:
use Test::Nginx::Socket;
use Cwd qw(cwd);
plan tests => repeat_each() * (blocks() * 4);
my $pwd = cwd();
our $HttpConfig = qq{
lua_package_path "$pwd/lib/?.lua;;";
error_log logs/error.log debug;
};
$ENV{TEST_NGINX_RESOLVER} = '8.8.8.8';
no_long_string();
#no_diff();
run_tests();
__DATA__
=== TEST 1 Test that pipelined reqests can be read correctly.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local responses = httpc:request_pipeline{
{
path = "/b",
},
{
path = "/c",
},
{
path = "/d",
}
}
for i,r in ipairs(responses) do
if r.status then
ngx.say(r.status)
ngx.say(r.headers["X-Res"])
ngx.say(r:read_body())
end
end
';
}
location = /b {
content_by_lua '
ngx.status = 200
ngx.header["X-Res"] = "B"
ngx.print("B")
';
}
location = /c {
content_by_lua '
ngx.status = 404
ngx.header["X-Res"] = "C"
ngx.print("C")
';
}
location = /d {
content_by_lua '
ngx.status = 200
ngx.header["X-Res"] = "D"
ngx.print("D")
';
}
--- request
GET /a
--- response_body
200
B
B
404
C
C
200
D
D
--- no_error_log
[error]
[warn]
=== TEST 2: Test we can handle timeouts on reading the pipelined requests.
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
httpc:set_timeout(1)
local responses = httpc:request_pipeline{
{
path = "/b",
},
{
path = "/c",
},
}
for i,r in ipairs(responses) do
if r.status then
ngx.say(r.status)
ngx.say(r.headers["X-Res"])
ngx.say(r:read_body())
end
end
';
}
location = /b {
content_by_lua '
ngx.status = 200
ngx.header["X-Res"] = "B"
ngx.print("B")
';
}
location = /c {
content_by_lua '
ngx.status = 404
ngx.header["X-Res"] = "C"
ngx.sleep(1)
ngx.print("C")
';
}
--- request
GET /a
--- response_body
200
B
B
--- no_error_log
[warn]
--- error_log eval
[qr/timeout/]

59
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/t/09-ssl.t vendored
View file

@ -1,59 +0,0 @@
# vim:set ft= ts=4 sw=4 et:
use Test::Nginx::Socket;
use Cwd qw(cwd);
plan tests => repeat_each() * (blocks() * 4);
my $pwd = cwd();
our $HttpConfig = qq{
lua_package_path "$pwd/lib/?.lua;;";
error_log logs/error.log debug;
};
$ENV{TEST_NGINX_RESOLVER} = '8.8.8.8';
no_long_string();
#no_diff();
run_tests();
__DATA__
=== TEST 1: parse_uri returns port 443 for https URIs
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
local parsed = httpc:parse_uri("https://www.google.com/foobar")
ngx.say(parsed[3])
';
}
--- request
GET /a
--- response_body
443
--- no_error_log
[error]
[warn]
=== TEST 2: parse_uri returns port 80 for http URIs
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
local parsed = httpc:parse_uri("http://www.google.com/foobar")
ngx.say(parsed[3])
';
}
--- request
GET /a
--- response_body
80
--- no_error_log
[error]
[warn]

57
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/t/10-clientbodyreader.t vendored
View file

@ -1,57 +0,0 @@
# vim:set ft= ts=4 sw=4 et:
use Test::Nginx::Socket;
use Cwd qw(cwd);
plan tests => repeat_each() * (blocks() * 4);
my $pwd = cwd();
our $HttpConfig = qq{
lua_package_path "$pwd/lib/?.lua;;";
};
$ENV{TEST_NGINX_RESOLVER} = '8.8.8.8';
no_long_string();
#no_diff();
run_tests();
__DATA__
=== TEST 1: Issue a notice (but do not error) if trying to read the request body in a subrequest
--- http_config eval: $::HttpConfig
--- config
location = /a {
echo_location /b;
}
location = /b {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/c",
headers = {
["Content-Type"] = "application/x-www-form-urlencoded",
}
}
if not res then
ngx.say(err)
end
ngx.print(res:read_body())
httpc:close()
';
}
location /c {
echo "OK";
}
--- request
GET /a
--- response_body
OK
--- no_error_log
[error]
[warn]

152
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/t/11-proxy.t vendored
View file

@ -1,152 +0,0 @@
# vim:set ft= ts=4 sw=4 et:
use Test::Nginx::Socket;
use Cwd qw(cwd);
plan tests => repeat_each() * (blocks() * 5);
my $pwd = cwd();
our $HttpConfig = qq{
lua_package_path "$pwd/lib/?.lua;;";
error_log logs/error.log debug;
};
$ENV{TEST_NGINX_RESOLVER} = '8.8.8.8';
no_long_string();
#no_diff();
run_tests();
__DATA__
=== TEST 1: Proxy GET request and response
--- http_config eval: $::HttpConfig
--- config
location = /a_prx {
rewrite ^(.*)_prx$ $1 break;
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
httpc:proxy_response(httpc:proxy_request())
httpc:set_keepalive()
';
}
location = /a {
content_by_lua '
ngx.status = 200
ngx.header["X-Test"] = "foo"
ngx.say("OK")
';
}
--- request
GET /a_prx
--- response_body
OK
--- response_headers
X-Test: foo
--- error_code: 200
--- no_error_log
[error]
[warn]
=== TEST 2: Proxy POST request and response
--- http_config eval: $::HttpConfig
--- config
location = /a_prx {
rewrite ^(.*)_prx$ $1 break;
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
httpc:proxy_response(httpc:proxy_request())
httpc:set_keepalive()
';
}
location = /a {
lua_need_request_body on;
content_by_lua '
ngx.status = 404
ngx.header["X-Test"] = "foo"
local args, err = ngx.req.get_post_args()
ngx.say(args["foo"])
ngx.say(args["hello"])
';
}
--- request
POST /a_prx
foo=bar&hello=world
--- response_body
bar
world
--- response_headers
X-Test: foo
--- error_code: 404
--- no_error_log
[error]
[warn]
=== TEST 3: Proxy multiple headers
--- http_config eval: $::HttpConfig
--- config
location = /a_prx {
rewrite ^(.*)_prx$ $1 break;
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
httpc:proxy_response(httpc:proxy_request())
httpc:set_keepalive()
';
}
location = /a {
content_by_lua '
ngx.status = 200
ngx.header["Set-Cookie"] = { "cookie1", "cookie2" }
ngx.say("OK")
';
}
--- request
GET /a_prx
--- response_body
OK
--- raw_response_headers_like: .*Set-Cookie: cookie1\r\nSet-Cookie: cookie2\r\n
--- error_code: 200
--- no_error_log
[error]
[warn]
=== TEST 4: Proxy still works with spaces in URI
--- http_config eval: $::HttpConfig
--- config
location = "/a_ b_prx" {
rewrite ^(.*)_prx$ $1 break;
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
httpc:proxy_response(httpc:proxy_request())
httpc:set_keepalive()
';
}
location = "/a_ b" {
content_by_lua '
ngx.status = 200
ngx.header["X-Test"] = "foo"
ngx.say("OK")
';
}
--- request
GET /a_%20b_prx
--- response_body
OK
--- response_headers
X-Test: foo
--- error_code: 200
--- no_error_log
[error]
[warn]

160
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/t/12-case_insensitive_headers.t vendored
View file

@ -1,160 +0,0 @@
# vim:set ft= ts=4 sw=4 et:
use Test::Nginx::Socket;
use Cwd qw(cwd);
plan tests => repeat_each() * (blocks() * 4);
my $pwd = cwd();
our $HttpConfig = qq{
lua_package_path "$pwd/lib/?.lua;;";
error_log logs/error.log debug;
};
$ENV{TEST_NGINX_RESOLVER} = '8.8.8.8';
no_long_string();
#no_diff();
run_tests();
__DATA__
=== TEST 1: Test header normalisation
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http_headers = require "resty.http_headers"
local headers = http_headers.new()
headers.x_a_header = "a"
headers["x-b-header"] = "b"
headers["X-C-Header"] = "c"
headers["X_d-HEAder"] = "d"
ngx.say(headers["X-A-Header"])
ngx.say(headers.x_b_header)
for k,v in pairs(headers) do
ngx.say(k, ": ", v)
end
';
}
--- request
GET /a
--- response_body
a
b
x-b-header: b
x-a-header: a
X-d-HEAder: d
X-C-Header: c
--- no_error_log
[error]
[warn]
=== TEST 2: Test headers can be accessed in all cases
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b"
}
ngx.status = res.status
ngx.say(res.headers["X-Foo-Header"])
ngx.say(res.headers["x-fOo-heaDeR"])
ngx.say(res.headers.x_foo_header)
httpc:close()
';
}
location = /b {
content_by_lua '
ngx.header["X-Foo-Header"] = "bar"
ngx.say("OK")
';
}
--- request
GET /a
--- response_body
bar
bar
bar
--- no_error_log
[error]
[warn]
=== TEST 3: Test request headers are normalised
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
httpc:connect("127.0.0.1", ngx.var.server_port)
local res, err = httpc:request{
path = "/b",
headers = {
["uSeR-AgENT"] = "test_user_agent",
x_foo = "bar",
},
}
ngx.status = res.status
ngx.print(res:read_body())
httpc:close()
';
}
location = /b {
content_by_lua '
ngx.say(ngx.req.get_headers()["User-Agent"])
ngx.say(ngx.req.get_headers()["X-Foo"])
';
}
--- request
GET /a
--- response_body
test_user_agent
bar
--- no_error_log
[error]
=== TEST 4: Test that headers remain unique
--- http_config eval: $::HttpConfig
--- config
location = /a {
content_by_lua '
local http_headers = require "resty.http_headers"
local headers = http_headers.new()
headers["x-a-header"] = "a"
headers["X-A-HEAder"] = "b"
for k,v in pairs(headers) do
ngx.log(ngx.DEBUG, k, ": ", v)
ngx.header[k] = v
end
';
}
--- request
GET /a
--- response_headers
x-a-header: b
--- no_error_log
[error]
[warn]
[warn]

52
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/t/13-default-path.t vendored
View file

@ -1,52 +0,0 @@
# vim:set ft= ts=4 sw=4 et:
use Test::Nginx::Socket;
use Cwd qw(cwd);
plan tests => repeat_each() * (blocks() * 3);
my $pwd = cwd();
our $HttpConfig = qq{
lua_package_path "$pwd/lib/?.lua;;";
error_log logs/error.log debug;
};
$ENV{TEST_NGINX_RESOLVER} = '8.8.8.8';
no_long_string();
#no_diff();
run_tests();
__DATA__
=== TEST 1: request_uri (check the default path)
--- http_config eval: $::HttpConfig
--- config
location /lua {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
local res, err = httpc:request_uri("http://127.0.0.1:"..ngx.var.server_port)
if res and 200 == res.status then
ngx.say("OK")
else
ngx.say("FAIL")
end
';
}
location =/ {
content_by_lua '
ngx.print("OK")
';
}
--- request
GET /lua
--- response_body
OK
--- no_error_log
[error]

161
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/t/14-host-header.t vendored
View file

@ -1,161 +0,0 @@
# vim:set ft= ts=4 sw=4 et:
use Test::Nginx::Socket;
use Cwd qw(cwd);
plan tests => repeat_each() * (blocks() * 3);
my $pwd = cwd();
our $HttpConfig = qq{
lua_package_path "$pwd/lib/?.lua;;";
error_log logs/error.log debug;
resolver 8.8.8.8;
};
$ENV{TEST_NGINX_RESOLVER} = '8.8.8.8';
$ENV{TEST_NGINX_PWD} ||= $pwd;
sub read_file {
my $infile = shift;
open my $in, $infile
or die "cannot open $infile for reading: $!";
my $cert = do { local $/; <$in> };
close $in;
$cert;
}
our $TestCertificate = read_file("t/cert/test.crt");
our $TestCertificateKey = read_file("t/cert/test.key");
no_long_string();
#no_diff();
run_tests();
__DATA__
=== TEST 1: Default HTTP port is not added to Host header
--- http_config eval: $::HttpConfig
--- config
location /lua {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
local res, err = httpc:request_uri("http://www.google.com")
';
}
--- request
GET /lua
--- no_error_log
[error]
--- error_log
Host: www.google.com
=== TEST 2: Default HTTPS port is not added to Host header
--- http_config eval: $::HttpConfig
--- config
location /lua {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
local res, err = httpc:request_uri("https://www.google.com:443", { ssl_verify = false })
';
}
--- request
GET /lua
--- no_error_log
[error]
--- error_log
Host: www.google.com
=== TEST 3: Non-default HTTP port is added to Host header
--- http_config
lua_package_path "$TEST_NGINX_PWD/lib/?.lua;;";
error_log logs/error.log debug;
resolver 8.8.8.8;
server {
listen *:8080;
}
--- config
location /lua {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
local res, err = httpc:request_uri("http://127.0.0.1:8080")
';
}
--- request
GET /lua
--- no_error_log
[error]
--- error_log
Host: 127.0.0.1:8080
=== TEST 4: Non-default HTTPS port is added to Host header
--- http_config
lua_package_path "$TEST_NGINX_PWD/lib/?.lua;;";
error_log logs/error.log debug;
resolver 8.8.8.8;
server {
listen *:8080;
listen *:8081 ssl;
ssl_certificate ../html/test.crt;
ssl_certificate_key ../html/test.key;
}
--- config
location /lua {
content_by_lua '
local http = require "resty.http"
local httpc = http.new()
local res, err = httpc:request_uri("https://127.0.0.1:8081", { ssl_verify = false })
';
}
--- user_files eval
">>> test.key
$::TestCertificateKey
>>> test.crt
$::TestCertificate"
--- request
GET /lua
--- no_error_log
[error]
--- error_log
Host: 127.0.0.1:8081
=== TEST 5: No host header on a unix domain socket returns a useful error.
--- http_config eval: $::HttpConfig
--- config
location /a {
content_by_lua_block {
local http = require "resty.http"
local httpc = http.new()
local res, err = httpc:connect("unix:test.sock")
if not res then
ngx.log(ngx.ERR, err)
end
local res, err = httpc:request({ path = "/" })
if not res then
ngx.say(err)
else
ngx.say(res:read_body())
end
}
}
--- tcp_listen: test.sock
--- tcp_reply: OK
--- request
GET /a
--- no_error_log
[error]
--- response_body
Unable to generate a useful Host header for a unix domain socket. Please provide one.

24
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/t/cert/test.crt vendored
View file

@ -1,24 +0,0 @@
-----BEGIN CERTIFICATE-----
MIID8DCCAtigAwIBAgIJALL9eJPZ6neGMA0GCSqGSIb3DQEBBQUAMFgxCzAJBgNV
BAYTAkdCMQ0wCwYDVQQIEwRUZXN0MQ0wCwYDVQQHEwRUZXN0MQ0wCwYDVQQKEwRU
ZXN0MQ0wCwYDVQQLEwRUZXN0MQ0wCwYDVQQDEwR0ZXN0MB4XDTE1MTAyMTE2MjQ1
NloXDTE1MTEyMDE2MjQ1NlowWDELMAkGA1UEBhMCR0IxDTALBgNVBAgTBFRlc3Qx
DTALBgNVBAcTBFRlc3QxDTALBgNVBAoTBFRlc3QxDTALBgNVBAsTBFRlc3QxDTAL
BgNVBAMTBHRlc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDz/AoE
c+TPdm+Aqcchq8fLNWksFQZqbsCBGnq8rUG1b6MsVlAOkDUQGRlNPs9v0/+pzgX7
IYXPCFcV7YONNsTUfvBYTq43mfOycmAdb3SX6kBygxdhYsDRZR+vCAIkjoRmRB20
meh1motqM58spq3IcT8VADTRJl1OI48VTnxmXdCtmkOymU948DcauMoxm03eL/hU
6eniNEujbnbB305noNG0W5c3h6iz9CvqUAD1kwyjick+f1atB2YYn1bymA+db6YN
3iTo0v2raWmIc7D+qqpkNaCRxgMb2HN6X3/SfkijtNJidjqHMbs2ftlKJ5/lODPZ
rCPQOcYK6TT8MIZ1AgMBAAGjgbwwgbkwHQYDVR0OBBYEFFUC1GrAhUp7IvJH5iyf
+fJQliEIMIGJBgNVHSMEgYEwf4AUVQLUasCFSnsi8kfmLJ/58lCWIQihXKRaMFgx
CzAJBgNVBAYTAkdCMQ0wCwYDVQQIEwRUZXN0MQ0wCwYDVQQHEwRUZXN0MQ0wCwYD
VQQKEwRUZXN0MQ0wCwYDVQQLEwRUZXN0MQ0wCwYDVQQDEwR0ZXN0ggkAsv14k9nq
d4YwDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQUFAAOCAQEAtaUQOr3Qn87KXmmP
GbSvCLSl+bScE09VYZsYaB6iq0pGN9y+Vh4/HjBUUsFexopw1dY25MEEJXEVi1xV
2krLYAsfKCM6c1QBVmdqfVuxUvxpXwr+CNRNAlzz6PhjkeY/Ds/j4sg7EqN8hMmT
gu8GuogX7+ZCgrzRSMMclWej+W8D1xSIuCC+rqv4w9SZdtVb3XGpCyizpTNsQAuV
ACXvq9KXkEEj+XNvKrNdWd4zG715RdMnVm+WM53d9PLp63P+4/kwhwHULYhXygQ3
DzzVPaojBBdw3VaHbbPHnv73FtAzOb7ky6zJ01DlmEPxEahCFpklMkY9T2uCdpj9
oOzaNA==
-----END CERTIFICATE-----

27
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/t/cert/test.key vendored
View file

@ -1,27 +0,0 @@
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA8/wKBHPkz3ZvgKnHIavHyzVpLBUGam7AgRp6vK1BtW+jLFZQ
DpA1EBkZTT7Pb9P/qc4F+yGFzwhXFe2DjTbE1H7wWE6uN5nzsnJgHW90l+pAcoMX
YWLA0WUfrwgCJI6EZkQdtJnodZqLajOfLKatyHE/FQA00SZdTiOPFU58Zl3QrZpD
splPePA3GrjKMZtN3i/4VOnp4jRLo252wd9OZ6DRtFuXN4eos/Qr6lAA9ZMMo4nJ
Pn9WrQdmGJ9W8pgPnW+mDd4k6NL9q2lpiHOw/qqqZDWgkcYDG9hzel9/0n5Io7TS
YnY6hzG7Nn7ZSief5Tgz2awj0DnGCuk0/DCGdQIDAQABAoIBAGjKc7L94+SHRdTJ
FtILacCJrCZW0W6dKulIajbnYzV+QWMlnzTiEyha31ciBw5My54u8rqt5z7Ioj60
yK+6OkfaTXhgMsuGv/iAz29VE4q7/fow+7XEKHTHLhiLJAB3hb42u1t6TzFTs1Vl
3pPa8wEIQsPOVuENzT1mYGoST7PW+LBIMr9ScMnRHfC0MNdV/ntQiXideOAd5PkA
4O7fNgYZ8CTAZ8rOLYTMFF76/c/jLiqfeghqbIhqMykk36kd7Lud//FRykVsn1aJ
REUva/SjVEth5kITot1hpMC4SIElWpha2YxiiZFoSXSaUbtHpymiUGV01cYtMWk0
MZ5HN3ECgYEA/74U8DpwPxd4up9syKyNqOqrCrYnhEEC/tdU/W5wECi4y5kppjdd
88lZzICVPzk2fezYXlCO9HiSHU1UfcEsY3u16qNCvylK7Qz1OqXV/Ncj59891Q5Z
K0UBcbnrv+YD6muZuhlHEbyDPqYO091G9Gf/BbL5JIBDzg1qFO9Dh9cCgYEA9Drt
O9PJ5Sjz3mXQVtVHpwyhOVnd7CUv8a1zkUQCK5uQeaiF5kal1FIo7pLOr3KAvG0C
pXbm/TobwlfAfcERQN88aPN8Z/l1CB0oKV6ipBMD2/XLzDRtx8lpTeh/BB8jIhrz
+FDJY54HCzLfW0P5kT+Cyw51ofjziPnFdO/Z6pMCgYEAon17gEchGnUnWCwDSl2Y
hELV+jBSW02TQag/b+bDfQDiqTnfpKR5JXRBghYQveL0JH5f200EB4C0FboUfPJH
6c2ogDTLK/poiMU66tCDbeqj/adx+fTr4votOL0QdRUIV+GWAxAcf8BvA1cvBJ4L
fy60ckKM2gxFCJ6tUC/VkHECgYBoMDNAUItSnXPbrmeAg5/7naGxy6qmsP6RBUPF
9tNOMyEhJUlqAT2BJEOd8zcFFb3hpEd6uwyzfnSVJcZSX2iy2gj1ZNnvqTXJ7lZR
v7N2dz4wOd1lEgC7OCsaN1LoOThNtl3Z0uz2+FVc66jpUEhJNGThpxt7q66JArS/
vAqkzQKBgFkzqA6QpnH5KhOCoZcuLQ4MtvnNHOx1xSm2B0gKDVJzGkHexTmOJvwM
ZhHXRl9txS4icejS+AGUXNBzCWEusfhDaZpZqS6zt6UxEjMsLj/Te7z++2KQn4t/
aI77jClydW1pJvICtqm5v+sukVZvQTTJza9ujta6fj7u2s671np9
-----END RSA PRIVATE KEY-----

63
controllers/nginx/rootfs/etc/nginx/lua/vendor/lua-resty-http/util/lua-releng vendored
View file

@ -1,63 +0,0 @@
#!/usr/bin/env perl
use strict;
use warnings;
sub file_contains ($$);
my $version;
for my $file (map glob, qw{ *.lua lib/*.lua lib/*/*.lua lib/*/*/*.lua lib/*/*/*/*.lua lib/*/*/*/*/*.lua }) {
# Check the sanity of each .lua file
open my $in, $file or
die "ERROR: Can't open $file for reading: $!\n";
my $found_ver;
while (<$in>) {
my ($ver, $skipping);
if (/(?x) (?:_VERSION) \s* = .*? ([\d\.]*\d+) (.*? SKIP)?/) {
my $orig_ver = $ver = $1;
$found_ver = 1;
# $skipping = $2;
$ver =~ s{^(\d+)\.(\d{3})(\d{3})$}{join '.', int($1), int($2), int($3)}e;
warn "$file: $orig_ver ($ver)\n";
} elsif (/(?x) (?:_VERSION) \s* = \s* ([a-zA-Z_]\S*)/) {
warn "$file: $1\n";
$found_ver = 1;
last;
}
if ($ver and $version and !$skipping) {
if ($version ne $ver) {
# die "$file: $ver != $version\n";
}
} elsif ($ver and !$version) {
$version = $ver;
}
}
if (!$found_ver) {
warn "WARNING: No \"_VERSION\" or \"version\" field found in `$file`.\n";
}
close $in;
print "Checking use of Lua global variables in file $file ...\n";
system("luac -p -l $file | grep ETGLOBAL | grep -vE 'require|type|tostring|error|ngx|ndk|jit|setmetatable|getmetatable|string|table|io|os|print|tonumber|math|pcall|xpcall|unpack|pairs|ipairs|assert|module|package|coroutine|[gs]etfenv|next|select|rawset|rawget|debug'");
#file_contains($file, "attempt to write to undeclared variable");
system("grep -H -n -E --color '.{120}' $file");
}
sub file_contains ($$) {
my ($file, $regex) = @_;
open my $in, $file
or die "Cannot open $file fo reading: $!\n";
my $content = do { local $/; <$in> };
close $in;
#print "$content";
return scalar ($content =~ /$regex/);
}
if (-d 't') {
for my $file (map glob, qw{ t/*.t t/*/*.t t/*/*/*.t }) {
system(qq{grep -H -n --color -E '\\--- ?(ONLY|LAST)' $file});
}
}

736
controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl
View file

@ -1,8 +1,11 @@
{{ $all := . }}
{{ $servers := .Servers }}
{{ $cfg := .Cfg }}
{{ $IsIPV6Enabled := .IsIPV6Enabled }}
{{ $healthzURI := .HealthzURI }}
{{ $backends := .Backends }}
{{ $proxyHeaders := .ProxySetHeaders }}
{{ $addHeaders := .AddHeaders }}
daemon off;
worker_processes {{ $cfg.WorkerProcesses }};
@ -11,6 +14,10 @@ pid /run/nginx.pid;
worker_rlimit_nofile {{ .MaxOpenFiles }};
{{ end}}
{{/* http://nginx.org/en/docs/ngx_core_module.html#worker_shutdown_timeout */}}
{{/* avoid waiting too long during a reload */}}
worker_shutdown_timeout {{ $cfg.WorkerShutdownTimeout }} ;
events {
multi_accept on;
worker_connections {{ $cfg.MaxWorkerConnections }};
@ -20,14 +27,15 @@ events {
http {
{{/* we use the value of the header X-Forwarded-For to be able to use the geo_ip module */}}
{{ if $cfg.UseProxyProtocol }}
set_real_ip_from {{ $cfg.ProxyRealIPCIDR }};
real_ip_header proxy_protocol;
{{ else }}
set_real_ip_from {{ $cfg.ProxyRealIPCIDR }};
real_ip_header X-Forwarded-For;
real_ip_header {{ $cfg.ForwardedForHeader }};
{{ end }}
real_ip_recursive on;
{{ range $trusted_ip := $cfg.ProxyRealIPCIDR }}
set_real_ip_from {{ $trusted_ip }};
{{ end }}
{{/* databases used to determine the country depending on the client IP address */}}
{{/* http://nginx.org/en/docs/http/ngx_http_geoip_module.html */}}
@ -41,14 +49,11 @@ http {
vhost_traffic_status_filter_by_set_key $geoip_country_code country::*;
{{ end }}
# lua section to return proper error codes when custom pages are used
lua_package_path '.?.lua;/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/lua-resty-http/lib/?.lua;';
init_by_lua_block {
require("error_page")
}
sendfile on;
aio threads;
aio_write on;
tcp_nopush on;
tcp_nodelay on;
@ -60,8 +65,10 @@ http {
keepalive_requests {{ $cfg.KeepAliveRequests }};
client_header_buffer_size {{ $cfg.ClientHeaderBufferSize }};
client_header_timeout {{ $cfg.ClientHeaderTimeout }}s;
large_client_header_buffers {{ $cfg.LargeClientHeaderBuffers }};
client_body_buffer_size {{ $cfg.ClientBodyBufferSize }};
client_body_timeout {{ $cfg.ClientBodyTimeout }}s;
http2_max_field_size {{ $cfg.HTTP2MaxFieldSize }};
http2_max_header_size {{ $cfg.HTTP2MaxHeaderSize }};
@ -71,6 +78,9 @@ http {
server_names_hash_bucket_size {{ $cfg.ServerNameHashBucketSize }};
map_hash_bucket_size {{ $cfg.MapHashBucketSize }};
proxy_headers_hash_max_size {{ $cfg.ProxyHeadersHashMaxSize }};
proxy_headers_hash_bucket_size {{ $cfg.ProxyHeadersHashBucketSize }};
variables_hash_bucket_size {{ $cfg.VariablesHashBucketSize }};
variables_hash_max_size {{ $cfg.VariablesHashMaxSize }};
@ -88,12 +98,21 @@ http {
gzip_proxied any;
{{ end }}
# Custom headers for response
{{ range $k, $v := $addHeaders }}
add_header {{ $k }} "{{ $v }}";
{{ end }}
server_tokens {{ if $cfg.ShowServerTokens }}on{{ else }}off{{ end }};
# disable warnings
uninitialized_variable_warn off;
log_format upstreaminfo {{ if $cfg.LogFormatEscapeJson }}escape=json {{ end }}'{{ buildLogFormatUpstream $cfg }}';
# Additional available variables:
# $namespace
# $ingress_name
# $service_name
log_format upstreaminfo {{ if $cfg.LogFormatEscapeJSON }}escape=json {{ end }}'{{ buildLogFormatUpstream $cfg }}';
{{/* map urls that should not appear in access.log */}}
{{/* http://nginx.org/en/docs/http/ngx_http_log_module.html#access_log */}}
@ -106,9 +125,9 @@ http {
{{ if $cfg.DisableAccessLog }}
access_log off;
{{ else }}
access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
access_log {{ $cfg.AccessLogPath }} upstreaminfo if=$loggable;
{{ end }}
error_log /var/log/nginx/error.log {{ $cfg.ErrorLogLevel }};
error_log {{ $cfg.ErrorLogPath }} {{ $cfg.ErrorLogLevel }};
{{ buildResolvers $cfg.Resolver }}
@ -135,21 +154,28 @@ http {
'' $server_port;
}
map $pass_access_scheme $the_x_forwarded_for {
default $remote_addr;
https $proxy_protocol_addr;
map {{ buildForwardedFor $cfg.ForwardedForHeader }} $the_real_ip {
default {{ buildForwardedFor $cfg.ForwardedForHeader }};
"~*(?<ip>[0-9\.]+).*" $ip;
{{ if $cfg.UseProxyProtocol }}
'' $proxy_protocol_addr;
{{ else }}
'' $realip_remote_addr;
{{ end }}
}
map $pass_access_scheme $the_real_ip {
default $remote_addr;
https $proxy_protocol_addr;
}
# map port 442 to 443 for header X-Forwarded-Port
{{ if $all.IsSSLPassthroughEnabled }}
# map port {{ $all.ListenPorts.SSLProxy }} to 443 for header X-Forwarded-Port
map $pass_server_port $pass_port {
442 443;
{{ $all.ListenPorts.SSLProxy }} 443;
default $pass_server_port;
}
{{ else }}
map $pass_server_port $pass_port {
443 443;
default $pass_server_port;
}
{{ end }}
# Map a response error watching the header Content-Type
map $http_accept $httpAccept {
@ -167,11 +193,16 @@ http {
}
# Obtain best http host
map $http_host $best_http_host {
map $http_host $this_host {
default $http_host;
'' $host;
}
map $http_x_forwarded_host $best_http_host {
default $http_x_forwarded_host;
'' $this_host;
}
server_name_in_redirect off;
port_in_redirect off;
@ -214,37 +245,52 @@ http {
{{ range $errCode := $cfg.CustomHTTPErrors }}
error_page {{ $errCode }} = @custom_{{ $errCode }};{{ end }}
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504{{ if $cfg.RetryNonIdempotent }} non_idempotent{{ end }};
proxy_ssl_session_reuse on;
{{ if $cfg.AllowBackendServerHeader }}
proxy_pass_header Server;
{{ end }}
{{range $name, $upstream := $backends}}
upstream {{$upstream.Name}} {
{{ if eq $upstream.SessionAffinity.AffinityType "cookie" }}
sticky hash={{$upstream.SessionAffinity.CookieSessionAffinity.Hash}} name={{$upstream.SessionAffinity.CookieSessionAffinity.Name}} httponly;
{{ else }}
{{ range $name, $upstream := $backends }}
{{ if eq $upstream.SessionAffinity.AffinityType "cookie" }}
upstream sticky-{{ $upstream.Name }} {
sticky hash={{ $upstream.SessionAffinity.CookieSessionAffinity.Hash }} name={{ $upstream.SessionAffinity.CookieSessionAffinity.Name }} httponly;
{{ if (gt $cfg.UpstreamKeepaliveConnections 0) }}
keepalive {{ $cfg.UpstreamKeepaliveConnections }};
{{ end }}
{{ range $server := $upstream.Endpoints }}server {{ $server.Address | formatIP }}:{{ $server.Port }} max_fails={{ $server.MaxFails }} fail_timeout={{ $server.FailTimeout }};
{{ end }}
}
{{ end }}
upstream {{ $upstream.Name }} {
# Load balance algorithm; empty for round robin, which is the default
{{ if ne $cfg.LoadBalanceAlgorithm "round_robin" }}
{{ $cfg.LoadBalanceAlgorithm }};
{{ end }}
{{ if (gt $cfg.UpstreamKeepaliveConnections 0) }}
keepalive {{ $cfg.UpstreamKeepaliveConnections }};
{{ end }}
{{ range $server := $upstream.Endpoints }}server {{ $server.Address | formatIP }}:{{ $server.Port }} max_fails={{ $server.MaxFails }} fail_timeout={{ $server.FailTimeout }};
{{ end }}
}
{{ end }}
{{/* build the maps that will be use to validate the Whitelist */}}
{{ range $index, $server := .Servers }}
{{ range $index, $server := $servers }}
{{ range $location := $server.Locations }}
{{ $path := buildLocation $location }}
{{ if isLocationAllowed $location }}
{{ if gt (len $location.Whitelist.CIDR) 0 }}
# Deny for {{ print $server.Hostname $path }}
geo $the_real_ip {{ buildDenyVariable (print $server.Hostname "_" $path) }} {
default 1;
@ -256,225 +302,77 @@ http {
{{ end }}
{{ end }}
{{ range $rl := (filterRateLimits $servers ) }}
# Ratelimit {{ $rl.Name }}
geo $the_real_ip $whitelist_{{ $rl.ID }} {
default 0;
{{ range $ip := $rl.Whitelist }}
{{ $ip }} 1;{{ end }}
}
# Ratelimit {{ $rl.Name }}
map $whitelist_{{ $rl.ID }} $limit_{{ $rl.ID }} {
0 {{ $cfg.LimitConnZoneVariable }};
1 "";
}
{{ end }}
{{/* build all the required rate limit zones. Each annotation requires a dedicated zone */}}
{{/* 1MB -> 16 thousand 64-byte states or about 8 thousand 128-byte states */}}
{{ range $zone := (buildRateLimitZones .Servers) }}
{{ range $zone := (buildRateLimitZones $servers) }}
{{ $zone }}
{{ end }}
{{ $backlogSize := .BacklogSize }}
{{ range $index, $server := .Servers }}
{{/* Build server redirects (from/to www) */}}
{{ range $hostname, $to := .RedirectServers }}
server {
{{ range $address := $all.Cfg.BindAddressIpv4 }}
listen {{ $address }}:{{ $all.ListenPorts.HTTP }}{{ if $all.Cfg.UseProxyProtocol }} proxy_protocol{{ end }};
listen {{ $address }}:{{ if $all.IsSSLPassthroughEnabled }}{{ $all.ListenPorts.SSLProxy }} proxy_protocol{{ else }}{{ $all.ListenPorts.HTTPS }}{{ if $all.Cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ end }} ssl;
{{ else }}
listen {{ $all.ListenPorts.HTTP }}{{ if $all.Cfg.UseProxyProtocol }} proxy_protocol{{ end }};
listen {{ if $all.IsSSLPassthroughEnabled }}{{ $all.ListenPorts.SSLProxy }} proxy_protocol{{ else }}{{ $all.ListenPorts.HTTPS }}{{ if $all.Cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ end }} ssl;
{{ end }}
{{ if $IsIPV6Enabled }}
{{ range $address := $all.Cfg.BindAddressIpv6 }}
listen {{ $address }}:{{ $all.ListenPorts.HTTP }}{{ if $all.Cfg.UseProxyProtocol }} proxy_protocol{{ end }};
listen {{ $address }}:{{ if $all.IsSSLPassthroughEnabled }}{{ $all.ListenPorts.SSLProxy }} proxy_protocol{{ else }}{{ $all.ListenPorts.HTTPS }}{{ if $all.Cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ end }};
{{ else }}
listen [::]:{{ $all.ListenPorts.HTTP }}{{ if $all.Cfg.UseProxyProtocol }} proxy_protocol{{ end }};
listen [::]:{{ if $all.IsSSLPassthroughEnabled }}{{ $all.ListenPorts.SSLProxy }} proxy_protocol{{ else }}{{ $all.ListenPorts.HTTPS }}{{ if $all.Cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ end }};
{{ end }}
{{ end }}
server_name {{ $hostname }};
return 301 $scheme://{{ $to }}$request_uri;
}
{{ end }}
{{ range $index, $server := $servers }}
server {
server_name {{ $server.Hostname }};
listen 80{{ if $cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $backlogSize }}{{end}};
{{ if $IsIPV6Enabled }}listen [::]:80{{ if $cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $backlogSize }}{{ end }};{{ end }}
set $proxy_upstream_name "-";
{{/* Listen on 442 because port 443 is used in the TLS sni server */}}
{{/* This listener must always have proxy_protocol enabled, because the SNI listener forwards on source IP info in it. */}}
{{ if not (empty $server.SSLCertificate) }}listen 442 proxy_protocol{{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $backlogSize }}{{end}} ssl {{ if $cfg.UseHTTP2 }}http2{{ end }};
{{ if $IsIPV6Enabled }}{{ if not (empty $server.SSLCertificate) }}listen [::]:442 proxy_protocol{{ end }} {{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $backlogSize }}{{end}} ssl {{ if $cfg.UseHTTP2 }}http2{{ end }};{{ end }}
{{/* comment PEM sha is required to detect changes in the generated configuration and force a reload */}}
# PEM sha: {{ $server.SSLPemChecksum }}
ssl_certificate {{ $server.SSLCertificate }};
ssl_certificate_key {{ $server.SSLCertificate }};
{{ end }}
{{ if (and (not (empty $server.SSLCertificate)) $cfg.HSTS) }}
more_set_headers "Strict-Transport-Security: max-age={{ $cfg.HSTSMaxAge }}{{ if $cfg.HSTSIncludeSubdomains }}; includeSubDomains{{ end }};{{ if $cfg.HSTSPreload }} preload{{ end }}";
{{ end }}
{{ if $cfg.EnableVtsStatus }}vhost_traffic_status_filter_by_set_key $geoip_country_code country::$server_name;{{ end }}
{{ if not (empty $server.ServerSnippet) }}
{{ $server.ServerSnippet }}
{{ end }}
{{ range $location := $server.Locations }}
{{ $path := buildLocation $location }}
{{ $authPath := buildAuthLocation $location }}
{{ if not (empty $location.CertificateAuth.AuthSSLCert.CAFileName) }}
# PEM sha: {{ $location.CertificateAuth.AuthSSLCert.PemSHA }}
ssl_client_certificate {{ $location.CertificateAuth.AuthSSLCert.CAFileName }};
ssl_verify_client on;
ssl_verify_depth {{ $location.CertificateAuth.ValidationDepth }};
{{ end }}
{{ if (or $location.Redirect.ForceSSLRedirect (and (not (empty $server.SSLCertificate)) $location.Redirect.SSLRedirect)) }}
# enforce ssl on server side
if ($pass_access_scheme = http) {
return 301 https://$best_http_host$request_uri;
}
{{ end }}
{{ if not (empty $location.Redirect.AppRoot)}}
if ($uri = /) {
return 302 {{ $location.Redirect.AppRoot }};
}
{{ end }}
{{ if not (empty $authPath) }}
location = {{ $authPath }} {
internal;
set $proxy_upstream_name "internal";
{{ if not $location.ExternalAuth.SendBody }}
proxy_pass_request_body off;
proxy_set_header Content-Length "";
{{ end }}
{{ if not (empty $location.ExternalAuth.Method) }}
proxy_method {{ $location.ExternalAuth.Method }};
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
{{ end }}
proxy_pass_request_headers on;
proxy_set_header Host {{ $location.ExternalAuth.Host }};
proxy_ssl_server_name on;
client_max_body_size "{{ $location.Proxy.BodySize }}";
{{ template "SERVER" serverConfig $all $server }}
set $target {{ $location.ExternalAuth.URL }};
proxy_pass $target;
}
{{ end }}
location {{ $path }} {
set $proxy_upstream_name "{{ $location.Backend }}";
{{ if isLocationAllowed $location }}
{{ if gt (len $location.Whitelist.CIDR) 0 }}
if ({{ buildDenyVariable (print $server.Hostname "_" $path) }}) {
return 403;
}
{{ end }}
port_in_redirect {{ if $location.UsePortInRedirects }}on{{ else }}off{{ end }};
{{ if not (empty $authPath) }}
# this location requires authentication
auth_request {{ $authPath }};
{{- range $idx, $line := buildAuthResponseHeaders $location }}
{{ $line }}
{{- end }}
{{ end }}
{{ if not (empty $location.ExternalAuth.SigninURL) }}
error_page 401 = {{ $location.ExternalAuth.SigninURL }};
{{ end }}
{{/* if the location contains a rate limit annotation, create one */}}
{{ $limits := buildRateLimit $location }}
{{ range $limit := $limits }}
{{ $limit }}{{ end }}
{{ if $location.BasicDigestAuth.Secured }}
{{ if eq $location.BasicDigestAuth.Type "basic" }}
auth_basic "{{ $location.BasicDigestAuth.Realm }}";
auth_basic_user_file {{ $location.BasicDigestAuth.File }};
{{ else }}
auth_digest "{{ $location.BasicDigestAuth.Realm }}";
auth_digest_user_file {{ $location.BasicDigestAuth.File }};
{{ end }}
proxy_set_header Authorization "";
{{ end }}
{{ if $location.EnableCORS }}
{{ template "CORS" }}
{{ end }}
client_max_body_size "{{ $location.Proxy.BodySize }}";
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
{{ if not (empty $location.CertificateAuth.AuthSSLCert.CAFileName) }}
proxy_set_header ssl-client-cert $ssl_client_cert;
{{ end }}
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Real-IP $the_real_ip;
proxy_set_header X-Forwarded-For $the_x_forwarded_for;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers
{{ range $k, $v := $proxyHeaders }}
proxy_set_header {{ $k }} "{{ $v }}";
{{ end }}
proxy_connect_timeout {{ $location.Proxy.ConnectTimeout }}s;
proxy_send_timeout {{ $location.Proxy.SendTimeout }}s;
proxy_read_timeout {{ $location.Proxy.ReadTimeout }}s;
proxy_redirect off;
proxy_buffering off;
proxy_buffer_size "{{ $location.Proxy.BufferSize }}";
proxy_buffers 4 "{{ $location.Proxy.BufferSize }}";
proxy_http_version 1.1;
proxy_cookie_domain {{ $location.Proxy.CookieDomain }};
proxy_cookie_path {{ $location.Proxy.CookiePath }};
{{/* rewrite only works if the content is not compressed */}}
{{ if $location.Redirect.AddBaseURL }}
proxy_set_header Accept-Encoding "";
{{ end }}
{{/* Add any additional configuration defined */}}
{{ $location.ConfigurationSnippet }}
{{ buildProxyPass $backends $location }}
{{ else }}
#{{ $location.Denied }}
return 503;
{{ end }}
}
{{ end }}
{{ if eq $server.Hostname "_" }}
# health checks in cloud providers require the use of port 80
location {{ $healthzURI }} {
access_log off;
return 200;
}
# this is required to avoid error if nginx is being monitored
# with an external software (like sysdig)
location /nginx_status {
allow 127.0.0.1;
{{ if $IsIPV6Enabled }}allow ::1;{{ end }}
deny all;
access_log off;
stub_status on;
}
{{ end }}
{{ template "CUSTOM_ERRORS" $cfg }}
{{ template "CUSTOM_ERRORS" $all }}
}
{{ if $server.Alias }}
server {
server_name {{ $server.Alias }};
{{ template "SERVER" serverConfig $all $server }}
{{ template "CUSTOM_ERRORS" $all }}
}
{{ end }}
{{ end }}
# default server, used for NGINX healthcheck and access to nginx stats
server {
# Use the port 18080 (random value just to avoid known ports) as default port for nginx.
# Use the port {{ $all.ListenPorts.Status }} (random value just to avoid known ports) as default port for nginx.
# Changing this value requires a change in:
# https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/nginx/command.go#L104
listen 18080 default_server reuseport backlog={{ .BacklogSize }};
{{ if $IsIPV6Enabled }}listen [::]:18080 default_server reuseport backlog={{ .BacklogSize }};{{ end }}
# https://github.com/kubernetes/ingress/blob/master/controllers/nginx/pkg/cmd/controller/nginx.go
listen {{ $all.ListenPorts.Status }} default_server reuseport backlog={{ $all.BacklogSize }};
{{ if $IsIPV6Enabled }}listen [::]:{{ $all.ListenPorts.Status }} default_server reuseport backlog={{ $all.BacklogSize }};{{ end }}
set $proxy_upstream_name "-";
location {{ $healthzURI }} {
@ -494,41 +392,15 @@ http {
{{ end }}
}
# this location is used to extract nginx metrics
# using prometheus.
# TODO: enable extraction for vts module.
location /internal_nginx_status {
set $proxy_upstream_name "internal";
allow 127.0.0.1;
{{ if not $cfg.DisableIpv6 }}allow ::1;{{ end }}
deny all;
access_log off;
stub_status on;
}
location / {
set $proxy_upstream_name "upstream-default-backend";
proxy_pass http://upstream-default-backend;
}
{{ template "CUSTOM_ERRORS" $cfg }}
}
# default server for services without endpoints
server {
listen 8181;
set $proxy_upstream_name "-";
location / {
{{ if .CustomErrors }}
content_by_lua_block {
openURL(ngx.req.get_headers(0), 503)
}
{{ else }}
return 503;
proxy_set_header X-Code 404;
{{ end }}
set $proxy_upstream_name "upstream-default-backend";
proxy_pass http://upstream-default-backend;
}
{{ template "CUSTOM_ERRORS" $all }}
}
}
@ -538,51 +410,81 @@ stream {
{{ if $cfg.DisableAccessLog }}
access_log off;
{{ else }}
access_log /var/log/nginx/access.log log_stream;
access_log {{ $cfg.AccessLogPath }} log_stream;
{{ end }}
error_log /var/log/nginx/error.log;
error_log {{ $cfg.ErrorLogPath }};
# TCP services
{{ range $i, $tcpServer := .TCPBackends }}
upstream tcp-{{ $tcpServer.Backend.Namespace }}-{{ $tcpServer.Backend.Name }}-{{ $tcpServer.Backend.Port }} {
upstream tcp-{{ $tcpServer.Port }}-{{ $tcpServer.Backend.Namespace }}-{{ $tcpServer.Backend.Name }}-{{ $tcpServer.Backend.Port }} {
{{ range $j, $endpoint := $tcpServer.Endpoints }}
server {{ $endpoint.Address }}:{{ $endpoint.Port }};
{{ end }}
}
server {
listen {{ $tcpServer.Port }};
{{ if $IsIPV6Enabled }}listen [::]:{{ $tcpServer.Port }};{{ end }}
proxy_pass tcp-{{ $tcpServer.Backend.Namespace }}-{{ $tcpServer.Backend.Name }}-{{ $tcpServer.Backend.Port }};
{{ range $address := $all.Cfg.BindAddressIpv4 }}
listen {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.UseProxyProtocol }} proxy_protocol{{ end }};
{{ else }}
listen {{ $tcpServer.Port }}{{ if $tcpServer.Backend.UseProxyProtocol }} proxy_protocol{{ end }};
{{ end }}
{{ if $IsIPV6Enabled }}
{{ range $address := $all.Cfg.BindAddressIpv6 }}
listen {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.UseProxyProtocol }} proxy_protocol{{ end }};
{{ else }}
listen [::]:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.UseProxyProtocol }} proxy_protocol{{ end }};
{{ end }}
{{ end }}
proxy_timeout {{ $cfg.ProxyStreamTimeout }};
proxy_pass tcp-{{ $tcpServer.Port }}-{{ $tcpServer.Backend.Namespace }}-{{ $tcpServer.Backend.Name }}-{{ $tcpServer.Backend.Port }};
}
{{ end }}
# UDP services
{{ range $i, $udpServer := .UDPBackends }}
upstream udp-{{ $udpServer.Backend.Namespace }}-{{ $udpServer.Backend.Name }}-{{ $udpServer.Backend.Port }} {
upstream udp-{{ $udpServer.Port }}-{{ $udpServer.Backend.Namespace }}-{{ $udpServer.Backend.Name }}-{{ $udpServer.Backend.Port }} {
{{ range $j, $endpoint := $udpServer.Endpoints }}
server {{ $endpoint.Address }}:{{ $endpoint.Port }};
{{ end }}
}
server {
{{ range $address := $all.Cfg.BindAddressIpv4 }}
listen {{ $address }}:{{ $udpServer.Port }} udp;
{{ else }}
listen {{ $udpServer.Port }} udp;
{{ if $IsIPV6Enabled }}listen [::]:{{ $udpServer.Port }} udp;{{ end }}
{{ end }}
{{ if $IsIPV6Enabled }}
{{ range $address := $all.Cfg.BindAddressIpv6 }}
listen {{ $address }}:{{ $udpServer.Port }} udp;
{{ else }}
listen [::]:{{ $udpServer.Port }} udp;
{{ end }}
{{ end }}
proxy_responses 1;
proxy_pass udp-{{ $udpServer.Backend.Namespace }}-{{ $udpServer.Backend.Name }}-{{ $udpServer.Backend.Port }};
proxy_timeout {{ $cfg.ProxyStreamTimeout }};
proxy_pass udp-{{ $udpServer.Port }}-{{ $udpServer.Backend.Namespace }}-{{ $udpServer.Backend.Name }}-{{ $udpServer.Backend.Port }};
}
{{ end }}
}
{{/* definition of templates to avoid repetitions */}}
{{ define "CUSTOM_ERRORS" }}
{{ range $errCode := .CustomHTTPErrors }}
{{ $proxySetHeaders := .ProxySetHeaders }}
{{ range $errCode := .Cfg.CustomHTTPErrors }}
location @custom_{{ $errCode }} {
internal;
content_by_lua_block {
openURL(ngx.req.get_headers(0), {{ $errCode }})
}
proxy_set_header X-Code {{ $errCode }};
proxy_set_header X-Format $http_accept;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Namespace $namespace;
proxy_set_header X-Ingress-Name $ingress_name;
proxy_set_header X-Service-Name $service_name;
proxy_pass http://upstream-default-backend;
}
{{ end }}
{{ end }}
@ -595,7 +497,7 @@ stream {
# Om nom nom cookies
#
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, PUT, POST, DELETE, OPTIONS';
add_header 'Access-Control-Allow-Methods' 'GET, PUT, POST, DELETE, PATCH, OPTIONS';
#
# Custom headers and headers various browsers *should* be OK with but aren't
#
@ -621,11 +523,281 @@ stream {
if ($request_method = 'DELETE') {
set $cors_method 1;
}
if ($request_method = 'PATCH') {
set $cors_method 1;
}
if ($cors_method = 1) {
add_header 'Access-Control-Allow-Origin' '*' always;
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, PUT, POST, DELETE, OPTIONS';
add_header 'Access-Control-Allow-Methods' 'GET, PUT, POST, DELETE, PATCH, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization';
}
{{ end }}
{{/* definition of server-template to avoid repetitions with server-alias */}}
{{ define "SERVER" }}
{{ $all := .First }}
{{ $server := .Second }}
{{ range $address := $all.Cfg.BindAddressIpv4 }}
listen {{ $address }}:{{ $all.ListenPorts.HTTP }}{{ if $all.Cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $all.BacklogSize }}{{end}};
{{ else }}
listen {{ $all.ListenPorts.HTTP }}{{ if $all.Cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $all.BacklogSize }}{{end}};
{{ end }}
{{ if $all.IsIPV6Enabled }}
{{ range $address := $all.Cfg.BindAddressIpv6 }}
listen {{ $address }}:{{ $all.ListenPorts.HTTP }}{{ if $all.Cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $all.BacklogSize }}{{ end }};
{{ else }}
listen [::]:{{ $all.ListenPorts.HTTP }}{{ if $all.Cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $all.BacklogSize }}{{ end }};
{{ end }}
{{ end }}
set $proxy_upstream_name "-";
{{/* Listen on {{ $all.ListenPorts.SSLProxy }} because port {{ $all.ListenPorts.HTTPS }} is used in the TLS sni server */}}
{{/* This listener must always have proxy_protocol enabled, because the SNI listener forwards on source IP info in it. */}}
{{ if not (empty $server.SSLCertificate) }}
{{ range $address := $all.Cfg.BindAddressIpv4 }}
listen {{ $address }}:{{ if $all.IsSSLPassthroughEnabled }}{{ $all.ListenPorts.SSLProxy }} proxy_protocol {{ else }}{{ $all.ListenPorts.HTTPS }}{{ if $all.Cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ end }} {{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $all.BacklogSize }}{{end}} ssl {{ if $all.Cfg.UseHTTP2 }}http2{{ end }};
{{ else }}
listen {{ if $all.IsSSLPassthroughEnabled }}{{ $all.ListenPorts.SSLProxy }} proxy_protocol {{ else }}{{ $all.ListenPorts.HTTPS }}{{ if $all.Cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ end }} {{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $all.BacklogSize }}{{end}} ssl {{ if $all.Cfg.UseHTTP2 }}http2{{ end }};
{{ end }}
{{ if $all.IsIPV6Enabled }}
{{ range $address := $all.Cfg.BindAddressIpv6 }}
{{ if not (empty $server.SSLCertificate) }}listen {{ $address }}:{{ if $all.IsSSLPassthroughEnabled }}{{ $all.ListenPorts.SSLProxy }} proxy_protocol{{ else }}{{ $all.ListenPorts.HTTPS }}{{ if $all.Cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ end }}{{ end }} {{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $all.BacklogSize }}{{end}} ssl {{ if $all.Cfg.UseHTTP2 }}http2{{ end }};
{{ else }}
{{ if not (empty $server.SSLCertificate) }}listen [::]:{{ if $all.IsSSLPassthroughEnabled }}{{ $all.ListenPorts.SSLProxy }} proxy_protocol{{ else }}{{ $all.ListenPorts.HTTPS }}{{ if $all.Cfg.UseProxyProtocol }} proxy_protocol{{ end }}{{ end }}{{ end }} {{ if eq $server.Hostname "_"}} default_server reuseport backlog={{ $all.BacklogSize }}{{end}} ssl {{ if $all.Cfg.UseHTTP2 }}http2{{ end }};
{{ end }}
{{ end }}
{{/* comment PEM sha is required to detect changes in the generated configuration and force a reload */}}
# PEM sha: {{ $server.SSLPemChecksum }}
ssl_certificate {{ $server.SSLCertificate }};
ssl_certificate_key {{ $server.SSLCertificate }};
{{ end }}
{{ if (and (not (empty $server.SSLCertificate)) $all.Cfg.HSTS) }}
more_set_headers "Strict-Transport-Security: max-age={{ $all.Cfg.HSTSMaxAge }}{{ if $all.Cfg.HSTSIncludeSubdomains }}; includeSubDomains{{ end }};{{ if $all.Cfg.HSTSPreload }} preload{{ end }}";
{{ end }}
{{ if $all.Cfg.EnableVtsStatus }}vhost_traffic_status_filter_by_set_key $geoip_country_code country::$server_name;{{ end }}
{{ if not (empty $server.CertificateAuth.CAFileName) }}
# PEM sha: {{ $server.CertificateAuth.PemSHA }}
ssl_client_certificate {{ $server.CertificateAuth.CAFileName }};
ssl_verify_client on;
ssl_verify_depth {{ $server.CertificateAuth.ValidationDepth }};
{{ if not (empty $server.CertificateAuth.ErrorPage)}}
error_page 495 496 = {{ $server.CertificateAuth.ErrorPage }};
{{ end }}
{{ end }}
{{ if not (empty $server.ServerSnippet) }}
{{ $server.ServerSnippet }}
{{ end }}
{{ range $location := $server.Locations }}
{{ $path := buildLocation $location }}
{{ $authPath := buildAuthLocation $location }}
{{ if not (empty $location.Rewrite.AppRoot)}}
if ($uri = /) {
return 302 {{ $location.Rewrite.AppRoot }};
}
{{ end }}
{{ if not (empty $authPath) }}
location = {{ $authPath }} {
internal;
set $proxy_upstream_name "internal";
{{ if not $location.ExternalAuth.SendBody }}
proxy_pass_request_body off;
proxy_set_header Content-Length "";
{{ end }}
{{ if not (empty $location.ExternalAuth.Method) }}
proxy_method {{ $location.ExternalAuth.Method }};
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
{{ end }}
proxy_pass_request_headers on;
proxy_set_header Host {{ $location.ExternalAuth.Host }};
proxy_ssl_server_name on;
client_max_body_size "{{ $location.Proxy.BodySize }}";
{{ if isValidClientBodyBufferSize $location.ClientBodyBufferSize }}
client_body_buffer_size {{ $location.ClientBodyBufferSize }};
{{ end }}
set $target {{ $location.ExternalAuth.URL }};
proxy_pass $target;
}
{{ end }}
location {{ $path }} {
set $proxy_upstream_name "{{ buildUpstreamName $server.Hostname $all.Backends $location }}";
{{ $ing := (getIngressInformation $location.Ingress $path) }}
{{/* $ing.Metadata contains the Ingress metadata */}}
set $namespace "{{ $ing.Namespace }}";
set $ingress_name "{{ $ing.Rule }}";
set $service_name "{{ $ing.Service }}";
{{ if (or $location.Rewrite.ForceSSLRedirect (and (not (empty $server.SSLCertificate)) $location.Rewrite.SSLRedirect)) }}
# enforce ssl on server side
if ($pass_access_scheme = http) {
return 301 https://$best_http_host$request_uri;
}
{{ end }}
{{ if isLocationAllowed $location }}
{{ if gt (len $location.Whitelist.CIDR) 0 }}
if ({{ buildDenyVariable (print $server.Hostname "_" $path) }}) {
return 403;
}
{{ end }}
port_in_redirect {{ if $location.UsePortInRedirects }}on{{ else }}off{{ end }};
{{ if not (empty $authPath) }}
# this location requires authentication
auth_request {{ $authPath }};
auth_request_set $auth_cookie $upstream_http_set_cookie;
add_header Set-Cookie $auth_cookie;
{{- range $idx, $line := buildAuthResponseHeaders $location }}
{{ $line }}
{{- end }}
{{ end }}
{{ if not (empty $location.ExternalAuth.SigninURL) }}
error_page 401 = {{ buildAuthSignURL $location.ExternalAuth.SigninURL }};
{{ end }}
{{/* if the location contains a rate limit annotation, create one */}}
{{ $limits := buildRateLimit $location }}
{{ range $limit := $limits }}
{{ $limit }}{{ end }}
{{ if $location.BasicDigestAuth.Secured }}
{{ if eq $location.BasicDigestAuth.Type "basic" }}
auth_basic "{{ $location.BasicDigestAuth.Realm }}";
auth_basic_user_file {{ $location.BasicDigestAuth.File }};
{{ else }}
auth_digest "{{ $location.BasicDigestAuth.Realm }}";
auth_digest_user_file {{ $location.BasicDigestAuth.File }};
{{ end }}
proxy_set_header Authorization "";
{{ end }}
{{ if $location.EnableCORS }}
{{ template "CORS" }}
{{ end }}
{{ if not (empty $location.Redirect.URL) }}
if ($uri ~* {{ $path }}) {
return {{ $location.Redirect.Code }} {{ $location.Redirect.URL }};
}
{{ end }}
client_max_body_size "{{ $location.Proxy.BodySize }}";
{{ if isValidClientBodyBufferSize $location.ClientBodyBufferSize }}
client_body_buffer_size {{ $location.ClientBodyBufferSize }};
{{ end }}
{{/* By default use vhost as Host to upstream, but allow overrides */}}
{{ if not (empty $location.UpstreamVhost) }}
proxy_set_header Host "{{ $location.UpstreamVhost }}";
{{ else }}
proxy_set_header Host $best_http_host;
{{ end }}
# Pass the extracted client certificate to the backend
{{ if not (empty $server.CertificateAuth.CAFileName) }}
proxy_set_header ssl-client-cert $ssl_client_cert;
{{ end }}
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Real-IP $the_real_ip;
proxy_set_header X-Forwarded-For $the_real_ip;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
{{ range $k, $v := $all.ProxySetHeaders }}
proxy_set_header {{ $k }} "{{ $v }}";
{{ end }}
proxy_connect_timeout {{ $location.Proxy.ConnectTimeout }}s;
proxy_send_timeout {{ $location.Proxy.SendTimeout }}s;
proxy_read_timeout {{ $location.Proxy.ReadTimeout }}s;
proxy_redirect off;
proxy_buffering off;
proxy_buffer_size "{{ $location.Proxy.BufferSize }}";
proxy_buffers 4 "{{ $location.Proxy.BufferSize }}";
proxy_request_buffering "{{ $location.Proxy.RequestBuffering }}";
proxy_http_version 1.1;
proxy_cookie_domain {{ $location.Proxy.CookieDomain }};
proxy_cookie_path {{ $location.Proxy.CookiePath }};
# In case of errors try the next upstream server before returning an error
proxy_next_upstream {{ buildNextUpstream $location.Proxy.NextUpstream }}{{ if $all.Cfg.RetryNonIdempotent }} non_idempotent{{ end }};
{{/* rewrite only works if the content is not compressed */}}
{{ if $location.Rewrite.AddBaseURL }}
proxy_set_header Accept-Encoding "";
{{ end }}
{{/* Add any additional configuration defined */}}
{{ $location.ConfigurationSnippet }}
{{/* if we are sending the request to a custom default backend, we add the required headers */}}
{{ if (hasPrefix $location.Backend "custom-default-backend-") }}
proxy_set_header X-Code 503;
proxy_set_header X-Format $http_accept;
proxy_set_header X-Namespace $namespace;
proxy_set_header X-Ingress-Name $ingress_name;
proxy_set_header X-Service-Name $service_name;
{{ end }}
{{ buildProxyPass $server.Hostname $all.Backends $location }}
{{ else }}
# Location denied. Reason: {{ $location.Denied }}
return 503;
{{ end }}
}
{{ end }}
{{ if eq $server.Hostname "_" }}
# health checks in cloud providers require the use of port {{ $all.ListenPorts.HTTP }}
location {{ $all.HealthzURI }} {
access_log off;
return 200;
}
# this is required to avoid error if nginx is being monitored
# with an external software (like sysdig)
location /nginx_status {
allow 127.0.0.1;
{{ if $all.IsIPV6Enabled }}allow ::1;{{ end }}
deny all;
access_log off;
stub_status on;
}
{{ end }}
{{ end }}

2
controllers/nginx/rootfs/ingress-controller/clean-nginx-conf.sh
View file

@ -5,4 +5,4 @@
# first sed removes empty lines
# second sed command replaces the empty lines
sed -e 's/^ *$/\'$'\n/g' | sed -e '/^$/{N;/^\n$/d;}'
sed -e 's/\r//g' | sed -e 's/^ *$/\'$'\n/g' | sed -e '/^$/{N;/^\n$/d;}'

8
controllers/nginx/test/data/config.json
View file

@ -38,8 +38,10 @@
"sslSessionTimeout": "10m",
"useGzip": true,
"useHttp2": true,
"proxyStreamTimeout": "600s",
"vtsStatusZoneSize": "10m",
"workerProcesses": 1
"workerProcesses": 1,
"limitConnZoneVariable": "$the_real_ip"
},
"customErrors": true,
"defResolver": "",
@ -94,7 +96,7 @@
"sslRedirect": true
},
"whitelist": {
"cidr": []
"cidr": ["1.1.1.1"]
},
"proxy": {
"conectTimeout": 5,
@ -144,7 +146,7 @@
"sslRedirect": false
},
"whitelist": {
"cidr": null
"cidr": ["1.1.1.1"]
},
"proxy": {
"conectTimeout": 5,

12
core/pkg/base64/base64.go Normal file
View file

@ -0,0 +1,12 @@
package base64
import (
"encoding/base64"
"strings"
)
// Encode encodes a string to base64 removing the equals character
func Encode(s string) string {
str := base64.URLEncoding.EncodeToString([]byte(s))
return strings.Replace(str, "=", "", -1)
}

19
core/pkg/file/file.go Normal file
View file

@ -0,0 +1,19 @@
package file
import (
"crypto/sha1"
"encoding/hex"
"io/ioutil"
)
// SHA1 returns the SHA1 of a file.
func SHA1(filename string) string {
hasher := sha1.New()
s, err := ioutil.ReadFile(filename)
if err != nil {
return ""
}
hasher.Write(s)
return hex.EncodeToString(hasher.Sum(nil))
}

41
core/pkg/ingress/annotations/alias/main.go Normal file
View file

@ -0,0 +1,41 @@
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package alias
import (
extensions "k8s.io/api/extensions/v1beta1"
"k8s.io/ingress/core/pkg/ingress/annotations/parser"
)
const (
annotation = "ingress.kubernetes.io/server-alias"
)
type alias struct {
}
// NewParser creates a new Alias annotation parser
func NewParser() parser.IngressAnnotation {
return alias{}
}
// Parse parses the annotations contained in the ingress rule
// used to add an alias to the provided hosts
func (a alias) Parse(ing *extensions.Ingress) (interface{}, error) {
return parser.GetStringAnnotation(annotation, ing)
}

60
core/pkg/ingress/annotations/alias/main_test.go Normal file
View file

@ -0,0 +1,60 @@
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package alias
import (
"testing"
api "k8s.io/api/core/v1"
extensions "k8s.io/api/extensions/v1beta1"
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
func TestParse(t *testing.T) {
ap := NewParser()
if ap == nil {
t.Fatalf("expected a parser.IngressAnnotation but returned nil")
}
testCases := []struct {
annotations map[string]string
expected string
}{
{map[string]string{annotation: "www.example.com"}, "www.example.com"},
{map[string]string{annotation: "*.example.com www.example.*"}, "*.example.com www.example.*"},
{map[string]string{annotation: `~^www\d+\.example\.com$`}, `~^www\d+\.example\.com$`},
{map[string]string{annotation: ""}, ""},
{map[string]string{}, ""},
{nil, ""},
}
ing := &extensions.Ingress{
ObjectMeta: meta_v1.ObjectMeta{
Name: "foo",
Namespace: api.NamespaceDefault,
},
Spec: extensions.IngressSpec{},
}
for _, testCase := range testCases {
ing.SetAnnotations(testCase.annotations)
result, _ := ap.Parse(ing)
if result != testCase.expected {
t.Errorf("expected %v but returned %v, annotations: %s", testCase.expected, result, testCase.annotations)
}
}
}

34
core/pkg/ingress/annotations/auth/main.go
View file

@ -24,9 +24,10 @@ import (
"regexp"
"github.com/pkg/errors"
api "k8s.io/client-go/pkg/api/v1"
extensions "k8s.io/client-go/pkg/apis/extensions/v1beta1"
api "k8s.io/api/core/v1"
extensions "k8s.io/api/extensions/v1beta1"
"k8s.io/ingress/core/pkg/file"
"k8s.io/ingress/core/pkg/ingress/annotations/parser"
ing_errors "k8s.io/ingress/core/pkg/ingress/errors"
"k8s.io/ingress/core/pkg/ingress/resolver"
@ -51,6 +52,34 @@ type BasicDigest struct {
Realm string `json:"realm"`
File string `json:"file"`
Secured bool `json:"secured"`
FileSHA string `json:"fileSha"`
}
// Equal tests for equality between two BasicDigest types
func (bd1 *BasicDigest) Equal(bd2 *BasicDigest) bool {
if bd1 == bd2 {
return true
}
if bd1 == nil || bd2 == nil {
return false
}
if bd1.Type != bd2.Type {
return false
}
if bd1.Realm != bd2.Realm {
return false
}
if bd1.File != bd2.File {
return false
}
if bd1.Secured != bd2.Secured {
return false
}
if bd1.FileSHA != bd2.FileSHA {
return false
}
return true
}
type auth struct {
@ -116,6 +145,7 @@ func (a auth) Parse(ing *extensions.Ingress) (interface{}, error) {
Realm: realm,
File: passFile,
Secured: true,
FileSHA: file.SHA1(passFile),
}, nil
}

Some files were not shown because too many files have changed in this diff Show more