* clean prometheus metrics
- add new histogram metrics with consistent names
- deprecate summary metrics with inconsistent names
* update prometheus metrics tests
* remove ingress_upstream_header_seconds metric
It hasn't been released so it is safe. Use header_duration_seconds metric.
* add documentation on prometheus metrics
This commit introduces a backwards compatible command line option
--report-status-classes which will enable reporting response status classes
(2xx, 3xx..) instead of status codes in exported metrics.
* Initial work on chrooting nginx process
* More improvements in chroot
* Fix charts and some file locations
* Fix symlink on non chrooted container
* fix psp test
* Add e2e tests to chroot image
* Fix logger
* Add internal logger in controller
* Fix overlay for chrooted tests
* Fix tests
* fix boilerplates
* Fix unittest to point to the right pid
* Fix PR review
When the ingress controller loads certificates (new ones or following a
secret update), it performs a series of check to ensure its validity.
In our systems, we detected a case where, when the secret object is
compromised, for example when the certificate does not match the secret
key, different pods of the ingress controller are serving a different
version of the certificate.
This behaviour is due to the cache mechanism of the ingress controller,
keeping the last known certificate in case of corruption. When this
happens, old ingress-controller pods will keep serving the old one,
while new pods, by failing to load the corrupted certificates, would
use the default certificate, causing invalid certificates for its
clients.
This generates a random error on the client side, depending on the
actual pod instance it reaches.
In order to allow detecting occurences of those situations, add a metric
to expose, for all ingress controlller pods, detailed informations of
the currently loaded certificate.
This will, for example, allow setting an alert when there is a
certificate discrepency across all ingress controller pods using a query
similar to `sum(nginx_ingress_controller_ssl_certificate_info{host="name.tld"})by(serial_number)`
This also allows to catch other exceptions loading certificates (failing
to load the certificate from the k8s API, ...
Co-authored-by: Daniel Ricart <danielricart@users.noreply.github.com>
Co-authored-by: Daniel Ricart <danielricart@users.noreply.github.com>
* fix inconsistent-label-cardinality
for prometheus metrics: nginx_ingress_controller_requests
* add host to collectorLabels only if metricsPerHost is true
In case some ingress have a syntax error in the snippet configuration,
the freshly generated configuration will not be reloaded to prevent tearing down existing rules.
Although, once inserted, this configuration is preventing from any other valid configuration to be inserted as it remains in the ingresses of the cluster.
To solve this problem, implement an optional validation webhook that simulates the addition of the ingress to be added together with the rest of ingresses.
In case the generated configuration is not validated by nginx, deny the insertion of the ingress.
In case certificates are mounted using kubernetes secrets, when those
changes, keys are automatically updated in the container volume, and the
controller reloads it using the filewatcher.
Related changes:
- Update vendors
- Extract useful functions to check configuration with an additional ingress
- Update documentation for validating webhook
- Add validating webhook examples
- Add a metric for each syntax check success and errors
- Add more certificate generation examples
There was a goroutine started to log things upon a test that could be
ended at the time `cmd.Wait()` ends.
To solve the problem, when the sub-test ends, ensure we wait until the
command ends when ending the test
The output of `make test` before the fix shows:
```
=== RUN TestNewUDPLogListener
==================
WARNING: DATA RACE
Read at 0x00c0002a8643 by goroutine 74:
testing.(*common).logDepth()
/usr/local/go/src/testing/testing.go:629 +0x132
testing.(*common).Logf()
/usr/local/go/src/testing/testing.go:614 +0x90
k8s.io/ingress-nginx/internal/ingress/metric/collectors.TestProcessCollector.func1.1()
/go/src/k8s.io/ingress-nginx/internal/ingress/metric/collectors/process_test.go:54 +0x140
Previous write at 0x00c0002a8643 by goroutine 72:
testing.tRunner.func1()
/usr/local/go/src/testing/testing.go:856 +0x33e
testing.tRunner()
/usr/local/go/src/testing/testing.go:869 +0x17f
Goroutine 74 (running) created at:
k8s.io/ingress-nginx/internal/ingress/metric/collectors.TestProcessCollector.func1()
/go/src/k8s.io/ingress-nginx/internal/ingress/metric/collectors/process_test.go:50 +0x218
testing.tRunner()
/usr/local/go/src/testing/testing.go:865 +0x163
Goroutine 72 (finished) created at:
testing.(*T).Run()
/usr/local/go/src/testing/testing.go:916 +0x699
testing.runTests.func1()
/usr/local/go/src/testing/testing.go:1157 +0xa8
testing.tRunner()
/usr/local/go/src/testing/testing.go:865 +0x163
testing.runTests()
/usr/local/go/src/testing/testing.go:1155 +0x523
testing.(*M).Run()
/usr/local/go/src/testing/testing.go:1072 +0x2eb
main.main()
_testmain.go:52 +0x222
==================
--- PASS: TestNewUDPLogListener (0.00s)
```
after the patch:
```
=== RUN TestNewUDPLogListener
--- PASS: TestNewUDPLogListener (0.01s)
```
Change-Id: I8ea246d14f5f80b330be19dd5b8299c6762f6d6b
When serving many hosts from one nginx, the metrics may become too
numerous for Prometheus. Add a flag to disable the host label, so
that metrics are totalled across all hosts.
