When the ingress controller loads certificates (new ones or following a
secret update), it performs a series of check to ensure its validity.
In our systems, we detected a case where, when the secret object is
compromised, for example when the certificate does not match the secret
key, different pods of the ingress controller are serving a different
version of the certificate.
This behaviour is due to the cache mechanism of the ingress controller,
keeping the last known certificate in case of corruption. When this
happens, old ingress-controller pods will keep serving the old one,
while new pods, by failing to load the corrupted certificates, would
use the default certificate, causing invalid certificates for its
clients.
This generates a random error on the client side, depending on the
actual pod instance it reaches.
In order to allow detecting occurences of those situations, add a metric
to expose, for all ingress controlller pods, detailed informations of
the currently loaded certificate.
This will, for example, allow setting an alert when there is a
certificate discrepency across all ingress controller pods using a query
similar to `sum(nginx_ingress_controller_ssl_certificate_info{host="name.tld"})by(serial_number)`
This also allows to catch other exceptions loading certificates (failing
to load the certificate from the k8s API, ...
Co-authored-by: Daniel Ricart <danielricart@users.noreply.github.com>
Co-authored-by: Daniel Ricart <danielricart@users.noreply.github.com>
In case some ingress have a syntax error in the snippet configuration,
the freshly generated configuration will not be reloaded to prevent tearing down existing rules.
Although, once inserted, this configuration is preventing from any other valid configuration to be inserted as it remains in the ingresses of the cluster.
To solve this problem, implement an optional validation webhook that simulates the addition of the ingress to be added together with the rest of ingresses.
In case the generated configuration is not validated by nginx, deny the insertion of the ingress.
In case certificates are mounted using kubernetes secrets, when those
changes, keys are automatically updated in the container volume, and the
controller reloads it using the filewatcher.
Related changes:
- Update vendors
- Extract useful functions to check configuration with an additional ingress
- Update documentation for validating webhook
- Add validating webhook examples
- Add a metric for each syntax check success and errors
- Add more certificate generation examples
