Consistent prometheus metric names and documentation (#8728)
* clean prometheus metrics - add new histogram metrics with consistent names - deprecate summary metrics with inconsistent names * update prometheus metrics tests * remove ingress_upstream_header_seconds metric It hasn't been released so it is safe. Use header_duration_seconds metric. * add documentation on prometheus metrics
This commit is contained in:
Dmitry Bashkatov
GitHub
parent
7cb91ef9c1
commit
e6ff97fbbe
4 changed files with 321 additions and 102 deletions
|
|
@ -32,6 +32,7 @@ They are set in the container spec of the `ingress-nginx-controller` Deployment
|
|||
| `--ingress-class-by-name` | Define if Ingress Controller should watch for Ingress Class by Name together with Controller Class. (default false) |
|
||||
| `--internal-logger-address` | Define the internal logger address to use when chroot images is used. (default 127.0.0.1:11514) |
|
||||
| `--kubeconfig` | Path to a kubeconfig file containing authorization and API server information. |
|
||||
| `--length-buckets` | Set of buckets which will be used for prometheus histogram metrics such as RequestLength, ResponseLength. (default `[10, 20, 30, 40, 50, 60, 70, 80, 90, 100]`) |
|
||||
| `--log_backtrace_at` | when logging hits line file:N, emit a stack trace (default :0) |
|
||||
| `--log_dir` | If non-empty, write log files in this directory |
|
||||
| `--log_file` | If non-empty, use this log file |
|
||||
|
|
@ -52,6 +53,7 @@ They are set in the container spec of the `ingress-nginx-controller` Deployment
|
|||
| `--skip_headers` | If true, avoid header prefixes in the log messages |
|
||||
| `--skip_log_headers` | If true, avoid headers when opening log files |
|
||||
| `--ssl-passthrough-proxy-port` | Port to use internally for SSL Passthrough. (default 442) |
|
||||
| `--size-buckets` | Set of buckets which will be used for prometheus histogram metrics such as BytesSent. (default `[10, 100, 1000, 10000, 100000, 1e+06, 1e+07]`) |
|
||||
| `--status-port` | Port to use for the lua HTTP endpoint configuration. (default 10246) |
|
||||
| `--status-update-interval` | Time interval in seconds in which the status should check if an update is required. Default is 60 seconds (default 60) |
|
||||
| `--stderrthreshold` | logs at or above this threshold go to stderr (default 2) |
|
||||
|
|
@ -59,6 +61,7 @@ They are set in the container spec of the `ingress-nginx-controller` Deployment
|
|||
| `--sync-period` | Period at which the controller forces the repopulation of its local object stores. Disabled by default. |
|
||||
| `--sync-rate-limit` | Define the sync frequency upper limit (default 0.3) |
|
||||
| `--tcp-services-configmap` | Name of the ConfigMap containing the definition of the TCP services to expose. The key in the map indicates the external port to be used. The value is a reference to a Service in the form "namespace/name:port", where "port" can either be a port number or name. TCP ports 80 and 443 are reserved by the controller for servicing HTTP traffic. |
|
||||
| `--time-buckets` | Set of buckets which will be used for prometheus histogram metrics such as RequestTime, ResponseTime. (default `[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]`) |
|
||||
| `--udp-services-configmap` | Name of the ConfigMap containing the definition of the UDP services to expose. The key in the map indicates the external port to be used. The value is a reference to a Service in the form "namespace/name:port", where "port" can either be a port name or number. |
|
||||
| `--update-status` | Update the load-balancer status of Ingress objects this controller satisfies. Requires setting the publish-service parameter to a valid Service reference. (default true) |
|
||||
| `--update-status-on-shutdown` | Update the load-balancer status of Ingress objects when the controller shuts down. Requires the update-status parameter. (default true) |
|
||||
|
|
|
|||
|
|
@ -1,9 +1,11 @@
|
|||
# Prometheus and Grafana installation
|
||||
Two different methods to install and configure Prometheus and Grafana are described in this doc.
|
||||
- Prometheus and Grafana installation using Pod Annotations. This installs Prometheus and Grafana in the same namespace as NGINX Ingress
|
||||
- Prometheus and Grafana installation using Service Monitors. This installs Prometheus and Grafana in two different namespaces. This is the preferred method, and helm charts supports this by default.
|
||||
# Monitoring
|
||||
|
||||
Two different methods to install and configure Prometheus and Grafana are described in this doc.
|
||||
* Prometheus and Grafana installation using Pod Annotations. This installs Prometheus and Grafana in the same namespace as NGINX Ingress
|
||||
* Prometheus and Grafana installation using Service Monitors. This installs Prometheus and Grafana in two different namespaces. This is the preferred method, and helm charts supports this by default.
|
||||
|
||||
## Prometheus and Grafana installation using Pod Annotations
|
||||
|
||||
## PROMETHEUS AND GRAFANA INSTALLATION USING POD ANNOTATIONS
|
||||
This tutorial will show you how to install [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/) for scraping the metrics of the NGINX Ingress controller.
|
||||
|
||||
!!! important
|
||||
|
|
@ -179,7 +181,7 @@ According to the above example, this URL will be http://10.192.0.3:31086
|
|||
- create a ingress resource with backend as "grafana" and port as "3000"
|
||||
- Similarly, you can edit the service "prometheus-server" and add a ingress resource.
|
||||
|
||||
## PROMETHEUS AND GRAFANA INSTALLATION USING SERVICE MONITORS
|
||||
## Prometheus and Grafana installation using Service Monitors
|
||||
This document assumes you're using helm and using the kube-prometheus-stack package to install Prometheus and Grafana.
|
||||
|
||||
### Verify NGINX Ingress controller is installed
|
||||
|
|
@ -346,3 +348,148 @@ This document assumes you're using helm and using the kube-prometheus-stack pack
|
|||
|
||||
|
||||
|
||||
|
||||
## Exposed metrics
|
||||
|
||||
Prometheus metrics are exposed on port 10254.
|
||||
|
||||
### Request metrics
|
||||
|
||||
* `nginx_ingress_controller_request_duration_seconds` Histogram
|
||||
|
||||
The request processing time in milliseconds (affected by client speed)
|
||||
|
||||
nginx var: `request_time`
|
||||
|
||||
* `nginx_ingress_controller_response_duration_seconds` Histogram
|
||||
|
||||
The time spent on receiving the response from the upstream server (affected by client speed)
|
||||
|
||||
nginx var: `upstream_response_time`
|
||||
|
||||
* `nginx_ingress_controller_header_duration_seconds` Histogram
|
||||
|
||||
The time spent on receiving first header from the upstream server
|
||||
|
||||
nginx var: `upstream_header_time`
|
||||
|
||||
* `nginx_ingress_controller_connect_duration_seconds` Histogram
|
||||
|
||||
The time spent on establishing a connection with the upstream server
|
||||
|
||||
nginx var: `upstream_connect_time`
|
||||
|
||||
* `nginx_ingress_controller_response_size` Histogram
|
||||
|
||||
The response length (including request line, header, and request body)
|
||||
|
||||
nginx var: `bytes_sent`
|
||||
|
||||
* `nginx_ingress_controller_request_size` Histogram
|
||||
|
||||
The request length (including request line, header, and request body)
|
||||
|
||||
nginx var: `request_length`
|
||||
|
||||
* `nginx_ingress_controller_requests` Counter
|
||||
|
||||
The total number of client requests
|
||||
|
||||
* `nginx_ingress_controller_bytes_sent` Histogram
|
||||
|
||||
The number of bytes sent to a client. **Deprecated**, use `nginx_ingress_controller_response_size`
|
||||
|
||||
nginx var: `bytes_sent`
|
||||
|
||||
* `nginx_ingress_controller_ingress_upstream_latency_seconds` Summary
|
||||
|
||||
Upstream service latency per Ingress. **Deprecated**, use `nginx_ingress_controller_connect_duration_seconds`
|
||||
|
||||
nginx var: `upstream_connect_time`
|
||||
|
||||
```
|
||||
# HELP nginx_ingress_controller_bytes_sent The number of bytes sent to a client. DEPRECATED! Use nginx_ingress_controller_response_size
|
||||
# TYPE nginx_ingress_controller_bytes_sent histogram
|
||||
# HELP nginx_ingress_controller_connect_duration_seconds The time spent on establishing a connection with the upstream server
|
||||
# TYPE nginx_ingress_controller_connect_duration_seconds nginx_ingress_controller_connect_duration_seconds
|
||||
* HELP nginx_ingress_controller_header_duration_seconds The time spent on receiving first header from the upstream server
|
||||
# TYPE nginx_ingress_controller_header_duration_seconds histogram
|
||||
# HELP nginx_ingress_controller_ingress_upstream_latency_seconds Upstream service latency per Ingress DEPRECATED! Use nginx_ingress_controller_connect_duration_seconds
|
||||
# TYPE nginx_ingress_controller_ingress_upstream_latency_seconds summary
|
||||
# HELP nginx_ingress_controller_request_duration_seconds The request processing time in milliseconds
|
||||
# TYPE nginx_ingress_controller_request_duration_seconds histogram
|
||||
# HELP nginx_ingress_controller_request_size The request length (including request line, header, and request body)
|
||||
# TYPE nginx_ingress_controller_request_size histogram
|
||||
# HELP nginx_ingress_controller_requests The total number of client requests.
|
||||
# TYPE nginx_ingress_controller_requests counter
|
||||
# HELP nginx_ingress_controller_response_duration_seconds The time spent on receiving the response from the upstream server
|
||||
# TYPE nginx_ingress_controller_response_duration_seconds histogram
|
||||
# HELP nginx_ingress_controller_response_size The response length (including request line, header, and request body)
|
||||
# TYPE nginx_ingress_controller_response_size histogram
|
||||
```
|
||||
|
||||
|
||||
### Nginx process metrics
|
||||
```
|
||||
# HELP nginx_ingress_controller_nginx_process_connections current number of client connections with state {active, reading, writing, waiting}
|
||||
# TYPE nginx_ingress_controller_nginx_process_connections gauge
|
||||
# HELP nginx_ingress_controller_nginx_process_connections_total total number of connections with state {accepted, handled}
|
||||
# TYPE nginx_ingress_controller_nginx_process_connections_total counter
|
||||
# HELP nginx_ingress_controller_nginx_process_cpu_seconds_total Cpu usage in seconds
|
||||
# TYPE nginx_ingress_controller_nginx_process_cpu_seconds_total counter
|
||||
# HELP nginx_ingress_controller_nginx_process_num_procs number of processes
|
||||
# TYPE nginx_ingress_controller_nginx_process_num_procs gauge
|
||||
# HELP nginx_ingress_controller_nginx_process_oldest_start_time_seconds start time in seconds since 1970/01/01
|
||||
# TYPE nginx_ingress_controller_nginx_process_oldest_start_time_seconds gauge
|
||||
# HELP nginx_ingress_controller_nginx_process_read_bytes_total number of bytes read
|
||||
# TYPE nginx_ingress_controller_nginx_process_read_bytes_total counter
|
||||
# HELP nginx_ingress_controller_nginx_process_requests_total total number of client requests
|
||||
# TYPE nginx_ingress_controller_nginx_process_requests_total counter
|
||||
# HELP nginx_ingress_controller_nginx_process_resident_memory_bytes number of bytes of memory in use
|
||||
# TYPE nginx_ingress_controller_nginx_process_resident_memory_bytes gauge
|
||||
# HELP nginx_ingress_controller_nginx_process_virtual_memory_bytes number of bytes of memory in use
|
||||
# TYPE nginx_ingress_controller_nginx_process_virtual_memory_bytes gauge
|
||||
# HELP nginx_ingress_controller_nginx_process_write_bytes_total number of bytes written
|
||||
# TYPE nginx_ingress_controller_nginx_process_write_bytes_total counter
|
||||
```
|
||||
|
||||
### Controller metrics
|
||||
```
|
||||
# HELP nginx_ingress_controller_build_info A metric with a constant '1' labeled with information about the build.
|
||||
# TYPE nginx_ingress_controller_build_info gauge
|
||||
# HELP nginx_ingress_controller_check_success Cumulative number of Ingress controller syntax check operations
|
||||
# TYPE nginx_ingress_controller_check_success counter
|
||||
# HELP nginx_ingress_controller_config_hash Running configuration hash actually running
|
||||
# TYPE nginx_ingress_controller_config_hash gauge
|
||||
# HELP nginx_ingress_controller_config_last_reload_successful Whether the last configuration reload attempt was successful
|
||||
# TYPE nginx_ingress_controller_config_last_reload_successful gauge
|
||||
# HELP nginx_ingress_controller_config_last_reload_successful_timestamp_seconds Timestamp of the last successful configuration reload.
|
||||
# TYPE nginx_ingress_controller_config_last_reload_successful_timestamp_seconds gauge
|
||||
# HELP nginx_ingress_controller_ssl_certificate_info Hold all labels associated to a certificate
|
||||
# TYPE nginx_ingress_controller_ssl_certificate_info gauge
|
||||
# HELP nginx_ingress_controller_success Cumulative number of Ingress controller reload operations
|
||||
# TYPE nginx_ingress_controller_success counter
|
||||
```
|
||||
|
||||
### Admission metrics
|
||||
```
|
||||
# HELP nginx_ingress_controller_admission_config_size The size of the tested configuration
|
||||
# TYPE nginx_ingress_controller_admission_config_size gauge
|
||||
# HELP nginx_ingress_controller_admission_render_duration The processing duration of ingresses rendering by the admission controller (float seconds)
|
||||
# TYPE nginx_ingress_controller_admission_render_duration gauge
|
||||
# HELP nginx_ingress_controller_admission_render_ingresses The length of ingresses rendered by the admission controller
|
||||
# TYPE nginx_ingress_controller_admission_render_ingresses gauge
|
||||
# HELP nginx_ingress_controller_admission_roundtrip_duration The complete duration of the admission controller at the time to process a new event (float seconds)
|
||||
# TYPE nginx_ingress_controller_admission_roundtrip_duration gauge
|
||||
# HELP nginx_ingress_controller_admission_tested_duration The processing duration of the admission controller tests (float seconds)
|
||||
# TYPE nginx_ingress_controller_admission_tested_duration gauge
|
||||
# HELP nginx_ingress_controller_admission_tested_ingresses The length of ingresses processed by the admission controller
|
||||
# TYPE nginx_ingress_controller_admission_tested_ingresses gauge
|
||||
```
|
||||
|
||||
### Histogram buckets
|
||||
|
||||
You can configure buckets for histogram metrics using these command line options (here are their default values):
|
||||
* `--time-buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]`
|
||||
* `--length-buckets=[10, 20, 30, 40, 50, 60, 70, 80, 90, 100]`
|
||||
* `--size-buckets=[10, 100, 1000, 10000, 100000, 1e+06, 1e+07]`
|
||||
|
|
|
|||
|
|
@ -29,14 +29,6 @@ import (
|
|||
"k8s.io/klog/v2"
|
||||
)
|
||||
|
||||
type upstream struct {
|
||||
Latency float64 `json:"upstreamLatency"`
|
||||
HeaderTime float64 `json:"upstreamHeaderTime"`
|
||||
ResponseLength float64 `json:"upstreamResponseLength"`
|
||||
ResponseTime float64 `json:"upstreamResponseTime"`
|
||||
//Status string `json:"upstreamStatus"`
|
||||
}
|
||||
|
||||
type socketData struct {
|
||||
Host string `json:"host"`
|
||||
Status string `json:"status"`
|
||||
|
|
@ -48,7 +40,11 @@ type socketData struct {
|
|||
RequestLength float64 `json:"requestLength"`
|
||||
RequestTime float64 `json:"requestTime"`
|
||||
|
||||
upstream
|
||||
Latency float64 `json:"upstreamLatency"`
|
||||
HeaderTime float64 `json:"upstreamHeaderTime"`
|
||||
ResponseTime float64 `json:"upstreamResponseTime"`
|
||||
//ResponseLength float64 `json:"upstreamResponseLength"`
|
||||
//Status string `json:"upstreamStatus"`
|
||||
|
||||
Namespace string `json:"namespace"`
|
||||
Ingress string `json:"ingress"`
|
||||
|
|
@ -68,16 +64,15 @@ type HistogramBuckets struct {
|
|||
type SocketCollector struct {
|
||||
prometheus.Collector
|
||||
|
||||
upstreamLatency *prometheus.SummaryVec // TODO: DEPRECATED, remove
|
||||
connectTime *prometheus.HistogramVec
|
||||
headerTime *prometheus.HistogramVec
|
||||
requestTime *prometheus.HistogramVec
|
||||
requestLength *prometheus.HistogramVec
|
||||
|
||||
responseTime *prometheus.HistogramVec
|
||||
|
||||
requestLength *prometheus.HistogramVec
|
||||
responseLength *prometheus.HistogramVec
|
||||
|
||||
upstreamHeaderTime *prometheus.SummaryVec
|
||||
upstreamLatency *prometheus.SummaryVec
|
||||
|
||||
bytesSent *prometheus.HistogramVec
|
||||
bytesSent *prometheus.HistogramVec // TODO: DEPRECATED, remove
|
||||
|
||||
requests *prometheus.CounterVec
|
||||
|
||||
|
|
@ -89,8 +84,6 @@ type SocketCollector struct {
|
|||
|
||||
metricsPerHost bool
|
||||
reportStatusClasses bool
|
||||
|
||||
buckets HistogramBuckets
|
||||
}
|
||||
|
||||
var (
|
||||
|
|
@ -145,6 +138,26 @@ func NewSocketCollector(pod, namespace, class string, metricsPerHost, reportStat
|
|||
metricsPerHost: metricsPerHost,
|
||||
reportStatusClasses: reportStatusClasses,
|
||||
|
||||
connectTime: prometheus.NewHistogramVec(
|
||||
prometheus.HistogramOpts{
|
||||
Name: "connect_duration_seconds",
|
||||
Help: "The time spent on establishing a connection with the upstream server",
|
||||
Namespace: PrometheusNamespace,
|
||||
ConstLabels: constLabels,
|
||||
Buckets: buckets.TimeBuckets,
|
||||
},
|
||||
requestTags,
|
||||
),
|
||||
headerTime: prometheus.NewHistogramVec(
|
||||
prometheus.HistogramOpts{
|
||||
Name: "header_duration_seconds",
|
||||
Help: "The time spent on receiving first header from the upstream server",
|
||||
Namespace: PrometheusNamespace,
|
||||
ConstLabels: constLabels,
|
||||
Buckets: buckets.TimeBuckets,
|
||||
},
|
||||
requestTags,
|
||||
),
|
||||
responseTime: prometheus.NewHistogramVec(
|
||||
prometheus.HistogramOpts{
|
||||
Name: "response_duration_seconds",
|
||||
|
|
@ -155,17 +168,6 @@ func NewSocketCollector(pod, namespace, class string, metricsPerHost, reportStat
|
|||
},
|
||||
requestTags,
|
||||
),
|
||||
responseLength: prometheus.NewHistogramVec(
|
||||
prometheus.HistogramOpts{
|
||||
Name: "response_size",
|
||||
Help: "The response length (including request line, header, and request body)",
|
||||
Namespace: PrometheusNamespace,
|
||||
ConstLabels: constLabels,
|
||||
Buckets: buckets.LengthBuckets,
|
||||
},
|
||||
requestTags,
|
||||
),
|
||||
|
||||
requestTime: prometheus.NewHistogramVec(
|
||||
prometheus.HistogramOpts{
|
||||
Name: "request_duration_seconds",
|
||||
|
|
@ -176,6 +178,17 @@ func NewSocketCollector(pod, namespace, class string, metricsPerHost, reportStat
|
|||
},
|
||||
requestTags,
|
||||
),
|
||||
|
||||
responseLength: prometheus.NewHistogramVec(
|
||||
prometheus.HistogramOpts{
|
||||
Name: "response_size",
|
||||
Help: "The response length (including request line, header, and request body)",
|
||||
Namespace: PrometheusNamespace,
|
||||
ConstLabels: constLabels,
|
||||
Buckets: buckets.LengthBuckets,
|
||||
},
|
||||
requestTags,
|
||||
),
|
||||
requestLength: prometheus.NewHistogramVec(
|
||||
prometheus.HistogramOpts{
|
||||
Name: "request_size",
|
||||
|
|
@ -190,7 +203,7 @@ func NewSocketCollector(pod, namespace, class string, metricsPerHost, reportStat
|
|||
requests: prometheus.NewCounterVec(
|
||||
prometheus.CounterOpts{
|
||||
Name: "requests",
|
||||
Help: "The total number of client requests.",
|
||||
Help: "The total number of client requests",
|
||||
Namespace: PrometheusNamespace,
|
||||
ConstLabels: constLabels,
|
||||
},
|
||||
|
|
@ -200,7 +213,7 @@ func NewSocketCollector(pod, namespace, class string, metricsPerHost, reportStat
|
|||
bytesSent: prometheus.NewHistogramVec(
|
||||
prometheus.HistogramOpts{
|
||||
Name: "bytes_sent",
|
||||
Help: "The number of bytes sent to a client",
|
||||
Help: "DEPRECATED The number of bytes sent to a client",
|
||||
Namespace: PrometheusNamespace,
|
||||
Buckets: buckets.SizeBuckets,
|
||||
ConstLabels: constLabels,
|
||||
|
|
@ -208,21 +221,10 @@ func NewSocketCollector(pod, namespace, class string, metricsPerHost, reportStat
|
|||
requestTags,
|
||||
),
|
||||
|
||||
upstreamHeaderTime: prometheus.NewSummaryVec(
|
||||
prometheus.SummaryOpts{
|
||||
Name: "ingress_upstream_header_seconds",
|
||||
Help: "The time spent on receiving first header from the upstream server",
|
||||
Namespace: PrometheusNamespace,
|
||||
ConstLabels: constLabels,
|
||||
Objectives: defObjectives,
|
||||
},
|
||||
[]string{"ingress", "namespace", "service", "canary"},
|
||||
),
|
||||
|
||||
upstreamLatency: prometheus.NewSummaryVec(
|
||||
prometheus.SummaryOpts{
|
||||
Name: "ingress_upstream_latency_seconds",
|
||||
Help: "Upstream service latency per Ingress",
|
||||
Help: "DEPRECATED Upstream service latency per Ingress",
|
||||
Namespace: PrometheusNamespace,
|
||||
ConstLabels: constLabels,
|
||||
Objectives: defObjectives,
|
||||
|
|
@ -233,15 +235,17 @@ func NewSocketCollector(pod, namespace, class string, metricsPerHost, reportStat
|
|||
|
||||
sc.metricMapping = map[string]interface{}{
|
||||
prometheus.BuildFQName(PrometheusNamespace, "", "requests"): sc.requests,
|
||||
prometheus.BuildFQName(PrometheusNamespace, "", "request_duration_seconds"): sc.requestTime,
|
||||
prometheus.BuildFQName(PrometheusNamespace, "", "request_size"): sc.requestLength,
|
||||
|
||||
prometheus.BuildFQName(PrometheusNamespace, "", "connect_duration_seconds"): sc.connectTime,
|
||||
prometheus.BuildFQName(PrometheusNamespace, "", "header_duration_seconds"): sc.headerTime,
|
||||
prometheus.BuildFQName(PrometheusNamespace, "", "response_duration_seconds"): sc.responseTime,
|
||||
prometheus.BuildFQName(PrometheusNamespace, "", "request_duration_seconds"): sc.requestTime,
|
||||
|
||||
prometheus.BuildFQName(PrometheusNamespace, "", "request_size"): sc.requestLength,
|
||||
prometheus.BuildFQName(PrometheusNamespace, "", "response_size"): sc.responseLength,
|
||||
|
||||
prometheus.BuildFQName(PrometheusNamespace, "", "bytes_sent"): sc.bytesSent,
|
||||
|
||||
prometheus.BuildFQName(PrometheusNamespace, "", "ingress_upstream_header_seconds"): sc.upstreamHeaderTime,
|
||||
prometheus.BuildFQName(PrometheusNamespace, "", "ingress_upstream_latency_seconds"): sc.upstreamLatency,
|
||||
}
|
||||
|
||||
|
|
@ -309,6 +313,13 @@ func (sc *SocketCollector) handleMessage(msg []byte) {
|
|||
}
|
||||
|
||||
if stats.Latency != -1 {
|
||||
connectTimeMetric, err := sc.connectTime.GetMetricWith(requestLabels)
|
||||
if err != nil {
|
||||
klog.ErrorS(err, "Error fetching connect time metric")
|
||||
} else {
|
||||
connectTimeMetric.Observe(stats.Latency)
|
||||
}
|
||||
|
||||
latencyMetric, err := sc.upstreamLatency.GetMetricWith(latencyLabels)
|
||||
if err != nil {
|
||||
klog.ErrorS(err, "Error fetching latency metric")
|
||||
|
|
@ -318,7 +329,7 @@ func (sc *SocketCollector) handleMessage(msg []byte) {
|
|||
}
|
||||
|
||||
if stats.HeaderTime != -1 {
|
||||
headerTimeMetric, err := sc.upstreamHeaderTime.GetMetricWith(latencyLabels)
|
||||
headerTimeMetric, err := sc.headerTime.GetMetricWith(requestLabels)
|
||||
if err != nil {
|
||||
klog.ErrorS(err, "Error fetching header time metric")
|
||||
} else {
|
||||
|
|
@ -460,32 +471,34 @@ func (sc *SocketCollector) RemoveMetrics(ingresses []string, registry prometheus
|
|||
|
||||
// Describe implements prometheus.Collector
|
||||
func (sc SocketCollector) Describe(ch chan<- *prometheus.Desc) {
|
||||
sc.connectTime.Describe(ch)
|
||||
sc.headerTime.Describe(ch)
|
||||
sc.responseTime.Describe(ch)
|
||||
sc.requestTime.Describe(ch)
|
||||
|
||||
sc.requestLength.Describe(ch)
|
||||
sc.responseLength.Describe(ch)
|
||||
|
||||
sc.requests.Describe(ch)
|
||||
|
||||
sc.upstreamLatency.Describe(ch)
|
||||
sc.upstreamHeaderTime.Describe(ch)
|
||||
|
||||
sc.responseTime.Describe(ch)
|
||||
sc.responseLength.Describe(ch)
|
||||
|
||||
sc.bytesSent.Describe(ch)
|
||||
}
|
||||
|
||||
// Collect implements the prometheus.Collector interface.
|
||||
func (sc SocketCollector) Collect(ch chan<- prometheus.Metric) {
|
||||
sc.connectTime.Collect(ch)
|
||||
sc.headerTime.Collect(ch)
|
||||
sc.responseTime.Collect(ch)
|
||||
sc.requestTime.Collect(ch)
|
||||
|
||||
sc.requestLength.Collect(ch)
|
||||
sc.responseLength.Collect(ch)
|
||||
|
||||
sc.requests.Collect(ch)
|
||||
|
||||
sc.upstreamLatency.Collect(ch)
|
||||
sc.upstreamHeaderTime.Collect(ch)
|
||||
|
||||
sc.responseTime.Collect(ch)
|
||||
sc.responseLength.Collect(ch)
|
||||
|
||||
sc.bytesSent.Collect(ch)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -98,6 +98,8 @@ func TestCollector(t *testing.T) {
|
|||
"path":"/admin",
|
||||
"requestLength":300.0,
|
||||
"requestTime":60.0,
|
||||
"upstreamLatency":1.0,
|
||||
"upstreamHeaderTime":5.0,
|
||||
"upstreamName":"test-upstream",
|
||||
"upstreamIP":"1.1.1.1:8080",
|
||||
"upstreamResponseTime":200,
|
||||
|
|
@ -122,6 +124,8 @@ func TestCollector(t *testing.T) {
|
|||
"path":"/admin",
|
||||
"requestLength":300.0,
|
||||
"requestTime":60.0,
|
||||
"upstreamLatency":1.0,
|
||||
"upstreamHeaderTime":5.0,
|
||||
"upstreamName":"test-upstream",
|
||||
"upstreamIP":"1.1.1.1:8080",
|
||||
"upstreamResponseTime":200,
|
||||
|
|
@ -131,8 +135,44 @@ func TestCollector(t *testing.T) {
|
|||
"service":"test-app",
|
||||
"canary":""
|
||||
}]`},
|
||||
metrics: []string{"nginx_ingress_controller_response_duration_seconds"},
|
||||
metrics: []string{
|
||||
"nginx_ingress_controller_connect_duration_seconds",
|
||||
"nginx_ingress_controller_header_duration_seconds",
|
||||
"nginx_ingress_controller_response_duration_seconds",
|
||||
},
|
||||
wantBefore: `
|
||||
# HELP nginx_ingress_controller_connect_duration_seconds The time spent on establishing a connection with the upstream server
|
||||
# TYPE nginx_ingress_controller_connect_duration_seconds histogram
|
||||
nginx_ingress_controller_connect_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="0.005"} 0
|
||||
nginx_ingress_controller_connect_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="0.01"} 0
|
||||
nginx_ingress_controller_connect_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="0.025"} 0
|
||||
nginx_ingress_controller_connect_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="0.05"} 0
|
||||
nginx_ingress_controller_connect_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="0.1"} 0
|
||||
nginx_ingress_controller_connect_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="0.25"} 0
|
||||
nginx_ingress_controller_connect_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="0.5"} 0
|
||||
nginx_ingress_controller_connect_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="1"} 1
|
||||
nginx_ingress_controller_connect_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="2.5"} 1
|
||||
nginx_ingress_controller_connect_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="5"} 1
|
||||
nginx_ingress_controller_connect_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="10"} 1
|
||||
nginx_ingress_controller_connect_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="+Inf"} 1
|
||||
nginx_ingress_controller_connect_duration_seconds_sum{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200"} 1
|
||||
nginx_ingress_controller_connect_duration_seconds_count{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200"} 1
|
||||
# HELP nginx_ingress_controller_header_duration_seconds The time spent on receiving first header from the upstream server
|
||||
# TYPE nginx_ingress_controller_header_duration_seconds histogram
|
||||
nginx_ingress_controller_header_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="0.005"} 0
|
||||
nginx_ingress_controller_header_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="0.01"} 0
|
||||
nginx_ingress_controller_header_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="0.025"} 0
|
||||
nginx_ingress_controller_header_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="0.05"} 0
|
||||
nginx_ingress_controller_header_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="0.1"} 0
|
||||
nginx_ingress_controller_header_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="0.25"} 0
|
||||
nginx_ingress_controller_header_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="0.5"} 0
|
||||
nginx_ingress_controller_header_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="1"} 0
|
||||
nginx_ingress_controller_header_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="2.5"} 0
|
||||
nginx_ingress_controller_header_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="5"} 1
|
||||
nginx_ingress_controller_header_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="10"} 1
|
||||
nginx_ingress_controller_header_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="+Inf"} 1
|
||||
nginx_ingress_controller_header_duration_seconds_sum{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200"} 5
|
||||
nginx_ingress_controller_header_duration_seconds_count{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200"} 1
|
||||
# HELP nginx_ingress_controller_response_duration_seconds The time spent on receiving the response from the upstream server
|
||||
# TYPE nginx_ingress_controller_response_duration_seconds histogram
|
||||
nginx_ingress_controller_response_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200",le="0.005"} 0
|
||||
|
|
@ -164,6 +204,8 @@ func TestCollector(t *testing.T) {
|
|||
"path":"/admin",
|
||||
"requestLength":300.0,
|
||||
"requestTime":60.0,
|
||||
"upstreamLatency":1.0,
|
||||
"upstreamHeaderTime":5.0,
|
||||
"upstreamName":"test-upstream",
|
||||
"upstreamIP":"1.1.1.1:8080",
|
||||
"upstreamResponseTime":200,
|
||||
|
|
@ -175,7 +217,7 @@ func TestCollector(t *testing.T) {
|
|||
}]`},
|
||||
metrics: []string{"nginx_ingress_controller_requests"},
|
||||
wantBefore: `
|
||||
# HELP nginx_ingress_controller_requests The total number of client requests.
|
||||
# HELP nginx_ingress_controller_requests The total number of client requests
|
||||
# TYPE nginx_ingress_controller_requests counter
|
||||
nginx_ingress_controller_requests{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="200"} 1
|
||||
`,
|
||||
|
|
@ -193,6 +235,8 @@ func TestCollector(t *testing.T) {
|
|||
"path":"/admin",
|
||||
"requestLength":300.0,
|
||||
"requestTime":60.0,
|
||||
"upstreamLatency":1.0,
|
||||
"upstreamHeaderTime":5.0,
|
||||
"upstreamName":"test-upstream",
|
||||
"upstreamIP":"1.1.1.1:8080",
|
||||
"upstreamResponseTime":200,
|
||||
|
|
@ -236,6 +280,8 @@ func TestCollector(t *testing.T) {
|
|||
"path":"/admin",
|
||||
"requestLength":300.0,
|
||||
"requestTime":60.0,
|
||||
"upstreamLatency":1.0,
|
||||
"upstreamHeaderTime":5.0,
|
||||
"upstreamName":"test-upstream",
|
||||
"upstreamIP":"1.1.1.1:8080",
|
||||
"upstreamResponseTime":200,
|
||||
|
|
@ -252,6 +298,8 @@ func TestCollector(t *testing.T) {
|
|||
"path":"/admin",
|
||||
"requestLength":300.0,
|
||||
"requestTime":60.0,
|
||||
"upstreamLatency":1.0,
|
||||
"upstreamHeaderTime":5.0,
|
||||
"upstreamName":"test-upstream",
|
||||
"upstreamIP":"1.1.1.1:8080",
|
||||
"upstreamResponseTime":200,
|
||||
|
|
@ -268,6 +316,8 @@ func TestCollector(t *testing.T) {
|
|||
"path":"/admin",
|
||||
"requestLength":300.0,
|
||||
"requestTime":60.0,
|
||||
"upstreamLatency":1.0,
|
||||
"upstreamHeaderTime":5.0,
|
||||
"upstreamName":"test-upstream",
|
||||
"upstreamIP":"1.1.1.1:8080",
|
||||
"upstreamResponseTime":200,
|
||||
|
|
@ -323,6 +373,8 @@ func TestCollector(t *testing.T) {
|
|||
"path":"/admin",
|
||||
"requestLength":300.0,
|
||||
"requestTime":60.0,
|
||||
"upstreamLatency":1.0,
|
||||
"upstreamHeaderTime":5.0,
|
||||
"upstreamName":"test-upstream",
|
||||
"upstreamIP":"1.1.1.1:8080",
|
||||
"upstreamResponseTime":200,
|
||||
|
|
@ -340,6 +392,8 @@ func TestCollector(t *testing.T) {
|
|||
"path":"/admin",
|
||||
"requestLength":300.0,
|
||||
"requestTime":60.0,
|
||||
"upstreamLatency":1.0,
|
||||
"upstreamHeaderTime":5.0,
|
||||
"upstreamName":"test-upstream",
|
||||
"upstreamIP":"1.1.1.1:8080",
|
||||
"upstreamResponseTime":100,
|
||||
|
|
@ -382,6 +436,8 @@ func TestCollector(t *testing.T) {
|
|||
"path":"/admin",
|
||||
"requestLength":300.0,
|
||||
"requestTime":60.0,
|
||||
"upstreamLatency":1.0,
|
||||
"upstreamHeaderTime":5.0,
|
||||
"upstreamName":"test-upstream",
|
||||
"upstreamIP":"1.1.1.1:8080",
|
||||
"upstreamResponseTime":200,
|
||||
|
|
|
|||
Loading…
Reference in a new issue