160 lines
6.4 KiB
Markdown
160 lines
6.4 KiB
Markdown
# process-exporter
|
|
Prometheus exporter that mines /proc to report on selected processes.
|
|
|
|
The premise for this exporter is that sometimes you have apps that are
|
|
impractical to instrument directly, either because you don't control the code
|
|
or they're written in a language that isn't easy to instrument with Prometheus.
|
|
A fair bit of information can be gleaned from /proc, especially for
|
|
long-running programs.
|
|
|
|
For most systems it won't be beneficial to create metrics for every process by
|
|
name: there are just too many of them and most don't do enough to merit it.
|
|
Various command-line options are provided to control how processes are grouped
|
|
and the groups are named. Run "process-exporter -man" to see a help page
|
|
giving details.
|
|
|
|
Metrics available currently include CPU usage, bytes written and read, and
|
|
number of processes in each group.
|
|
|
|
Bytes read and written come from /proc/[pid]/io in recent enough kernels.
|
|
These correspond to the fields `read_bytes` and `write_bytes` respectively.
|
|
These IO stats come with plenty of caveats, see either the Linux kernel
|
|
documentation or man 5 proc.
|
|
|
|
CPU usage comes from /proc/[pid]/stat fields utime (user time) and stime (system
|
|
time.) It has been translated into fractional seconds of CPU consumed. Since
|
|
it is a counter, using rate() will tell you how many fractional cores were running
|
|
code from this process during the interval given.
|
|
|
|
An example Grafana dashboard to view the metrics is available at https://grafana.net/dashboards/249
|
|
|
|
## Instrumentation cost
|
|
|
|
process-exporter will consume CPU in proportion to the number of processes in
|
|
the system and the rate at which new ones are created. The most expensive
|
|
parts - applying regexps and executing templates - are only applied once per
|
|
process seen. If you have mostly long-running processes process-exporter
|
|
should be lightweight: each time a scrape occurs, parsing of /proc/$pid/stat
|
|
and /proc/$pid/cmdline for every process being monitored and adding a few
|
|
numbers.
|
|
|
|
## Config
|
|
|
|
To select and group the processes to monitor, either provide command-line
|
|
arguments or use a YAML configuration file.
|
|
|
|
To avoid confusion with the cmdline YAML element, we'll refer to the
|
|
null-delimited contents of `/proc/<pid>/cmdline` as the array `argv[]`.
|
|
|
|
Each item in `process_names` gives a recipe for identifying and naming
|
|
processes. The optional `name` tag defines a template to use to name
|
|
matching processes; if not specified, `name` defaults to `{{.ExeBase}}`.
|
|
|
|
Template variables available:
|
|
- `{{.ExeBase}}` contains the basename of the executable
|
|
- `{{.ExeFull}}` contains the fully qualified path of the executable
|
|
- `{{.Matches}}` map contains all the matches resulting from applying cmdline regexps
|
|
|
|
Each item in `process_names` must contain one or more selectors (`comm`, `exe`
|
|
or `cmdline`); if more than one selector is present, they must all match. Each
|
|
selector is a list of strings to match against a process's `comm`, `argv[0]`,
|
|
or in the case of `cmdline`, a regexp to apply to the command line.
|
|
|
|
For `comm` and `exe`, the list of strings is an OR, meaning any process
|
|
matching any of the strings will be added to the item's group.
|
|
|
|
For `cmdline`, the list of regexes is an AND, meaning they all must match. Any
|
|
capturing groups in a regexp must use the `?P<name>` option to assign a name to
|
|
the capture, which is used to populate `.Matches`.
|
|
|
|
A process may only belong to one group: even if multiple items would match, the
|
|
first one listed in the file wins.
|
|
|
|
Other performance tips: give an exe or comm clause in addition to any cmdline
|
|
clause, so you avoid executing the regexp when the executable name doesn't
|
|
match.
|
|
|
|
```
|
|
|
|
process_names:
|
|
# comm is the second field of /proc/<pid>/stat minus parens.
|
|
# It is the base executable name, truncated at 15 chars.
|
|
# It cannot be modified by the program, unlike exe.
|
|
- comm:
|
|
- bash
|
|
|
|
# exe is argv[0]. If no slashes, only basename of argv[0] need match.
|
|
# If exe contains slashes, argv[0] must match exactly.
|
|
- exe:
|
|
- postgres
|
|
- /usr/local/bin/prometheus
|
|
|
|
# cmdline is a list of regexps applied to argv.
|
|
# Each must match, and any captures are added to the .Matches map.
|
|
- name: "{{.ExeFull}}:{{.Matches.Cfgfile}}"
|
|
exe:
|
|
- /usr/local/bin/process-exporter
|
|
cmdline:
|
|
- -config.path\\s+(?P<Cfgfile>\\S+)
|
|
|
|
|
|
```
|
|
|
|
Here's the config I use on my home machine:
|
|
|
|
```
|
|
|
|
process_names:
|
|
- comm:
|
|
- chromium-browse
|
|
- bash
|
|
- prometheus
|
|
- gvim
|
|
- exe:
|
|
- /sbin/upstart
|
|
cmdline:
|
|
- --user
|
|
name: upstart:-user
|
|
|
|
```
|
|
|
|
## Docker
|
|
|
|
A docker image can be created with
|
|
|
|
```
|
|
make docker
|
|
```
|
|
|
|
Then run the docker, e.g.
|
|
|
|
```
|
|
docker run --privileged --name pexporter -d -v /proc:/host/proc -p 127.0.0.1:9256:9256 process-exporter:master -procfs /host/proc -procnames chromium-browse,bash,prometheus,gvim,upstart:-user -namemapping "upstart,(-user)"
|
|
```
|
|
|
|
This will expose metrics on http://localhost:9256/metrics. Leave off the
|
|
`127.0.0.1:` to publish on all interfaces. Leave off the --priviliged and
|
|
add the --user docker run argument if you only need to monitor processes
|
|
belonging to a single user.
|
|
|
|
## History
|
|
|
|
An earlier version of this exporter had options to enable auto-discovery of
|
|
which processes were consuming resources. This functionality has been removed.
|
|
These options were based on a percentage of resource usage, e.g. if an
|
|
untracked process consumed X% of CPU during a scrape, start tracking processes
|
|
with that name. However during any given scrape it's likely that most
|
|
processes are idle, so we could add a process that consumes minimal resources
|
|
but which happened to be active during the interval preceding the current
|
|
scrape. Over time this means that a great many processes wind up being
|
|
scraped, which becomes unmanageable to visualize. This could be mitigated by
|
|
looking at resource usage over longer intervals, but ultimately I didn't feel
|
|
this feature was important enough to invest more time in at this point. It may
|
|
re-appear at some point in the future, but no promises.
|
|
|
|
Another lost feature: the "other" group was used to count usage by non-tracked
|
|
procs. This was useful to get an idea of what wasn't being monitored. But it
|
|
comes at a high cost: if you know what processes you care about, you're wasting
|
|
a lot of CPU to compute the usage of everything else that you don't care about.
|
|
The new approach is to minimize resources expended on non-tracked processes and
|
|
to require the user to whitelist the processes to track.
|