PCP, a computer/network performance monitoring toolkit, features a separation between data collection and logging/processing/monitoring. The separation is present in installable packages, the programming API, and on the network. It makes it easy to let a sysadmin decide what data to log, stored where, originated where, at what sampling intervals. Then such decisions then need to be encoded into concrete configurations. pmmgr is a daemon included with PCP since early 2014 and can serve as a one-stop receptacle for that configuration.

As the name suggests, pmmgr is a manager: it creates, looks after, and kills pmlogger and/or pmie worker processes. (pmlogger logs specified PCP metrics to archive files on disk; pmie searches for specified conditions in live data streams to trigger alarms.) Since pmlogger can generate large data sets that require regular rotation / aging, pmmgr handles that part too. With the pmmgr service activated, there is no need to run the pmlogger/pmie pcp services separately.

The basic engine of pmmgr is based on periodic polling to identify running pmcd (pcp data collector/server) instances. It can look for them just on the local host, it can listen for their self-announcements via avahi, or it can actively scan a network near or far. Once pmcd connects to a running pmcd instance, it extracts uniquely-identifying information for its host. This information is used to rank/merge duplicate access paths for the same host (say, across different network interfaces or network protocols), and to notice whether a host is new, stable, moved, or dead. When pmmgr takes notice of pmcd lifecycle changes, it launches or stops correspondingly targeted pmlogger and/or pmie instances. The result is one pmlogger and/or pmie process running exactly matching each target host, with metric archives kept identified and managed.

configuring pmmgr

There is plenty of configuration possible. A sysadmin may not want pmie running at all. Or she may need a custom configuration for the each host logger. Or she may just want a simple common default that can be activated against a network. This needs flexibility + simplicity.

The options are selected by individual files in a configuration directory: usually /etc/pcp/pmmgr/, instead of a single parsed configuration file. It is in the style of *.d directories for other tools, or the fine tools by D. J. Bernstein, where the presence or absence of a file functions as a switch; lines of text in some files represent parameters. Each configuration directory represents a common configuration for one or more hosts. A single pmmgr instance can support multiple configuration directories at once, so can manage a heterogenous fleet of daemons.

Let's take a look at the default configuration directory.

% ls -l /etc/pcp/pmmgr
-rw-r--r-- 1 root root    0 Mar  2 20:21 pmie
-rw-r--r-- 1 root root    0 Mar  2 20:21 pmieconf
-rw-r--r-- 1 root root    0 Mar  2 20:21 pmlogconf
-rw-r--r-- 1 root root    0 Mar  2 20:21 pmlogger
-rw-r--r-- 1 root root    0 Mar  2 20:21 pmlogmerge
-rw-r--r-- 1 root root    0 Mar  2 20:21 pmlogmerge-granular
-rw-r--r-- 1 root root    0 Mar  2 20:21 pmlogmerge-rewrite
-rw-r--r-- 1 root root  604 Mar  2 20:21 pmmgr.options
-rw-r--r-- 1 root root  656 Mar  2 20:21 README
-rw-r--r-- 1 root root   18 Mar  2 20:21 target-discovery.example-avahi

Note all the empty files: that is all that might be needed for a common default.

Going into a bit more detail, the first few mean that pmmgr should start basic pmie instances for each host found, should run the pmieconf program to generate a configuration for each pmie. If one added some text to the /pmie file, those bits of text would be added to the pmie command line. If one removed the /etc/pcp/pmmgr/pmie file, no pmie would be started at all. Without the /pmieconf file, pmmgr will expect a user to specify the pmie configuration as a -c PATH_TO_FILE in the /pmie file.

The /pmlogger and /pmlogconf files are analogous for the pmlogger daemons. Remove a file, and no pmlogger. Add options to a file, and they get transcribed to the corresponding process. Normally, log archives will collect under /var/log/pcp/pmmgr/$HOSTNAME/. (The PCP web applications associated with pmwebd happen to look for historical archives under that directory. Consider installing the PCP webapps, and give http://localhost:44323/ a browse.)

The /pmlogmerge* files govern management of the log archives created by the pmlogger processes: how long each archive file should be, how old data to throw away, whether to merge old ones together into a single one or granular-periodic ones, that sort of thing. If the files are non-empty, they may override default time or size limits, or set options to subordinate log-management programs.

There are several more optional files; all are documented in the pmmgr man page. The most important ones that remain relate to designation of target hosts to connect to: /target-host and /target-discovery. The former contains a list of network host names (or IP addresses, or general PCP pmcd host specification strings that may include authentication information); pmmgr will scan them all. The latter contains a list of directives for interpretation the PCP pmDiscoverServices function that can locate pmcd instances by discovery or probing; pmmgr will scan them all.

That's all there is: no magic. Just a few files driving pmmgr to scan for pmcd instances, and to keep pmlogger and/or pmie instances attached to them, like a hungry dog seeking then latching onto some fine beefy miracle.

configuration ideas

How about a few specific interesting configurations, modifying files in the /etc/pcp/pmmgr/ directory:

centralized logging

Populate the /target-host with all the hosts near you where pmcd might run. Populate the /target-discovery file with

avahi
probe=192.168.1.0/24
  

to scan your network. Each host with a network-accessible pmcd will get its own logging subdirectory under /var/log/pcp/pmmgr/.

higher rate logging

The pmlogger default is to sample metrics every 60 seconds. If you want them more frequent, add

-t 20

to the line in the /pmlogger configuration file.

longer data retention

The log archive default retention is 14 days. If you want it longer, put

56days

into /pmlogmerge-retain.

heterogenous manual configuration

If none of the defaults work for you - if you want a custom hand-configured pmlogger, but still would like pmmgr to manage the log archives, create multiple configuration directories.

for host in h1 h2 h3; do
  d=/etc/pcp/pmmgr/$host
  mkdir $d
  touch $d/pmlogger
  echo $h1 > $d/target-host      
  echo 'log advisory on default { filesys.avail }' > $d/hand-made.conf # etc.
  echo "-c $1/hand-made.conf" > $d/pmlogger
  touch $d/pmlogmerge
  touch $d/pmlogmerge-granular
  # arrange for pmmgr service to also read each new configuration subdirectory
  echo "-c $d" >> /etc/pcp/pmmgr/pmmgr.options      
done

container logging

If you'd like all containers running on a host to automatically get treated as "sub-hosts", each with its own individualized pmlogger, wait until this patch is merged into PCP, then create one little file:

touch /etc/pcp/pmmgr/subtarget-containers

Then individual logs will start showing up under /var/log/pcp/pmmgr/$HOST--$CONTAINER_ID.

additional requests welcome!