zhmc-prometheus-exporter - A prometheus.io exporter for the IBM Z Hardware Management Console¶

Introduction¶

What this package provides¶

zhmc_prometheus_exporter adapts metrics from the IBM Z Hardware Management Console (HMC) for the Prometheus monitoring system. It is written in Python and uses the zhmcclient package.

Supported environments¶

Operating systems: It has been tested on Linux. Other operating systems are not officially supported.
Python versions: It has been tested with versions 3.4 through 3.7. Later versions are not officially supported. Earlier versions, including Python 2, are not supported.
HMC versions: 2.11.1 and higher

Installation¶

Due to the language of the exporter and the required dependencies, you are going to need Python & Pip.

Installation with pip¶

The traditional way is to install with pip. Run

$ pip3 install zhmc-prometheus-exporter

Or, locally run

$ pip3 install .

from the cloned repository.

Installation with make¶

If you have make on your system, the quickest way is to simply run

$ make install

from the cloned repository.

Install as little as possible¶

If you do not want to install the package itself, you can also just run

$ pip3 install -r requirements.txt

or, with make:

$ make setup

Instead of running zhmc_prometheus_exporter, you will then run python3 zhmc_prometheus_exporter/zhmc_prometheus_exporter.py from the repository.

Do not install at all: Run as a Docker container¶

You will have to configure it before you can run it in Docker container virtualisation. Please refer to The exporter in Docker.

Quickstart¶

The exporter itself¶

To sign into the HMC, you have to provide credentials in the YAML format. The file hmccreds.yaml could look something like this (see also the sample in the examples folder):

metrics:
  hmc: 10.10.10.10
  userid: user
  password: password

Furthermore, the file metrics.yaml defines details about the metrics fetching. An example file is in the repository root, for more information on its anatomy see chapter The metrics YAML file.

Put both of these files into /etc/zhmc-prometheus-exporter (or link them). You can then run

$ zhmc_prometheus_exporter

The default port is 9291, you can change it with -p. If you do not want to put hmccreds.yaml and metrics.yaml into /etc/zhmc-prometheus-exporter, you can also specify them with -c and -m respectively.

The exporter in Docker¶

Follow the normal quickstart guide, but ensure that your hmccreds.yaml and metrics.yaml are in the project home directory. You can then run

$ docker build . -t zhmcexporter
$ docker run -p 9291:9291 zhmcexporter

Demo setup¶

Beware that using Prometheus and a possible graphical frontend, Grafana, is not the scope of this project. This is a very sparse guide. Consult their documentations if you want anything more complicated than a “three simple metrics” setup.

The Prometheus server scrapes the metrics from the exporter. Get it from the Prometheus download page. A sample configuration YAML is provided in the examples folder. Fill in the IP and port the exporter will run on. If you left it at default, the port will be 9291. From the downloaded directory, you can then run:
```
$ ./prometheus --config.file=prometheus.yaml
```
See also Prometheus’ guide.

The Grafana server is a more versatile option to visualise the metrics scraped from metrics. Get it from Grafana. From the downloaded directory, you can then run:
```
$ ./bin/grafana-server web
```
By default it will be on localhost:3000. You will have to set IP and port of the Prometheus server. If you didn’t change it, it’s localhost:9090. See also Prometheus’ guide on Grafana.

Create the dashboard in Grafana. A sample JSON is provided in the examples folder. If you want it to work natively, you will have to name your source ZHMC_Prometheus. If you want to change something, you might find it easier to change it in the Web GUI instead of the JSON file.

The following image illustrates what the setup described above could look like.

Reporting issues¶

If you encounter a problem, please report it as an issue on GitHub.

License¶

This package is licensed under the Apache 2.0 License.

Usage¶

This page describes how to use the exporter beyond following the quickstart guide.

Revision: Quickstart¶

To sign into the HMC, you have to provide credentials in the YAML format. The file hmccreds.yaml could look something like this (see also the sample in the examples folder):

metrics:
  hmc: 10.10.10.10
  userid: user
  password: password

Furthermore, the file metrics.yaml defines details about the metrics fetching. An example file is in the repository root, for more information on its anatomy see chapter The metrics YAML file.

Put both of these files into /etc/zhmc-prometheus-exporter (or link them). You can then run

$ zhmc_prometheus_exporter

The default port is 9291, you can change it with -p. If you do not want to put hmccreds.yaml and metrics.yaml into /etc/zhmc-prometheus-exporter, you can also specify them with -c and -m respectively.

Output anatomy¶

All metrics are of the Prometheus specific type Gauge. They are grouped into prefixes, e.g. dpm for general metrics read in the Dynamic Partition Manager (DPM) mode, or lpar for logical partitions in classic mode. The suffix of a metric represents the unit of measurement; for instance, a percent value will usually end with usage_ratio, while a temperature would end in celsius.

In the case of many of the metrics, they apply to multiple devices. The dpm group can have several CPCs, the partition group will usually have several partitions, etc. These devices are separated with labels, the label being resource and the value being the CPC name, the partition name, etc.

Available metrics¶

This is an easier to read version of the relevant parts from the HMC Web Services API Documentation. On the first level, you can read the metric groups and their prefixes. They will additionally be prefixed with zhmc_. On the second level, you can read the metrics and their exporter name without the prefix.

Only in classic mode

cpc-usage-overview as cpc
logical-partition-usage as lpar
channel-usage as channel

Only in DPM mode

dpm-system-usage-overview as dpm
- network-usage as network_usage_ratio
- temperature-celsius as temperature_celsius
- storage-usage as storage_usage_ratio
- crypto-usage as crypto_usage_ratio
- processor-usage as processor_usage_ratio
- accelerator-usage as accelerator_usage_ratio
- all-shared-processor-usage as shared_processor_usage_ratio
- power-consumption-watts as power_watts
- ifl-shared-processor-usage as ifl_shared_processor_usage_ratio
- ifl-all-processor-usage as ifl_total_processor_usage_ratio
- cp-shared-processor-usage as cp_shared_usage_ratio
- cp-all-processor-usage as cp_total_usage_ratio
partition-usage as partition
- accelerator-usage as accelerator_usage_ratio
- crypto-usage as crypto_usage_ratio
- network-usage as network_usage_ratio
- processor-usage as processor_usage_ratio
- storage-usage as storage_usage_ratio
adapter-usage as adapter
crypto-usage as crypto
flash-memory-usage as flash
roce-usage as roce

Only in ensemble mode

virtualization-host-cpu-memory-usage as virtualized

The metrics YAML file¶

Various properties about scraping are collected from the metrics.yaml file that is given to the exporter with the -m option.

The metric groups section¶

contains the metric groups, as seen on the first level of the lists in Available metrics.

Example:

dpm-system-usage-overview:
  prefix: dpm
  fetch: True

Within one section, the metric prefix and the fetch True/False value is stored. Note that the former will additionally be prefixed with zhmc_. The latter is due to runtime concerns: Some metric groups take over a second to be scraped.

The metrics section¶

contains the metrics themselves, as seen on the second level of the lists in Available metrics.

Example:

dpm-system-usage-overview:
  network-usage:
    percent: True
    exporter_name: network_usage_ratio
    exporter_desc: DPM total network usage

The first level section is the metric group, the second level section is the metric. Within one metric section, a percent True/False value is stored, as well as the name and description for the exporter. The former is required because for the HMC, 100% means 100, whereas for Prometheus, 100% means 1. The latter two are requirements for an exporter, the exporter_name will be prepended with the group prefix and an underscore.

Error messages¶

This page further describes error messages if you have trouble understanding them. If an error is not listed here, see Reporting issues.

Permission error¶

Example:

$ zhmc_prometheus_exporter
Permission error. Make sure you have appropriate permissions to read from /etc/zhmc-prometheus-exporter/hmccreds.yaml.

You don’t have permission to read from a YAML file. Change the permissions with chmod, check man chmod if you are unfamiliar with it.

File not found¶

Example:

$ zhmc_prometheus_exporter
Error: File not found. It seems that /etc/zhmc-prometheus-exporter/hmccreds.yaml does not exist.

A required YAML file (hmccreds.yaml and metrics.yaml) does not exist. Make sure that you specify paths, relative or absolute, with -c or -m if the file is not in etc/zhmc-prometheus-exporter/. You have to copy the credentials file from the examples folder and fill in your own credentials, see Quickstart for more information.

Section not found¶

Example:

$ zhmc_prometheus_exporter
Section metric_groups not found in file /etc/zhmc-prometheus-exporter/metrics.yaml.

At least one of the sections metric_groups and metrics in your metrics.yaml or metrics in hmccreds.yaml is missing in its entirety. See chapter The metrics YAML file for more information.

Doesn’t follow the YAML syntax¶

Example:

$ zhmc_prometheus_exporter
/etc/zhmc-prometheus-exporter/metrics.yaml does not follow the YAML syntax

A YAML file you specified breaks the syntax rules of the YAML specification. If you derive your YAML files from the existing examples (see chapter Quickstart), this error should not occur, you can also check the YAML specification.

You did not specify¶

Example:

$ zhmc_prometheus_exporter
You did not specify the IP address of the HMC in /etc/zhmc-prometheus-exporter/hmccreds.yaml.

There is a lot of mandatory information in the two YAML files that might be missing if you improperly filled the credentials file (see Quickstart) or made bad changes to the metrics file (see The metrics YAML file).

All of these values could in some way be missing or incorrect:

In the credentials YAML file, in the section “metrics”

hmc, the IP address of the HMC (it must be a correct IP address as well!)
userid, a username for the HMC
password, the respective password

In the metrics YAML file, in the section “metric_groups”, for each metric group

prefix, the prefix for the metrics to be exported
fetch, specifying whether the group should be fetched (it must be one of True or False as well!)

In the metrics YAML file, in the section “metrics”, for each metric group

The group must also exist in the metric_groups section
percent, specifying whether the metric is a percent value (it must be one of True or False as well!)
exporter_name, the name for the exporter (minus the prefix)
exporter_desc, the mandatory description for the exporter

Time out¶

Example:

$ zhmc_prometheus_exporter
Time out. Ensure that you have access to the HMC and that you have stored the correct IP address in /etc/zhmc-prometheus-exporter/hmccreds.yaml.

There is a certain timeout threshold if the HMC cannot be found. Check that you have access to the HMC on the IP address that you specified in the credentials file.

Authentication error¶

Example:

$ zhmc_prometheus_exporter
Authentication error. Ensure that you have stored a correct user ID-password combination in /etc/zhmc-prometheus-exporter/hmccreds.yaml.

Wrong username or password in the credentials file. Check if you can regularly access the HMC with this username-password combniation.

Warning: Metric not found¶

Example:

$ zhmc_prometheus_exporter
...: UserWarning: Metric network-usage was not found. Consider adding it to your metrics.yaml.
  warnings.warn(warning_str % (metric, filename))

It might occur that within a known metric group, the HMC exposes a metric previously unknown. Some generic formatting will automatically be added, but it is recommended that you actually edit this metric in.

Development¶

This page covers the relevant aspects for developers.

Repository¶

The exporter is on GitHub.

Code of Conduct¶

Contribution must follow the Code of Conduct as defined by the Contributor Covenant.

Contributing¶

Third party contributions to this project are welcome!

In order to contribute, create a Git pull request, considering this:

Test is required.
Each commit should only contain one “logical” change.
A “logical” change should be put into one commit, and not split over multiple commits.
Large new features should be split into stages.
The commit message should not only summarize what you have done, but explain why the change is useful.
The commit message must follow the format explained below.

What comprises a “logical” change is subject to sound judgement. Sometimes, it makes sense to produce a set of commits for a feature (even if not large). For example, a first commit may introduce a (presumably) compatible API change without exploitation of that feature. With only this commit applied, it should be demonstrable that everything is still working as before. The next commit may be the exploitation of the feature in other components.

For further discussion of good and bad practices regarding commits, see:

Format of commit messages¶

A commit message must start with a short summary line, followed by a blank line.

Optionally, the summary line may start with an identifier that helps identifying the type of change or the component that is affected, followed by a colon.

It can include a more detailed description after the summary line. This is where you explain why the change was done, and summarize what was done.

It must end with the DCO (Developer Certificate of Origin) sign-off line in the format shown in the example below, using your name and a valid email address of yours. The DCO sign-off line certifies that you followed the rules stated in DCO 1.1. In short, you certify that you wrote the patch or otherwise have the right to pass it on as an open-source patch.

We use GitCop during creation of a pull request to check whether the commit messages in the pull request comply to this format. If the commit messages do not comply, GitCop will add a comment to the pull request with a description of what was wrong.

Example commit message:

cookies: Add support for delivering cookies

Cookies are important for many people. This change adds a pluggable API for
delivering cookies to the user, and provides a default implementation.

Signed-off-by: Random J Developer <random@developer.org>

Use git commit --amend to edit the commit message, if you need to.

Use the --signoff (-s) option of git commit to append a sign-off line to the commit message with your name and email as known by Git.

If you like filling out the commit message in an editor instead of using the -m option of git commit, you can automate the presence of the sign-off line by using a commit template file:

Create a file outside of the repo (say, ~/.git-signoff.template) that contains, for example:

<one-line subject>

<detailed description>

Signed-off-by: Random J Developer <random@developer.org>

Configure Git to use that file as a commit template for your repo:
```
git config commit.template ~/.git-signoff.template
```

Releasing a version¶

This section shows the steps for releasing a version to PyPI.

Switch to your work directory of the python-zhmcclient Git repo (this is where the Makefile is), and perform the following steps in that directory:

Set a shell variable for the version to be released, e.g.:
```
$ MNU='0.11.0'
```
Verify that your working directory is in a Git-wise clean state:
```
$ git status
```
Check out the master branch, and update it from upstream:
```
$ git checkout master
$ git pull
```
Create a topic branch for the release, based upon the master branch:
```
git checkout -b release-$MNU
```
Edit the change log (docs/changes.rst) and perform the following changes in the top-most section (that is the section for the version to be released):
- If needed, change the version in the section heading to the version to be released, e.g.:
```
Version 0.11.0
^^^^^^^^^^^^^^
```
- Change the release date to today’s date, e.g.:
```
Released: 2018-08-20
```
- Make sure that the change log entries reflect all changes since the previous version, and make sure they are relevant for and understandable by users.
- In the “Known issues” list item, remove the link to the issue tracker and add any known issues you want users to know about. Just linking to the issue tracker quickly becomes incorrect for released versions:
```
**Known issues:**

* ...
```
- Remove all empty list items in the change log section for this release.

Commit your changes and push them upstream:

$ git add docs/changes.rst
$ git commit -sm "Updated change log for $MNU release."
$ git push --set-upstream origin release-$MNU

On GitHub, create a pull request for branch release-$MNU.
Perform a complete test:
```
$ make test
```
This should not fail because the same tests have already been run in the Travis CI. However, run it for additional safety before the release.
- If this test fails, fix any issues until the test succeeds. Commit the changes and push them upstream:
```
$ git add <changed-files>
$ git commit -sm "<change description with details>"
$ git push
```
  Wait for the automatic tests to show success for this change.
Once the CI tests on GitHub are complete, merge the pull request.
Update your local master branch:
```
$ git checkout master
$ git pull
```
Tag the master branch with the release label and push the tag upstream:
```
$ git tag $MNU
$ git push --tags
```
On GitHub, edit the new tag, and create a release description on it. This will cause it to appear in the Release tab.

You can see the tags in GitHub via Code -> Releases -> Tags.
Upload the package to PyPI:
```
$ make upload
```
This will show the package version and will ask for confirmation.

Attention! This only works once for each version. You cannot release the same version twice to PyPI.
Verify that the released version is shown on PyPI.
On GitHub, close milestone M.N.U.

Build a package¶

You can build a binary and a source distribution with

$ make build

You will find the files zhmc_prometheus_exporter-VERSION_NUMBER-py2.py3-none-any.whl and zhmc_prometheus_exporter-VERSION_NUMBER.tar.gz in the dist folder, the former being the binary and the latter being the source distribution.

The binary could then be installed with

$ pip3 install zhmc_prometheus_exporter-VERSION_NUMBER-py2.py3-none-any.whl

The source distribution (a more minimal version of the repository) can be unpacked with

$ tar xfz zhmc_prometheus_exporter-VERSION_NUMBER.tar.gz

Build the documentation¶

You can build the documentation as HTML with

$ make builddoc

The root for the built documentation will be docs/_build/index.html.

Unit & lint testing¶

You can perform unit tests based on unittest with

$ make test

If you want to speed up test time, you can remove the timeout test.

You can perform lint tests based on flake8 with

$ make lint

Cleanup processes¶

The package can be uninstalled with

$ make uninstall

The unnecessary files from the build process can be removed with

$ make clean

Appendix¶

Glossary¶

Exporter: A server application for exposing metrics to Prometheus
IBM Z: IBM’s mainframe product line
Prometheus: A server application for monitoring and alerting
Z HMC: Hardware Management Console for IBM Z

Change log¶

Version 0.4.0¶

Released: 2019-08-21

Bug fixes: - Avoid exception in case of a connection drop error handling. - Replace yaml.load() by yaml.safe_load(). In PyYAML before 5.1,

the yaml.load() API could execute arbitrary code if used with untrusted data (CVE-2017-18342).

Known issues: See the `list of open issues`_.

Version 0.3.0¶

Released: 2019-08-11

Bug fixes: - Reconnect in case of a connection drop.

Known issues: See the `list of open issues`_.

Version 0.2.0¶

Released: 2018-08-24

Incompatible changes: All metrics now have a zhmc_ prefix.

Bug fixes: Uses Grafana 5.2.2.

Known issues: See the `list of open issues`_.

Version 0.1.2¶

Released: 2018-08-23

Enhancements: The description now instructs the user to pip3 install zhmc-prometheus-exporter instead of running a local install from the cloned repository. It also links to the stable version of the documentation rather than to the latest build.

Version 0.1.1¶

Released: 2018-08-23

Initial PyPI release (0.1.0 was for testing purposes)

Version 0.1.0¶

Released: Only on GitHub, never on PyPI

Initial release