monitord ... know how happy your systemd is! 😊



monitord ... know how happy your systemd is! 😊

Requirements

What does monitord monitor?

monitord collects systemd health metrics via D-Bus (and optionally Varlink) and outputs them as JSON. It provides visibility into:

Run Modes

We offer the following run modes:

Open to more formats / run methods ... Open an issue to discuss. Depends on the dependencies basically.

monitord is a config driven binary. We plan to keep CLI arguments to a minimum.

INFO level logging is enabled to stderr by default. Use -l LEVEL to increase or decrease logging.

Quick Start

  1. Install monitord: bash cargo install monitord

  2. Create a minimal config at /etc/monitord.conf: ```ini [monitord] output_format = json-pretty

[units] enabled = true

[pid1] enabled = true ```

  1. Run it: bash monitord

This will collect unit counts and PID 1 stats, then print JSON to stdout and exit. Enable additional collectors in the config as needed (see Configuration below).

Install

Pre-built binaries

Download pre-built binaries from GitHub Releases:

# Example: download and install the latest release (x86_64)
curl -L -o /usr/local/bin/monitord \
  https://github.com/cooperlees/monitord/releases/latest/download/monitord-linux-amd64
chmod +x /usr/local/bin/monitord

From crates.io

Install via cargo or use as a dependency in your Cargo.toml.

crl-linux:monitord cooper$ monitord --help
monitord: Know how happy your systemd is! 😊

Usage: monitord [OPTIONS]

Options:
  -c, --config <CONFIG>
          Location of your monitord config

          [default: /etc/monitord.conf]

  -l, --log-level <LOG_LEVEL>
          Adjust the console log-level

          [default: Info]
          [possible values: error, warn, info, debug, trace]

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version

Config

monitord can have the different components monitored. To enable / disabled set the following in our monitord.conf. This file is ini format to match systemd unit files.

# Pure ini - no yes/no for bools

[monitord]
# Set a custom dbus address to connect to
# OPTIONAL: If not set, we default to the Unix socket below
dbus_address = unix:path=/run/dbus/system_bus_socket
# Timeout in seconds for dbus connection/collections
# OPTIONAL: default is 30 seconds
dbus_timeout = 30
# Run as a daemon or 1 time
daemon = false
# Time to refresh systemd stats in seconds
# Daemon mode only
daemon_stats_refresh_secs = 60
# Prefix flat-json key with this value
# The value automatically gets a '.' appended (so don't put here)
key_prefix = monitord
# cron/systemd timer output format
# Supported: json, json-flat, json-pretty
output_format = json

# Grab as much stats from DBus GetStats call
# we can from running dbus daemon
# More tested on dbus-broker daemon
[dbus]
# Summary counters - both dbus-broker + dbus-daemon
enabled = false
# Count stale pidfds held by the system dbus-broker from procfs
stale_fd_stats = true
# dbus.user.* metrics: user stats as reported by dbus-broker
user_stats = false
# dbus.peer.* metrics: peer stats as reported by dbus-broker
peer_stats = false
# Only report peers that own a well-known bus name when peer_stats is on
peer_well_known_names_only = false
# dbus.cgroup.* stats is an aggregation of peer_stats by cgroup
# by dbus-broker
cgroup_stats = false

# Grab networkd stats from files + networkctl
[networkd]
enabled = true
link_state_dir = /run/systemd/netif/links

# Enable grabbing PID 1 stats via procfs
[pid1]
enabled = true

# Services to grab extra stats for
# .service is important as that's what DBus returns from `list_units`
[services]
foo.service

# systemd version and system state (e.g. running, degraded)
[system-state]
enabled = true

[timers]
enabled = true

[timers.allowlist]
foo.timer

[timers.blocklist]
bar.timer

# Grab unit status counts via dbus
[units]
enabled = true
state_stats = true
# Also record how long each unit has been in its current state
state_stats_time_in_state = true
ignore_inactive_oneshot_services = true
# Varlink only: reuse each [services] unit's io.systemd.Unit.List reply for up
# to this many seconds. It is refetched sooner when the unit changes state or
# systemd reloads; cgroup counters are still read every cycle. 0 disables.
service_details_cache_max_age_secs = 300

# Filter what services you want collect state stats for
# If both lists are configured blocklist is preferred
# If neither exist all units state will generate counters
[units.state_stats.allowlist]
foo.service

[units.state_stats.blocklist]
bar.service

# machines config
[machines]
enabled = true

# Same rules apply as state_stats lists above
[machines.allowlist]
foo

[machines.blocklist]
bar

# Collect via systemd varlink APIs instead of D-Bus where available
# (see the Varlink section below). Falls back to D-Bus unless no_fallback = true
[varlink]
enabled = false
no_fallback = false

# Boot blame metrics - shows the N slowest units at boot
# Similar to `systemd-analyze blame`
# Disabled by default
[boot]
enabled = false
# Cache boot blame stats in <cache_dir>/<boot_id>.boot_blame.bin
# Enabled by default; set false to force recalculation every run
cache_enabled = true
# Directory where boot blame cache files are stored (optional)
# Defaults to /run/monitord if not set; avoid world-writable dirs like /tmp when running privileged
# cache_dir = /tmp
# Number of slowest units to report
num_slowest_units = 5

# Optional: only include specific units in boot blame (if empty, all units are checked)
# Same rules apply as state_stats lists above
[boot.allowlist]
# slow-startup.service

# Optional: exclude specific units from boot blame
[boot.blocklist]
# noisy-but-expected.service

# systemd-report (systemd v260+), two independent switches:
# - enabled: serve monitord's own health (last run healthy, per-collector
#   timings and failure counters) on /run/systemd/report/io.monitord. Daemon
#   mode only. Pairs with monitord-report.socket; see "systemd-report
#   integration" in the README.
# - collect: pass through metrics from other systemd-report sources:
#   every io.systemd.Metrics socket in /run/systemd/report/ except
#   io.systemd.Manager and io.systemd.Network (already parsed by the units and
#   networkd collectors), io.monitord (monitord itself), io.systemd.CGroup
#   (too many keys until per-cgroup filtering lands) and systemd-report's
#   io.systemd.Report.* signers. A source may publish thousands of metrics.
#   Always varlink: there is no D-Bus fallback, and [varlink] enabled does
#   not gate it.
[report]
enabled = false
collect = false

# Optional: only collect these sources (if empty, all sources are collected)
[report.sources.allowlist]
# io.systemd.Basic

# Optional: never collect these sources (preferred over the allowlist)
[report.sources.blocklist]
# io.systemd.Files

# Never collect these metrics from any source. An entry also covers every
# metric under it (io.systemd.Basic.SMBIOS covers ...SMBIOS.SystemProductName).
# By default this drops identifiers that would otherwise be shipped with every
# sample: they identify a host or its hardware but say nothing about health
# (see "systemd-report sources" in the README). Delete lines you want back.
[report.metrics.blocklist]
io.systemd.Basic.BootID
io.systemd.Basic.Hostname
io.systemd.Basic.MachineID
io.systemd.Basic.SMBIOS
io.systemd.Basic.TPM2
io.systemd.Journal.HighPriorityMessage

# Unit verification using systemd-analyze verify
# Disabled by default as it can be slow on large systems
[verify]
enabled = false

# Optional: only verify specific units (if empty, all units are checked)
[verify.allowlist]
# example.service
# example.timer

# Optional: skip verification for specific units
[verify.blocklist]
# noisy.service
# broken.timer

When using the provided monitord.service, systemd creates /run/monitord via RuntimeDirectory=monitord and assigns ownership to the configured service User/Group. If you run monitord another way and use the default cache_dir (/run/monitord), ensure the directory exists and is writable by the monitord process user so boot cache files can be created. Alternatively, set cache_dir to a location like /tmp that is always writable.

systemd-report integration

systemd 260+ ships systemd-report, which gathers metrics from every io.systemd.Metrics socket in /run/systemd/report/. With [report] enabled = true in daemon mode, monitord serves its own health there as the io.monitord source, so systemd-report metrics shows whether monitord is collecting. These metrics are about monitord itself. The stats it collects are not re-exported: systemd's own sources already publish them.

Metric Type Object Value
io.monitord.collectionErrorsTotal counter runs where at least one collector failed
io.monitord.collectionRunTimeMs gauge wall time of the last run
io.monitord.collectionRunsTotal counter runs since monitord started
io.monitord.collectorElapsedMs gauge collector wall time of each collector in the last run
io.monitord.collectorFailuresTotal counter collector failed runs of each collector
io.monitord.collectorSuccess gauge collector 1 if the collector succeeded in the last run
io.monitord.healthy gauge 1 if the last run had no failures (0 before the first run)
io.monitord.lastCollectionTimestamp gauge unix seconds when the last run finished. Use it to spot a stalled daemon
io.monitord.varlinkUsage gauge collector 1 if varlink served the collector, 0 for D-Bus
io.monitord.version string monitord version
$ systemd-report metrics io.monitord
$ varlinkctl call --more /run/systemd/report/io.monitord io.systemd.Metrics.List '{}'

The socket name is fixed, because systemd-report drops any metric whose name does not start with the socket's name. monitord gets the socket in one of two ways:

ini # /etc/systemd/system/monitord.service.d/report.conf [Service] Type=exec Restart=on-failure

Then run systemctl enable --now monitord-report.socket monitord.service. - Self-bind. Without a passed socket, monitord binds /run/systemd/report/io.monitord itself (mode 0666). That needs write access to the directory, so in practice root. A socket file someone is still serving (another monitord, or monitord-report.socket) is never replaced: monitord logs a warning and keeps collecting. A stale one left by a process that died is replaced. monitord never creates the directory: if it is missing, systemd is older than 260, so monitord logs a warning and keeps collecting.

Do not use RuntimeDirectory=systemd/report to get write access. systemd would delete its own report directory, with the other sources' sockets in it, whenever monitord stops.

Library users with their own collection loop (e.g. monitord-exporter) can serve the same metrics with monitord::report::spawn_server.

Machines support

From version >=0.11 monitord supports obtaining the same set of key from systemd 'machines' (i.e. machinectl --list). Only machines of class container are collected. They are enumerated via machined's io.systemd.Machine.List varlink API (systemd v257+) when varlink is enabled, falling back to machined's D-Bus API.

The keys are the same format as below in json_flat output but are prefixed with the machines keyword and machine name. For example:

# $KEY_PREFIX.machines.$MACHINE_NAME
{
  ...
  "monitord.machines.foo.pid1.fd_count": 69,
  ...
}

Permissions

Machine collection is the only part of monitord that needs extra privileges, so skip this if you don't monitor machines. When running as a non-root user (like the shipped monitord.service), what it needs depends on how machines are collected:

Machines collected over Capabilities needed
D-Bus (the default, or [machines] varlink = false) CAP_SYS_PTRACE
varlink ([varlink] enabled = true and [machines] varlink = true) CAP_SYS_PTRACE + CAP_SYS_ADMIN
# /etc/systemd/system/monitord.service.d/machines.conf
# Drop CAP_SYS_ADMIN if you collect machines over D-Bus.
[Service]
AmbientCapabilities=CAP_SYS_PTRACE CAP_SYS_ADMIN
CapabilityBoundingSet=CAP_SYS_PTRACE CAP_SYS_ADMIN

Both capabilities are broad: CAP_SYS_PTRACE also allows reading other processes' memory, and CAP_SYS_ADMIN is close to root. If you'd rather not grant CAP_SYS_ADMIN, set [machines] varlink = false to collect machines over D-Bus with CAP_SYS_PTRACE only, or run monitord inside each machine instead of collecting from the host.

Machines started with user namespacing (e.g. machinectl start defaults to PrivateUsers=pick) also reject monitord's D-Bus connection, as host users are not mapped inside the machine. Set PrivateUsers=no in /etc/systemd/nspawn/<machine>.nspawn to collect from them.

Output Formats

json

Normal serde_json non pretty JSON. All on one line. Most compact format.

json-flat

Move all key value pairs to the top level and . notate components + sub values. Is semi pretty too + custom. All unittested ...

stat_collection_run_time_ms is emitted in milliseconds (with _ms suffix) to follow Prometheus metric naming conventions for duration units, which keeps unit semantics clear and consistent when these keys are transformed into Prometheus metric names.

This sample is from a host with no containers, so it deliberately omits the machines.<name>.* key space. Those keys duplicate the top level ones documented here, just prefixed per machine β€” see Machines support β€” so they are not repeated below.

{
  "boot.blame.cpe_chef.service": 103.05,
  "boot.blame.dev-ttyS0.device": 15.809,
  "boot.blame.dnf5-automatic.service": 204.159,
  "boot.blame.sys-module-fuse.device": 16.21,
  "boot.blame.systemd-networkd-wait-online.service": 1.674,
  "collection_timings.list_units_ms": 5.26,
  "collection_timings.per_unit_loop_ms": 42.99,
  "collection_timings.service_dbus_fetches": 0,
  "collection_timings.slowest_units.0.chronyd.service": 5.248485,
  "collection_timings.slowest_units.1.fstrim.timer": 5.173884,
  "collection_timings.slowest_units.2.systemd-logind.service": 0.042048,
  "collection_timings.slowest_units.3.dev-nvme0n1.device": 0.035938,
  "collection_timings.slowest_units.4.sys-module-fuse.device": 0.035106,
  "collection_timings.state_dbus_fetches": 0,
  "collection_timings.timer_dbus_fetches": 24,
  "collection_timings.varlink_service_cache_hits": 0,
  "collection_timings.varlink_unit_list_fetches": 0,
  "collection_timings.varlink_unit_list_max_ms": 0.0,
  "collection_timings.varlink_unit_list_ms": 0.0,
  "collector_timings.boot_blame.elapsed_ms": 53.36,
  "collector_timings.boot_blame.start_offset_ms": 0.08,
  "collector_timings.boot_blame.success": 1,
  "collector_timings.dbus_stats.elapsed_ms": 1.84,
  "collector_timings.dbus_stats.start_offset_ms": 0.11,
  "collector_timings.dbus_stats.success": 1,
  "collector_timings.machines.elapsed_ms": 0.51,
  "collector_timings.machines.start_offset_ms": 0.14,
  "collector_timings.machines.success": 0,
  "collector_timings.networkd.elapsed_ms": 8.48,
  "collector_timings.networkd.start_offset_ms": 0.1,
  "collector_timings.networkd.success": 1,
  "collector_timings.pid1.elapsed_ms": 0.94,
  "collector_timings.pid1.start_offset_ms": 0.1,
  "collector_timings.pid1.success": 1,
  "collector_timings.system_state.elapsed_ms": 2.38,
  "collector_timings.system_state.start_offset_ms": 0.12,
  "collector_timings.system_state.success": 1,
  "collector_timings.units.elapsed_ms": 53.24,
  "collector_timings.units.start_offset_ms": 0.06,
  "collector_timings.units.success": 1,
  "collector_timings.verify.elapsed_ms": 31.07,
  "collector_timings.verify.start_offset_ms": 0.13,
  "collector_timings.verify.success": 1,
  "collector_timings.version.elapsed_ms": 2.25,
  "collector_timings.version.start_offset_ms": 0.09,
  "collector_timings.version.success": 1,
  "dbus.active_connections": 10,
  "dbus.bus_names": 16,
  "dbus.cgroup.system.slice-systemd-logind.service.activation_request_bytes": 0,
  "dbus.cgroup.system.slice-systemd-logind.service.activation_request_fds": 0,
  "dbus.cgroup.system.slice-systemd-logind.service.incoming_bytes": 16,
  "dbus.cgroup.system.slice-systemd-logind.service.incoming_fds": 0,
  "dbus.cgroup.system.slice-systemd-logind.service.match_bytes": 6942,
  "dbus.cgroup.system.slice-systemd-logind.service.matches": 5,
  "dbus.cgroup.system.slice-systemd-logind.service.name_objects": 1,
  "dbus.cgroup.system.slice-systemd-logind.service.outgoing_bytes": 0,
  "dbus.cgroup.system.slice-systemd-logind.service.outgoing_fds": 0,
  "dbus.cgroup.system.slice-systemd-logind.service.reply_objects": 0,
  "dbus.incomplete_connections": 0,
  "dbus.match_rules": 26,
  "dbus.peak_bus_names": 33,
  "dbus.peak_bus_names_per_connection": 2,
  "dbus.peak_match_rules": 33,
  "dbus.peak_match_rules_per_connection": 13,
  "dbus.peer.org.freedesktop.systemd1.activation_request_bytes": 0,
  "dbus.peer.org.freedesktop.systemd1.activation_request_fds": 0,
  "dbus.peer.org.freedesktop.systemd1.incoming_bytes": 16,
  "dbus.peer.org.freedesktop.systemd1.incoming_fds": 0,
  "dbus.peer.org.freedesktop.systemd1.match_bytes": 46533,
  "dbus.peer.org.freedesktop.systemd1.matches": 33,
  "dbus.peer.org.freedesktop.systemd1.name_objects": 1,
  "dbus.peer.org.freedesktop.systemd1.outgoing_bytes": 0,
  "dbus.peer.org.freedesktop.systemd1.outgoing_fds": 0,
  "dbus.peer.org.freedesktop.systemd1.reply_objects": 0,
  "dbus.stale_fds": 3,
  "dbus.user.cooper.bytes": 919236,
  "dbus.user.cooper.fds": 78,
  "dbus.user.cooper.matches": 510,
  "dbus.user.cooper.objects": 80,
  "dbus.user.root.stale_fds": 3,
  "networkd.eno4.address_state": 3,
  "networkd.eno4.admin_state": 4,
  "networkd.eno4.carrier_state": 5,
  "networkd.eno4.ipv4_address_state": 3,
  "networkd.eno4.ipv6_address_state": 2,
  "networkd.eno4.oper_state": 9,
  "networkd.eno4.required_for_online": 1,
  "networkd.managed_interfaces": 2,
  "networkd.wg0.address_state": 3,
  "networkd.wg0.admin_state": 4,
  "networkd.wg0.carrier_state": 5,
  "networkd.wg0.ipv4_address_state": 3,
  "networkd.wg0.ipv6_address_state": 3,
  "networkd.wg0.oper_state": 9,
  "networkd.wg0.required_for_online": 1,
  "pid1.cpu_time_kernel": 48,
  "pid1.cpu_user_kernel": 41,
  "pid1.fd_count": 245,
  "pid1.memory_usage_bytes": 19165184,
  "pid1.tasks": 1,
  "report.io.systemd.Basic.colliding_metrics": 0,
  "report.io.systemd.Basic.dropped_metrics": 0,
  "report.io.systemd.Basic.metrics.CPUUsage.type_system": 177191060000000,
  "report.io.systemd.Basic.metrics.CPUUsage.type_user": 360232390000000,
  "report.io.systemd.Basic.metrics.DiskReadBytes./dev/sda": 176618894336,
  "report.io.systemd.Basic.metrics.KernelVersion": "6.12.4-200.fc41.x86_64",
  "report.io.systemd.Basic.metrics.LoadAverage1Min": 0.29541015625,
  "report.io.systemd.Basic.metrics.OSRelease.ID": "fedora",
  "report.io.systemd.Basic.metrics.PressureAvg10.resource_cpu.type_some": 0.92041015625,
  "report.io.systemd.Basic.up": 1,
  "services.chronyd.service.active_enter_timestamp": 1683556542382710,
  "services.chronyd.service.active_exit_timestamp": 0,
  "services.chronyd.service.cpuusage_nsec": 328951000,
  "services.chronyd.service.inactive_exit_timestamp": 1683556541360626,
  "services.chronyd.service.ioread_bytes": 18446744073709551615,
  "services.chronyd.service.ioread_operations": 18446744073709551615,
  "services.chronyd.service.memory_available": 18446744073709551615,
  "services.chronyd.service.memory_current": 5214208,
  "services.chronyd.service.nrestarts": 0,
  "services.chronyd.service.processes": 1,
  "services.chronyd.service.restart_usec": 100000,
  "services.chronyd.service.state_change_timestamp": 1683556542382710,
  "services.chronyd.service.status_errno": 0,
  "services.chronyd.service.tasks_current": 1,
  "services.chronyd.service.timeout_clean_usec": 18446744073709551615,
  "services.chronyd.service.watchdog_usec": 0,
  "stat_collection_run_time_ms": 87.4013,
  "system-state": 3,
  "timers.fstrim.timer.accuracy_usec": 3600000000,
  "timers.fstrim.timer.fixed_random_delay": 0,
  "timers.fstrim.timer.last_trigger_usec": 1743397269608978,
  "timers.fstrim.timer.last_trigger_usec_monotonic": 0,
  "timers.fstrim.timer.next_elapse_usec_monotonic": 0,
  "timers.fstrim.timer.next_elapse_usec_realtime": 1744007133996149,
  "timers.fstrim.timer.persistent": 1,
  "timers.fstrim.timer.randomized_delay_usec": 6000000000,
  "timers.fstrim.timer.remain_after_elapse": 1,
  "timers.fstrim.timer.service_unit_last_state_change_usec": 1743517244700135,
  "timers.fstrim.timer.service_unit_last_state_change_usec_monotonic": 639312703,
  "unit_files.root.generated.mount_units": 6,
  "unit_files.root.generated.service_units": 1,
  "unit_files.root.generated.socket_units": 1,
  "unit_files.root.generated.swap_units": 1,
  "unit_files.root.transient.scope_units": 1,
  "unit_files.root.transient.service_units": 1,
  "unit_files.user.transient.scope_units": 19,
  "unit_files.user.transient.service_units": 15,
  "unit_states.chronyd.service.active_state": 1,
  "unit_states.chronyd.service.load_state": 1,
  "unit_states.chronyd.service.time_in_state_usecs": 2176062087185,
  "unit_states.chronyd.service.unhealthy": 0,
  "units.activating_units": 0,
  "units.active_units": 403,
  "units.automount_units": 1,
  "units.device_units": 150,
  "units.failed_units": 0,
  "units.inactive_units": 159,
  "units.jobs_queued": 0,
  "units.loaded_units": 497,
  "units.masked_units": 25,
  "units.mount_units": 52,
  "units.not_found_units": 38,
  "units.path_units": 4,
  "units.scope_units": 17,
  "units.service_units": 199,
  "units.slice_units": 7,
  "units.socket_units": 28,
  "units.target_units": 54,
  "units.timer_persistent_units": 1,
  "units.timer_remain_after_elapse": 1,
  "units.timer_units": 20,
  "units.total_units": 562,
  "varlink_usage.boot_blame": 1,
  "varlink_usage.machines": 1,
  "varlink_usage.networkd": 1,
  "varlink_usage.system_state": 1,
  "varlink_usage.units": 1,
  "varlink_usage.verify": 1,
  "varlink_usage.version": 1,
  "verify.failing.device": 43,
  "verify.failing.mount": 15,
  "verify.failing.service": 31,
  "verify.failing.slice": 1,
  "verify.failing.total": 97,
  "version": "255.7-1.fc40"
}

json-pretty

Normal serde_json pretty representations of each components structs.

systemd-report sources

systemd v260+ ships systemd-report, which reads metrics from every io.systemd.Metrics varlink socket in /run/systemd/report/ (systemd's own systemd-report-*.socket sources such as io.systemd.Basic and io.systemd.Journal, plus any daemon that installs a socket there). With [report] collect = true, monitord reads the same sources and passes their metrics through, so anything a source publishes reaches monitord's output without per-source code.

Default metric denylist

io.systemd.Basic publishes some values that identify a host or its hardware and say nothing about its health, and io.systemd.Journal publishes log entries. Passed through, they would be copied into every sample monitord ships to your metrics pipeline, and they are hard to remove from long-retention storage later. The example monitord.conf therefore lists them in [report.metrics.blocklist]:

Entry Why it's blocked
io.systemd.Basic.MachineID /etc/machine-id is a stable, unique host ID. systemd documents it as confidential and says it shouldn't be exposed to untrusted environments (machine-id(5))
io.systemd.Basic.BootID Unique per boot, so it's a host ID until the next reboot and tracks the host across restarts in between. It also makes the time series change on every boot
io.systemd.Basic.Hostname Identifies the host and is usually already a label in your pipeline. Duplicated here, it's a string that changes if the host is renamed
io.systemd.Basic.SMBIOS Firmware vendor, version and date, board and product names, SKU and asset tags. Together they fingerprint the hardware, and asset tags are often organization-specific. Covers every SMBIOS.* metric, including fields future systemd versions add (such as serial numbers)
io.systemd.Basic.TPM2 TPM manufacturer and vendor strings: more hardware fingerprint, no health signal
io.systemd.Journal.HighPriorityMessage The 10 most recent crit-or-higher journal entries, each a whole entry object (MESSAGE, _CMDLINE, _MACHINE_ID, _BOOT_ID, …), with object set to SYSLOG_IDENTIFIER. That's log text, not a metric: it would put command lines (which can carry secrets) and the IDs above into metric keys, the keys change with every new message, and two entries from one identifier collide on one key

The list lives only in the example config, so a config without [report.metrics.blocklist] blocks nothing. Remove an entry if you want that value, for example Hostname if your pipeline doesn't label hosts itself.

Flat keys are report.<source>.up, report.<source>.dropped_metrics, report.<source>.colliding_metrics, and report.<source>.metrics.<metric>[.<object>][.<field>_<value>...]. <metric> is the metric name without the <source>. prefix, and fields are sorted by name. Booleans become 0/1 at any depth, and object or array values are flattened into .<member> / .<index> keys. json and json-pretty output the same data under report.<source>.metrics, keyed the same way.

Keys stay readable, so objects keep their dots (foo.service), the same as monitord's other per-unit keys. As a result, two different metrics can map to the same flat key. For example, fields {x: "a", y: "b"} and {x: "a.y_b"} both render as Value.x_a.y_b, and an object value {"a": {"b": 1}} collides with a member named "a.b". When that happens the first metric keeps the key. Each later one is left out, counted in colliding_metrics and logged as a warning, rather than silently overwriting it. A non-zero colliding_metrics means a source needs a closer look.

Per-collector timing metrics

monitord records the wall time each collector future spends inside a single stat_collector cycle and exposes the result on MonitordStats::collector_timings, plus an inner phase breakdown for the units collector (SystemdUnitStats::collection_timings).

Varlink usage gauges

Each collector that can use varlink reports which transport served it on the last run: varlink_usage.<collector> is 1 when the varlink attempt succeeded and 0 when the collector fell back to D-Bus (or file-based collection for networkd). Collectors with no varlink path (pid1, dbus_stats) and disabled collectors emit no gauge, so the present gauges are exactly the enabled set β€” no separate enabled-collectors counter is needed.

As the systemd under monitord upgrades past each endpoint's minimum version (networkd v257+, system state/version v258+, units v260+, unit details v261+), collectors silently flip from 0 to 1 with no config change, so watching these gauges over time shows varlink adoption climbing across the fleet. Downstream consumers such as monitord-exporter can aggregate them (share of collectors reporting 1) from there; monitord itself only makes the gauges available in its output formats. Per-container gauges are emitted under machines.<name>.varlink_usage.<collector>. The host machines gauge covers enumeration only (machined's io.systemd.Machine.List, v257+); per-container collection reports its own gauges.

Field Meaning
collector_timings.<name>.start_offset_ms ms from the top of the cycle until the spawned future was first polled. Should be sub-ms when collectors are running in parallel; a non-trivial value means the spawn loop or runtime is delaying first poll.
collector_timings.<name>.elapsed_ms ms from first poll to completion for that collector.
collector_timings.<name>.success 1 if the collector returned Ok, 0 otherwise.
collection_timings.list_units_ms ms for the systemd ListUnits D-Bus call (one batched call). On the varlink path, ms for the io.systemd.Metrics.List stream.
collection_timings.per_unit_loop_ms ms spent walking each listed unit, including any per-unit D-Bus calls (timer/state/service).
collection_timings.timer_dbus_fetches Count of timer D-Bus property fetches this run.
collection_timings.state_dbus_fetches Count of unit-state D-Bus fetches (only when state_stats_time_in_state is enabled).
collection_timings.service_dbus_fetches Count of per-service D-Bus property fetches.
collection_timings.slowest_units The units.slowest_units_count slowest units this run (unit name, duration ms), descending. Empty when slowest_units_count = 0. D-Bus path only (see parity note below).
collection_timings.varlink_unit_list_fetches Varlink path: count of io.systemd.Unit.List calls to PID 1 this run.
collection_timings.varlink_unit_list_ms Varlink path: total ms spent in those Unit.List calls. Included in per_unit_loop_ms, which also covers cgroupfs reads and local parsing.
collection_timings.varlink_unit_list_max_ms Varlink path: slowest single Unit.List call this run, in ms. A high value points at a slow PID 1.
collection_timings.varlink_service_cache_hits Varlink path: [services] entries served from the cross-cycle cache (see units.service_details_cache_max_age_secs) instead of a Unit.List call.

Comparing sum(collector_timings.*.elapsed_ms) against stat_collection_run_time_ms gives an effective parallelism ratio (sum / wall β‰ˆ N means N-way parallelism, β‰ˆ 1 means effectively serial).

The per-collector lines are also emitted to logs at debug! level. The end-of-cycle "stat collection run took {}ms" summary stays at info!.

Varlink-vs-D-Bus parity

collection_timings is populated identically by the D-Bus path (units::parse_unit_state) and the varlink path (varlink_units::parse_metrics plus the io.systemd.Unit.List detail pass). In the varlink case, list_units_ms is the bulk varlink List call on io.systemd.Manager and per_unit_loop_ms covers the local parse loop plus the per-unit detail pass that fills service and timer stats.

All three *_dbus_fetches counters stay at zero on the varlink path, since nothing there touches D-Bus any more: per-service stats, timer properties and service types come from io.systemd.Unit.List, and time-in-state from the StateChangeTimestamp metric. They only become nonzero if that socket is unusable and monitord falls back. This makes varlink.enabled = true vs false directly comparable on the same host.

Convention for new collectors moved to varlink: when porting a collector from D-Bus to varlink, add the equivalent inner timings so the two implementations remain comparable. The minimum is wall time of the bulk fetch (analogous to list_units_ms) and the local parse loop (analogous to per_unit_loop_ms), recorded onto a struct nested inside the collector's public stats type. Single-shot varlink calls (e.g. networkd Describe) do not need an inner split β€” the outer collector_timings.<name>.elapsed_ms already covers them.

Metric Value Reference

Many metrics are serialized as integers. Here are the enum mappings:

system-state

Value State
0 unknown
1 initializing
2 starting
3 running
4 degraded
5 maintenance
6 stopping
7 offline

active_state (unit_states.*.active_state)

Value State
0 unknown
1 active
2 reloading
3 inactive
4 failed
5 activating
6 deactivating

load_state (unit_states.*.load_state)

Value State
0 unknown
1 loaded
2 error
3 masked
4 not-found

networkd address_state / ipv4_address_state / ipv6_address_state

Value State
0 unknown
1 off
2 degraded
3 routable

networkd admin_state

Value State
0 unknown
1 pending
2 failed
3 configuring
4 configured
5 unmanaged
6 linger

networkd carrier_state

Value State
0 unknown
1 off
2 no-carrier
3 dormant
4 degraded-carrier
5 carrier
6 enslaved

networkd oper_state

Value State
0 unknown
1 missing
2 off
3 no-carrier
4 dormant
5 degraded-carrier
6 carrier
7 degraded
8 enslaved
9 routable

networkd required_for_online

Value State
0 false
1 true
255 unknown

Note the unknown sentinel is 255 (u8::MAX), not 0 β€” 0 means an explicit false. Alerting rules should treat 255 as "no data", not as "not required for online".

dbus stats

You're going to need to be root or allow permissiong to pull dbus stats. For dbus-broker here is example config allow a user monitord to query getStats

When stale_fd_stats = true, monitord also inspects /proc/<dbus-broker>/fdinfo for stale pidfds reported by the kernel as Pid: -1. These counters are emitted as dbus.stale_fds and, for metric compatibility, dbus.user.root.stale_fds; the per-user root value is the unattributed system-broker count, not proof that root owns those descriptors.

[cooper@l33t ~]# cat /etc/dbus-1/system.d/allow_monitord_stats.conf
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE busconfig PUBLIC
 "-//freedesktop//DTD D-BUS Bus Configuration 1.0//EN"
 "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd">
<busconfig>
  <policy user="monitord">
    <allow send_destination="org.freedesktop.DBus"
           send_interface="org.freedesktop.DBus.Debug.Stats"
           send_member="GetStats"
           send_path="/org/freedesktop/DBus"
           send_type="method_call"/>
  </policy>
</busconfig>

Development

To do test runs (requires systemd and systemd-networkd installed)

Ensure the following pass before submitting a PR (CI checks):

Releasing a new version

  1. Increment the version in Cargo.toml
  2. Run ./build_docs.sh to regenerate docs
  3. Commit with message: Move to version X.Y.Z for release + update docs
  4. If you have commit bit, push directly to main. Otherwise, push a branch and open a PR.
  5. Cut a GitHub release: gh release create X.Y.Z --title "X.Y.Z" --generate-notes

Generate codegen APIs

Then add the following macros to tell clippy to go away:

#![allow(warnings)]
#![allow(clippy)]

Non Linux development

Sometimes I develop from my Mac OS X laptop. So I thought I'd document and add the way I build a Fedora Rawhide container and mount the local repo to /repo in the container to run monitord and test.

You can now log into the container to build + run tests and run the binary now against systemd.

Integration test

varlink_integration_test.py is what the varlink-integration GitHub Action runs, so a CI failure reproduces locally with the same command:

It rebuilds the image on every run (a no-op when the Dockerfile has not changed) and recreates its container whenever the image changes, so a local run uses the same image the Action builds from scratch instead of a stale one. It reuses the running container otherwise; --fresh rebuilds without the cache. The image must carry everything the test shells out to, which is why the Dockerfile names systemd-container, systemd-networkd and shadow-utils explicitly β€” see REQUIRED_CONTAINER_TOOLS, which fails the run up front with one message if any of them is missing.

Troubleshooting

"Connection refused" or D-Bus connection errors

The system bus connection is created lazily: monitord starts fine without a bus as long as no enabled collector needs D-Bus. Collectors served entirely over varlink, cgroupfs, state files, sysfs or procfs never connect β€” the networkd file fallback maps ifindexes to names via /sys/class/net/*/ifindex and only reaches for Manager.ListLinks when sysfs yields nothing usable. A collector that does need the bus β€” including any varlink fallback β€” reports its own per-collector error and the run continues with the remaining collectors. If a D-Bus collector is failing, ensure the system D-Bus daemon is running and the socket exists at /run/dbus/system_bus_socket. If using a custom address, set dbus_address in [monitord] config. Increase dbus_timeout if running on slow systems.

Empty or missing networkd metrics

systemd-networkd must be installed and running (systemctl start systemd-networkd). If networkd is not in use on your system, disable the collector with enabled = false in [networkd].

Permission denied for machines / containers

A non-root monitord needs CAP_SYS_PTRACE to reach machines via /proc/<leader_pid>/root, plus CAP_SYS_ADMIN to collect them over varlink. See Permissions under Machines support.

Permission denied for D-Bus stats

The [dbus] collector requires permission to call org.freedesktop.DBus.Debug.Stats.GetStats. Either run monitord as root or add a D-Bus policy file β€” see the dbus stats section.

PID 1 stats unavailable

PID 1 stats require Linux with procfs mounted at /proc. This collector is compiled out on non-Linux targets. If /proc is not available (some container runtimes), disable with enabled = false in [pid1].

Collector errors don't crash monitord

When an individual collector fails (e.g., networkd not running, D-Bus timeout), monitord logs a warning and continues with the remaining collectors. Check stderr output or increase the log level (-l debug) to see which collectors had issues.

Large u64 values (18446744073709551615) in output

These represent u64::MAX and mean "not available" or "not tracked" for that metric. This is how systemd reports fields that are unsupported or not configured for the unit. The cgroup-derived service fields report u64::MAX only when neither cgroupfs nor the IPC fallback has data (e.g. ioread_bytes with the io controller disabled); memory_available in particular is computed even without limits, as host MemAvailable.

Library API

monitord can be used as a Rust library. See the full API documentation at monitord.xyz.

DBus

All monitord's dbus is done via async (tokio) zbus crate.

systemd Dbus APIs are in use in the following modules:

Some of these modules can be disabled via configuration. Due to this, monitord might not always be running / calling all these DBus calls per run.

Varlink

monitord supports collecting unit statistics via systemd's Varlink metrics API, available in systemd v260+. When enabled, monitord connects to the io.systemd.Metrics interface at /run/systemd/report/io.systemd.Manager to collect unit counts, active/load states, and restart counts.

Enabling Varlink

Set enabled = true in the [varlink] section of monitord.conf:

[varlink]
enabled = true

When varlink is enabled, monitord will attempt to collect stats via the varlink APIs first, automatically falling back to D-Bus or file-based collection when a varlink socket is unavailable (e.g., older systemd versions).

Set no_fallback = true alongside it to turn any varlink failure into a loud per-collector error instead of falling back. That is a verification mode for proving a collector set is varlink-clean in CI β€” not a hardening flag: tripped collectors still report success=0 in collector_timings and the run exits 0 (per-collector failures are deliberately non-fatal, especially in daemon mode), so CI must assert on the success gauges, not the exit status. no_fallback only fires inside varlink code paths, so it has no effect while [varlink] enabled=false (a warning is logged) or on [dbus] stats, which have no varlink path (they are statistics about the D-Bus daemon itself). Partial varlink data that parses with warnings (e.g. a skipped metric) is not a fallback either β€” pair no_fallback with the completeness assertions, not as a substitute for them.

Each varlink-capable collector ([units], [networkd], [system-state], [boot], [verify], [machines] for machine enumeration and container collection) also has its own varlink toggle, defaulting to true. A collector uses varlink only when both the global switch and its section toggle are true, so collectors can be moved to varlink one at a time by setting a section toggle to false. Container collection additionally requires [machines] varlink. Timers have no toggle of their own: they ride the units path and follow [units] varlink. [system-state] varlink also gates version collection, which shares the Manager.Describe call β€” including when [system-state] itself is disabled.

Metrics collected via Varlink

Units (io.systemd.Metrics β€” systemd v260+, v261+ for jobs queued, exact totals, load-state totals, time-in-state, service errno, and the ActiveTimestamp/InactiveExitTimestamp families boot blame reads): - Unit counts by type (service, mount, socket, target, device, automount, timer, path, slice, scope) - Unit counts by state (activating, active, failed, inactive) - Unit counts by load state (UnitsByLoadStateTotal, v261+: loaded, masked, not-found; counted from per-unit load states on v260) - Exact total unit count (UnitsTotal, v261+; approximated from mapped per-type counts on v260) - Queued job count (JobsQueued, v261+; unavailable (0) on v260) - Per-unit active state and load state (with allowlist/blocklist filtering) - Per-unit time in state (StateChangeTimestamp, v261+; unavailable on v260) - Per-unit health status (computed from active + load state) - Per-service restart counts (nrestarts) - Per-service errno status (StatusErrno, v261+; unavailable (0) on v260) - Full per-service stats for units in [services] via io.systemd.Unit.List (systemd v261+): timestamps, restart/timeout/watchdog settings from the reply, and CPU, memory, tasks, IO plus process count read from cgroupfs (same reader as the D-Bus path, so both agree). A field with no cgroupfs data falls back to the reply's runtime.CGroup value, defaulting to u64::MAX when the reply omits it too - Per-timer stats via io.systemd.Unit.List (systemd v261+): accuracy, delays, next elapse, last trigger, and the triggered unit's state change β€” no D-Bus backfill - Falls back to D-Bus collection if the socket is unavailable

System state and version (io.systemd.Manager.Describe β€” systemd v258+): - Overall systemd system state (running, degraded, …) - Version of the running systemd manager (which can trail the installed package until PID 1 re-execs) - Falls back to the D-Bus SystemState/Version properties if the socket is unavailable

Boot blame (io.systemd.Metrics β€” systemd v261+): - Slowest units at boot, from the per-unit ActiveTimestamp/InactiveExitTimestamp metrics rather than a D-Bus property read per unit - Falls back to D-Bus if the socket is unavailable

Verify (unit names from UnitLoadState metric objects β€” systemd v260+): - Unit enumeration for systemd-analyze verify without D-Bus ListUnits; the metric objects are the same set ListUnits returns (verified live, including not-found units) - The analyze run itself is unchanged and shared by both paths - Falls back to D-Bus if the socket is unavailable

Networkd interfaces (io.systemd.Network.Describe β€” systemd v257+): - Per-interface operational, carrier, admin, and address states - Falls back to parsing /run/systemd/netif/links state files if the socket is unavailable

Machines (io.systemd.Machine.List on /run/systemd/machine/io.systemd.Machine β€” systemd v257+): - Enumeration of containers and their leader PIDs, with the [machines] allowlist/blocklist - Needs no extra privileges (unlike per-container collection, see below) - Falls back to machined's D-Bus ListMachines if the socket is unavailable

systemd-report sources (io.systemd.Metrics.List on every other socket in /run/systemd/report/ β€” systemd v260+, opt-in with [report] collect = true): - Generic pass-through of every source's metrics (see systemd-report sources) - No D-Bus fallback: a source that is missing collects nothing, and one that fails reports up = 0

Containers

For systemd-nspawn containers, monitord uses the same varlink endpoints as on the host, inside the container: units from /run/systemd/report/io.systemd.Manager plus io.systemd.Unit.List on /run/systemd/io.systemd.Manager, system state and version from /run/systemd/io.systemd.Manager, and networkd from /run/systemd/netif/io.systemd.Network (with the same file-based fallback). Unit file and cgroup data are read below /proc/<leader_pid>/root.

systemd's credential-checking varlink servers (PID 1, networkd) refuse callers outside their PID namespace (#211, systemd/systemd#43807), so these connections are made by a per-machine helper inside the container's PID namespace. That needs CAP_SYS_ADMIN in addition to CAP_SYS_PTRACE; without it monitord warns once and collects containers over D-Bus. See Permissions.

varlink 101

varlink might one day replace our DBUS usage. Here are some notes on how to work with systemd varlink as there isn't really documentation outside man pages.

Checking interfaces

Here is an example with networkd's interfaces. monitord uses Describe, which returns per-interface states (GetStates returns the manager-level aggregate instead):

varlinkctl info unix:/run/systemd/netif/io.systemd.Network
varlinkctl introspect unix:/run/systemd/netif/io.systemd.Network io.systemd.Network

cooper@au:~$ varlinkctl call unix:/run/systemd/netif/io.systemd.Network io.systemd.Network.Describe '{}' | jq '.Interfaces[0]'
{
  "Index": 2,
  "Name": "eth0",
  "AdministrativeState": "configured",
  "OperationalState": "routable",
  "CarrierState": "carrier",
  "AddressState": "routable",
  "IPv4AddressState": "routable",
  "IPv6AddressState": "routable",
  "OnlineState": "online",
  "NetworkFile": "/etc/systemd/network/69-eno4.network",
  "RequiredForOnline": true
}