Cross-Domain Resiliency and Ansible Monitoring

Cross domain resiliency uses the mdadm monitoring and reporting mechanism. The KumoScale Provisioner forwards events and commands from the hosts and the Ansible to the Syslog server. The mdadm periodically polls the md arrays and reports any detected events to a configured Syslog server (rsyslog).

6.1 Configuration and Activation

  1. Ensure that mdadm monitoring is activated by the configure_mdadm_syslog
  2. Ensure that Host commands and events monitoring is activated when a Syslog server is configured for backend use by using the ks_syslog
  3. Configure the Syslog parameters in the vars.yml file:

Parameter

Description

Example

syslog_server

The IP address of the Syslog server

syslog_server: 172.28.10.90

syslog_port

The port of the Syslog server

syslog_port: 6514

6.2 Event Notifications

The mdadm monitors the arrays and generates events that can be sent to a Syslog server. These events are sent along with the following parameters:

  1. The name of the event (see the table below)
  2. The affected md device name
  3. A related device, if it exists (e.g., a component device that has failed)

These events are categorized by the level of severity: critical, warning, and info.

Event

Description

Severity

DegradedArray

A replica has disconnected (this is not generated when mdadm notices a drive failure).

Critical

Fail

An active component device of an array was marked faulty.

Critical

RebuildNN

The progress of a replica rebuild process in percentage (NN is a zero-based, two-digit number, e.g. 05, 48).

Warning

RebuildFinished

The reconstruction of a replica finished (successfully or aborted).

Warning

NewArray

A new md array has been detected in the /proc/mdstat file.

Info

TestMessage

An array was detected at boot and the --test flag was specified.

Info

Table 8 – Linux Events Reported by the Cross-Domain Resiliency Solution

Notes:

  • Each event has an associated array device (e.g. /dev/md1) and possibly an additional device. For Fail, the second device is the relevant component device.
  • Refer to mdadm documentation for additional information regarding the various event states.

The following host events are forwarded to the Syslog server if a Syslog server is configured in KumoScale:

Event

Description

Parameters

Severity

Session Established

A host connected to a target.

The host’s (initiator) NQN and the target’s NQN

Info

Session closed

A host disconnected from a target.

The host’s (initiator) NQN and the target’s NQN

Info

Table 9 – Host Events Reported by KumoScale to the Syslog server