CSI System Management

This chapter explains how to perform several management operations.

Self-Healing

Note - This operation is executed by the CSI driver as part of its operation and is described here for informative purposes only.

The KumoScale CSI driver monitors the mdraid[1] (MD) of each volume every 30 seconds and repairs inconsistencies and incomplete states.

There are several scenarios which may cause these states. The driver handles the scenarios described in the following sections.

Restoring a Replica that was Down

Replicas may go down temporarily as a result of a network failure or a maintenance operation over the rack. In this case, the orchestrator will migrate the container to a node on another rack containing one of the other replicas of the volume and reconnect to it.

When the appliance of the first replica is back up, it is not connected and added to the MD anymore. The agent will discover that the replica belongs to the volume, add it to the target as a namespace and add its block device to the MD. The MD will then synchronize the changes to the restored replica.

Completing Deletion of a Replica

A replica may be in a state where it was deleted, but not removed from the MD, due to a temporary disconnection or failure.

In this case, the provisioner service will report the replica was deleted but is still connected to the MD. The driver will remove it from the MD, attempt to disconnect it from the target, and mark it as ‘removed.’

Autonomous Self-Healing

Configure the maxReplicaDownTime parameter (see "Storage Class Parameters" in CSI Storage Provisioning) for the number of minutes the CSI driver will wait when a degraded replicated volume is detected until autonomously initiating a healing process (provided it has less than four replicas).

The self-healing will initiate a process in which a new replica will be allocated in the most appropriate location, according to the volume’s storage class parameters. The self-healing will also connect to the new replica and synchronize it. The missing replica will be removed from the volume configuration.

CSI Event Forwarding

KumoScale software forwards CSI events and configuration commands from the Kubernetes master and node to the KumoScale Provisioner service.

Configure a syslog server on a KumoScale appliance if you want to log these events and commands.

The following table lists the events reported by the node to the syslog server:

Table 2. CSI Events Reported to the Syslog Server

Event

Level

Description

Host Disconnected

Fatal

The host NVMe-oF™ connection to the storage node was disconnected.

Host Reconnected

Info

The host NVMe-oF connection to the storage node was reconnected.

Replicated Volume Degraded

Fatal

A replicated volume is in a degraded state.

Replicated Volume Healed

Info

A replicated volume healed.

Replicated Volume Synch Started

Warning

The synchronization of a non-synchronized replica began.

Replicated Volume Synch Ended

Warning

The synchronization of a non-synchronized replica has completed.

Report Auditing

Info

The node reports these auditing commands to the KumoScale provisioner service:

1.      Session Established – an initiator connected to KumoScale software via the NVMe-oF standard.

2.     Session Closed – the NVMe-oF connection between an initiator and KumoScale software was disconnected.

 

Collecting KumoScale CSI Driver Logs

Use kubectl logs to collect the KumoScale CSI Driver logs for each of the containers
(ks-csi-plugin, ks-provisioner).

For example, run the following command to collect the logs:

  • KumoScale Provisioner service logs:
    • kubectl logs -n kube-system csi-kumoscale-controller-0 ks-provisioner > provisioner.log
  • Controller logs:
    • kubectl logs -n kube-system csi-kumoscale-controller-0 ks-csi-plugin > controller.log

[1] ‘mdraid’ is a Linux OS component that controls storage devices. It is referred to as Linux software RAID as it makes RAID use possible without a hardware RAID controller.

 

Next: Return to KumoScale 3.20 documentation list