Maintainance for KumoScale in Appliance Mode

This article explains how to perform maintenance activities in KumoScale in Appliance mode.

KumoScale Software Cluster Upgrade

This section describes the requirements and process for upgrading an existing cluster from KumoScale software version 3.20[1]. The process will upgrade the following components in the order below:

  1. Storage Nodes: The management engine, target module, CLI, and local registry catalogue will be upgraded with all new containers.
  2. KumoScale software and Kubernetes internal services: All services automatically installed with the KumoScale software, such as the operators, CSI, and Provisioner will be upgraded.

The upgrade process maintains configuration and user data between versions. If you have a high-availability KumoScale Installation, the upgrade will happen without service disruption. It does NOT support the following:

  1. Upgrades to services that were manually installed by the user. For example, Prometheus, Fluentd™, and Loki™, must be manually upgraded through the CRD.
  2. Upgrades to the:
    • Private Kubernetes cluster for the KumoScale storage cluster.
    • Storage node’s Operating System (OS) and kernel versions.

3.  Upgrades to external components and agents deployed using the OpenStack Cinder driver, Nova Connector, Ansible Client, Ansible modules, and Customer-developed CSI deployments in Kubernetes.


4.  Adding a new storage node or removing an existing one during the cluster upgrade.

A syslog event is generated at the beginning and end of all upgrades so that you can review the results of a successful or failed upgrade.

NOTE: If you need to upgrade a specific component independently, you must contact technical support for guidance. The process varies from the one described and supported in this document 

Requirements for Upgrading

To upgrade an existing cluster, you will need to confirm the following:

1. Access to a server with ftp, http, or https in order to copy the image used for upgrading including the required credentials, if any.


2. The cluster is healthy. In particular:

  • All storage nodes are up and running.
  • The time is synchronized across all nodes in the cluster.
  • The KumoScale Provisioner service has a valid license.
  • The KumoScale secret file is deployed and active.
  • There is at least one initialized storage node.

3. All KumoScale storage nodes and components are running the same version.


4. No node or replica is in a synchronizing state.


5. No storage node has a replica of a degraded resilient volume. Such a node cannot be rebooted; thus you must wait to upgrade until this condition is resolved.

Upgrade Process

Once you have confirmed all requirements for upgrading have been met, you are ready to complete the following steps to perform the upgrade.

  1. Copy the software upgrade package to an ftp, http, or https server. You may specify any location as part of the URL. Below is an example for each of the three server types:
ftp://kioxia:welcome@###.##.##.##/ftp/upload/upgrade_nvmf_<version>-<sub_version>.tar.gpg
http://###.##.##.##/ks-upgrades/upgrade_nvmf_<version>-<sub_version>.tar.gpg
https://###.##.##.##/ks-upgrades/upgrade_nvmf_<version>-<sub_version>.tar.gpg
2. Modify the Software upgrade secret yaml file as follows:
  • Insert the URL from step 1.
  • If username and password are required for URL access, encode them into base 64 and add to the file under data.

For example, using the secret yaml file called softwareupgrade-secret.yaml:

apiVersion: v1

kind: Secret

metadata:

name: softwareupgrade-secret

namespace: default

type: Opaque

stringData:

url: ftp://<server_IP_address>:/ftp/upgrade_nvmf_<version>-<subversion>.tar.gpg

data:

username: dGVzdHVzZXI=

password: QUJDRDEyMzQ=
3. Create the KumoScale software upgrade secret CR with the command:
kubectl create -f softwareupgrade-secret.yaml

You should receive the message

secret/softwareupgrade-secret created

You can also verify the secret was created with

/usr/bin/kubectl get secret 

to see the name, type, and age of the secret.

4. Modify the software upgrade CR yaml file with values for:
    1. upgradeSecretName: The name of the secret file from step 2 containing the ftp, http, or https connection details.
    2. waitPeriod: The time to wait between storage nodes upgrades in minutes. The default (and recommended) value is 5 minutes. The value can range between 1 and 1440. 1440 minutes is equivalent to 1 day.
    3. adminState: This should be equal to either RUN or STOP to indicate whether the Admin user can resume (RUN) or stop (STOP) the upgrade process. The default value is RUN. Admin users can change adminState (from STOP to RUN or from RUN to STOP) as long as the state of the upgrade is equal to Pending, Upgrading, or Stopped.

Below is an example of the upgrade CR file kumoscale_v1_softwareupgrade_cr.yaml:

apiVersion: kumoscale.kioxia.com/v1

kind: SoftwareUpgrade

metadata:

name: softwareupgrade

spec:

upgradeSecretName: softwareupgrade-secret

waitPeriod: 5

adminState: RUN
5. Start the upgrade process by creating the software upgrade CR. For example, using  kumoscale_v1_softwareupgrade_cr. yaml:
kubectl create -f kumoscale_v1_softwareupgrade_cr.yaml

The nodes will be rebooted once in a cycle.

6. Verify the upgrade was successful by checking the version returned by either of the following commands.
kubectl get softwareupgrade -o wide

kubectl describe softwareupgrade <name>

Update

This section explains how to perform updates on various components of KumoScale software. These user operations need to be done externally from a remote host via kubectl commands.

Update the License

Edit the license CR with the new license key and execute the following:

kubectl apply -f kumoscale.kioxia.com_v1_license_cr.yaml

Update a Syslog

Edit the Syslog CR with the new version information and execute the following:

kubectl apply -f kumoscale.kioxia.com_syslogs.yaml

Upgrading the SSD Firmware

NOTE: Selecting a missing SSD invalidates the process.

 

The SSD firmware upgrade is done using SSD Custom Resources. See the User Manual for complete instructions on how to upgrade firmware.

NVMe Patches on Application Initiators

This section describes how to setup a patch for NVMe module application initiators with an example (kernel 4.18.0-193.el8), and resolves timing issues seen when an initiator tries to connect to a target.

To verify the patch is installed, run the following on the initiator machine

# modinfo nvme-core|grep description

If the result is nvme host KIOXIA patch (compiled 2020-12-01_12-13-59), then the patch is installed, otherwise the initiator runs with original kernel module.

To install the NVMe initiator patch:

# rpm -i nvme-centos82-patch-2020_12_01_12_13_59-1.x86_64.rpm

To uninstall the NVMe initiator patch:

# rpm -e nvme-centos82-patch-2020_12_01_12_13_59-1.x86_64

[1] To upgrade KumoScale cluster running software versions older than 3.18, please contact your KIOXIA support team.

 

 

1    Maintenance

1.1   KumoScale Software Cluster Upgrade

This section describes the requirements and process for upgrading an existing cluster from KumoScale software version 3.20[1]. The process will upgrade the following components in the order below:

  1. Storage Nodes: The management engine, target module, CLI, and local registry catalogue will be upgraded with all new containers.
  2. KumoScale software and Kubernetes internal services: All services automatically installed with the KumoScale software, such as the operators, CSI, and Provisioner will be upgraded.

The upgrade process maintains configuration and user data between versions. If you have a high-availability KumoScale Installation, the upgrade will happen without service disruption. It does NOT support the following:

  1. Upgrades to services that were manually installed by the user. For example, Prometheus, Fluentd™, and Loki™, must be manually upgraded through the CRD.
  2. Upgrades to the:
  • Private Kubernetes cluster for the KumoScale storage cluster.
  • Storage node’s Operating System (OS) and kernel versions.
  1. Upgrades to external components and agents deployed using the OpenStack Cinder driver, Nova Connector, Ansible Client, Ansible modules, and Customer-developed CSI deployments in Kubernetes.
  2. Adding a new storage node or removing an existing one during the cluster upgrade.

A syslog event is generated at the beginning and end of all upgrades so that you can review the results of a successful or failed upgrade.

 

NOTE

If you need to upgrade a specific component independently, you must contact technical support for guidance. The process varies from the one described and supported in this document.

 

 

 

 

1.1.1  Requirements for Upgrading

To upgrade an existing cluster, you will need to confirm the following:3

  1. Access to a server with ftp, http, or https in order to copy the image used for upgrading including the required credentials, if any.
  2. The cluster is healthy. In particular:
  • All storage nodes are up and running.
  • The time is synchronized across all nodes in the cluster.
  • The KumoScale Provisioner service has a valid license.
  • The KumoScale secret file is deployed and active.
  • There is at least one initialized storage node.
  1. All KumoScale storage nodes and components are running the same version.
  2. No node or replica is in a synchronizing state.
  3. No storage node has a replica of a degraded resilient volume. Such a node cannot be rebooted; thus you must wait to upgrade until this condition is resolved.

1.2   Upgrade Process

Once you have confirmed all requirements for upgrading have been met, you are ready to complete the following steps to perform the upgrade.

  1. Copy the software upgrade package to an ftp, http, or https server. You may specify any location as part of the URL. Below is an example for each of the three server types:

ftp://kioxia:welcome@###.##.##.##/ftp/upload/upgrade_nvmf_<version>-<sub_version>.tar.gpg

 

http://###.##.##.##/ks-upgrades/upgrade_nvmf_<version>-<sub_version>.tar.gpg

 

https://###.##.##.##/ks-upgrades/upgrade_nvmf_<version>-<sub_version>.tar.gpg

 

  1. Modify the Software upgrade secret yaml file as follows:
  • Insert the URL from step 1.
  • If username and password are required for URL access, encode them into base 64 and add to the file under data.

 

 

For example, using the secret yaml file called softwareupgrade-secret.yaml:

apiVersion: v1

kind: Secret

metadata:

name: softwareupgrade-secret

namespace: default

type: Opaque

stringData:

url: ftp://<server_IP_address>:/ftp/upgrade_nvmf_<version>-<subversion>.tar.gpg

data:

username: dGVzdHVzZXI=

password: QUJDRDEyMzQ=

 

  1. Create the KumoScale software upgrade secret CR with the command:

kubectl create -f softwareupgrade-secret.yaml

You should receive the message secret/softwareupgrade-secret created.

You can also verify the secret was created with /usr/bin/kubectl get secret to see the name, type, and age of the secret.

 

  1. Modify the software upgrade CR YAML file with values for:
    1. upgradeSecretName: The name of the secret file from step 2 containing the ftp, http, or https connection details.
    2. waitPeriod: The time to wait between storage nodes upgrades in minutes. The default (and recommended) value is 5 minutes. The value can range between 1 and 1440. 1440 minutes is equivalent to 1 day.
    3. adminState: This should be equal to either RUN or STOP to indicate whether the Admin user can resume (RUN) or stop (STOP) the upgrade process. The default value is RUN. Admin users can change adminState (from STOP to RUN or from RUN to STOP) as long as the state of the upgrade is equal to Pending, Upgrading, or Stopped.

Below is an example of the upgrade CR file kumoscale_v1_softwareupgrade_cr.yaml:

apiVersion: kumoscale.kioxia.com/v1

kind: SoftwareUpgrade

metadata:

name: softwareupgrade

spec:

upgradeSecretName: softwareupgrade-secret

waitPeriod: 5

adminState: RUN

  1. Start the upgrade process by creating the software upgrade CR. For example, using yaml:

kubectl create -f kumoscale_v1_softwareupgrade_cr.yaml

The nodes will be rebooted once in a cycle.

 

  1. Verify the upgrade was successful by checking the version returned by either of the following commands.

kubectl get softwareupgrade -o wide

kubectl describe softwareupgrade <name>

 

1.3   Update

This section explains how to perform updates on various components of KumoScale software. As mentioned in Access the KumoScale Storage Cluster from a Remote Administrative Host, these user operations need to be done externally from a remote host via kubectl commands.

 

1.3.1  Update the License

Edit the license CR with the new license key and execute the following:

kubectl apply -f kumoscale.kioxia.com_v1_license_cr.yaml

 

1.3.2  Update a Syslog

Edit the Syslog CR with the new version information and execute the following:

kubectl apply -f kumoscale.kioxia.com_syslogs.yaml

 

1.4   Upgrading the SSD Firmware

 

NOTE

Selecting a missing SSD invalidates the process.

 

The SSD firmware upgrade is done using SSD Custom Resources. See the User Manual for complete instructions on how to upgrade firmware.

 

1.5   NVMe Patches on Applications Initiators

This section describes how to setup a patch for NVMe module application initiators with an example
(kernel 4.18.0-193.el8), and resolves timing issues seen when an initiator tries to connect to a target.

To verify the patch is installed, run the following on the initiator machine

# modinfo nvme-core|grep description

 

If the result is nvme host KIOXIA patch (compiled 2020-12-01_12-13-59), then the patch is installed, otherwise the initiator runs with original kernel module.

To install the NVMe initiator patch:

# rpm -i nvme-centos82-patch-2020_12_01_12_13_59-1.x86_64.rpm

 

To uninstall the NVMe initiator patch:

# rpm -e nvme-centos82-patch-2020_12_01_12_13_59-1.x86_64

 

[1] To upgrade KumoScale cluster running software versions older than 3.18, please contact your KIOXIA support team.

 

password: QUJDRDEyMzQ=

 

  1. Create the KumoScale software upgrade secret CR with the command:

kubectl create -f softwareupgrade-secret.yaml

You should receive the message secret/softwareupgrade-secret created.

You can also verify the secret was created with /usr/bin/kubectl get secret to see the name, type, and age of the secret.

 

  1. Modify the software upgrade CR YAML file with values for:
    1. upgradeSecretName: The name of the secret file from step 2 containing the ftp, http, or https connection details.
    2. waitPeriod: The time to wait between storage nodes upgrades in minutes. The default (and recommended) value is 5 minutes. The value can range between 1 and 1440. 1440 minutes is equivalent to 1 day.
    3. adminState: This should be equal to either RUN or STOP to indicate whether the Admin user can resume (RUN) or stop (STOP) the upgrade process. The default value is RUN. Admin users can change adminState (from STOP to RUN or from RUN to STOP) as long as the state of the upgrade is equal to Pending, Upgrading, or Stopped.

Below is an example of the upgrade CR file kumoscale_v1_softwareupgrade_cr.yaml:

apiVersion: kumoscale.kioxia.com/v1

kind: SoftwareUpgrade

metadata:

name: softwareupgrade

spec:

upgradeSecretName: softwareupgrade-secret

waitPeriod: 5

adminState: RUN

  1. Start the upgrade process by creating the software upgrade CR. For example, using yaml:

kubectl create -f kumoscale_v1_softwareupgrade_cr.yaml

The nodes will be rebooted once in a cycle.

 

  1. Verify the upgrade was successful by checking the version returned by either of the following commands.

kubectl get softwareupgrade -o wide

kubectl describe softwareupgrade <name>

 

1.3   Update

This section explains how to perform updates on various components of KumoScale software. As mentioned in Access the KumoScale Storage Cluster from a Remote Administrative Host, these user operations need to be done externally from a remote host via kubectl commands.

 

1.3.1  Update the License

Edit the license CR with the new license key and execute the following:

kubectl apply -f kumoscale.kioxia.com_v1_license_cr.yaml

 

1.3.2  Update a Syslog

Edit the Syslog CR with the new version information and execute the following:

kubectl apply -f kumoscale.kioxia.com_syslogs.yaml

 

1.4   Upgrading the SSD Firmware

 

NOTE

Selecting a missing SSD invalidates the process.

 

The SSD firmware upgrade is done using SSD Custom Resources. See the User Manual for complete instructions on how to upgrade firmware.

 

1.5   NVMe Patches on Applications Initiators

This section describes how to setup a patch for NVMe module application initiators with an example
(kernel 4.18.0-193.el8), and resolves timing issues seen when an initiator tries to connect to a target.

To verify the patch is installed, run the following on the initiator machine

# modinfo nvme-core|grep description

 

If the result is nvme host KIOXIA patch (compiled 2020-12-01_12-13-59), then the patch is installed, otherwise the initiator runs with original kernel module.

To install the NVMe initiator patch:

# rpm -i nvme-centos82-patch-2020_12_01_12_13_59-1.x86_64.rpm

 

To uninstall the NVMe initiator patch:

# rpm -e nvme-centos82-patch-2020_12_01_12_13_59-1.x86_64

 

[1] To upgrade KumoScale cluster running software versions older than 3.18, please contact your KIOXIA support team.