Release Notes

1      KumoScale Storage Node Software

KumoScale storage node Software Release 3.19 delivers new features improvements and several bug fixes.

1.1    Released Software Components

Item

Description

Revision

KumoScale ISO Network Installation

KumoScale storage node software ISO Network Installation Package

3.19-14660

KumoScale ISO USB Installation

KumoScale storage node software ISO USB Installation Package

3.19-14660

KumoScale Virtual CD Installation

KumoScale storage node software Virtual CD Installation Package

3.19-14660

1.2    Hardware Requirements

See the Hardware Compatibility List (HCL) document.

1.3    New Features

  • Added support for Evict SSD allowing sub-volume migration, control over SSD allocation and support for online migration of a sub-volume which belongs to a thin-provisions volume.
  • Added support for NVMe-oF™ RoCEv2 using Mellanox™ MT2892 Family NIC [ConnectX™-6 Dx].
  • Added support for makeusb.sh to work with Ubuntu Desktop 20.04LTS
  • Added Support for CNI flannel pod upgrade.
  • Add ethtool output to collect logs operation.

1.4    Modified Features

  • Added SSD and volume utilization, latency, IOPs and bandwidth to the telemetry data feed.
  • NVMe-oF Target complies with NVMe™ 1.4.

1.5    Deprecated Features

  • None

1.6   Issues Fixed Since Previous Release

KumoScale Storage Software v. 3.19-14660, released in December, has been updated to fix NOF-6874 below.  It now uses Apache™ Log4j version 2.16.0 which addresses Apache Log4j Common Vulnerabilities and Exposures (CVE) CVE-2021-44228 and CVE-2021-45046. For more information on these vulnerabilities, see

https://logging.apache.org/log4j/2.x/security.html. Updates to Apache Log4j v. 2.16.0 were made to both the KumoScale Storage Node and KumoScale Provisioner. Please see the KumoScale Installation Guide for instructions on how to install or upgrade to the latest version.

Bug ID

Description

NOF-6874

Fix Engine security vulnerability in Apache Log4j.

1.7    Open Issues and Known Limitations

IDs of new issues are marked in bold.

1.7.1   High-Impact Issues

Bug ID

Description

NOF-5384

NOF-5240

In rare instances, a new deployment on a SuperMicro™ 2029 system may fail. The Last reported message shown is Couldn’t get size 0x800000000000000e.

Mitigation: A second deployment attempt will succeed.

NOF-3366 

NOF-3196

SSD hot-plug is not always recognized by Linux™ OS on a SuperMicro 2029U-TN24R4T server.

Mitigation: Power cycle the server to ensure that all SSDs are discovered.

NOF-2460

NOF-570 

NOF-426

SSD hot-plug may fail on a SuperMicro 2028U-TN24R4T server.

Mitigation: Please power cycle the server to ensure that all SSDs are discovered.

NOF-516

Removing an SSD via Baseboard Management Controller (BMC)[1] while running IO commands to it, may cause KumoScale software to malfunction.

Mitigation: Do not remove an SSD via the BMC.

1.7.2   Medium-Impact Issues

Bug ID

Description

NOF-4319

Transmission Control Protocol (TCP) transport session disconnection might lead to a small memory leak.

Mitigation: This issue is most likely due to a problem in the NVMe-TCP initiator code. A similar memory leak was detected in the open source NVMe-TCP target implementation. The issue was reported to the Linux OS NVMe protocol open-source community. No mitigation currently exists for this issue.

NOF-3023

All KumoScale software targets disconnect, leading to an initiator crash, if portal IP addresses clash.

Mitigation: When configuring portals, make sure they have unique IP addresses.

NOF-2487

A KumoScale software target with large number of initiators may disconnect all initiators if one of the initiators disconnects.

Mitigation: No mitigation found. It is advised not to connect a large number of initiators to a single target.

NOF-1482

 

Renewing the Dynamic Host Configuration Protocol (DHCP) lease does not work if the new portal IP address is different.

Mitigation: Set the DHCP lease expiry time to infinite (usually -1) or configure the DHCP server to bind the MAC address to the IP address.

1.7.3   Low-Impact Issues

Bug ID

Description

NOF-5392

Deployment on a Dell EMC™ PowerEdge™ R7415 Rack Server will print IO_PAGE_FAULT messages during a PXE installation.

Mitigation: Add iommu=soft to the grub command line options.

NOF-5098

Storage node logs report repeated kubelet errors.

Mitigation: Add the storage node to a KumoScale cluster.

NOF-5332

In case a deployment was done via USB with a static management IP and all of the network interfaces are not connected, the Cluster VIP might not be accessible if the management interface was connected after the deployment of the first storage node has completed.

Mitigation: Make sure the management interface is connected during the deployment.

NOF-4842

Moving bootable NVMe disks to a non-bootable NVMe slot results in the system booting from SATA SSDs.

Mitigation: Do not use non-bootable NVMe slots for storage node OS.

NOF-4758

OpenStack Live Migration fails in the case of rebooting a node during a Live Migration operation.

Mitigation: Do not reboot nodes during Live Migration operations.

2      KumoScale Provisioner Service

KumoScale Provisioner service 3.19 provides new functionality and improvements.

2.1    Released Components

Item   

Description

Revision

KumoScale Provisioner Software

The KumoScale software suite includes a distributed, resilient provisioner service to more efficiently manage disaggregated flash storage at data center scale. The KumoScale Provisioner service performs intelligent and dynamic mapping of user-specified storage volumes to physical SSDs over the data center network, utilizing the NVMe over Fabrics (NVMe-oF) standard.

3.19-14655

2.2    System Requirements

See KumoScale Installation Guide.

2.3    Compliance with Other KumoScale Components

The KumoScale Provisioner service is deployed in an environment with KumoScale storage nodes, as part of the KumoScale Management Cluster. It maintains an inventory of the different storage nodes and selects the most suitable storage node (or nodes, for a resilient volume) for the newly created volume.

Interfacing with the KumoScale Provisioner service may be done in one of four ways, depending on the customer’s environment and orchestration platform:

  • Directly, by calling the KumoScale Provisioner REST APIs (see the KumoScale REST APIs documentation for details).
  • Using the KumoScale Provisioner’s CLI (see KumoScale CLI documentation for more details).
  • Via the Ansible modules (for a bare-metal KumoScale software installation).
  • Via the KumoScale software Kubernetes CSI driver, for customers using the Kubernetes orchestration platform.

The following table describes the different component versions compliant with the current KumoScale Provisioner service releases:

KumoScale Provisioner Version

KumoScale Software Version

KumoScale CSI Driver Version

KumoScale Ansible Version

3.19-1294

3.19-14655

3.19-1614

3.19-397

2.4    New Features

The KumoScale Provisioner service 3.19-1294 software release includes the following new features:

  • Added Get Volume Extended to the KumoScale REST API which provides SSDs and connectivity information of Volumes.

2.5    Modified Features

  • Added publish validations for local volume.
  • Added host name field to the syslog events.

2.6    Deprecated Features

None.

2.7   Issues Fixed Since Previous Release

KumoScale Provisioner Software  has been updated to fix NOF-6875 below.  It now uses Apache™ Log4j version 2.16.0 which addresses Apache Log4j Common Vulnerabilities and Exposures (CVE) CVE-2021-44228 and CVE-2021-45046. For more information on these vulnerabilities, see

https://logging.apache.org/log4j/2.x/security.html. Updates to Apache Log4j v. 2.16.0 were made to both the KumoScale Storage Node and KumoScale Provisioner. Please see the KumoScale Installation Guide for instructions on how to install or upgrade to the latest version.

Bug ID

Description

NOF-6875

Fix Provisioner security vulnerability in Apache Log4j.

2.8    Open Issues and Known Limitations

IDs of new issues are marked in bold.

2.8.1   High-Impact Issues

Bug ID

Description

NOF-6422

When trying to concurrently delete multiple snapshots taken from the same source volume operation might fail for some of the snapshots.

Mitigation: Re-Issue the delete operation for any failed deleted Snapshot

2.8.2   Medium-Impact Issues

Bug ID

Description

NOF-5938

Until all storage nodes discovered, the Tenants Volumes Budget may not be enforced during provisioner service restart.

Mitigation: wait patiently for discovery.

2.8.3   Low-Impact Issues

Bug ID

Description

NOF-4358

KumoScale Provisioner service reports the initially configured reserved space as its reserved space.

Mitigation: Query the KumoScale storage node that the volume is allocated on to display the actual reserved space.

3      KumoScale Operators for KumoScale Management

KumoScale Operators is based on version 3.18 and provides new functionality and improvements.

3.1    Released Components

Item   

Description

Revision

KumoScale Operators Software

KumoScale Software Kubernetes Operators Software for KumoScale Management

3.19-1622

3.2    New Features

Release 3.19 enhances KumoScale’s operators for the installation and configuration of the storage nodes. Operators are used to/for:

  • SSD maintenance and firmware upgrades via SSD Operation CR.
  • Tenant configuration and monitoring via Tenant CR.

3.3    Modified Features

  • Verify the IP and MASK parameters while configuring Static MgmIP via StorageNode CR.
  • In case of Software Upgrade failure the Software Upgrade Status description field presents additional information about the reason of
  • "kubectl getsoftwareupgrades" shows the following value “0” in case of failure in Upload or Upgrade.

3.4    Deprecated Software

None.

3.5    Open Issues and Known Limitations

3.5.1   High- Impact Issues

Bug ID

Description

NOF-6398

When trying to add a Storage Node that was previously part of the KumoScale Cluster, rarely, such operation might fail with "The storage node already exists" error.

Mitigation: In such case delete the Storage Node and redeploy it via ISO prior a second join attempt.

3.5.2   Medium-Impact Issues

None.

3.5.3   Low-Impact Issues

None.

4      KumoScale CSI Driver for Kubernetes Orchestration

The KumoScale CSI driver for Kubernetes orchestration is based on KumoScale CSI version 3.18 and provides improvements.

4.1    Released Components

Item   

Description

Revision

KumoScale CSI software

KumoScale CSI driver binary container for Kubernetes orchestration

3.19-1416

4.2    Software Requirements

  • Kubernetes Orchestration Version Number 14 or greater.

4.3    Compliance with other KumoScale Components

Below are the versions of KumoScale software, Provisioner, and CSI Driver that have been validated together; other combinations have not been tested and are not supported with version 3.19.

KumoScale Software Version

KumoScale Provisioner Version

KumoScale CSI Driver Version

3.19-14655

3.19-1294

3.19-1614

4.4    New Features

  • Added support for Kubernetes clone volume.
  • Support Kubernetes versions 1.20 and 1.21.
  • Support for NVMe MultiPath based kernel used by Ubuntu 20.04.

4.5    Modified Features

  • Update support for Kubernetes Snapshot and snapshot volumes.
  • Bare-Metal Agent supports multi attach volumes and cluster MD.

4.6    Deprecated Features

None.

4.7    Fixed Issues from Previous Release

None.

4.8    Open Issues and Known Limitations

None.

5      KumoScale Software Ansible Modules

KumoScale software Ansible modules is based on KumoScale Software Ansible Modules version 3.18 and provides improvements. The modules include Ansible playbooks which may be used as a design reference for configuring and provisioning KumoScale storage nodes via the KumoScale Provisioner service.

5.1    Released Components

Item   

Description

Revision

KumoScale Software Ansible Modules

KumoScale software Ansible modules for bare-metal installation

3.19-397

5.2    System Requirements

The Ansible client may run on a physical or virtual machine (VM) and connect to KumoScale storage nodes via the machines' management ports.

  • Python™ Version: Python version on the hosts and the Ansible client ≥ 3.
  • Ansible Module Version: KumoScale Software Ansible Modules assume version 3.

5.3    Compliance with Other KumoScale Components

Below are the versions of KumoScale software, Provisioner, and Ansible Modules that have been validated together; other combinations have not been tested and are not supported with version 3.19.

KumoScale Software Version

KumoScale Provisioner Version

KumoScale Ansible Modules

3.19-14655

3.19-1294

3.19-397

5.4    New Features

  • Improved the ks_mirror module to handle Cluster MD, for GPFS support.

5.5    Modified Features

  • Removed "ks_portal” and “ks_group” modules and playbooks.
  • Instead of “ - - limit” , users should use “hosts_group” parameter available in vars.yaml file.
  • Naming convention: host_main parameter replaces the host

5.6    Deprecated Features

None.

5.7    Fixed Issues from Previous Release

None.

5.8    Open Issues and Known Limitations

5.8.1   High-Impact Issues

Bug ID

Description

NOF-6421

When using GPFS with ClusterMD , the Self-Healing process  might fail to recover a replica located on storage node that is permanently inaccessible.

Mitigation: None.

5.8.2   Medium-Impact Issues

Bug ID

Description

NOF-6366

After reboot of a  complete GPFS Cluster, some GPFS cluster nodes might not discover all ClusterMD volumes

Mitigation: manual assemble using “mdadm -A –scan"  command OR reboot of the failed node.

5.8.3   Low-Impact Issues

None.

6      KumoScale Software for OpenStack Platforms

KumoScale OpenStack Wallaby Software includes a KumoScale NVMe-oF Connector, Cinder Driver and Agent module for OpenStack Wallaby Release. KumoScale software for OpenStack platforms integrates KumoScale software into the customer’s OpenStack Cluster.

This version is based on KumoScale software for the OpenStack platform and is tested with OpenStack version Wallaby. It enables the OpenStack cluster to provision KumoScale volumes and includes a KumoScale Cinder driver for the control plane as well as a KumoScale software NVMe-oF connector that is used by the NOVA OS brick layer.

6.1    Released Components

Item   

Description

Revision

KumoScale software OpenStack platform software agent

KumoScale OpenStack Wallaby Software

3.19.179

6.2    New Features

  • Support for NVMe MultiPath based kernel used by Ubuntu 20.04.

6.3    Open Issues and Known Limitations

Bug ID

Description

NOF-6422

When trying to concurrently delete multiple snapshots taken from the same source volume operation might fail for some of the snapshots.

Mitigation: Re-Issue the delete operation for any failed deleted Snapshot

7      Initiator Server Software Requirements

  • Operating System

For NVMe-oF RoCEv2

For NVMe-oF TCP

Linux OS with kernel 4.9.64 x86 or newer.

LINUX OS kernel must support NVMe over TCP, 5.0 and above.

Please consult with KIOXIA Technical Support regarding the exact distribution you plan to use.

  • NVMe Command Line Interface (CLI): NVMe CLI 1.6 or newer.

[1] On Dell EMC servers BMC is called iDRAC. On HPE servers it is called iLO.