Release Notes
      Back to home
      1. KumoScale Documentation
      2. KumoScale™ Software v3.19
      3. Release Notes

      Release Notes

      1      KumoScale Storage Node Software

      KumoScale storage node Software Release 3.19 delivers new features improvements and several bug fixes.

      1.1    Released Software Components

      Item

      Description

      Revision

      KumoScale ISO Network Installation

      KumoScale storage node software ISO Network Installation Package

      3.19-14660

      KumoScale ISO USB Installation

      KumoScale storage node software ISO USB Installation Package

      3.19-14660

      KumoScale Virtual CD Installation

      KumoScale storage node software Virtual CD Installation Package

      3.19-14660

      1.2    Hardware Requirements

      See the Hardware Compatibility List (HCL) document.

      1.3    New Features

      • Added support for Evict SSD allowing sub-volume migration, control over SSD allocation and support for online migration of a sub-volume which belongs to a thin-provisions volume.
      • Added support for NVMe-oF™ RoCEv2 using Mellanox™ MT2892 Family NIC [ConnectX™-6 Dx].
      • Added support for makeusb.sh to work with Ubuntu Desktop 20.04LTS
      • Added Support for CNI flannel pod upgrade.
      • Add ethtool output to collect logs operation.

      1.4    Modified Features

      • Added SSD and volume utilization, latency, IOPs and bandwidth to the telemetry data feed.
      • NVMe-oF Target complies with NVMe™ 1.4.

      1.5    Deprecated Features

      • None

      1.6   Issues Fixed Since Previous Release

      KumoScale Storage Software v. 3.19-14660, released in December, has been updated to fix NOF-6874 below.  It now uses Apache™ Log4j version 2.16.0 which addresses Apache Log4j Common Vulnerabilities and Exposures (CVE) CVE-2021-44228 and CVE-2021-45046. For more information on these vulnerabilities, see

      https://logging.apache.org/log4j/2.x/security.html. Updates to Apache Log4j v. 2.16.0 were made to both the KumoScale Storage Node and KumoScale Provisioner. Please see the KumoScale Installation Guide for instructions on how to install or upgrade to the latest version.

      Bug ID

      Description

      NOF-6874

      Fix Engine security vulnerability in Apache Log4j.

      1.7    Open Issues and Known Limitations

      IDs of new issues are marked in bold.

      1.7.1   High-Impact Issues

      Bug ID

      Description

      NOF-5384

      NOF-5240

      In rare instances, a new deployment on a SuperMicro™ 2029 system may fail. The Last reported message shown is Couldn’t get size 0x800000000000000e.

      Mitigation: A second deployment attempt will succeed.

      NOF-3366 

      NOF-3196

      SSD hot-plug is not always recognized by Linux™ OS on a SuperMicro 2029U-TN24R4T server.

      Mitigation: Power cycle the server to ensure that all SSDs are discovered.

      NOF-2460

      NOF-570 

      NOF-426

      SSD hot-plug may fail on a SuperMicro 2028U-TN24R4T server.

      Mitigation: Please power cycle the server to ensure that all SSDs are discovered.

      NOF-516

      Removing an SSD via Baseboard Management Controller (BMC)[1] while running IO commands to it, may cause KumoScale software to malfunction.

      Mitigation: Do not remove an SSD via the BMC.

      1.7.2   Medium-Impact Issues

      Bug ID

      Description
      NOF-4319

      Transmission Control Protocol (TCP) transport session disconnection might lead to a small memory leak.

      Mitigation: This issue is most likely due to a problem in the NVMe-TCP initiator code. A similar memory leak was detected in the open source NVMe-TCP target implementation. The issue was reported to the Linux OS NVMe protocol open-source community. No mitigation currently exists for this issue.

      NOF-3023

      All KumoScale software targets disconnect, leading to an initiator crash, if portal IP addresses clash.

      Mitigation: When configuring portals, make sure they have unique IP addresses.

      NOF-2487

      A KumoScale software target with large number of initiators may disconnect all initiators if one of the initiators disconnects.

      Mitigation: No mitigation found. It is advised not to connect a large number of initiators to a single target.

      NOF-1482

       

      Renewing the Dynamic Host Configuration Protocol (DHCP) lease does not work if the new portal IP address is different.

      Mitigation: Set the DHCP lease expiry time to infinite (usually -1) or configure the DHCP server to bind the MAC address to the IP address.

      1.7.3   Low-Impact Issues

      Bug ID

      Description

      NOF-5392

      Deployment on a Dell EMC™ PowerEdge™ R7415 Rack Server will print IO_PAGE_FAULT messages during a PXE installation.

      Mitigation: Add iommu=soft to the grub command line options.

      NOF-5098

      Storage node logs report repeated kubelet errors.

      Mitigation: Add the storage node to a KumoScale cluster.

      NOF-5332

      In case a deployment was done via USB with a static management IP and all of the network interfaces are not connected, the Cluster VIP might not be accessible if the management interface was connected after the deployment of the first storage node has completed.

      Mitigation: Make sure the management interface is connected during the deployment.

      NOF-4842

      Moving bootable NVMe disks to a non-bootable NVMe slot results in the system booting from SATA SSDs.

      Mitigation: Do not use non-bootable NVMe slots for storage node OS.

      NOF-4758

      OpenStack Live Migration fails in the case of rebooting a node during a Live Migration operation.

      Mitigation: Do not reboot nodes during Live Migration operations.

      2      KumoScale Provisioner Service

      KumoScale Provisioner service 3.19 provides new functionality and improvements.

      2.1    Released Components

      Item   

      Description

      Revision

      KumoScale Provisioner Software

      The KumoScale software suite includes a distributed, resilient provisioner service to more efficiently manage disaggregated flash storage at data center scale. The KumoScale Provisioner service performs intelligent and dynamic mapping of user-specified storage volumes to physical SSDs over the data center network, utilizing the NVMe over Fabrics (NVMe-oF) standard.

      3.19-14655

      2.2    System Requirements

      See KumoScale Installation Guide.

      2.3    Compliance with Other KumoScale Components

      The KumoScale Provisioner service is deployed in an environment with KumoScale storage nodes, as part of the KumoScale Management Cluster. It maintains an inventory of the different storage nodes and selects the most suitable storage node (or nodes, for a resilient volume) for the newly created volume.

      Interfacing with the KumoScale Provisioner service may be done in one of four ways, depending on the customer’s environment and orchestration platform:

      • Directly, by calling the KumoScale Provisioner REST APIs (see the KumoScale REST APIs documentation for details).
      • Using the KumoScale Provisioner’s CLI (see KumoScale CLI documentation for more details).
      • Via the Ansible modules (for a bare-metal KumoScale software installation).
      • Via the KumoScale software Kubernetes CSI driver, for customers using the Kubernetes orchestration platform.

      The following table describes the different component versions compliant with the current KumoScale Provisioner service releases:

      KumoScale Provisioner Version

      KumoScale Software Version

      KumoScale CSI Driver Version

      KumoScale Ansible Version

      3.19-1294

      3.19-14655

      3.19-1614

      3.19-397

      2.4    New Features

      The KumoScale Provisioner service 3.19-1294 software release includes the following new features:

      • Added Get Volume Extended to the KumoScale REST API which provides SSDs and connectivity information of Volumes.

      2.5    Modified Features

      • Added publish validations for local volume.
      • Added host name field to the syslog events.

      2.6    Deprecated Features

      None.

      2.7   Issues Fixed Since Previous Release

      KumoScale Provisioner Software  has been updated to fix NOF-6875 below.  It now uses Apache™ Log4j version 2.16.0 which addresses Apache Log4j Common Vulnerabilities and Exposures (CVE) CVE-2021-44228 and CVE-2021-45046. For more information on these vulnerabilities, see

      https://logging.apache.org/log4j/2.x/security.html. Updates to Apache Log4j v. 2.16.0 were made to both the KumoScale Storage Node and KumoScale Provisioner. Please see the KumoScale Installation Guide for instructions on how to install or upgrade to the latest version.

      Bug ID

      Description

      NOF-6875

      Fix Provisioner security vulnerability in Apache Log4j.

      2.8    Open Issues and Known Limitations

      IDs of new issues are marked in bold.

      2.8.1   High-Impact Issues

      Bug ID

      Description

      NOF-6422

      When trying to concurrently delete multiple snapshots taken from the same source volume operation might fail for some of the snapshots.

      Mitigation: Re-Issue the delete operation for any failed deleted Snapshot

      2.8.2   Medium-Impact Issues

      Bug ID

      Description

      NOF-5938

      Until all storage nodes discovered, the Tenants Volumes Budget may not be enforced during provisioner service restart.

      Mitigation: wait patiently for discovery.

      2.8.3   Low-Impact Issues

      Bug ID

      Description

      NOF-4358

      		 KumoScale Provisioner service reports the initially configured reserved space as its reserved space.

      Mitigation: Query the KumoScale storage node that the volume is allocated on to display the actual reserved space.

      3      KumoScale Operators for KumoScale Management

      KumoScale Operators is based on version 3.18 and provides new functionality and improvements.

      3.1    Released Components

      Item   

      Description

      Revision

      KumoScale Operators Software

      KumoScale Software Kubernetes Operators Software for KumoScale Management

      3.19-1622

      3.2    New Features

      Release 3.19 enhances KumoScale’s operators for the installation and configuration of the storage nodes. Operators are used to/for:

      • SSD maintenance and firmware upgrades via SSD Operation CR.
      • Tenant configuration and monitoring via Tenant CR.

      3.3    Modified Features

      • Verify the IP and MASK parameters while configuring Static MgmIP via StorageNode CR.
      • In case of Software Upgrade failure the Software Upgrade Status description field presents additional information about the reason of
      • "kubectl getsoftwareupgrades" shows the following value “0” in case of failure in Upload or Upgrade.

      3.4    Deprecated Software

      None.

      3.5    Open Issues and Known Limitations

      3.5.1   High- Impact Issues

      Bug ID

      Description

      NOF-6398

      When trying to add a Storage Node that was previously part of the KumoScale Cluster, rarely, such operation might fail with "The storage node already exists" error.

      Mitigation: In such case delete the Storage Node and redeploy it via ISO prior a second join attempt.

      3.5.2   Medium-Impact Issues

      None.

      3.5.3   Low-Impact Issues

      None.

      4      KumoScale CSI Driver for Kubernetes Orchestration

      The KumoScale CSI driver for Kubernetes orchestration is based on KumoScale CSI version 3.18 and provides improvements.

      4.1    Released Components

      Item   

      Description

      Revision

      KumoScale CSI software

      KumoScale CSI driver binary container for Kubernetes orchestration

      3.19-1416

      4.2    Software Requirements

      • Kubernetes Orchestration Version Number 14 or greater.

      4.3    Compliance with other KumoScale Components

      Below are the versions of KumoScale software, Provisioner, and CSI Driver that have been validated together; other combinations have not been tested and are not supported with version 3.19.

      KumoScale Software Version

      KumoScale Provisioner Version

      KumoScale CSI Driver Version

      3.19-14655

      3.19-1294

      3.19-1614

      4.4    New Features

      • Added support for Kubernetes clone volume.
      • Support Kubernetes versions 1.20 and 1.21.
      • Support for NVMe MultiPath based kernel used by Ubuntu 20.04.

      4.5    Modified Features

      • Update support for Kubernetes Snapshot and snapshot volumes.
      • Bare-Metal Agent supports multi attach volumes and cluster MD.

      4.6    Deprecated Features

      None.

      4.7    Fixed Issues from Previous Release

      None.

      4.8    Open Issues and Known Limitations

      None.

      5      KumoScale Software Ansible Modules

      KumoScale software Ansible modules is based on KumoScale Software Ansible Modules version 3.18 and provides improvements. The modules include Ansible playbooks which may be used as a design reference for configuring and provisioning KumoScale storage nodes via the KumoScale Provisioner service.

      5.1    Released Components

      Item   

      Description

      Revision

      KumoScale Software Ansible Modules

      KumoScale software Ansible modules for bare-metal installation

      3.19-397

      5.2    System Requirements

      The Ansible client may run on a physical or virtual machine (VM) and connect to KumoScale storage nodes via the machines' management ports.

      • Python™ Version: Python version on the hosts and the Ansible client ≥ 3.
      • Ansible Module Version: KumoScale Software Ansible Modules assume version 3.

      5.3    Compliance with Other KumoScale Components

      Below are the versions of KumoScale software, Provisioner, and Ansible Modules that have been validated together; other combinations have not been tested and are not supported with version 3.19.

      KumoScale Software Version

      KumoScale Provisioner Version

      KumoScale Ansible Modules

      3.19-14655

      3.19-1294

      3.19-397

      5.4    New Features

      • Improved the ks_mirror module to handle Cluster MD, for GPFS support.

      5.5    Modified Features

      • Removed "ks_portal” and “ks_group” modules and playbooks.
      • Instead of “ - - limit” , users should use “hosts_group” parameter available in vars.yaml file.
      • Naming convention: host_main parameter replaces the host

      5.6    Deprecated Features

      None.

      5.7    Fixed Issues from Previous Release

      None.

      5.8    Open Issues and Known Limitations

      5.8.1   High-Impact Issues

      Bug ID

      Description

      NOF-6421

      When using GPFS with ClusterMD , the Self-Healing process  might fail to recover a replica located on storage node that is permanently inaccessible.

      Mitigation: None.

      5.8.2   Medium-Impact Issues

      Bug ID

      Description

      NOF-6366

      After reboot of a  complete GPFS Cluster, some GPFS cluster nodes might not discover all ClusterMD volumes

      Mitigation: manual assemble using “mdadm -A –scan"  command OR reboot of the failed node.

      5.8.3   Low-Impact Issues

      None.

      6      KumoScale Software for OpenStack Platforms

      KumoScale OpenStack Wallaby Software includes a KumoScale NVMe-oF Connector, Cinder Driver and Agent module for OpenStack Wallaby Release. KumoScale software for OpenStack platforms integrates KumoScale software into the customer’s OpenStack Cluster.

      This version is based on KumoScale software for the OpenStack platform and is tested with OpenStack version Wallaby. It enables the OpenStack cluster to provision KumoScale volumes and includes a KumoScale Cinder driver for the control plane as well as a KumoScale software NVMe-oF connector that is used by the NOVA OS brick layer.

      6.1    Released Components

      Item   

      Description

      Revision

      KumoScale software OpenStack platform software agent

      KumoScale OpenStack Wallaby Software

      3.19.179

      6.2    New Features

      • Support for NVMe MultiPath based kernel used by Ubuntu 20.04.

      6.3    Open Issues and Known Limitations

      Bug ID

      Description

      NOF-6422

      When trying to concurrently delete multiple snapshots taken from the same source volume operation might fail for some of the snapshots.

      Mitigation: Re-Issue the delete operation for any failed deleted Snapshot

      7      Initiator Server Software Requirements

      • Operating System

      For NVMe-oF RoCEv2

      For NVMe-oF TCP

      Linux OS with kernel 4.9.64 x86 or newer.

      LINUX OS kernel must support NVMe over TCP, 5.0 and above.

      Please consult with KIOXIA Technical Support regarding the exact distribution you plan to use.

      • NVMe Command Line Interface (CLI): NVMe CLI 1.6 or newer.

      [1] On Dell EMC servers BMC is called iDRAC. On HPE servers it is called iLO.