Release Notes

KUMOSCALE™ STORAGE SOFTWARE V. 3.20

KumoScale Storage Software includes a target kernel component, an embedded engine, services, a REST API, and a Command Line Interface (CLI) for management and control.

The software can be installed and deployed in one the following modes:

  • Appliance Mode, where all components are integrated as a single image with a customized Linux™ kernel 18.0-240.22.1 (RHEL 8.3) and Kubernetes (v1.17.5)
  • Managed Mode, where components are installed and configured on the customer’s Linux OS and kernel.

Released Software Components

Appliance Mode

Item

Description

Revision

KumoScale ISO Network Installation

KumoScale storage node software ISO Network Installation Package for Appliance Mode deployment

3.20-15206

KumoScale ISO USB Installation

KumoScale storage node software ISO USB Installation Package for Appliance Mode deployment

3.20-15206

KumoScale Virtual CD Installation

KumoScale storage node software Virtual CD Installation Package for Appliance Mode deployment

3.20-15206

Managed Mode

Item

Description

Revision

KumoScale Storage Node Engine Software

KumoScale Storage Node Engine software for Managed Mode deployment

3.20-15206

KumoScale Storage Node CLI Software

KumoScale Storage Node CLI software for Managed Mode deployment

3.20-15206

KumoScale Storage Node Target Software

KumoScale Storage Node Target Source Code software for Managed Mode deployment

3.20-2235

Linux NVMe™ host module patches

Linux NVMe host module patch for Ubuntu™ 20.04 LTS (kernel 5.4.0-91-generic)

1.0

Linux NVMe host module patch for CentOS™ 8.5 (kernel 4.18.0-348.2.1.el8_5)

1.1

Linux NVMe host module patch for kernel-5.10.61

1.1

KumoScale License Generator for Windows™

KumoScale License Generator Tool for Windows

3.13.7

Hardware Requirements

See the Hardware Compatibility List (HCL) document.

New Features

KumoScale Storage Node 3.20-15206 software release is based on KumoScale Storage Node 3.19-14660 software and include the following changes:

  • Added support for the KumoScale Storage Node Software Deployment on Customer Managed Server Platforms. KumoScale StorageNode Software components, independently deployed on a Bare metal Linux OS server (Centos/RHEL/UBUNTU) with Managed Mode Platforms Deployment Target kernel software , provided as source code, to be compiled by the customer to their specific platform kernel version.
  • Added security enhancements for KumoScale Storage Nodes to support integration with an external OpenID Connect (OIDC) authentication server.
  • Added support for automatic target core affinity for improved NVMe-oF™ performance.
  • Added support for a configurable Storage Node REST API management port reporting to the Provisioner.
  • Added support for reporting the Storage Node target driver Version.
  • Added support for any network interface reported with the vendor “Mellanox™” to support NVMe-oF RoCEv2 transport.
  • Added support for IPv6 Address in Managed Mode.

Modified Features

  • Updated REST API and CLI for Managed Platforms Deployment.
  • KS 3.19 Bug Fixing and small improvements.

Deprecated Features

  • None

Fixed Issues from Previous Release

Bug ID

Description

NOF-6946

log4j-1.2.17.jar file found to be included on the local disk as part of KumoScale Engine

NOF-6993

Upgrade process do not keep the original CIDR and set it to it’s default - 192.168.0.0/16

NOF-6649

Volume usage information is erroneous

NOF-6637

KS CLI - get log operation failed and KS CLI crashed

Open Issues and Known Limitations

Critical Issues

Bug ID

Description

NOF-2083

In rare situations, QCT™ QuantaGrid™ D42A-2U platform (AMD EPYC™) hangs during a reboot done via REST/CLI. The following error is seen via IPMI: Mlx5_core: mlx5_enter_error_state 128 (pid 1)

This issue was not reproducible with any other platform and may be related to PCIe errors reported with the current server samples we use.

Mitigation: Manual reboot will be required to recover

High-Impact Issues

Bug ID

Description

NOF-5384

NOF-5240

In rare instances, a new deployment on a SuperMicro™ 2029 system may fail. The Last reported message shown is Couldn’t get size 0x800000000000000e.

Mitigation: A second deployment attempt will succeed.

NOF-4817

When Working with Dell™ PowerEdge™ R7525 server SSD hot plug is not always recognized by Linux OS.

Mitigation: Server Power cycle might be needed for all SSDs to be discovered.

NOF-3366 NOF-3196

SSD hot-plug is not always recognized by Linux™ OS on a SuperMicro 2029U-TN24R4T server.

Mitigation: Power cycle the server to ensure that all SSDs are discovered.

NOF-2460 NOF-570 NOF-426

SSD hot-plug may fail on a SuperMicro 2028U-TN24R4T server.

Mitigation: Please power cycle the server to ensure that all SSDs are discovered.

NOF-1507

 

KumoScale ISO Installation of Lenovo™ ThinkSystem™ SRxxx-POC can’t be done via PXE.

Mitigation: Installation should be done via USB only.

NOF-1045

 

When working with Tyan B8026T70AE24HR server and C-M5 SSD, SSD hot plug is not always recognized by Linux OS.

Mitigation: Server Power cycle might be needed for all SSDs to be discovered.

NOF-516

Removing an SSD via Baseboard Management Controller (BMC)[1] while running IO commands to it, may cause KumoScale software to malfunction.

Mitigation: Do not remove an SSD via the BMC.

Medium-Impact Issues

Bug ID

Description

NOF-7031

KumoScale target install script might failed to load KS modules but exit with a successfully install message.

Mitigation: When the modprobe ERROR is reported, a reboot is required to properly load KS modules.

NOF-5849

Solarflare™ XtremeScale™ X2541-100G Adapter is not supported with Latest KumoScale kernel (Centos 8.3 based kernel).

NOF-4319

Small Memory leak as a result of TCP session transport disconnection.

No Mitigation yet found for this issue.

Issue was reported to the Linux NVMe open-source community as it was seen also with open source NVMe-TCP target implementation.

NOF-3023

When duplicate portal IP is get detected on another appliance, all the targets gets disconnected and initiator crashes.

Mitigation: Avoid configuration of duplicated portal IP.

NOF-2487

When simulating 2000 initiators using 4 compute nodes that get connected to a single KumoScale target with ~14,000 sessions, When one of the compute node that had ~11,000 sessions crash target lost also sessions from other compute node to this target.

No Mitigation was found yet for this issue.

It is advised not to connect so many initiators to a single target.

NOF-1482 

DHCP IP Portal change as a result of DHCP IP renew is not reflected until KS get rebooted or the user manually deleted and created the Portal.

Mitigation: If DHCP portal changes, remove the portal and create it again with the new IP.

Low-Impact Issues

Bug ID

Description

NOF-6479NOF-6390NOF-5883

NOF-5881

Seldom, during boot of an AMD EPYC based server platform , APEI Generic Hardware Error get reported and system might hang.

Mitigation: A system reboot should overcome this issue.

NOF-5392

Deployment on a Dell EMC™ PowerEdge™ R7415 Rack Server will print IO_PAGE_FAULT messages during a PXE installation.

Mitigation: Add iommu=soft to the grub command line options.

NOF-5332

In case a deployment was done via USB with a static management IP and all of the network interfaces are not connected, the Cluster VIP might not be accessible if the management interface was connected after the deployment of the first storage node has completed.

Mitigation: Make sure the management interface is connected during the deployment.

NOF-5098

Storage node logs report repeated kubelet errors.

Mitigation: Add the storage node to a KumoScale cluster.

NOF-4842

Moving bootable NVMe disks to a non-bootable NVMe slot results in the system booting from SATA SSDs.

Mitigation: Do not use non-bootable NVMe slots for storage node OS.

NOF-1062

When using Tyan B8026T70AE24HR server, alert of “The system temperature is too high. Above: 75” can be seen.

As AMD EPYC CPU on this platform report a temperature value of 82 degrees Celsius.

KumoScale Provisioner Service

The KumoScale Provisioner Software for Kubernetes and for bare metal servers provides a central storage provisioning service across KumoScale Server platforms. It assists a large-scale data center orchestrator to select the best KumoScale appliance to store a required volume.

The KumoScale Provisioner Software is delivered to customers as a binary container for Kubernetes deployment or as an RPM package for installation on a bare metal server.

Released Components

Item   

Description

Revision

KumoScale Provisioner Software

KumoScale Provisioner Software for provisioning with a Kubernetes cluster or a bare metal server with CentOS or Ubuntu

3.20-1497

System Requirements

See KumoScale Installation Guide.

New Features

KumoScale Provisioner 3.20 software release is based on KumoScale Provisioner 3.19 software and includes the following changes:

  • Added support for Provisioner Software deployment on a customer Managed server platform. In this mode the Provisioner is deployed and upgraded as an RPM.
  • Updated Provisioner REST API and CLI for Managed platforms deployment.
  • Added support for reporting storage node target driver version.
  • Added security enhancements for the KumoScale Provisioner to support integration with an external OpenID Connect (OIDC) authentication server.

Modified Features

  • Provisioner 19 Bug Fixing and small improvements

Deprecated Features

None

Fixed Issues from Previous Release

Bug ID

Description

NOF-6966

Provisioner pod stopped responding to REST calls due to deadlocks related to parallel provisioning requests.

Add Liveness and Readiness probes for provisioner pod and install operator.

NOF-6478

No actionable error message when snapshot creation fails.

NOF-6446

Provisioner cli "show tenants"- BW values are wrong.

NOF-6422

OpenStack: failed to delete one snapshot during multiple snapshot deletion.

Open Issues and Known Limitations

High-Impact Issues

None

Medium-Impact Issues

Bug ID

Description

NOF-5938

Until all storage nodes discovered, the Tenants Volumes Budget may not be enforced during provisioner service restart.

Mitigation: Wait patiently for discovery.

Low-Impact Issues

Bug ID

Description

NOF-7044

makeusb.sh script might failed on Ubuntu 20.04 with error /dev is not a block device.

Mitigation : consider using makeusb with Centos OS.

NOF-7038

Provisioner show-volumes CLI command when having invalid token, return with empty volume list instead of Bad credentials error message.

NOF-4358

KumoScale Provisioner service reports the initially configured reserved space as its reserved space.

Mitigation: Query the KumoScale storage node that the volume is allocated on to display the actual reserved space.

KumoScale Operators for KumoScale Management

The KumoScale Operator Software for Kubernetes provides operators for installing and configuring a KumoScale Kubernetes storage cluster. The KumoScale Operator Software delivered as a binary container for Kubernetes deployment.

Released Components

Item   

Description

Revision

KumoScale Operators Software

KumoScale Operator Software for Kubernetes

3.20-1871

New Features

KumoScale Operators 3.20 software based on KumoScale Operators 3.19 software and includes the following changes:

  • Added security enhancements for KumoScale Operators to support integration with an external OpenID Connect (OIDC) authentication server.
  • Added Operators support for storage nodes working in Managed mode.
  • Added Operators support for IPv6 Address in Managed mode.
  • Add support to use KumoScale Operators on Kubernetes control cluster used in Managed mode.

Modified Features

  • Operator 19 Bug Fixing and small improvements

Deprecated Software

None

Fixed Issues from Previous Release

Fixed Issues from version 3.19 (version 3.19-14660):

Bug ID

Description

NOF-6398

When trying to add a Storage Node that was previously part of the KumoScale Cluster, rarely, such operation might fail with "The storage node already exists" error.

Open Issues and Known Limitations

High- Impact Issues

None

Medium-Impact Issues

None

Low-Impact Issues

Bug ID

Description

NOF-7019

Software upgrade from KumoScale 3.19 to 3.20 completed successfully although the error “secrets kumoscale-provisioner not found” is reported.

Please ignore the error.

KumoScale CSI Software for Kubernetes Orchestration

KumoScale CSI Software for Kubernetes provides a

  • KumoScale CSI plugin driver for accessing both KumoScale Server platforms and the KumoScale Provisioner. The KumoScale CSI plugin driver is written in the go language, released as open-source code, and delivered to customers as a binary container that can be downloaded from the KIOXIA web site and loaded into a customer’s local image repository using the Docker load command.
  • KumoScale Agent for bare metal or VM provides the CSI functionality for non-Kubernetes deployments.

Released Components

Item   

Description

Revision

KumoScale CSI software

KumoScale CSI Software for Kubernetes and agent for bare metal and VM.

3.20-1843

Software Requirements

  • Kubernetes Orchestration Version Number 1.14 or greater
  • CentOS/RHEL 7.x/8.x or Ubuntu 18.x/20.x

New Features

The KumoScale CSI 3.20 software release is based on KumoScale CSI 3.19 software and includes the following changes:

  • Added support for Kubernetes CSI raw block volumes.
  • Added security enhancement for KumoScale CSI to support integration with an external OpenID Connect (OIDC) authentication server.
  • Support for Kubernetes version 1.22.

Modified Features

  • CSI 19 Bug Fixing and small improvements

Deprecated Features

None

Fixed Issues from Previous Release

Bug ID

Description

NOF-6933

Fixed default of spanAllowed=true in CSI

NOF-6895

Initiator got hang on trying to run 'nvme discover' command - Added fix in csi/ks-agent - Don't disconnect a target if its controller state is not 'live'.

NOF-6366

After reboot of a complete GPFS Cluster, some GPFS cluster nodes might not discover all ClusterMD volumes.

NOF-6421

When using GPFS with ClusterMD , the Self-Healing process   might fail to recover a replica located on storage node that is permanently inaccessible.

Mitigation: None.

Open Issues and Known Limitations

High- Impact Issues

None

Medium-Impact Issues

Bug ID

Description

NOF-7047

CSI failed to create the volume when trying to move back from openIDC to LOCAL.

Mitigation: Delete the CSI driver and install it again to allow it to reload the token from its updated secret.

Low- Impact Issues

None

KumoScale Software Ansible Modules

KumoScale Software Ansible Modules includes Ansible playbooks which may be used as a design reference for configuring and provisioning KumoScale storage nodes via the KumoScale Provisioner service. The KumoScale Ansible software is packaged in a tar format.

Released Components

Item   

Description

Revision

KumoScale Software Ansible Modules

KumoScale software Ansible modules for bare-metal installation

3.20-441

System Requirements

The Ansible client may run on a physical or virtual machine (VM) and connect to KumoScale storage nodes via the machines' management ports.

  • Python™ Version: Python version on the initiators and the Ansible client ≥ 3.
  • Ansible Module Version: KumoScale Software Ansible Modules assume version 3.

New Features

KumoScale Ansible 3.20 software release is based on KumoScale Ansible 3.19 software and includes the following changes:

  • Added security enhancements for KumoScale Ansible to support integration with an external OpenID Connect (OIDC) authentication server.

Modified Features

  • Ansible 19 Bug Fixing and small improvements

Deprecated Features

None

Fixed Issues from Previous Release

Bug ID

Description

NOF-6518

Ansible : create fs failed during simple volume creation because of unknown device name

NOF-6497

Ansible playbook can't accept -e hosts_group=giant

NOF-6494

Ansible delete_volume task fails- error while evaluating conditional

NOF-6895

Ansible playbook - single replicated volume that get attached to a multiple hosts (initiators) , file system is being created for each host instead of only the first host

NOF-6459

Ansible: Check host (initiator) name module fails to login, wrong error message

NOF-6441

Ansible fails to create token, running install_host.yaml playbook

NOF-6798

Fix ansible install_host.yml playbook for Ubuntu 20.04

NOF-6798

Ansible install_host.yml script failures

NOF-6495

Delete_volume_simple.yml contains unnecessary parameters for the ks_volume task

NOF-6421

When using GPFS with ClusterMD , the Self-Healing process   might fail to recover a replica located on storage node that is permanently inaccessible.

Open Issues and Known Limitations

High-Impact Issues

None

Medium-Impact Issues

Bug ID

Description

NOF-7022

Ansible install_host.yml sets the token for OIDC instead of the OIDC configuration.

Mitigation: To work with Ansible in OIDC mode, after executing install_host.yml modify the host file /lib/systemd/system/ks-agent.conf with the OIDC configuration and restart ks-agent.

Low-Impact Issues

None

KumoScale Software for OpenStack Platforms

KumoScale OpenStack Software includes a KumoScale NVMe-oF Connector, Cinder Driver and Agent module for the OpenStack Xena release. It enables the OpenStack cluster to provision KumoScale volumes and includes a KumoScale Cinder driver for the control plane as well as a KumoScale software NVMe-oF connector that is used by the NOVA OS brick layer.

Released Components

Item   

Description

Revision

KumoScale software OpenStack platform software agent

KumoScale OpenStack Xena Software

3.20.199

New Features

KumoScale OpenStack Xena 3.20 Software release is based on OpenStack Xena release and KumoScale OpenStack 3.19 and includes bug fixes and small improvements.

Open Issues and Known Limitations

High-Impact Issues

Bug ID

Description

NOF-4758

Rebooting an initiator during live/cold VM migration might cause VM instance to go into an "error state " and leave OpenStack databases related to this VM instance in an inconsistent state.

This seems to be a known limitation for OpenStack Stein release:
https://bugs.launchpad.net/nova/+bug/1859496https://access.redhat.com/solutions/4420801

Medium-Impact Issues

Bug ID

Description

NOF-4757

During long term tests of recurrent live VM migrations a failed migration was reported once and the VM   was inaccessible.

Low-Impact Issues

Bug ID

Description

NOF-6422

When trying to concurrently delete multiple snapshots taken from the same source volume operation might fail for some of the snapshots.

Mitigation: Re-Issue the delete operation for any failed deleted Snapshot.

Initiator Server Software Requirements

  • Operating System

For NVMe-oF RoCEv2

For NVMe-oF TCP

Linux OS with kernel 4.9.64 x86 or newer.

LINUX OS kernel must support NVMe over TCP, Linux kernel version 5.0 and above.

Please consult with KIOXIA Technical Support regarding the exact distribution you plan to use.

  • NVMe Command Line Interface (CLI): NVMe CLI 1.6 or newer

[1] On Dell EMC servers BMC is called iDRAC. On HPE servers it is called iLO.