Install - Multinode / Production

This chapter explains how to install KumoScale software in Appliance mode for an environment supporting multiple nodes.

Before you Begin Installation

We recommend that you install and configure servers one at a time starting with the storage cluster masters.  The figure below illustrates the process for installing KumoScale in your environment and configuring it for storage.

installman-fig2

Figure: KumoScale Node Installation and Configuration Process

At a high level, the process consists of:

  • Installing KumoScale software on a physical server, steps 1- 5.
  • Configure the masters for the storage cluster, steps 6- 10. As explained in the overview, this is a private Kubernetes cluster that supports services and operators for provisioning, monitoring, and support. You will use the remote host set up in Administrative Host with Support for Kubernetes kubectl section to complete the configuration steps. You will use custom resource files to configure the cluster for your deployment environment. The User Manual has complete information on how custom resources are used in KumoScale software and what parameters you will need to specify the master nodes.
  • Configure storage nodes as workers of the storage cluster, step 11. As with the master you will use Custom Resource (CR) files to configure all worker nodes.

Upon completing each step, you will have a KumoScale software environment configured with storage nodes as shown in the figure  Example of a KumoScale Software Deployment.  Note that you will need all of the information discussed in Configuration Requirements in order to complete these steps successfully.

We recommend that you install and configure servers one at a time starting with the masters. Although you may be able to install KumoScale software on all servers, you must configure all masters, before configuring additional storage.

  1. Connect your platform to the network and prepare files for installation as defined in the Prepare the Installation File and Environment
  2. Determine values for any command line (CLI) parameter that needs to be set for KumoScale software to install in your environment. Note that you will need to specify the VIP for the masters when installing on the first master. See Installation Command Line Parameters for details.
  3. Configure your platform to use the UEFI and to boot from either of the appropriate devices. See Configure the Boot Order.
  4. Start the installation process by rebooting or powering up the server and booting according to the preconfigured option specified in Step 3 above (network/USB/Virtual CD). The OS, and all required packages and dependencies, along with KumoScale software, are installed using any parameters you choose to provide per Step 2 above. This process is explained in Start the Installation.
  5. Verify that installation was successful as instructed in Verify Installation Success.
  6. Access the KumoScale storage cluster from a remote host as explained in Access the KumoScale Storage Cluster from a Remote Administrative Host.
  7. Configure the first master for the KumoScale storage cluster. You will need the information gathered in the Configuration Requirements to complete this step. See Configure the First Master.
  8. Repeat Steps 1-6 above on the servers to be added as masters to the KumoScale storage cluster and configure as instructed in Install and Deploy Remaining Masters.
  9. Install the internal components on the KumoScale storage cluster. You will need these tools for internal analytics. See Install Internal Components on Storage Cluster.
  10. Verify that the KumoScale storage cluster is running and set up as expected. See Verify KumoScale Storage Cluster Configuration.
  11. Install and configure storage nodes as workers as explained in Install and Configure Storage Nodes.

Prepare the Installation File and Environment

Connect the Platform to the Network

KumoScale software will sort the connected interfaces according to maximum speed (from lowest to highest), bus information, and Media Access Controller (MAC) address, and choose the slowest for the management port. Connect the management port according to these guidelines, or connect the data port(s) after the installation has completed. Certain hardware platforms do not contain a dedicated management interface. In such a case, KumoScale software supports utilizing the data port as a shared data/management interface.

Prepare the Installation Files

Prepare the installation files according to the type of installation you are performing.

NOTE: The first server to be installed and configured must be the first master of the storage cluster and you will need to specify the VIP for that node

Network Installation using a PXE Server

1) Installation requires the following files:
a. The compressed image for deployment (this file includes gpg checksum):
kumoscale-<KumoScale release>-img.raw.tar.gpg.
For example, kumoscale-20-15206-img.raw.tar.gpg.
b. The upgrade file: upgrade_nvmf_<KumoScale release>.gpg. For example, upgrade_nvmf_3.20-15206.tar.gpg.
c. The diskless file system image: kioxiadiskless-<KumoScale release>.cpio.gz.
For example, kioxia.diskless-3.20-15206.cpio.gz.
d. The CentOS kernel file
/tftpboot/vmlinuz-10.0-957.27.2.el7.centos.plus.bug16242.x86_64


2. Save the files 1a and 1b in the PXE server images You can use HTTP or HTTPS for the PXE server URL. For example, http://192.0.2.1/images or https://192.0.2.1/images.

3. Copy the diskless file system imageinto one of the PXE server’s boot image locations, e.g. /tftpboot. If choosing a folder different than /tftpboot, be sure to replace it in the next section as well.


4. Ensure that the CentOS kernel file is located in the /tftpboot folder:
vmlinuz-3.10.0-957.27.2.el7.centos.plus.bug16242.x86_64.


5. For installation of KumoScale on the first node of the storage cluster, the first master node, set the value of kx_cluster_vip to the value of the master VIP. You do not need to supply this parameter for any other node.

6. Add the pxelinux menu entry below to your PXE cfg menu. For example:
menuentry 'KUMOSCALE Diskless deployer' {

       linuxefi /vmlinuz-3.10.0-957.27.2.el7.centos.plus.bug16242.x86_64 ro copytoram=y panic=30 ip=::::::dhcp ipv6.disable=1 biosdevname=0 net.ifnames=0 quiet kx_imagepath=<image_path> kx_mode=deploy kx_cluster_vip=192.0.2.0

   initrdefi /kioxia.diskless-<KumoScale_release>.cpio.gz

}
7. For serial port deployment, add the console redirection as shown in the example below:
menuentry 'KUMOSCALE Diskless deployer' {

linuxefi /vmlinuz-3.10.0-957.27.2.el7.centos.plus.bug16242.x86_64 ro copytoram=y panic=30 ip=::::::dhcp ipv6.disable=1 biosdevname=0 net.ifnames=0 quiet kx_imagepath=<image_path> kx_mode=deploy console=tty1 console=ttyS0,115200n8

initrdefi /kioxia.diskless-< KumoScale_release>.cpio.gz

}

8. Add any other parameters according to Installation Command Line Parameters.

9. Verify that the server is connected to the PXE server’s network.

Virtual CD Installation

1. Virtual CD installation requires the ISO image: kumoscale-<KumoScale release>.iso.
For example, kumoscale-3.20-15206.iso.
2. Mount or attach the ISO image from either one of the IPMI, iLO or iDRAC of the deployed appliance.

USB Installation

This section describes creation of a USB device from which you can boot.

WARNING: Do not connect the bootable USB drive to a production server, since it might delete its content without a prior warning

1. The installation requires the following files:
a. The compressed image for deployment:
kumoscale-<KumoScale release>-img.raw.tar.gpg.
For example, kumoscale-3.20-15206-img.raw.tar.gpg.
b. The file system image: kumoscale-<KumoScale release>-usb.raw.gpg.
For example, kumoscale-3.20-15206-usb.raw.gpg.
c.  The upgrade file: upgrade_nvmf_<KumoScale release>.tar.gpg.
For example, upgrade_nvmf-3.20-15206.tar.gpg.
d. The installation script: makeusb.sh.


2. Save all the files above in the same folder locally on a RHEL Linux platform with a connected USB device.

3. Deploy the USB device /dev/sdx by executing the following:
./makeusb.sh /dev/sdx /path/to/files

NOTE 1: You may need to set the access rights of the script for execution.

NOTE 2: If the USB is auto-mounted by the operating system, the command above will not function properly. If so, unmount the USB device prior to this step

Installation Command Line Parameters

NOTE: The first server to be installed and configured must be the first master of the storage cluster, so you will need to use kx_cluster_vip for the first install.

Several installation settings can be changed via boot parameters passed in as command line arguments. Some examples are provided in this section to help you determine when to use these parameters.

How to use the parameters:

 

Table. Installation Command Line Parameters

Parameter

Options
(default in boldface)

How Used

kx_cluster_vip

 

Master VIP. The virtual IP of the KumoScale storage cluster must be provided during the installation of the first node for the cluster. It is bound to the management interface and must be on the same subnet as the KumoScale management IP. Do NOT provide it for any other node.

For example, kx_cluster_vip=192.0.2.0

kx_static_ip

 

Static IP address. When a value for this parameter is specified, the Static IP will be automatically configured based on the following algorithm for choosing the management interface:

1.      Use the first online physical network interface (OPERSTATE= up and CARRIER=1) found by sorting by minimum speed, PCI Bus, and MAC address.

2.     Configure the Static IP on the found interface:

  • PXE and Virtual CD installations: Check the connectivity to the http server provided by the kx_image parameter. If the server unreachable, then clear the current interface configuration, exclude the current interface from the list, and return to step 1.
  • USB installation: The system does not check for network connectivity and the Static IP will be configured on first online interface. If there is no interface on line, it will select the first interface from a list sorted by minimum speed, PCI bus, and MAC address.

Note: If the Static IP is configured, the Dynamic Host Configuration Protocol (DHCP) network mode will be disabled.

kx_subnet_mask

255.255.0.0

Subnet mask. This parameter should be used only when a value for kx_static_ip is specified.

kx_gateway

 

Default gateway. This parameter should be used only when a value is specified for kx_static_ip.

kx_mode

deploy

purge

·         kx_mode=deploy will delete the contents of the system disk and deploy a new image.

·         kx_mode=purge will completely wipe out the contents of the devices specified in the kx_devices parameter. This is useful for wiping previous installations.

No default value.

kx_blocktype

ata

nvme

If specified, this indicates the type of device for deployment:
It is ignored if kx_mode=purge, and purging will be determined according to the value of kx_devices.

·         kx_blocktype=ata (default): Use only ATA devices for the deployment process, deploying to nvme, usb or any other device type will fail.

·         kx_blocktype=nvme: Use only NVMe devices for the deployment process, deploying to ata, usb or any other device type will fail.

kx_devices

 

Required when kx_mode=purge, otherwise optional.

This specifies the PCI slots of the drives that KumoScale software will be deployed to, as a comma-separated list of their slot numbers. The slot numbers should be provided in their full form, as they appear in the output of lspci -D.

You can specify slots that belong to either an ATA or NVMe device, and your selected devices will be verified against the specified kx_blocktype. For example:

·         If you specify two (2) devices and at the same time specify kx_raid=single the process will fail.

·         If you specify one (1) device and at the same time specify kx_raid=raid the process will fail.

·         If you specify a PCISLOT that matches more than one (1) disk - such as `0000`, the process will fail.

Examples:

·         kx_devices=0000:1a:00.0,0000:5e:00.0 : Use two (2) devices - 0000:1a:00.0 and 0000:5e:00.0 and do not automatically detect which devices to use.

·         kx_devices=0000:18:00.0: Use one (1) device - 0000:18:00.0 and do not automatically detect which device to use.

·         kx_devices=0000:00:1f.2/ata9,0000:00:1f.2/ata10: Use two (2) SATA devices - 0000:00:1f.2/ata9 and 0000:00:1f.2/ata10 and do not automatically detect which devices to use.

No default value.

kx_raid

auto

raid

single

·         kx_raid=auto (default): automatically discover the best RAID deployment and create a RAID over them.

·         kx_raid=raid: install over two drives (either detected or specified). Will fail if not enough devices were detected or specified in kx_devices.

·         kx_raid=single: install over a single drive. Will fail if more than one drive was specified in kx_devices.

kx_rootsize

Integer with units (string)

16 GiB[1]

Modify the size of the root partition to this value. If units are not specified, it will assume MiB.

Note: You must enter the same value during an upgrade or the upgrade will fail.

kx_debug

false

true

·         kx_debug=true: Disable default quiet mode and print extended debug information during deployment.

·         kx_debug=false: Default value. Does not print any debugging information.

kx_pods_cidr

Address in CIDR notation

This is an optional parameter for the first master node installation only. It allows the user to manually select the value of $POD_NETWORK_CIDR. When this parameter is not defined by the user, a /16 range is selected out of the free private range. The value must be in CIDR notation and must define a free private address range. The free private range is within:

1.      192.168.0.0/16 - entire range

2.     10.0.0.0/8 - excluding 10.96.0.0/12 used for k8s services (10.0.0.0/16 - 10.95.0.0/16, 10.112.0.0/16 - 10.255.0.0/16)

3.     172.16.0.0/12 - excluding 172.16.0.0/16 used by docker (172.17.0.0/16 - 172.31.0.0/16)

All the above excluding subnets configured on the machine (mgmt: static/dhcp).

 

Examples

Below are examples of a network installation where a new installation of KumoScale software is deployed. The content of each menu entry sets up a node that will be configured as a master or worker node:

Install-KumoScale-static-master: Delete the contents of the system disk and install a new image of KumoScale software on a server that will be used as the master node of the storage cluster with VIP 192.0.2.0. Deploy across a RAID created based on KumoScale software discovery.

menuentry 'Install-KumoScale-static-master' {
linuxefi ks/vmlinuz-3.10.0-957.27.2.el7.centos.plus.bug16242.x86_64 ro copytoram=y panic=30 ip=::::::dhcp ipv6.disable=1 biosdevname=0 net.ifnames=0 quiet kx_imagepath=http://xxx.xx.x.xx/files/Install/KumoScale/3.20_golden/kumoscale-3.20-15206-img.raw.tar.gpg kx_static_ip= xxx.xx.x.xx kx_subnet_mask= xxx.xx.x.xx kx_gateway= xxx.xx.x.xx kx_cluster_vip=192.0.2.0 kx_mode=deploy kx_raid=auto
initrdefi ks/kioxia.diskless-3.20-15206.cpio.gz
}

 

Install-KumosScale-single: Delete the contents of the system disk and install a new image of KumoScale software on a storage node. Deploy over a single drive.

menuentry 'Install-KumoScale-single' {
linuxefi ks/vmlinuz-3.10.0-957.27.2.el7.centos.plus.bug16242.x86_64 ro copytoram=y panic=30 ip=::::::dhcp ipv6.disable=1 biosdevname=0 net.ifnames=0 quiet kx_imagepath=http://xxx.xx.x.xx/files/Install/KumoScale/3.20_golden/kumoscale-3.20-15206-img.raw.tar.gpg kx_mode=deploy kx_raid=single
initrdefi ks/kioxia.diskless-3.20-15206.cpio.gz
}

 

Install-KumoScale-auto: Delete the contents of the system disk and install a new image of KumoScale software on a storage node over https. Deploy across a RAID created based on KumoScale software discovery.

menuentry 'Install-KumoScale-auto' {
linuxefi ks/vmlinuz-3.10.0-957.27.2.el7.centos.plus.bug16242.x86_64 ro copytoram=y panic=30 ip=::::::dhcp ipv6.disable=1 biosdevname=0 net.ifnames=0 quiet kx_imagepath=https://xxx.xx.x.xx/files/Install/KumoScale/3.20_golden/kumoscale-3.20-15206-img.raw.tar.gpg --noverifyssl kx_mode=deploy kx_raid=auto
initrdefi ks/kioxia.diskless-3.20-15206.cpio.gz

 

Install- KumoScale-auto-NVMe: Delete the contents of the system disk and install a new image of KumoScale software on a storage node. Deploy across a RAID using only NVMe, based on KumoScale software discovery.

menuentry 'Install-KumoScale-auto-NVMe )' {

linuxefi  ks/vmlinuz-3.10.0-957.27.2.el7.centos.plus.bug16242.x86_64 ro copytoram=y panic=30 ip=::::::dhcp ipv6.disable=1 biosdevname=0 net.ifnames=0 quiet kx_imagepath=http://xxx.xx.x.xx/files/Install/KumoScale/3.20_golden/kumoscale-3.20-15206-img.raw.tar.gpg kx_mode=deploy kx_raid=auto kx_blocktype=nvme

initrdefi ks/kioxia.diskless-3.20-15206.cpio.gz

}

 

Install- KumoScale-purge-NVMe: Delete the contents of the drives associated with the PCI slots 0000:40:01.1/0000:41:00.0/0000:42:02.0/0000:45:00.0.

menuentry 'Install- KumoScale-purge-NVMe' {

linuxefi  ks/vmlinuz-3.10.0-957.27.2.el7.centos.plus.bug16242.x86_64 ro copytoram=y panic=30 ip=::::::dhcp ipv6.disable=1 biosdevname=0 net.ifnames=0 quiet kx_imagepath=http://xxx.xx.x.xx/files/Install/KumoScale/3.20_golden/kumoscale-3.20-15206-img.raw.tar.gpg kx_mode=purge kx_blocktype=nvme kx_devices=0000:40:01.1/0000:41:00.0/0000:42:02.0/0000:45:00.0

initrdefi ks/kioxia.diskless-3.20-15206.cpio.gz

}

 

Install- KumoScale-purge SATA: Delete the contents of SATA SSDs.

menuentry 'Install- KumoScale-purge-SATA' {

linuxefi  ks/vmlinuz-3.10.0-957.27.2.el7.centos.plus.bug16242.x86_64 ro copytoram=y panic=30 ip=::::::dhcp ipv6.disable=1 biosdevname=0 net.ifnames=0 quiet kx_imagepath=http://xxx.xx.x.xx/files/Install/KumoScale/3.20_golden/kumoscale-3.20-15206-img.raw kx_mode=purge kx_devices=host9

initrdefi ks/kioxia.diskless-3.20-15206.cpio.gz

}

Configure the Boot Order

NOTE: In AMD™ server-based systems, the Input-Output Memory Management Unit (IOMMU) needs to be disabled. This is done by setting Virtual Technology to Disable. Specifically, in the certified Quanta™ Cloud Technology (QCT) systems, this BIOS setting is found under:

AMD CBS > NBIO Common Options > NB Configuration > IOMMU (default = Auto).

Change this value to Disabled.

You will need to configure the Basic Input/Output System (BIOS) to boot from the correct location. For details on how to configure the BIOS, refer to your platform’s user manual or vendor.

Complete the steps below to configure the boot order:

1. Set the UEFI.
Booting from the UEFI instead of legacy BIOS requires the following steps:
a. Enable the NIC UEFI setting via the boot menu.
b. Reboot the system.
c. Choose the UEFI boot setting in the BIOS menu.


2. Set the BIOS Boot Order.
Configure the server BIOS boot order settings according to the chosen installation type:

    1. For a network installation, set the following order:
      1. Local hard drive
      2. Network
    2. For a Virtual CD installation, set the following order:
      1. Local hard drive
      2. CDROM
    3. For a USB installation, set the following order:
      1. USB
      2. Local hard drive

3. Disable Hyper-Threading.
Hyper-threading must be disabled (or verify it is disabled).


4. Configure Serial Port Redirection.
To install via serial port, configure the BIOS for serial port redirection with the first device
(COM0 or COM1).

Start the Installation

To start the installation, complete the following steps for your install environment. Installation will take a few minutes; the exact time depends on your environment. You may observe multiple reboots, and even be prompted for a login before installation has completed. You must wait until you get a confirmation that the Installation has completed successfully before logging in as instructed in the next step.

Network Installation

Boot the server. The installation will run automatically. Any command line parameters added to the pxelinux menu entry of the PXE grub.cfg menu will be applied per the Installation Command Line Parameters.

Virtual CD Installation

1. Boot your platform from Virtual CD.
installman-fig9


2. Press the e key to edit the KIOXIA Auto Deploy grub menu item, and set the path to the image location.
3. Enter the options chosen in  Installation Command Line Parameters. If you are installing the first node of the storage cluster, you must use kx_cluster_vip to provide the VIP for the master. After providing all the options and entering ctrl-x, the installation will begin, and the CentOS installation screen will appear.

USB Drive Installation

NOTE: The management port will be configured to kx0 if all the ports were disconnected during the installation.

WARNING: Do not connect the bootable USB drive to a production server, since it might delete its content without a prior warning.

1. Prepare the USB drive using the process described in USB Installation.
2. Connect the USB drive and boot the appliance. A SYSLINUX boot menu will appear containing four options:
KumoScale Auto Deployer

KumoScale Static IP (Define Static IP parameters)

KumoScale Auto Deployer – serial console

KumoScale Live (internal usage for debug)

3. Choose KumoScale Auto Deployer or, if using serial port redirection, KumoScale Auto Deployer – serial console.

4. Press Tab to customize any of the installation parameters.

5. Enter the options chosen in Installation Command Line Parameters. If you are installing the first node of the storage cluster, you must use kx_cluster_vip to provide the VIP for the master. After providing all the options, installation will begin, and the CentOS installation screen will appear.

Verify Installation Success

At the end of a successful installation:

  • The masters are set up with the KumoScale Provisioner service, control operators for installation and configuration, and the CSI driver.
  • Storage nodes are configured with DHCP unless a static IP has been specified.

Do not proceed until you get the message stating installation success similar to the one shown below:

installman-fig15

 

[1] Definition of capacity - KIOXIA Corporation defines a megabyte (MB) as 1,000,000 bytes, a gigabyte (GB) as 1,000,000,000 bytes and a terabyte (TB) as 1,000,000,000,000 bytes. A computer operating system, however, reports storage capacity using powers of 2 for the definition of 1Gbit = 230 bits = 1,073,741,824 bits, 1GB = 230 bytes = 1,073,741,824 bytes and 1TB = 240 bytes = 1,099,511,627,776 bytes and therefore shows less storage capacity. Available storage capacity (including examples of various media files) will vary based on file size, formatting, settings, software and operating system, and/or pre-installed software applications, or media content. Actual formatted capacity may vary.

 

Next: Install: Configuration & Deploying Additional Nodes