Deploying Storage Nodes

Once the KumoScale software is installed and the master CR deployed with configuration information for the KumoScale storage cluster, you will use storage node custom CR files to configure the nodes to join as masters or workers.

To create a storage node that will be automatically joined as a master or added to the KumoScale Provisioner service, follow these steps:

  1. Create your customized storage node.
  2. Create the storage node.

When creating a KumoScale storage node, all existing data on its SSDs will be deleted.

Create your Customized Storage Node

KumoScale software provides a storage node CR file that you can use to define your own CRs for storage nodes in your environment. The storage node CRD includes parameters for defining system, network, and topology of the storage node. This section documents the parameters available and examples of how they are used.

You may expect to have a library of storage node CRs to reflect different applications and environments. Each one of these is referred to as a storage node. All of these parameters can be used for storage nodes on the KumoScale storage cluster or the KumoScale Provisioner service. However, KumoScale software recommends that all masters of the KumoScale storage cluster, specified by the numberOfMasters in the master CR, be established before you can configure nodes for storage.

  • Make a copy of kioxia.com_v1_storagenode_cr.yaml for editing, and save to a separate directory (e.g., deploy/crds/myapp_storagenode_cr.yaml).
  • For each storage node you want to create you will need to update yaml with values for the parameters listed in the following section.

Storage Node Parameters

This section defines all the parameters used with the storage node CR. Examples of storage nodes using these parameters is provided in Example Storage Node CRD.

Required Parameters

The table below shows system level parameters you should specify for each distinct storage node that you deploy. The apiVersion and kind parameters which appear at the top of the file should not be modified.

Parameter Name

Description

name

Unique name for the storage node. It must comply with the ‘Name’ rules (see KumoScale Field Types).

initMgmtIp

The initial management IP address.

adminSecretName

Name of the secret file created during installation.

groupName

Name of the group. It can remain with the default value.

timeSettings:
timeZoneID:

The time zone to use on the storage node.

   timeSettings:
     mode:

Mode can be NTP or Manual:

·       NTP: You must also specify a value for ntpServer. It must be reachable by ping otherwise the command will fail. It is advised to use this configuration when running a cluster of storage nodes, since all storage node’s time should be synchronized.

·       Manual: Time will be returned in YYYY-MM-DDThh:mm:ss format.

timeSettings:
ntpServer:

Specify only when mode:NTP as explained above.

See Example Storage Node CRD for examples using these parameters.

NOTE: KumoScale requires all nodes to be synchronized as some sites may be supported by external NTP servers, and others by an internal NTP server. To avoid issues related to node timers, validate synchronization across cluster components. Before deploying a storage node, all SSDs must be attached to it. 

Network Portal Parameters

You must configure at least one NVMe-oF portal before provisioning storage. Use this section to configure a data interface for the application hosts to connect to their targets. See the following sections for details on the configuration you need to support:

  • Consolidated Data and Management Ports
  • Network Management IP Parameters
  • Modifying Network Interfaces Parameters: Maximum Transmission Unit (MTU) or Link State
  • Network VLAN and LACP Configuration
  • Network VLAN Configuration
  • Network Border Gateway Protocol (BGP) Configuration
  • Topological Parameters

Consolidated Data and Management Ports

Most platforms provide for two sets of ports for connectivity:

  • Data ports are used for data traffic, and storage is accessed via block, file or object storage protocols.
  • Management ports are used by administrators to connect to, and manage, data or storage nodes. Some platforms provide one or more dedicated management ports, and these ports cannot be re-purposed for data traffic. On the other hand, some platforms do not provide any management ports, and administrators need to access node management capability via data ports.

KumoScale software has qualified certain platforms that do not have a dedicated management port. In this case, KumoScale software can use a consolidated data and management port, which performs both functions.

portals: Parameter Name

Description

Optional/Required

ip

The IP address for the interface to connect to the portal.

Required

subnet

The subnet for the interface to connect to the portal.

Required

interface

The interface to connect to the portal. A list of available interfaces may be obtained using the CLI show-interfaces command.

Required

port

The port for the interface to connect to the portal. For valid port numbers, see KumoScale Field Types.

Required

name

A unique portal name that must comply with the ‘Name’ rules (see KumoScale Field Types).

Required

transportType

TCP_IP or RoCEv2 (for RDMA).

Required

See Example Storage Node CRD for examples using these parameters.

Network Management IP Parameters

mgmtIps Parameter Name

Description

Optional/Required

Interface:

Interface name

Required

ipAddress:

IP address

Required for Static IP

mask

Mask

Required for Static IP

mode

Can be DHCP (default) or Static

(DHCP = Dynamic Host Configuration Protocol)

Required

defaultGateway

The storage node default gateway IP

Required for Static IP

dnsServer

DNS address

Required

See Example Storage Node CRD for examples using these parameters.

Modifying Network Interfaces Parameters: Maximum Transmission Unit (MTU) or Link State

You may also use the modify the network interface’s link state or the MTU. To do so, you need to specify the interface name and desired values.

You can set the MTU for external network interfaces. You cannot set the MTU internal interfaces which are used in HA systems.

interfaces

Parameter Name

Description

Optional/Required

name:

The interface name. It must comply with the ‘Name’ rules (see KumoScale Field Types).

Required

mtu

The interface MTU; may be 1500 (default), 4200, or 9000. For network interfaces that use:

  • RoCE transport type, the optional MTU values are: 1500, 4200, or 9000.
  • TCP_IP transport type, the optional MTU values may be: 1500 or 9000.

Optional

adminState

The current link state of the interface. The value is either UP or DOWN.

Optional

See Storage Node with MTU and Link State for an example using these parameters.

Network VLAN and LACP Configuration

KumoScale software supports the Link Aggregation Control Protocol (LACP) of data ports to provide network resiliency. This is implemented with data ports teaming for Transmission Control Protocol (TCP) transport and bonding NIC ports for Remote Direct Memory Access (RDMA). Teaming is done by setting team parameters in the storage node CR.

The following constraints must be followed when configuring LACP:

  • Port configuration must be constant during an active session.
  • Data ports in an application must all be teamed/bonded or not.
  • Teamed NIC ports must be identical (e.g. the same maximal speed, product, or vendor).
  • The team member interface name or interface id may be specified, but not both.

teams Parameter Name

Description

Optional/Required

name:

Team interface name, It must comply with the ‘Name’ rules (see KumoScale Field Types). You may specify the name or ID but not both; name is recommended.

Either name or id is required but not both; name is recommended

members:
id:

A list of at least two unused network interfaces, with its ID. You may specify the name or id but not both; name is recommended.

Either name or id is required but not both; name is recommended

tags

Supported tag names:

  • tx_hash: value should be a list of comma-separated values from:
    • eth: Uses source and destination MAC addresses.
    • vlan: Uses VLAN id.
    • ipv4: Uses source and destination IPv4 addresses.
    • ipv6: Uses source and destination IPv6 addresses.
    • ip: Uses source and destination IPv4 and IPv6 addresses.
    • l3: Uses source and destination IPv4 and IPv6 addresses.
    • tcp: Uses source and destination TCP ports.
    • udp : Uses source and destination UDP ports.
    • sctp: Uses source and destination SCTP ports.
    • l4: Uses source and destination TCP and UDP and SCTP ports.    
  • tx_balancer_name: If the user wants to configure a balancer, a balancer name should be provided. The only value supported for now is “basic”.
  • tx_balancing_interval: An integer specifying the balancing interval in tenths of seconds.

Optional

See Example Storage Node CRD for examples using these parameters.

Network VLAN Configuration

The user can configure a VLAN by specifying VLAN information. VLAN tagging may also be applied to a teamed interface, e.g., VLAN over LACP. KumoScale software allows up to sixteen (16) VLAN tags per storage node. Once the VLAN interface is created, a portal interface may be created over it.

The following constraints must be followed when configuring VLAN:

  • Port configuration must be constant during an active session.
  • VLAN configuration may not be changed while there is an active session.

vlans Parameter Name

Description

Optional/Required

interface

Name of the physical interface

Required

tag

The VLAN tag maximum: 4094, minimum:1

Required

See Example Storage Node CRs for examples using these parameters.

Network Border Gateway Protocol (BGP) Configuration

KumoScale includes support for the Border Gateway Protocol (BGP) implemented via integration of the Free Range Routing (FRR) network routing software. It delivers multipath networking for NVMe-oF storage over TCP/IP networks. A Clos network topology is often used by data center operators to build high performance, scalable, cost effective, and robust networks. Such networks use IP routing as the primary packet forwarding mechanism, and BGP is the most popular routing protocol used in this type of environment. Storage system support for the BGP protocol enables storage resources to participate as a first-class citizen in a Clos network, allowing resilient, high bandwidth connectivity between client initiators and storage targets. Traditional storage interconnect uses layer 2 technologies, such as port channels, to connect into IP networks. By participating instead at layer 3 (i.e. IP routing), KumoScale storage systems enter the modern data center network as a native cloud service. Running BGP as the routing protocol allows KumoScale storage systems to provide reliable and dynamically reroutable L3 level multipath network connectivity between client initiators and KumoScale storage targets.

BGP-based multipath routing enables a network to support resilient connectivity, higher capacity and performance through load balancing, improvement of the timeliness of response to network path changes, and enhancement of system resilience and security in the face of failures and attacks. 

KumoScale software supports two types of BGP configurations: unnumbered and numbered.

  • BGP unnumbered: KumoScale software storage nodes are configured with Free Range Routing (FRR) such that storage nodes can be deployed with copy-paste FRR configuration files that simply specify the nearest Tor neighbors. KumoScale implements target-side virtual IPs to provision volumes to these clients while allowing network paths to change beneath storage clients and targets.
  • BGP numbered: KumoScale software storage nodes are configured with static IPs; with each NIC mapped to a virtual IP. Note that you will need to add a masquerade rule to the IP tables of the host machine. The reason for this is that with numbered BGP, the source IP is the BGP peer IP which is internal; masquerade will make packets look like they were sent from the external IP. For example, if the BGP peers on the host machine are ###.##.##.1/31 on eth4 and ###.##.##.3/31 on eth5, the rules would be one of the following. For:
Interface level general:
iptables -t nat -A POSTROUTING –o eth4 -j MASQUERADE

iptables -t nat -A POSTROUTING –o eth5 -j MASQUERADE

IP level precise:

iptables -t nat -A POSTROUTING –s ###.##.##.1 -j MASQUERADE

iptables -t nat -A POSTROUTING –s ###.##.##.3 -j MASQUERADE

IP range less precise, shorter:

iptables -t nat -A POSTROUTING –s ###.##.##.0/24 -j MASQUERADE

The user can configure a BGP portal by specifying the information below in the storage node CR. Example configurations are provided in Storage Node with BGP Unnumbered and Storage Node with BGP Numbered.

bgpportals Parameter Name

Description

Optional/Required

asn

The portal Autonomous System Number (ASN). The value should be an integer in the range 1–23455, 23457–64495, 64512-65534, 131072–4294967294.

Required

ip

The IP address for the portal.

Required

members:
   interface:

The net interface name for the member of the BGP portal. This should adhere to name field requirements specified in KumoScale Field Types.

Required

members:
   ip

IP address for the member.

Required for numbered configurations

members:
   mask

Mask for the member.

Required for numbered configurations

members:
   neighborASN

The neighbor ASN.

Required for numbered configurations

members:
   neighborIP

The neighbor IP.

Required for numbered configurations

mode

The portal mode. Equal to either Unnumbered or Numbered.

Required

name

Name for the portal. This should adhere to name field requirements specified in KumoScale Field Types.

Required

port

Port number for the portal. An integer between 4420 and 65535. Default is 4420.

Required

subnet

The subnet for the portal IP.

Required

transportType

The transport type of the portal. Only TCP_IP is currently supported.

Optional

Topological Parameters

Topological parameters are used for volume placements and are specified as pairs. You can specify one or all of rack, zone, and region.

topology Parameter Name

Description

Optional/Required

name

Can be one of topology.kubernetes.io/rack, topology.kubernetes.io/zone, topology.kubernetes.io/region, or kubernetes.io/hostname.

Optional

value

Rack, zone, or region depending on value of name.

Optional

See Example Storage Node CRD for examples using these parameters.

Example Storage Node CRD

This section includes examples of storage node CRD and the purpose of specific settings. The IP addresses are for documentation purposes only.

Storage Node using NTP, TCP_IP Portal, Rack and Zone Topology

apiVersion: kumoscale.kioxia.com/v1
kind: StorageNode
metadata:

name: ks-node1-000c298c715f

spec:

initMgmtIp: 192.0.20.0

adminSecretName: kumoscale-secret

groupName: group1

timeSettings:

   timeZoneID: Asia/Jerusalem

   mode: NTP

   ntpServer: 172.###.###.###

network:

   portals:

     - ip: 192.0.20.1

       name: portal1

       subnet: 255.255.0.0

       interface: kx0

       port: 4420

       transportType: TCP_IP

topology:

   - name: topology.kubernetes.io/rack

     value: "RACK1"

   - name: topology.kubernetes.io/zone

     value: "LAB"

Storage Node with MTU and Link State

apiVersion: kumoscale.kioxia.com/v1

kind: StorageNode

metadata:

name: testnode

spec:

initMgmtIp: 192.0.2.0

adminSecretName: kumoscale-secret

groupName: group1

timeSettings:

   timeZoneID: Asia/Jerusalem

   mode: NTP

   ntpServer: time.google.com

network:

     interfaces:

     - name: kx2

       mtu: b9000

       adminState: UP

   portals:

     - ip: 192.0.2.1

       name: portal20

       subnet: 255.255.0.0

      interface: kx2

       port: 4420

       transportType: TCP_IP

topology:

   - name: topology.kubernetes.io/rack

     value: "RACK1"

   - name: topology.kubernetes.io/zone

     value: "LAB"

Storage Node with Management IP on Team

apiVersion: kumoscale.kioxia.com/v1

kind: StorageNode

metadata:

name: ks-node2-000c29693644

spec:

initMgmtIp: 192.0.2.0

adminSecretName: kumoscale-secret

groupName: group1

timeSettings:

   timeZoneID: Asia/Jerusalem

   mode: Manual

   ntpServer: None

network:

   portals:

     - ip: 192.0.2.1

       name: portal1

       subnet: 255.255.0.0

       interface: kx0

       port: 4420

       transportType: TCP_IP

   mgmtIps:

     - interface: team1

       ipAddress: 30.30.30.26

       mask: 255.255.255.0

       mode: STATIC

   teams:

     - members:

         - name: kx1

         - name: kx2

       name: team1

topology:

   - name: topology.kubernetes.io/rack

     value: "RACK1"

   - name: topology.kubernetes.io/zone

     value: "LAB"

Storage Node with Management IP on VLAN

apiVersion: kumoscale.kioxia.com/v1

kind: StorageNode

metadata:

name: ks-node2-000c29693644

spec:

initMgmtIp: 172.##.##.###

adminSecretName: kumoscale-secret

groupName: group1

timeSettings:

   timeZoneID: Asia/Jerusalem

   mode: Manual

   ntpServer: None

network:

   portals:

     - ip: 172.##.##.###

       name: portal1

       subnet: 255.255.0.0

       interface: kx0

       port: 4420

       transportType: TCP_IP

   mgmtIps:

     - interface: kx1.9

       ipAddress: 30.##.##.###

       mask: 255.255.255.0

       mode: STATIC

   vlans:

     - interface: kx1

       tag: 9

topology:

   - name: topology.kubernetes.io/rack

     value: "RACK1"

   - name: topology.kubernetes.io/zone

     value: "LAB"

Storage Node with Portal on Physical Interface

apiVersion: kumoscale.kioxia.com/v1

kind: StorageNode

metadata:

name: ks-node2-0cc47ad48560

spec:

initMgmtIp: 10.###.##.##

adminSecretName: kumoscale-secret

groupName: p1

timeSettings:

   timeZoneID: Asia/Kolkata

   mode: NTP

   ntpServer: 172.##.##.##

network:

   portals:

     - ip: 192.168.20.94

       name: portal1

       subnet: 255.255.255.0

       interface: kx4

       port: 4420

      transportType: RoCEv2

topology:

   - name : kubernetes.io/hostname

     value : "ks-node2-0cc47ad48560"

Storage Node with Portal on VLAN

apiVersion: kumoscale.kioxia.com/v1

kind: StorageNode

metadata:

name: ks-node2-0cc47ad48560

spec:

initMgmtIp: 10.###.##.##

adminSecretName: kumoscale-secret

groupName: p1

timeSettings:

   timeZoneID: Asia/Kolkata

   mode: NTP

   ntpServer: 172.##.##.##

network:

   teams:

   vlans:

     - interface: kx4

       tag: 9

   portals:

     - ip: 30.30.30.25

       name: portal1

       subnet: 255.255.255.0

       interface: kx4.9

       port: 4420

      transportType: TCP_IP

topology:

   - name : kubernetes.io/hostname

     value : "ks-node2-0cc47ad48560"

Storage Node with Portal on VLAN on Team (LACP)

apiVersion: kumoscale.kioxia.com/v1

kind: StorageNode

metadata:

name: ks-node2-0cc47ad48560

spec:

initMgmtIp: 10.116.86.25

adminSecretName: kumoscale-secret

groupName: p1

timeSettings:

   timeZoneID: Asia/Kolkata

   mode: NTP

   ntpServer: 172.##.##.##

network:

   teams:

     - name: team1

       members:

         - name: kx1

         - name: kx2

       tags:

         - name: tx_hash

           value: eth,ipv4,ipv6

         - name: tx_balancer_name

           value: basic

         - name: tx_balancing_interval

           value: "70"

   vlans:

     - interface: team1

       tag: 9

   portals:

     - ip: 30.30.30.25

       name: portal1

       subnet: 255.255.255.0

       interface: team1.9

       port: 4420

      transportType: TCP_IP

topology:

   - name : kubernetes.io/hostname

     value : "ks-node2-0cc47ad48560"

Storage Node with BGP Unnumbered

apiVersion: kumoscale.kioxia.com/v1

kind: StorageNode

metadata:

name: demo

spec:

initMgmtIp: 172.28.11.30

adminSecretName: kumoscale-secret

groupName: group1

timeSettings:

   timeZoneID: Asia/Jerusalem

   mode: Manual

   ntpServer: None

network:

   bgp_portals:

   - asn: 64005

     ip: 10.##.##.##

     members:

     - interface: kx1

     - interface: ens161

     mode: Unnumbered

     name: bgp_portal1

     port: 7980

     subnet: 255.255.0.0

     transportType: TCP_IP

Storage Node with BGP Numbered

Remember that you will also need to provide masquerade rules for the IP tables of the host machine.

kumoscale_v1_storagenode_cr_numbered.yaml:

Storage Node numbered

apiVersion: kumoscale.kioxia.com/v1

kind: StorageNode

metadata:

name: demo

spec:

initMgmtIp: 172.##.##.#

adminSecretName: kumoscale-secret

groupName: group1

timeSettings:

   timeZoneID: Asia/Jerusalem

   mode: Manual

   ntpServer: None

network:

   bgp_portals:

   - asn: 64005

     ip: 10.14.15.13

     members:

     - interface: kx1

       ip: 10.14.15.14

       mask: 255.255.0.0

       neighborASN: 63005

       neighborIP: 10.14.15.16

     - interface: ens161

       ip: 10.14.15.15

       mask: 255.255.0.0

       neighborASN: 62005

       neighborIP: 10.14.15.17

     mode: Numbered

     name: bgp_portal1

     port: 7980

     subnet: 255.255.0.0

     transportType: TCP_IP

Create Storage Nodes

The ‘create command’ creates the storage node and adds it to the KumoScale Provisioner service on the KumoScale storage cluster.

You will need both a valid license and a secret to create storage nodes. Ensure that you have entered a valid license key before proceeding with this step. If you have the KumoScale software secret but not a valid license, your storage node will be created but will not be registered with the KumoScale Provisioner service and you will not be able to use it for provisioning.

To create the Storage Node with name = myapp_storagenode_1 defined in the CR file myapp_storagenode_cr.yaml, enter the following:

kubectl create -f myapp_storagenode_cr.yaml

To verify that the node was created and added to the KumoScale Provisioner service, enter:

kubectl get storagenodes
kubectl describe storagenodes myapp_storagenode_1

The storage node will not be added to the KumoScale Provisioner service if one of the parameter values is invalid. In the event of a failure, you will get an error message identifying the parameters causing the failure. Update the relevant parameter in the CR and enter the following to apply the changes.

kubectl apply -f myapp_storagenode_cr.yaml

NOTE: Once the storage node is added to the KumoScale Provisioner service, you will not be able to modify it except for cases noted in the following section, Update Storage Nodes.  

Update Storage Nodes

You can change the configuration of a storage node by using kubectl to edit the CR file that defined the particular storage node and then apply the change. For example, to edit the storagenode myapp_storagenode_1 defined by myapp_storagenode_cr.yaml, enter:

kubectl edit -f myapp_storagenode_cr.yaml

Change the settings and save the file. Then enter the command:

kubectl apply -f myapp_storagenode_cr.yaml

Run following command to verify the node has the desired configuration::

kubectl describe storagenodes myapp_storagenode_1

State and Status Information

Kubectl commands to get information on storage nodes and services will return STATE and STATUS of the nodes. There is a difference between these two fields. The

  • STATE reflects the current state of the storage node. It is dynamically reported by the Provisioner and reflected by the operator.
  • STATUS reflects the status of the result of the last user operation done on a storage node CR. For example; create a new storage node CR or update an existing one.

For example, kubectl get storagenodes presents the STATE, which is the real STATE of the storage node:

kubectl get storagenodes –o wide presents the STATE of the storage node and the STATUS of the last user operation on this storage node:

kubectl describe storagenode <name> presents the complete latest STATUS info of the storage node.

Delete Storage Nodes

You can use kubectl to delete a storage node only when it is not currently undergoing an update and the status of the node is either:

  • Dead, or
  • Available and it has no volumes attached.

The system will remove any syslog configured on the server.

For example, to delete the storagenode myapp_storagenode_1 defined by myapp_storagenode_cr.yaml, enter:

kubectl delete -f myapp_storagenode_cr.yaml

 

Next: Role-based Access Control