Deploying Storage Nodes

When creating a KumoScale storage node, all existing data on its SSDs will be deleted.

Once the KumoScale software is installed and configured for KumoScale, you will use storage node custom CRD files to configure nodes to join as masters or workers.

To create a storage node that will be automatically joined as a master or added to the KumoScale Provisioner service, follow these steps:

Create your Customized Storage Node

KumoScale software provides a storage node CRD file that you can use to define your own CRD for storage nodes in your environment. The storage node CRD includes parameters for defining system, network, and topology of the storage node. This section documents the parameters available and examples of how they are used.

You may expect to have a library of storage node CRD to reflect different applications and environments. Each one of these is referred to as a storage node. All of these parameters can be used for storage nodes on the KumoScale storage cluster or the KumoScale Provisioner service. For appliance mode, KumoScale software recommends that all masters of the KumoScale storage cluster, specified by the numberOfMasters in the master CR, be established before you can configure nodes for storage.

  • Make a copy of kioxia.com_v1_storagenode_cr.yaml for editing, and save to a separate directory (e.g., deploy/crds/myapp_storagenode_cr.yaml).
  • For each storage node you want to create you will need to update yaml with values for the parameters listed in Storage Node Parameters.

Storage Node Parameters

This section defines all the parameters used with the storage node CRD. Examples of storage nodes using these parameters is provided in Example Storage Node CRD.

Required Parameters

The table below shows system level parameters you should specify for each distinct storage node that you deploy. The apiVersion and kind parameters which appear at the top of the file should not be modified.

Parameter Name

Description

name

Unique name for the storage node. It must comply with the ‘Name’ rules (see KumoScale Field Types).

initMgmtIp

The initial management IP address.

adminSecretName

Name of the secret file created during installation.

groupName

Name of the group. It can remain with the default value.

timeSettings:
timeZoneID:
  

The time zone to use on the storage node.

(Appliance Mode only; does not apply to Managed mode)

   timeSettings:
     mode:

Mode can be NTP or Manual (Appliance Mode only; does not apply to Managed mode)

·         NTP: You must also specify a value for ntpServer. It must be reachable by ping otherwise the command will fail. It is advised to use this configuration when running a cluster of storage nodes, since all storage node’s time should be synchronized.

·         Manual: Time will be returned in YYYY-MM-DDThh:mm:ss format.

timeSettings:
ntpServer:

Specify only when mode:NTP as explained above

(Appliance Mode only; does not apply to Managed mode)

AuthenticationMode

The authentication mode. One of Local, Ldap, OPENIDC

See Example Storage Node CRD for examples using these parameters.

KumoScale requires all nodes to be synchronized as some sites may be supported by external NTP servers, and others by an internal NTP server. To avoid issues related to node timers, validate synchronization across cluster components.

Before deploying a storage node, all SSDs must be attached to it.

    Network Portal Parameters

    You must configure at least one NVMe-oF portal before provisioning storage. Use this section to configure a data interface for the application hosts to connect to their targets. See the following sections for details on the configuration you need to support; note that not all configurations are available in Managed Mode:

    • Consolidated Data and Management Ports
    • Network Management IP Parameters
    • Modifying Network Interfaces Parameters: Maximum Transmission Unit (MTU) or Link State
    • Network VLAN and LACP Configuration
    • Network VLAN Configuration
    • Network Border Gateway Protocol (BGP) Configuration
    • Topological Parameters

    Consolidated Data and Management Ports

    Most platforms provide for two sets of ports for connectivity:

    • Data ports are used for data traffic, and storage is accessed via block, file or object storage protocols.
    • Management ports are used by administrators to connect to, and manage, data or storage nodes. Some platforms provide one or more dedicated management ports, and these ports cannot be re-purposed for data traffic. On the other hand, some platforms do not provide any management ports, and administrators need to access node management capability via data ports.

    KumoScale software has qualified certain platforms that do not have a dedicated management port. In this case, KumoScale software can use a consolidated data and management port, which performs both functions.

    portals: Parameter Name

    Description

    Optional/Required

    ip

    The IP address for the interface to connect to the portal.

    Required

    subnet

    The subnet for the interface to connect to the portal.

    Required

    interface

    The interface to connect to the portal. A list of available interfaces may be obtained using the CLI show-interfaces command.

    Required

    port

    The port for the interface to connect to the portal. For valid port numbers, see KumoScale Field Types.

    Required

    name

    A unique portal name that must comply with the ‘Name’ rules (see KumoScale Field Types).

    Required

    transportType

    TCP_IP or RoCEv2 (for RDMA).

    Required

    See Example Storage Node CRD for examples using these parameters.

     

    Network Management IP Parameters

    mgmtIps Parameter Name

    Description

    Optional/Required

    Interface:

    Interface name

    Required

    ipAddress:

    IP address

    Required for Static IP

    mask

    Mask

    Required for Static IP

    mode

    Can be DHCP (default) or Static

    (DHCP = Dynamic Host Configuration Protocol)

    Required

    defaultGateway

    The storage node default gateway IP

    Required for Static IP

    dnsServer

    DNS address

    Required

    See Example Storage Node CRD for examples using these parameters.

     

    Modifying Network Interfaces Parameters: Maximum Transmission Unit (MTU) or Link State (Appliance Mode only)

    You may also use the modify the network interface’s link state or the MTU. To do so, you need to specify the interface name and desired values.

    You can set the MTU for external network interfaces. You cannot set the MTU internal interfaces which are used in HA systems.

    interfaces

    Parameter Name

    Description

    Optional/Required

    name:

    The interface name. It must comply with the ‘Name’ rules (see KumoScale Field Types).

    Required

    mtu

    The interface MTU; may be 1500 (default), 4200, or 9000. For network interfaces that use:

    ·         RoCE transport type, the optional MTU values are: 1500, 4200, or 9000.

    ·         TCP_IP transport type, the optional MTU values may be: 1500 or 9000.

    Optional

    adminState

    The current link state of the interface. The value is either UP or DOWN.

    Optional

    See Storage Node with MTU and Link State for an example using these parameters.

     

    Network VLAN and LACP Configuration (Appliance Mode only)

    KumoScale software n Appliance Mode isupports the Link Aggregation Control Protocol (LACP) of data ports to provide network resiliency. This is implemented with data ports teaming for Transmission Control Protocol (TCP) transport and bonding NIC ports for Remote Direct Memory Access (RDMA). Teaming is done by setting team parameters in the storage node CRD.

    The following constraints must be followed when configuring LACP:

    • Port configuration must be constant during an active session.
    • Data ports in an application must all be teamed/bonded or not.
    • Teamed NIC ports must be identical (e.g. the same maximal speed, product, or vendor).
    • The team member interface name or interface id may be specified, but not both.

    teams Parameter Name

    Description

    Optional/Required

    name:

    Team interface name, It must comply with the ‘Name’ rules (see KumoScale Field Types). You may specify the name or ID but not both; name is recommended.

    Either name or id is required but not both; name is recommended

    members:
    id:

    A list of at least two unused network interfaces, with its ID. You may specify the name or id but not both; name is recommended.

    Either name or id is required but not both; name is recommended

    tags

    Supported tag names:

    ·         tx_hash: value should be a list of comma-separated values from:

    o   eth: Uses source and destination MAC addresses.

    o   vlan: Uses VLAN id.

    o   ipv4: Uses source and destination IPv4 addresses.

    o   ipv6: Uses source and destination IPv6 addresses.

    o   ip: Uses source and destination IPv4 and IPv6 addresses.

    o   l3: Uses source and destination IPv4 and IPv6 addresses.

    o   tcp: Uses source and destination TCP ports.

    o   udp : Uses source and destination UDP ports.

    o   sctp: Uses source and destination SCTP ports.

    o   l4: Uses source and destination TCP and UDP and SCTP ports.    

    ·         tx_balancer_name: If the user wants to configure a balancer, a balancer name should be provided. The only value supported for now is “basic”.

    ·         tx_balancing_interval: An integer specifying the balancing interval in tenths of seconds.

    Optional

    See Example Storage Node CRD for examples using these parameters.

     

    Network VLAN Configuration (Appliance mode only)

    KumoScale in Appliance mode allows you to configure a VLAN by specifying VLAN information. VLAN tagging may also be applied to a teamed interface, e.g., VLAN over LACP. KumoScale software allows up to sixteen (16) VLAN tags per storage node. Once the VLAN interface is created, a portal interface may be created over it.

    The following constraints must be followed when configuring VLAN:

    • Port configuration must be constant during an active session.
    • VLAN configuration may not be changed while there is an active session.

    vlans Parameter Name

    Description

    Optional/Required

    interface

    Name of the physical interface

    Required

    tag

    The VLAN tag maximum: 4094, minimum:1

    Required

    See Example Storage Node CRD for examples using these parameters.

     

    Network Border Gateway Protocol (BGP) Configuration (Appliance Mode only)

    KumoScale in Appliance mode includes support for the Border Gateway Protocol (BGP) implemented via integration of the Free Range Routing (FRR) network routing software. It delivers multipath networking for NVMe-oF storage over TCP/IP networks. A Clos network topology is often used by data center operators to build high performance, scalable, cost effective, and robust networks. Such networks use IP routing as the primary packet forwarding mechanism, and BGP is the most popular routing protocol used in this type of environment. Storage system support for the BGP protocol enables storage resources to participate as a first-class citizen in a Clos network, allowing resilient, high bandwidth connectivity between initiators and storage targets. Traditional storage interconnect uses layer 2 technologies, such as port channels, to connect into IP networks. By participating instead at layer 3 (i.e. IP routing), KumoScale storage systems enter the modern data center network as a native cloud service. Running BGP as the routing protocol allows KumoScale storage systems to provide reliable and dynamically reroutable L3 level multipath network connectivity between initiators and KumoScale storage targets.

    BGP-based multipath routing enables a network to support resilient connectivity, higher capacity and performance through load balancing, improvement of the timeliness of response to network path changes, and enhancement of system resilience and security in the face of failures and attacks. 

    KumoScale software supports two types of BGP configurations: unnumbered and numbered.

    • BGP unnumbered: KumoScale software storage nodes are configured with Free Range Routing (FRR) such that storage nodes can be deployed with copy-paste FRR configuration files that simply specify the nearest Tor neighbors. KumoScale implements target-side virtual IPs to provision volumes to these initiators while allowing network paths to change beneath storage initiators and targets.
    • BGP numbered: KumoScale software storage nodes are configured with static IPs; with each NIC mapped to a virtual IP. Note that you will need to add a masquerade rule to the IP tables of the initiator machine. The reason for this is that with numbered BGP, the source IP is the BGP peer IP which is internal; masquerade will make packets look like they were sent from the external IP. For example, if the BGP peers on the initiator are ###.##.##.1/31 on eth4 and ###.##.##.3/31 on eth5, the rules would be one of the following. For:

    Interface level general:

    iptables -t nat -A POSTROUTING –o eth4 -j MASQUERADE

    iptables -t nat -A POSTROUTING –o eth5 -j MASQUERADE

     

    IP level precise:

    iptables -t nat -A POSTROUTING –s ###.##.##.1 -j MASQUERADE

    iptables -t nat -A POSTROUTING –s ###.##.##.3 -j MASQUERADE

     

    IP range less precise, shorter:

    iptables -t nat -A POSTROUTING –s ###.##.##.0/24 -j MASQUERADE

     

    The user can configure a BGP portal by specifying the information below in the storage node CRD. Example configurations are provided in Storage Node with BGP Unnumbered and Storage Node with BGP Numbered.

    bgpportals Parameter Name

    Description

    Optional/Required

    asn

    The portal Autonomous System Number (ASN). The value should be an integer in the range 1–23455, 23457–64495, 64512-65534, 131072–4294967294.

    Required

    ip

    The IP address for the portal.

    Required

    members:
       interface:

    The net interface name for the member of the BGP portal. This should adhere to name field requirements specified in KumoScale Field Types.

    Required

    members:
       ip

    IP address for the member.

    Required for numbered configurations

    members:
       mask

    Mask for the member.

    Required for numbered configurations

    members:
       neighborASN

    The neighbor ASN.

    Required for numbered configurations

    members:
       neighborIP

    The neighbor IP.

    Required for numbered configurations

    mode

    The portal mode. Equal to either Unnumbered or Numbered.

    Required

    name

    Name for the portal. This should adhere to name field requirements specified in KumoScale Field Types.

    Required

    port

    Port number for the portal. An integer between 4420 and 65535. Default is 4420.

    Required

    subnet

    The subnet for the portal IP.

    Required

    transportType

    The transport type of the portal. Only TCP_IP is currently supported.

    Optional

     

    Topological Parameters

    Topological parameters are used for volume placements and are specified as pairs. You can specify one or all of rack, zone, and region.

    topology Parameter Name

    Description

    Optional/Required

    name

    Can be one of topology.kubernetes.io/rack, topology.kubernetes.io/zone, topology.kubernetes.io/region, or kubernetes.io/hostname.

    Optional

    value

    Rack, zone, or region depending on value of name.

    Optional

    See Example Storage Node CRD for examples using these parameters.

    Example Storage Node CRD

    This section includes examples of storage node CRD and the purpose of specific settings. The IP addresses are for documentation purposes only.

    Storage Node using NTP, TCP_IP Portal, Rack and Zone Topology

    apiVersion: kumoscale.kioxia.com/v1
    kind: StorageNode
    metadata:

    name: ks-node1-000c298c715f

    spec:

    initMgmtIp: 192.0.20.0

    adminSecretName: kumoscale-secret

    groupName: group1

    timeSettings:

       timeZoneID: Asia/Jerusalem

       mode: NTP

       ntpServer: 172.###.###.###

    network:

       portals:

         - ip: 192.0.20.1

           name: portal1

           subnet: 255.255.0.0

           interface: kx0

           port: 4420

           transportType: TCP_IP

    topology:

       - name: topology.kubernetes.io/rack

         value: "RACK1"

       - name: topology.kubernetes.io/zone

         value: "LAB"

     

    Storage Node with MTU and Link State

    apiVersion: kumoscale.kioxia.com/v1

    kind: StorageNode

    metadata:

    name: testnode

    spec:

    initMgmtIp: 192.0.2.0

    adminSecretName: kumoscale-secret

    groupName: group1

    timeSettings:

       timeZoneID: Asia/Jerusalem

       mode: NTP

       ntpServer: time.google.com

    network:

         interfaces:

         - name: kx2

           mtu: b9000

           adminState: UP

       portals:

         - ip: 192.0.2.1

           name: portal20

           subnet: 255.255.0.0

           interface: kx2

           port: 4420

           transportType: TCP_IP

    topology:

       - name: topology.kubernetes.io/rack

         value: "RACK1"

       - name: topology.kubernetes.io/zone

         value: "LAB"

     

    Storage Node with Management IP on Team (Appliance Mode only)

    apiVersion: kumoscale.kioxia.com/v1

    kind: StorageNode

    metadata:

    name: ks-node2-000c29693644

    spec:

    initMgmtIp: 192.0.2.0

    adminSecretName: kumoscale-secret

    groupName: group1

    timeSettings:

       timeZoneID: Asia/Jerusalem

       mode: Manual

       ntpServer: None

    network:

       portals:

         - ip: 192.0.2.1

           name: portal1

           subnet: 255.255.0.0

           interface: kx0

           port: 4420

           transportType: TCP_IP

       mgmtIps:

         - interface: team1

           ipAddress: 30.30.30.26

           mask: 255.255.255.0

           mode: STATIC

       teams:

         - members:

             - name: kx1

             - name: kx2

           name: team1

    topology:

       - name: topology.kubernetes.io/rack

         value: "RACK1"

       - name: topology.kubernetes.io/zone

         value: "LAB"

     

    Storage Node with Management IP on VLAN (Appliance Mode only)

    apiVersion: kumoscale.kioxia.com/v1

    kind: StorageNode

    metadata:

    name: ks-node2-000c29693644

    spec:

    initMgmtIp: 172.##.##.###

    adminSecretName: kumoscale-secret

    groupName: group1

    timeSettings:

       timeZoneID: Asia/Jerusalem

       mode: Manual

       ntpServer: None

    network:

       portals:

         - ip: 172.##.##.###

           name: portal1

           subnet: 255.255.0.0

           interface: kx0

           port: 4420

           transportType: TCP_IP

       mgmtIps:

         - interface: kx1.9

           ipAddress: 30.##.##.###

           mask: 255.255.255.0

           mode: STATIC

       vlans:

         - interface: kx1

           tag: 9

    topology:

       - name: topology.kubernetes.io/rack

         value: "RACK1"

       - name: topology.kubernetes.io/zone

         value: "LAB"

     

    Storage Node with Portal on Physical Interface

    apiVersion: kumoscale.kioxia.com/v1

    kind: StorageNode

    metadata:

    name: ks-node2-0cc47ad48560

    spec:

    initMgmtIp: 10.###.##.##

    adminSecretName: kumoscale-secret

    groupName: p1

    timeSettings:

       timeZoneID: Asia/Kolkata

       mode: NTP

       ntpServer: 172.##.##.##

    network:

       portals:

         - ip: 192.168.20.94

           name: portal1

           subnet: 255.255.255.0

           interface: kx4

           port: 4420

           transportType: RoCEv2

    topology:

       - name : kubernetes.io/hostname

         value : "ks-node2-0cc47ad48560"

     

    Storage Node with Portal on VLAN (Appliance mode only)

    apiVersion: kumoscale.kioxia.com/v1

    kind: StorageNode

    metadata:

    name: ks-node2-0cc47ad48560

    spec:

    initMgmtIp: 10.###.##.##

    adminSecretName: kumoscale-secret

    groupName: p1

    timeSettings:

       timeZoneID: Asia/Kolkata

       mode: NTP

       ntpServer: 172.##.##.##

    network:

       teams:

       vlans:

         - interface: kx4

           tag: 9

       portals:

         - ip: 30.30.30.25

           name: portal1

           subnet: 255.255.255.0

           interface: kx4.9

           port: 4420

           transportType: TCP_IP

    topology:

       - name : kubernetes.io/hostname

         value : "ks-node2-0cc47ad48560"

     

    Storage Node with Portal on VLAN on Team (LACP) (Appliance Mode only)

    apiVersion: kumoscale.kioxia.com/v1

    kind: StorageNode

    metadata:

    name: ks-node2-0cc47ad48560

    spec:

    initMgmtIp: ##.###.##.##

    adminSecretName: kumoscale-secret

    groupName: p1

    timeSettings:

       timeZoneID: Asia/Kolkata

       mode: NTP

       ntpServer: 172.##.##.##

    network:

       teams:

         - name: team1

           members:

             - name: kx1

             - name: kx2

           tags:

             - name: tx_hash

               value: eth,ipv4,ipv6

             - name: tx_balancer_name

               value: basic

             - name: tx_balancing_interval

             value: "70"

       vlans:

         - interface: team1

           tag: 9

       portals:

         - ip: 30.30.30.25

           name: portal1

           subnet: 255.255.255.0

           interface: team1.9

           port: 4420

           transportType: TCP_IP

    topology:

       - name : kubernetes.io/hostname

         value : "ks-node2-0cc47ad48560"

     

    Storage Node with BGP Unnumbered (Appliance mode)

    apiVersion: kumoscale.kioxia.com/v1

    kind: StorageNode

    metadata:

    name: demo

    spec:

    initMgmtIp: 172.28.11.30

    adminSecretName: kumoscale-secret

    groupName: group1

    timeSettings:

       timeZoneID: Asia/Jerusalem

       mode: Manual

       ntpServer: None

    network:

       bgp_portals:

       - asn: 64005

         ip: 10.##.##.##

         members:

         - interface: kx1

         - interface: ens161

         mode: Unnumbered

         name: bgp_portal1

         port: 7980

         subnet: 255.255.0.0

         transportType: TCP_IP

     

    Storage Node with BGP Numbered (Appliance mode)

    Remember that you will also need to provide masquerade rules for the IP tables of the initiator.

    kumoscale_v1_storagenode_cr_numbered.yaml:

    Storage Node numbered

    apiVersion: kumoscale.kioxia.com/v1

    kind: StorageNode

    metadata:

    name: demo

    spec:

    initMgmtIp: 172.##.##.#

    adminSecretName: kumoscale-secret

    groupName: group1

    timeSettings:

       timeZoneID: Asia/Jerusalem

       mode: Manual

       ntpServer: None

    network:

       bgp_portals:

       - asn: 64005

         ip: 10.14.15.13

         members:

         - interface: kx1

           ip: 10.14.15.14

           mask: 255.255.0.0

           neighborASN: 63005

           neighborIP: 10.14.15.16

         - interface: ens161

           ip: 10.14.15.15

           mask: 255.255.0.0

           neighborASN: 62005

           neighborIP: 10.14.15.17

         mode: Numbered

         name: bgp_portal1

         port: 7980

         subnet: 255.255.0.0

         transportType: TCP_IP

     

    Create Storage Nodes

    The ‘create command’ creates the storage node and adds it to the KumoScale Provisioner service on the KumoScale storage cluster.

    You will need both a valid license and a secret to create storage nodes. Ensure that you have entered a valid license key before proceeding with this step. If you have the KumoScale software secret but not a valid license, your storage node will be created but will not be registered with the KumoScale Provisioner service and you will not be able to use it for provisioning.

    To create the Storage Node with name = myapp_storagenode_1 defined in the CRD file myapp_storagenode_cr.yaml, enter the following:

    kubectl create -f myapp_storagenode_cr.yaml

    To verify that the node was created and added to the KumoScale Provisioner service, enter:

    kubectl get storagenodes
    kubectl describe storagenodes myapp_storagenode_1

    The storage node will not be added to the KumoScale Provisioner service if one of the parameter values is invalid. In the event of a failure, you will get an error message identifying the parameters causing the failure. Update the relevant parameter in the CRD and enter the following to apply the changes.

    kubectl apply -f myapp_storagenode_cr.yaml

    Once the storage node is added to the KumoScale Provisioner service, you will not be able to modify it except for cases noted in Updating Storage Nodes.

     Update Storage Nodes

    You can change the configuration of a storage node by using kubectl to edit the CRD file that defined the particular storage node and then apply the change. For example, to edit the storagenode myapp_storagenode_1 defined by myapp_storagenode_cr.yaml, enter:

    kubectl edit -f myapp_storagenode_cr.yaml

    Change the settings and save the file. Then enter the command:

    kubectl apply -f myapp_storagenode_cr.yaml

    Run following command to verify the node has the desired configuration::

    kubectl describe storagenodes myapp_storagenode_1

    State and Status Information

    Kubectl commands to get information on storage nodes and services will return STATE and STATUS of the nodes. There is a difference between these two fields. The

    • STATE reflects the current state of the storage node. It is dynamically reported by the Provisioner and reflected by the operator.
    • STATUS reflects the status of the result of the last user operation done on a storage node CRD. For example; create a new storage node CRD or update an existing one.

    Possible values for State and Status are Available, Dead, Pending, and Unavailable.

    For example, kubectl get storagenodes presents the STATE, which is the real STATE of the storage node:

    userman-fig4a

    kubectl get storagenodes –o wide presents the STATE of the storage node and the STATUS of the last user operation on this storage node:

    kubectl describe storagenode <name> presents the complete latest STATUS info of the storage node.

    Delete Storage Nodes

    You can use kubectl to delete a storage node only when it is not currently undergoing an update and the status of the node is either:

    • Dead, or
    • Available and it has no volumes attached.

    The system will remove any syslog configured on the server.

    For example, to delete the storagenode myapp_storagenode_1 defined by myapp_storagenode_cr.yaml, enter:

    kubectl delete -f myapp_storagenode_cr.yaml

     

    Next: Role Based Access Control (Appliance Mode)