CSI Storage Provisioning

This section explains how to install and configure your environment to support storage provisioning in a Kubernetes orchestration environment.

KumoScale Installation and Configuration

  1. Follow the instructions in KumoScale Installation Guide and User Manual to install and configure KumoScale software on storage nodes. This will also add the storage nodes to the provisioner service as worker nodes.
  2. You may further configure the storage nodes using REST API or CLI commands with advanced networking options (see the User Manual and REST API documents for further details).

Enable Topology Awareness

Edit the storage class and set the volume’s topology constraints to match a zone or region for resiliency purposes or to ensure pods have access to it.

  1. Set the VolumeBindingMode to “WaitForFirstConsumer
    (See Storage Class Parameters). This will ensure that provisioner service allocation will be done when a pod that refers to it starts running for the first time.
  2. Label each Kubernetes node with topology labels:
    1. topology.kubernetes.io/region
    2. topology.kubernetes.io/zone
    3. topology.kubernetes.io/rack
      1. Note - The CSI driver must be redeployed after labeling nodes.
  3. Set the storage node’s Region, Rack, and Zone with corresponding values.
    1. Note - A matching storage node with enough free capacity must exist to ensure that a KumoScale software volume is allocated in the required region and zone.

Define a Tenant in a Multi-Tenancy Environment

Create a tenant via a REST API or CLI with performance and capacity budgets. Specify the tenant’s ID in the secret file (see the example provisioner-secret.yaml).

Provisioning Storage from KumoScale Software

Follow these steps to create a volume based on a storage class.

Use the parameters defined in Storage Class Parameters to create storage classes in the steps below:

  1. Create a KumoScale storage class using provisioner-storage-class.yaml as an example:
    1. kubectl -f create provisioner-storage-class.yaml
  2. Create a Persistent Volume Claim (PVC) using the storage class. See yaml for an example.
  3. Create a pod based on the PVC. See yaml for an example.

Storage Class Parameters

The KumoScale CSI driver supports the volume properties shown in the following table.

Table 1. KumoScale CSI Storage Class Parameters

Parameter

Description and Possible Values

name

The storage class name - A string of up to 255 characters. The default name is default.

protocol

The volume’s protocol:

  • NVMe-oF protocol (default).
  • local (for hosted applications).
    If specified only one replica is allowed, and the hostId must be specified.
provisioningType

Volume type:

  • thin provisioned.
    The storage space will be allocated only upon use (i.e., when data is actually written).
  • thick (default).
numReplicas

The number of replications for a volume created with this storage class. When this parameter equals:

  • 1 (default): A non-resilient volume is created.
  • 2 or 3: A volume with 2 or 3 replicas is created accordingly. 3 is the maximum.
sameRackAllowed

Indicates whether more than one replica of a volume may be placed on the same rack.

Boolean (true or false). The default value is false.

volumeBindingMode

Indicates whether or not to wait for the first consumer before binding the volume. This will ensure the volume is accessible to the pod, since the binding will be done according to the node topology (if the node's ‘zone’ and ‘region’ labels are configured). WaitForFirstConsumer or Immediate.

The default is Immediate.

csi.storage.k8s.io/fstype

Selected filesystem.

The possible options are: ext2, ext3, ext4, or xfs.
The default value is ext4.

maxIOPSPerGB

Sets an upper limitation on the IOPs on the volume, per volume GB. The default value is 0 – no limit.

desiredIOPSPerGB

The desired amount of IOPs for the volume, per volume GB.

The default value is 0 – no specification.

maxBWPerGB

Sets an upper limitation on the bandwidth of the volume IO commands, per volume GB.

In kilobytes per second (KB/s). The default value is 0 – no limit.

desiredBWPerGB

The desired amount of bandwidth for the volume IO, per volume GB.

In kilobytes per second (KB/s). The default value is 0 – no specification.

blockSize

The block size of the volume and filesystem.

In bytes. The default is 4096.

allowVolumeExpansion

Indicates if the volume can be expanded.
Boolean (true or false). The default value is false.

reservedSpacePercentage

The space initially reserved for a snapshot volume change log, or for a thin-provisioned volume in percentage of the original volume capacity. The default value is 10.

For thin-provisioned volumes the actual reserved space should be larger than 20 gigabytes[1] (GB).

writable

Specifies whether a snapshot volume based on this storage class is writable or read-only.

Boolean (true or false). The default value is true.

maxReplicaDownTime

The maximum duration in minutes to wait when detecting degradation in the replicated volume before initiating the autonomous self-healing process.

Minimum: 5. Maximum: 1440 (1 day).

Default value: 0 (no automatic recovery).

allowSpan

Allow volumes to span more than one SSD.

Boolean (true or false). The default value is true.

shareSSDBetweenVolumes

Specifies whether several volumes with this storage class can reside on the same volume.

Boolean (true or false). The default value is true.

logSpace

Defines the required log space in case of snapshot volume as a percentage of the source volume capacity.

 

Deleting a Volume

A source volume can be deleted even if it has snapshot volumes or clone volumes. Before deleting a volume, verify no pod is currently using it. You can do that using the kubectl command to describe pods. Then execute the following command to delete the volume:

kubectl delete -f <create_yaml_file>

Note - create_yaml_file is the yaml file used to create the PVC.

Expanding a Volume

KumoScale software supports volume online expansion for any type of volume including snapshot volumes and clones. In order to expand a volume you must have set up the plug-in as described in "Prepare the Environment for Features in Beta" in CSI Installation, and the PVC must be attached to a pod by following the steps below:

  1. Edit the PVC using the following command:
    1. kubectl edit pvc <pvc name>
  2. Increase the PVC capacity (under spec: resources: requests: storage).
  3. Save and exit.

Allow 1-2 minutes for the actual expansion to take place: ‘kubectl’ needs to detect the change and execute the expansion.

Using Snapshots

The CSI driver can support up to 8 snapshots per volume. Up to 512 snapshot volumes can be created from each of these snapshots. Snapshot volumes can be either Read/Write or Read Only. It is assumed that the application layer will quiesce the IO (bring the data into a state suitable for backups) to maintain data coherency.

Before proceeding, if you are using Kubernetes version:

  • 20 or higher, you should have installed the plug-ins as explained in "Install the Snapshot Plug-ins" in CSI Installation.
  • 16- 1.19, you must complete the steps from "Prepare the Environment for Features in Beta" in CSI Installation.

To take a snapshot, follow the steps below:

  1. Create a snapshot.
  2. Create a snapshot volume.

Creating a Snapshot

  1. Create a VolumeSnapshotClass. Use SnapshotClass.yaml as an example.
  2. The reservedSpacePercentage field can be specified which indicates the percentage of the original volume’s capacity a snapshot may utilize (the default value is 10).
  3. Create a PVC for a non-resilient volume (numReplica=1 in the storage class). Use kube1PVC.yaml as an example.
  4. Create a snapshot with a snapshot yaml that refers to the PVC. See Snapshot.yaml for an example.

Creating a Snapshot Volume

Create a snapshot volume with a snapshot yaml. Use the SnapshotVolume.yaml as an example for a writable snapshot, or the ROSnapshotVolume.yaml (along with the ReadOnlyStorageClass.yaml) as an example of a read-only snapshot volume.

When creating a writable snapshot volume using the SnapshotVolume.yaml example, a different class can be provided to configure the reserved space with a value other than the value in SnapshotClass.

Listing Snapshots

You can list all snapshots taken from a volume using the list snapshot command:

  • Provisioner REST API: Get Snapshots by Volume.
  • Provisioner CLI: show-snapshots.

Deleting a Snapshot Volume

To delete a snapshot, you must delete components in this order:

  1. Delete all snapshot volumes.
  2. Delete the snapshot.
  3. Delete the original volume (PVC).

Creating a Clone

To clone a volume, you need to create a PVC with a data source that refers to an existing PVC. See CloneVolume.yaml for an example.

Snapshot volumes and clones are independent. This means that a source volume can be deleted even if it has snapshot volumes or clone volumes.

You can also create clones of equal or greater quantity to the original volume. This is due to the fact that CSI allows allocation via a capacity range. If the value is greater than the source volume, the Provisioner will extend the created volume to the required capacity.

Extended Resources

Kubernetes allows administrators to specify key, value pairs on each node as an extended resource. Administrators can then create pods with requests for a specific amount of extended resources. KumoScale uses this feature to influence where pods are scheduled depending on storage nodes constraints.

For example, the pod below asks for two (2) volumes of 200 GB on two (2) separate SSDs:

apiVersion: v1

kind: Pod

metadata:

name: extended-resource-demo2

spec:

containers:

- name: extended-resource-demo-ctr

   image: nginx

   resources:

     requests:

       kumoscale.kioxia.com/availableSpaceFor2VolumesInGB: 200

     limits:

       kumoscale.kioxia.com/availableSpaceFor2VolumesInGB: 200

 

[1] Definition of capacity - KIOXIA Corporation defines a megabyte (MB) as 1,000,000 bytes, a gigabyte (GB) as 1,000,000,000 bytes and a terabyte (TB) as 1,000,000,000,000 bytes. A computer operating system, however, reports storage capacity using powers of 2 for the definition of 1Gbit = 230 bits = 1,073,741,824 bits, 1GB = 230 bytes = 1,073,741,824 bytes and 1TB = 240 bytes = 1,099,511,627,776 bytes and therefore shows less storage capacity. Available storage capacity (including examples of various media files) will vary based on file size, formatting, settings, software and operating system, and/or pre-installed software applications, or media content. Actual formatted capacity may vary.