An "operator" is a design pattern for implementing a control system in software.  The operator approach was originally popularized by the Kubernetes® container orchestration system.  Since the KumoScale™ control plane is built on a Kubernetes platform, it is natural that the key control constructs are expressed in a Kubernetes native way.

Imperative vs. Declarative

To understand the elegance of the operator pattern, it's necessary to first emphasize the distinction between "imperative" and "declarative" and styles of expressing a user's intent.  Imperative programming consists of a list of commands, each of which directly change the state of the system, and the sequence of these commands is the direct embodiment of the algorithm.  An everyday example of actions specified in an imperative way is driving directions.  "Drive forward one kilometer, then turn left" is an imperative statement, and a list of such statements executed in order will progress the system state (position) from the start to the desired destination.

In this context, imperative expression can be seen to have two weaknesses:  First, the directions are valid only for one starting point.  Even for the same destination, a set of directions are useless if the current position is either unknown or different from the one they were written for.  Secondly, this method assumes that the user of the system knows the best route (algorithm) to accomplish the goal, and is aware of any transient circumstances (closed roads, traffic) that might change that.  The system executes imperative statements faithfully, whether they are in fact advisable or not.

By contrast, a declarative system is analogous to GPS navigation:  With a GPS system, the user supplies only the destination, and the control system itself contains the ability to determine current location, and the logic to find a safe and efficient path to to that destination.  The user need not know the current location (or system state), nor the best sequence of steps to reach the destination.  This makes the statement of the desired final state both independent of the current state, and resilient to unexpected changes in the environment that may occur along the way.

Components of an Operator

An operator, in a Kubernetes and in KumoScale environment, is a software construct which implements a declarative control system.  An operator consists of two key parts:

Custom Resource Definition (CRD)

A CRD is simply a template for an entry in a database used to keep track of system state.  A "resource" can be an atomic object (e.g. a Kubernetes pod, or a KumoScale SSD) or a composite object (a Kubernetes deployment containing many pods, a KumoScale storage node containing many SSDs).  Each entry is called a "custom resource", and represents one such object.

The salient aspect of a CRD is that for each controllable aspect of the resource, it contains two descriptors.  One, the "current state," is kept up to date by the system, and as its name implies, represents the current state of that variable for that object.  "Desired state" is writable by the user, and is used to express the user's intent.  "Current state" and "desired state" correspond to current location and destination in the GPS example.


The controller portion of an object is a software routine which embeds the business logic associated with changing the state of the system.  In the GPS example, the controller contains the street map, plus a navigation algorithm.   Whenever the current state of a CR does not match the desired state, the corresponding controller is triggered, and takes the actions necessary to bring them into alignment. So, the process of changing the configuration of KumoScale is largely accomplished by editing the "desired state" portion of various system objects.

KumoScale Operator-controlled Resources

The following table lists some of the KumoScale custom resources and their constituent configuration variables, which are supervised by controllers. This list is not exhaustive, and will grow over time as additional configuration resources are brought under operator control.


Management Cluster
Master CRD  
  numberOfMasters Number of master nodes in the cluster (should be an odd number)
  affinity/antiAffinity Controls which storage nodes can become masters based on topology constraints.
  Virtual IP Service IP at which the management cluster can be reached
License CRD  
  License key  
External Services CRD  
Telemetry CRD  
  name Name for the telemetry configuration
  telemetryTSDBType Which specific time series database is in use (e.g. Graphite®, Prometheus)
  transportType, ip, port Transport protocol (TCP or UDP), database IP address & port number
  pushState, intervalMin Whether the telemetry framework should spontaneously "push" data at some rate, or the telemetry server will poll for ("pull") the data.  And, time interval between telemetry push events (N/A for pull-mode systems.)
Syslog CRD  
  url, ip, port  
  tls, certFileName  
Storage Node CRD  
  initMgmtIp Initial management IP
  timeSettings Time zone, ntpServer
  portals Name, IP:port, and transport time for one or more logical network interfaces
  team, vlan Network portal teaming & vlan configuration
  topology Location label strings (rack, zone, region) that enable specification of topology-based constraints on the placement of volumes.