# Clustering configuration The following configuration items define how Transformer participates in a cluster of Transformers. **📌 NOTE**\ Transformer’s clustering configuration must be kept in a separate configuration file called `cluster.conf.` If you’re using the [Deployment Framework](../deployment-framework/index.md), put changes and additions to this configuration in `/global_config/overrides/servers/Transformer/etc/cluster.conf` where `` is the Deployment Framework’s topmost directory. **❗ IMPORTANT**\ The clustering configuration must be the same for every Transformer in the cluster, in order for the cluster to operate correctly, except (where used) the configuration in the file defined by [`primary-config-file`](#primary-config-file). **📌 NOTE**\ "this Transformer" means the Transformer for which you are defining the configuration. "this node" means the cluster node that this Transformer represents. ## add-cluster-node `add-cluster-node` identifies a Transformer (a "node") in the cluster. **❗ IMPORTANT**\ Ensure that every Transformer in the cluster has the same set of `add-cluster-node` configuration items, specified in the same order. **💡 TIP**\ For more about how to use `add-cluster-node`, see [Clustering Transformers without the Caplin Deployment Framework](transformer-set-up-a-cluster-of-transformers.md#clustering-transformers-without-the-caplin-deployment-framework) in How can I.. [Set up a cluster of Transformers](transformer-set-up-a-cluster-of-transformers.md). **Syntax:** ``` add-cluster-node xref:cluster-addr[cluster-addr] [string] xref:cluster-port[cluster-port] [string] end-cluster-node ``` | Option | Type | Default | Description | | --- | --- | --- | --- | | `cluster-addr` | string | `127.0.0.1` | Network interface address or hostname of the cluster node. In the [Caplin Platform Deployment Framework](../deployment-framework/index.md), the configuration macros `TRANSFORMER${THIS_LEG}_HOST` and `TRANSFORMER${OTHER_LEG}_HOST` specify the `cluster-addr` values. (Clustering is used in the Deployment Framework to support failover capability.) You normally use the [`./dfw hosts`](../deployment-framework/cdf-the-deployment-framework-command-utility-dfw.md#hosts) command to set up the hostnames in these macros. For more about this, see [Configuring server hostnames](../deployment-framework/cdf-set-up-server-failover-capability.md#configuring-server-hostnames) in How can I... [Set up server failover capability](../deployment-framework/cdf-set-up-server-failover-capability.md), and [Change server hostnames](../deployment-framework/cdf-change-server-specific-configuration.md#change-server-hostnames) in How can I ... [Change server-specific configuration](../deployment-framework/cdf-change-server-specific-configuration.md). Also see Deployment Framework [Configuration macros and items](../deployment-framework/cdf-configuration-macros-and-items.md). | | `cluster-port` | string | +[none]+ | Network port number of the cluster node. This port is used for cluster-related communication between the Transformer and the other Transformers in the cluster. In the [Caplin Platform Deployment Framework](../deployment-framework/index.md), the configuration macros `TRANSFORMER${THIS_LEG}_CLUSTER_PORT` and `TRANSFORMER${OTHER_LEG}_CLUSTER_PORT` specify the `cluster-port` values. For more about this, see [Clustering Transformers in the Caplin Deployment Framework](transformer-set-up-a-cluster-of-transformers.md#clustering-transformers-in-the-caplin-deployment-framework) in How can I ... [Set up a cluster of Transformers](transformer-set-up-a-cluster-of-transformers.md). Also see Deployment Framework [Configuration macros and items](../deployment-framework/cdf-configuration-macros-and-items.md). | ## always-auto-distribute `always-auto-distribute` specifies a space-separated list of subject names for which this Transformer automatically distributes updates to all other Transformers in the cluster. A subject name can be a Linux shell wildcard pattern, in which case updates for all subject names that match the pattern are forwarded. The pattern can contain any of the following wildcards in an appropriate combination: * `*` (matches zero, one or many characters) * `?` (matches just one character) * `[ ]` (matches any of the characters inside the brackets) **⚠️ WARNING**\ To prevent updates for a subject from being repeatedly "bounced" between Transformers, make sure that none of the subjects specified in the `always-auto-distribute` list for a Transformer (either explicitly specified or selected through a wildcard pattern) are also selected by the `always-auto-distribute` lists of any other Transformers in the cluster. Also see [`always-auto-distribute-ignore`](#always-auto-distribute-ignore) **Syntax:** `always-auto-distribute ... ` **Type:** array of strings **Default value:** (none, but this configuration item is optional) ## always-auto-distribute-ignore `always-auto-distribute-ignore` specifies a space-separated list of subject names that are excluded from the patterns specified in `[always-auto-distribute](#always-auto-distribute).` Updates are for subjects specified in `always-auto-distribute-ignore` are therefore not automatically distributed to any other Transformers in the cluster. A subject name in the list can be a Linux shell wildcard pattern, as specified in [`always-auto-distribute`](#always-auto-distribute) **Syntax:** `always-auto-distribute-ignore ... ` **Type:** array of strings **Default value:** (none, but this configuration item is optional) **Example:** ``` always-auto-distribute /I/*.L always-auto-distribute-ignore /I/VOD.L ``` In this example, all updates whose subjects have names of the form `/I/.L` are automatically distributed to the other Transformers in the cluster, except updates for the subject `/I/VOD.L` ## auto-distribute `auto-distribute` specifies a space-separated list of subject names for which all updates are automatically distributed to all other Transformers in the cluster, _provided this Transformer is the primary node in the cluster_. A subject name in the list can be a Linux shell wildcard pattern, as specified in [`always-auto-distribute`](#always-auto-distribute) Also see [`auto-distribute-ignore`](#auto-distribute-ignore) **Syntax:** `auto-distribute ... ` **Type:** array of strings **Default value:** (none, but this configuration item is optional) ## auto-distribute-ignore `auto-distribute-ignore` specifies a space-separated list of subject names that are excluded from the patterns specified in [`auto-distribute`](#auto-distribute) and for which updates are therefore not automatically distributed to any other Transformers in the cluster when this Transformer is the primary node of the cluster. A subject name in the list can be a Linux shell wildcard pattern, as specified in [`always-auto-distribute`](#always-auto-distribute) **Syntax:** `auto-distribute-ignore ... ` **Type:** array of strings **Default value:** (none, but this configuration item is optional) **Example:** ``` auto-distribute /I/*.L auto-distribute-ignore /I/VOD.L ``` In this example, all updates whose subjects have names of the form `/I/.L` are automatically distributed to the other Transformers in the cluster when this Transformer is the primary node of the cluster, except updates for the subject `/I/VOD.L` ## check-period `check-period` specifies the interval in seconds at which the status of the cluster’s primary node is checked. The status is exposed in the Transformer’s [JMX monitoring](../datasource/datasource-datasource-monitoring-and-management.md) interface. Also, at the end of every `check-period` this Transformer marks any auto-distributed data (see [`auto-distribute`](#auto-distribute)) that has changed since the last `check-period`, so that the updated data is saved to the object cache file at the end of the current [`write-cache-period`](transformer-memory-and-cache-configuration.md#write-cache-period). (The data’s only saved if this Transformer is the primary cluster node, or if it’s a secondary node and [`cluster-nosave-on-secondary`](#cluster-nosave-on-secondary) is `FALSE`.) **Syntax:** `check-period ` **Type:** float **Default value:** `1.0` second ## cluster-connect-timeout `cluster-connect-timeout` specifies this Transformer’s timeout on an attempt to connect to another Transformer in the cluster; for example, when this Transformer starts up. When the timeout expires, this Transformer waits one second and then attempts to reconnect. After ten attempts at reconnecting, the Transformer doubles the wait time before trying again. It doubles the delay every ten attempts up to a maximum delay of 32 seconds. after which it stops attempting to connect to the other Transformer. **Syntax:** `cluster-connect-timeout ` **Type:** float **Default value:** `10 seconds` ## cluster-index `cluster-index` specifies the index number of this Transformer cluster node. The index number uniquely identifies this Transformer within the cluster. It states which of the [add-cluster-node](#add-cluster-node) items this node represents. Index numbers start at 0 and correspond to the order of the [`add-cluster-node`](#add-cluster-node) items in the configuration file. **📌 NOTE**\ In the [Caplin Platform Deployment Framework](../deployment-framework/index.md), you use a configuration macro `TRANSFORMER_CLUSTER_NODE_INDEX` to specify Transformer’s `cluster-index`. See [Clustering Transformers in the Caplin Deployment Framework](transformer-set-up-a-cluster-of-transformers.md#clustering-transformers-in-the-caplin-deployment-framework) in How can I ... [Set up a cluster of Transformers](transformer-set-up-a-cluster-of-transformers.md), and Deployment Framework [Configuration macros and items](../deployment-framework/cdf-configuration-macros-and-items.md). **Syntax:** `cluster-index ` **Type:** integer **Default value:** `-1` (This transformer is not part of a cluster) ## cluster-nosave-on-secondary `cluster-nosave-on-secondary` specifies when `TRUE` that if this Transformer is a secondary cluster node (that is, not the primary node), it _won’t_ save updates for it’s auto-distributed data (see [`auto-distribute`](#auto-distribute)) to the object cache file (see [`check-period`](#check-period) and [`write-cache-period`](transformer-memory-and-cache-configuration.md#write-cache-period)). If this Transformer is the primary cluster node, it _always_ saves these updates to the object cache file, regardless of the setting of this configuration item. **Syntax:** `cluster-nosave-on-secondary ` **Type:** boolean **Default value:** `FALSE` (this Transformer’s auto-distributed data is always saved to the object cache file) ## cluster-packet-logfile `cluster-packet-logfile` specifies the name of the log file in which message packets passed between this Transformer and other nodes in the Transformer cluster are recorded. The filename can contain the parameter `%a`, which is replaced at run-time by this Transformer’s DataSource [`application-name`](../datasource/datasource-run-time-environment-configuration.md#application-name). The log file is located in Transformer’s `var` directory; in the [Deployment Framework](../deployment-framework/index.md), the full directory path is `/servers/Transformer/var` where `` is the Deployment Framework’s topmost directory. The log file’s (binary) format is the same as that of the standard DataSource packet log (see [`datasrc-pkt-log`](../datasource/datasource-datasource-peers-configuration-part-2.md#datasrc-pkt-log) in [DataSource peers configuration part 2](../datasource/datasource-datasource-peers-configuration-part-2.md)), so you need to use the [`logcat` utility](../datasource/datasource-view-packet-logs-using-logcat.md) to read it. **❗ IMPORTANT**\ Only configure `cluster-packet-logfile` when you need to troubleshoot your Transformer cluster. Cluster packet logs can quite quickly become very large, so you should only enable logging of this information for debugging purposes. **Syntax:** `cluster-packet-logfile ` **Type:** string **Default value:** (none, but this configuration item is optional - if it’s not present then cluster packets aren’t logged) ## connect-sync `connect-sync` specifies a space-separated list of subject names for which a snapshot of the each subject’s current data held by this Transformer is sent on to another Transformer in the cluster when that Transformer connects to this one. The data is only sent to a connecting node when either this Transformer is the primary node of the cluster, or [`connect-sync-always`](#connect-sync-always) is specified as `TRUE`. A subject name in the list can be a Linux shell wildcard pattern, as specified in [`always-auto-distribute`](#always-auto-distribute) Also see [`connect-sync-ignore`](#connect-sync-ignore) **Syntax:** `connect-sync ... ` **Type:** array of strings **Default value:** (none, but this configuration item is optional) ## connect-sync-always `connect-sync-always` specifies that this Transformer must always send to another Transformer in the cluster that connects to it a snapshot of the current data for each subject specified by [`connect-sync`](#connect-sync) (and filtered by [`connect-sync-ignore`](#connect-sync-ignore)), _even if this Transformer is not the primary node in the cluster_. **Syntax:** `connect-sync-always ` **Type:** boolean **Default value:** `FALSE` (data is only sent to a connecting Transformer if this Transformer is the primary node of the cluster) ## connect-sync-ignore `connect-sync-ignore` specifies a space-separated list of subject names that are excluded from the patterns specified in [`connect-sync`](#connect-sync) Therefore, a snapshot of the current data for these subjects held by this Transformer is _not_ sent on to any other Transformer in the cluster when that Transformer that connects to this one. A subject name in the list can be a Linux shell wildcard pattern, as specified in [`always-auto-distribute`](#always-auto-distribute) **Syntax:** `connect-sync-ignore ... ` **Type:** array of strings **Default value:** (none, but this configuration item is optional) **Example:** ``` connect-sync /I/*.L connect-sync-ignore /I/VOD.L ``` In this example, the current data for all subjects with names of the form `/I/.L` is automatically distributed to another connecting Transformer in the cluster, except the data for the subject `/I/VOD.L` ## force-primary `force-primary` specifies that this Transformer is forced to be the default primary node in the cluster. This configuration item only applies when [`primary-method`](#primary-method) has the value `forced`. **❗ IMPORTANT**\ Only set this configuration item in the file defined by the configuration item [`primary-config-file`](#primary-config-file). Once set to `TRUE`, the value of `force-primary` can subsequently be changed to `FALSE` if a [JMX monitoring](../datasource/datasource-datasource-monitoring-and-management.md) command is used to force a different Transformer in the cluster to become the primary; for more about this see [`primary-config-file`](#primary-config-file). **Syntax:** `force-primary ` **Type:** boolean **Default value:** `FALSE` (the cluster’s primary node is either selected at run time by the method specified by [`primary-method`](#primary-method), or, when `primary-method` is set to `forced`, by the configuration for some other Transformer in the cluster having `force-primary` set to `TRUE`) ## heartbeat-slack-time `heartbeat-slack-time` When this Transformer doesn’t receive an expected cluster heartbeat (see [`heartbeat-time`](#heartbeat-time)) from another Transformer in the cluster, it waits `heartbeat-slack-time` seconds before disconnecting from the cluster and trying to reconnect to it. When all the Transformers in the cluster have initially connected, they compare their `heartbeat-slack-time` values and use the lowest. **Syntax:** `heartbeat-slack-time ` **Type:** integer **Default value:** `5` seconds ## heartbeat-time `heartbeat-time` specifies the time in seconds between each cluster heartbeat. The Transformers forming a cluster exchange heartbeat messages at regular intervals, allowing each of them to check that all the others are still present. If the cluster heartbeat times out, this Transformer waits [`heartbeat-slack-time`](#heartbeat-slack-time) and then disconnects from the cluster and tries to reconnect to it. When all the Transformers in the cluster have initially connected, they compare their `heartbeat-time` values and use the lowest. **📌 NOTE**\ In the [Caplin Platform Deployment Framework](../deployment-framework/index.md), you use a configuration macro `TRANSFORMER_CLUSTER_HEARTBEAT_TIME` to specify Transformer’s `cluster-heartbeat-time`. See [Clustering Transformers in the Caplin Deployment Framework](transformer-set-up-a-cluster-of-transformers.md#clustering-transformers-in-the-caplin-deployment-framework) in How can I ... [Set up a cluster of Transformers](transformer-set-up-a-cluster-of-transformers.md), and Deployment Framework [Configuration macros and items](../deployment-framework/cdf-configuration-macros-and-items.md). **Syntax:** `heartbeat-time ` **Type:** integer **Default value:** `15` seconds The minimum allowed value is 1 second. ## logfile `logfile` specifies the name of the file to which cluster-related events are logged. The filename can contain the parameters `%a` and `%h` At run time, `%a` is replaced by the Transformer’s application name (see the DataSource configuration item [`application-name`](../datasource/datasource-run-time-environment-configuration.md#application-name)), and `%h` is replaced by the host name of the machine on which the Transformer is running. **Syntax:** `logfile ` **Type:** string **Default value:** `cluster.log` ## log-level `log-level` specifies the severity of the errors and events that this Transformer reports to the cluster-related events log file (see [`logfile`](#logfile)). **Syntax:** `log-level ` **Type:** string **Default value:** `INFO` **Values accepted:** | log-level-name | DESCRIPTION | | --- | --- | | `DEBUG` | Reports all errors and events. | | `INFO` | Reports events and information regarding normal operation, and all errors included in the `WARN`, `NOTIFY`, `ERROR` and `CRIT` log levels. | | `WARN` | Reports minor errors and all errors included in the `NOTIFY`, `ERROR` and `CRIT` log levels. | | `NOTIFY` | Reports errors regarding data corruptions and all errors included in the `ERROR` and `CRIT` log levels. | | `ERROR` | Reports serious errors and all errors included in the `CRIT` log level. | | `CRIT` | Reports critical errors that prevent this Transformer from participating in the cluster. | ## primary-config-file `primary-config-file` specifies the name, and optionally the directory path of the Transformer configuration file that records that this Transformer has been forced to become the primary node. When this Transformer becomes the primary node in the cluster, it sends a message to all the other Transformers in the cluster to inform them of this. If you _force_ this Transformer to become the primary node by sending it a [JMX monitoring](../datasource/datasource-datasource-monitoring-and-management.md) command (see the `forced` method in [`primary-method`](#primary-method)), the Transformer creates a file defined by `primary-config-file` that contains the following configuration: ``` xref:force-primary[force-primary] TRUE ``` When this Transformer is restarted, this configuration file ensures that the Transformer resumes the role of primary node. When another Transformer in the cluster receives this Transformer’s "I am now primary" message, it marks itself as a secondary node and clears out its own `primary-config-file`. This procedure ensures that once the administrators of a Transformer cluster have established a particular node as the primary, when the cluster is restarted, that node will continue to be the primary one. You can manually configure one Transformer of the cluster to be the primary, but still allow the primary to subsequently be changed through JMX. To do this: 1. In all the Transformers of the cluster, set [`primary-method`](#primary-method) to the value `forced` 2. For the primary Transformer, create a configuration file with the name as defined by `primary-config-file` 3. Put the configuration line `[force-primary](#force-primary) TRUE` in the file. 4. Include the file in the Transformer’s `cluster.conf` file. (In the Deployment Framework, add the `include-file` statement to the `cluster.conf` file located in `/global_config/overrides/servers/Transformer/etc/`) 5. Repeat steps 2 and 4 only for all the secondary Transformers in the cluster (so these Transformers have empty ``primary-config-file``s). The `primary-config-file` filename can contain the parameters `%a` and `%h` At run time, `%a` is replaced by the Transformer’s application name (see the DataSource configuration item [`application-name`](../datasource/datasource-run-time-environment-configuration.md#application-name)), and `%h` is replaced by the host name of the machine on which the Transformer is running. The directory path can contain the parameter `%r`, which is replaced at run time by the root directory ([`application-root`](../datasource/datasource-run-time-environment-configuration.md#application-root)) under which the DataSource application runs, **Syntax:** `primary-config-file ` **Type:** string **Default value:** `%r/etc/cluster-primary.conf` ## primary-method `primary-method` specifies how to determine which Transformer ("node") in the cluster takes on the role of the primary. **Syntax:** `primary-method ` **Type:** string **Default value:** `order` **Values accepted:** | <method-name> | Meaning | | --- | --- | | `forced` | The primary node is defined either: * by a [JMX monitoring](../datasource/datasource-datasource-monitoring-and-management.md) command sent to this Transformer (for example from the [Caplin Management Console](../cmc/index.md)), in which case `[force-primary](#force-primary) TRUE` is recorded in the file defined by [`primary-config-file`](#primary-config-file) * through a `[force-primary](#force-primary) TRUE` setting in the [`primary-config-file`](#primary-config-file) configuration file for this Transformer. (Only one of the Transformers in the cluster can have `force-primary` set to `TRUE`) This setting ensures that once the administrators of a Transformer cluster have established a particular node as the primary, when the cluster is restarted, that node will continue to be the primary one. For more about how this works, see [`primary-config-file`](#primary-config-file) | | `order` | The primary node is the one with the lowest non-negative [`cluster-index`](#cluster-index) value in its configuration. | | `persisted` | Same as for `forced`. | | `uptime` | The node that has been running for the longest time becomes the primary. | **❗ IMPORTANT**\ Make sure that `primary-method` is set to the same value for every Transformer within the cluster. ## stable-time `stable-time` specifies the minimum time that this Transformer has to have been running in order to be considered for promotion to the cluster’s primary node. **❗ IMPORTANT**\ Before attempting to change `stable-time` from its default value, please seek advice from **Caplin Support**. **Syntax:** `stable-time ` **Type:** integer **Default value:** `5` seconds --- **See also:** * [Transformer clustering features and concepts](transformer-clustering.md) * How can I ... [Set up a cluster of Transformers](transformer-set-up-a-cluster-of-transformers.md) * How can I... [View packet logs using logcat](../datasource/datasource-view-packet-logs-using-logcat.md) (enables you to read cluster packet log files written to [`cluster-packet-logfile`](#cluster-packet-logfile)) * Reference: [Transformer Heartbeats configuration](transformer-heartbeats-configuration.md)