# Data services configuration These DataSource configuration items define a DataSource application’s [data services](datasource-data-services.md). When the DataSource application requests an object programmatically (such as subscribing to a subject), the data services system determines, through the configuration, which peers the request goes to instead of you having to specify this explicitly in the application’s code. It also prevents a request from going to any peer more than once. Of course, if your DataSource application is an [Integration Adapter](../platform-architecture/putting-data-into-the-platform.md) that only sends data to its peers, you don’t need to define any data services for it. **📌 NOTE**\ Data services can’t be configured in Java-based DataSource applications. **📌 NOTE**\ "_this DataSource application_" means the DataSource application for which you are defining the configuration. "_DataSource peer_" or "_peer_" means a DataSource application that this DataSource application communicates with. For some examples of how data services are configured, see the [data services features and concepts page](datasource-data-services.md). ## add-data-service `add-data-service` specifies a data service that this DataSource application can use.Through the [add-source-group](#add-source-group) option and the [add-priority](#add-priority) option within `add-source-group`, it allows you to specify multiple peers that provide data for the service, load balance the provision of data across multiple peers, and configure alternate peers that your DataSource application can failover to. For examples of these setups, see the [data services features and concepts page](datasource-data-services.md). **📌 NOTE**\ Also see the note below about [default behaviour](#data-service-default-behaviour). **Use in:** C **Syntax:** ``` add-data-service add-source-group add-priority remote-label [value] end-priority ... add-priority remote-label [value] end-priority end-source-group ... add-source-group ... end-source-group affinity [array of strings] disable-auto-status [boolean] discard-timeout [float] exclude-pattern [array of strings] include-pattern [array of strings] nodata-ignore-request-timeout [boolean] request-timeout [float] required-state [integer/string] service-name [string] service-type [integer/string] end-data-service ``` | Option | Type | Default | Description | | --- | --- | --- | --- | | `[add-source-group](#add-source-group)` | group item | See [Default behaviour](#data-service-default-behaviour). | The [add-source-group](#add-source-group) group item specifies, via the [add-priority](#add-priority) option, one or more DataSource peers that provide the data for the data service. * By using multiple [add-source-group](#add-source-group) items, you can specify multiple sources of data for the data service. * By using multiple [add-priority](#add-priority) options within an [add-source-group](#add-source-group) item, you can specify alternative sets of peers to failover to. For examples of these setups, see the [data services features and concepts page](datasource-data-services.md) | | `affinity` | array of strings | +[none]+ | **Deprecated** from version 6.2.6+ of Liberator and version 6.2.5+ of Transformer; use the [affinity](#affinity-2) option within the [add-source-group](#add-source-group) group item instead. | | `disable-auto-status` | boolean | `FALSE` | If you don’t want this DataSource application to send [subject status](datasource-subject-status.md) changes for this data service to subscribing peers, set `disable-auto-status` to `TRUE` You’d typically do this in a Transformer, to prevent it from sending subject status changes unnecessarily on to subscribing Liberators. | | `discard-timeout` | float | For DataSource applications other than Liberator: `-1.0` (no timeout) For Liberator only: Value of `active-discard-timeout` | Specifies the time in seconds for which this DataSource application holds on to the data for a subscribed subject once the last client application has unsubscribed from that data. After this time, the object is deleted from the DataSource application’s cache, and the DataSource application sends a discard instruction to the data service’s peer(s) to cancel the subscription. For Liberator only: * This option overrides the setting of the global time out for active objects, `active-discard-timeout`. * If an object obtained by this data service was specified through the Liberator’s [add-object](../liberator/liberator-object-configuration-part-1.md#add-object) configuration item, the [discard-timeout](../liberator/liberator-object-configuration-part-1.md#discard-timeout) option of the `add-object` (if any) overrides this `discard-timeout`. | | `exclude-pattern` | array of strings | +[none]+ | One or more regular expressions that determine subjects for which this data service must not provide data. This option is optional. Use it in conjunction with [include-pattern](#include-pattern). The options [include-pattern](#include-pattern) and `exclude_pattern` are applied in the order they are defined, and the first matching rule wins. So, define all your `exclude_pattern` options _before_ their associated [include-pattern](#include-pattern) options. .Using exclude-pattern to refine include-pattern Match all subjects beginning /FX/MAJOR, but exclude /FX/MAJOR/AUDUSD and /FX/MAJOR/AUDGBP: | ``` exclude-pattern ^/FX/MAJOR/AUDUSD ^/FX/MAJOR/AUDGBP ① include-pattern ^/FX/MAJOR/ ② ` <1> This data service does _not_ provide data for subjects /FX/MAJOR/AUDUSD and /FX/MAJOR/AUDGBP <2> This data service provides data for subjects beginning /FX/MAJOR/ Note that if the `exclude-pattern` was specified _after_ the `include-pattern` rule, then the rules would not exclude /FX/MAJOR/AUDUSD and /FX/MAJOR/AUDGBP. For either of those two subjects, the first matching rule (the winning rule) would be the `include-pattern`. | You can have multiple occurrences of `exclude-pattern` in the same `add-data-service` item, so the above example could be written as: ``` exclude-pattern ^/FX/MAJOR/AUDUSD exclude-pattern ^/FX/MAJOR/AUDGBP include-pattern ^/FX/MAJOR/ ``` | `include-pattern` | array of strings | `^/` a| One or more regular expressions that determine the subjects for which this data service provides data. For example: `include-pattern ^/FX/MAJOR/ ^/FX/MINOR/` specifies that this data service is to provide data for subjects that begin with `/FX/MAJOR/` or `/FX/MINOR/` You can have multiple occurences of `include-pattern` in the same `add-source-group` item, so the above example could be written as: ``` include-pattern ^/FX/MAJOR/ include-pattern ^/FX/MINOR/ ``` You should normally specify an `include-pattern` option, otherwise the data service will return updates for all subjects (that is, subjects starting with `/`) that the service’s DataSource peers can provide. Also see the [exclude-pattern](#exclude-pattern) option. | `nodata-ignore-request-timeout` | boolean | `FALSE` a| A DataSource peer that provides a data service can send back a "no data" response if there is no data available for a requested subject. The data service then discards any outstanding requests for the subject. The `nodata-ignore-request-timeout` option applies when multiple peers can supply requested data for a service. When this option is `TRUE`, if any peer providing the service sends back a "no data" response, this response is _immediately_ propagated from the DataSource API to the custom code of the DataSource application - the [request-timeout](datasource-datasource-peers-configuration-part-1.md#request-timeout)s (or [source-request-timeout](datasource-datasource-peers-configuration-part-2.md#source-request-timeout)s) for the remaining peers are ignored. When this option is `FALSE`, if a "no data" response is received from a peer providing the service, the data service continues wait for the data from any of the other peers, applying the [request-timeout](datasource-datasource-peers-configuration-part-1.md#request-timeout)s (or [source-request-timeout](datasource-datasource-peers-configuration-part-2.md#source-request-timeout)s), until either the data is returned by at least one peer, or all the peers have timed out or returned a "no data" response. | `request-timeout` | float a| Value of [service-request-timeout](#service-request-timeout) a| Specifies the timeout in seconds for all subscription requests sent to this data service. If the DataSource peers providing the service don’t respond within this time, the requested subject is assumed to not be available and the application sends a discard message for the subject to the peers. A value of `0` means no timeout is set and this DataSource application then waits indefinitely for this data service to respond to a request. Use this setting in Liberator if you want to allow all Liberator’s subscription requests for this data service to be [open subscriptions](../liberator/liberator-open-subscriptions.md) (see How can I... [Keep subscriptions open when the peer is down](../liberator/liberator-keep-subscriptions-open-when-the-peer-is-down.md)). `request-timeout` is the master timeout; it overrides the global [service-request-timeout](#service-request-timeout) configuration item, the [retry-time](#retry-time) option of [add-source-group](#add-source-group), and the timeouts on individual peer requests ([request-timeout](datasource-datasource-peers-configuration-part-1.md#request-timeout) option of [add-peer](datasource-datasource-peers-configuration-part-1.md#add-peer), and the global [source-request-timeout](datasource-datasource-peers-configuration-part-2.md#source-request-timeout) item). | `required-state` a| integer/ string | `0` (=`down`) a| This option only applies to [Liberator](../liberator/index.md). Specifies the state that the data service must be in when Liberator starts up, in order for Liberator to accept client connections. Possible values are: * `0` or `down` or `optional`: Liberator accepts client connections regardless of the state of the data service. * `1` or `limited`: Liberator accepts client connections only if the data service has status `1` (`limited`) or `2` (`ok` or `up`). * `2` or `ok` or `up`: Liberator accepts client connections only if the data service has status `2` (`ok` or `up`). The default value of `down` means that, if no `required-state` options are specified for any data services, Liberator accepts client connections at start up, regardless of the state of the services. Liberator checks service status against the `required-state` option only at start up. For example, if `required-state` is `ok` and the data service status is "ok" at start up, client connections are accepted. Then if the service status subsequently changes to "down", the Liberator will continue to accept client connections. | `service-name` | string | +[none]+ | The name of the data service. | `service-type` a| integer/ string | `3` (= `active|broadcast`) a| Specifies whether the DataSource peers that supply data for the service are of type active, broadcast, or both. Possible values are: * `active` or `1`: This data service is provided by active DataSource peers. An active DataSource keeps track of which [subjects](datasource-subjects-symbols-and-fields.md) have been [requested](datasource-subscriptions.md) and sends updates for those subjects only (see [Active subscription model](datasource-active-subscription-model.md)). * `broadcast` or `2`: This data service is provided by [broadcast](datasource-broadcasting-data.md) DataSource peers. * `active|broadcast` or `3`: This data service is provided by both active and broadcast peers. You can use this option to override the [local-type](datasource-datasource-peers-configuration-part-1.md#local-type) option of a peer’s [add-peer](datasource-datasource-peers-configuration-part-1.md#add-peer) configuration. For example, assume a peer that supplies data for a data service is configured as an active peer and also is configured to broadcast data (that is, its ``add-peer`’s `local-type` option is set to `broadcast|active`). If you don’t want your data service to receive the peer’s broadcast data, set the ``add-data-service`’s `service-type` option to `active`. ## add-source-group `add-source-group` specifies, via the [add-priority](#add-priority) option one or more DataSource peers that provide the data for the data service defined by [add-data-service](#add-data-service). By using multiple `add-source-group items`, you can specify multiple sources of data for the data service. By using multiple [add-priority](#add-priority) options within an `add-source-group item`, you can specify alternative sets of peers to failover to. If you specify more than one peer within an [add-priority](#add-priority) item, subscriptions are load-balanced across them. For examples of these setups, see the [data services features and concepts page](datasource-data-services.md). **Use in:** C **Syntax:** ``` add-data-service ... add-source-group xref:add-priority-option[add-priority] remote-label [value] end-priority ... add-priority remote-label [value] end-priority xref:affinity-2[affinity] [array of strings] xref:required[required] [boolean] xref:retry-time[retry-time] [float] end-source-group end-data-service ``` | Option | Type | Default | Description | | --- | --- | --- | --- | | `[add-priority](#add-priority)` | group item | See [Default behaviour](#data-service-default-behaviour). | Specifies one or more DataSource peers that this DataSource application can use to provide data for a data service. If you specify more than one peer in an `add-priority` option, [subscriptions](datasource-active-subscription-model.md) are load-balanced across those peers. You can also use multiple instances of `add-priority` to implement failover to alternative peers. (But, because the [remote-label](#remote-label) option of [add-priority](#add-priority) points to an [add-peer](datasource-datasource-peers-configuration-part-1.md#add-peer) configuration item, this DataSource application will first try to fail over to any alternate peers that you’ve defined in the [addr](datasource-datasource-peers-configuration-part-1.md#addr) option of the `add-peer` item.) For examples of load balancing and failover configurations, see the [data services features and concepts page](datasource-data-services.md). | | `affinity` | array of strings | +[none]+ | Enables source affinity for load-balanced adapters in the source group. Source affinity distributes channel subscriptions evenly across a load-balanced set of adapters while maintaining the integrity of user sessions. Some integration adapters are designed to serve data on one channel as a result of activity on another. For example, trading adapters are designed to write entries to a user’s blotter channel as a result of trading activity on the user’s trade channel. A load-balancer can inadvertently break the operation of such adapters by distributing the user’s channels to different adapter instances. Source affinity stops a user’s channels from being distributed to different adapters in a load-balanced set by establishing a strong affinity between the user and one adapter in the set. To ensure that two source groups, under different data services, each select an identical adapter for a user: * both source groups should be assigned the same affinity key (see below). * both source groups should have be defined with an identical arrangement of adapters: the same adapters, in the same priority groups and in the same order. **Usage**: `affinity [affinity_key] [regular_expression]` `affinity_key`: an arbitrary string identifying a set of user-to-adapter affinities. Source groups with identical affinity keys will select the same adapter instance for any of a user’s channels served by them. Source groups with identical affinity keys must have the same number and order of adapters. `regular_expression`: a regular expression used to extract the username from the mapped subject of a channel served by a source group. Enables the source affinity algorithm to identify the owner (source) of a channel. **Example**: `affinity trading-adapters /PRIVATE/([^-]+).*/` For more information, see the following links: * [Source affinity overview](datasource-data-services.md#sourceaffinity) * [Load-balance adapters with source affinity](datasource-load-balance-adapters-using-source-affinity.md) for detailed configuration instructions with examples | | `required` | boolean | `FALSE` | When set to `TRUE`, if one of the peers that provides the service (see add-priority option) is down, updates from this service are marked as stale and the status of the service is changed to stale. | | `retry-time` | float | `30.0` | After finding that none of the peers defined by `add-priority` options in the source group have responded to a request, this DataSource application waits `retry-time` seconds before reissuing the request to the group. The minimum value allowed is `1.0` seconds. The DataSource application issues the request to each of the peers in the source group in turn. Each request is timed out according to the setting of [request-timeout](datasource-datasource-peers-configuration-part-1.md#request-timeout) in the [add-peer](datasource-datasource-peers-configuration-part-1.md#add-peer) configuration item. If none of the peers in the source group replies, the DataSource application waits `retry-time` and then tries each connection again in turn. It repeats this sequence until the master timeout for the service, defined by [service-request-timeout](#service-request-timeout), or the [request-timeout](#request-timeout) option of [add-data-service](#add-data-service), expires. If the `request-timeout` out of the `add-data-service` is set to `-1`, the DataSource reissues the request indefinitely. | ## add-priority `add-priority` specifies one or more DataSource peers that this DataSource application can use to provide data for a data service. By using multiple `add-priority` options within an [add-source-group](#add-source-group) item, you can specify alternative sets of peers to failover to. If you specify more than one peer within an `add-priority` item, [subscriptions](datasource-active-subscription-model.md) are load-balanced across them; for an example of this, see the [data services features and concepts page](datasource-data-services.md). **Use in:** C **Syntax:** ``` add-data-service ... add-source-group ... add-priority xref:remote-label[remote-label] [array of strings] remote-label [array-of-strings] ... xref:label[label] [array-of-strings] label [array-of-strings] ... end-priority end-source-group end-data-service ``` | Option | Type | Default | Description | | --- | --- | --- | --- | | `label` | array of strings | See [Default behaviour](#data-service-default-behaviour). | **Deprecated.** Use `remote-label` instead. One or more labels identifying DataSource peers that supply data for this DataService. Each label must match the [label](datasource-datasource-peers-configuration-part-1.md#label) option of an [add-peer](datasource-datasource-peers-configuration-part-1.md#add-peer) configuration item that defines a DataSource peer. If you specify more than one peer, either through multiple values of a `label` option or by multiple `label` options, subscriptions to the referenced peers are load-balanced across the peers. Also see [Note 1](#note-1). | | `remote-label` | array of strings | See [Default behaviour](#data-service-default-behaviour). | One or more labels identifying DataSource peers that supply data for this DataService. Each label must match the [remote-label](datasource-datasource-peers-configuration-part-1.md#remote-label) option of an [add-peer](datasource-datasource-peers-configuration-part-1.md#add-peer) configuration item that defines a DataSource peer. If you specify more than one peer, either through multiple values of a `remote-label` option or by multiple `remote-label` options, subscriptions to the referenced peers are load-balanced across the peers. Also see [Note 1](#note-1) below, and the [example](datasource-datasource-peers-configuration-part-1.md#example-of-datasrc-local-label-and-remote-label) in [DataSource peers configuration (part 1)](datasource-datasource-peers-configuration-part-1.md). | **Note 1:** Platform blades, and Liberator and Transformer, that run under the [Caplin Platform Deployment Framework](../deployment-framework/index.md) use the deprecated [label](#label) option in their configuration, rather than [remote-label](#remote-label). This will change in a future release of the Framework. ## broadcast-cleanup-age `broadcast-cleanup-age` specifies how old in minutes broadcast objects provided by this DataSource application need to be before they’re removed from the DataSource application’s cache. Every [broadcast-cleanup-period](#broadcast-cleanup-period) minutes past [broadcast-cleanup-time](#broadcast-cleanup-time), the application checks the cache and deletes any broadcast objects that are older than `broadcast-cleanup-age`. This configuration item is ignored if [broadcast-cleanup-time](#broadcast-cleanup-time) isn’t set. **Use in:** C **Syntax:** `broadcast-cleanup-age ` **Type:** integer **Default value:** `-1` (no timeout set, so broadcast items are never removed from the cache) ## broadcast-cleanup-period `broadcast-cleanup-period` specifies how often in minutes, relative to [broadcast-cleanup-time](#broadcast-cleanup-time), broadcast objects provided by this DataSource application are removed from the DataSource application’s cache. Every `broadcast-cleanup-period` minutes past `broadcast-cleanup-time`, the application checks the cache and deletes any broadcast objects that are older than [broadcast-cleanup-age](#broadcast-cleanup-age). This configuration item is ignored if [broadcast-cleanup-time](#broadcast-cleanup-time) isn’t set. **Use in:** C **Syntax:** `broadcast-cleanup-period ` **Type:** integer **Default value:** `1440` minutes (= 24 hours; that is, clean up every day at the time set by [broadcast-cleanup-time](#broadcast-cleanup-time).) ## broadcast-cleanup-time `broadcast-cleanup-time` specifies when broadcast objects provided by this DataSource application are first removed from the DataSource application’s cache. This time is the number of minutes after midnight (local time) on the day the Liberator was started, or if it’s greater than or equal to 1440 (24 hours), it’s the number of minutes past midnight on a Sunday. At `broadcast-cleanup-time`, and subsequently every [broadcast-cleanup-period](#broadcast-cleanup-period) minutes past `broadcast-cleanup-time`, the application checks the cache and deletes any broadcast objects that are older than [broadcast-cleanup-age](#broadcast-cleanup-age). By default, the DataSource application’s cache is not cleaned up on a timed basis. **Use in:** C **Syntax:** `broadcast-cleanup-time ` **Default value:** `-1` (Don’t clean up the DataSource application’s cache on a timed basis, so [broadcast-cleanup-age](#broadcast-cleanup-age) and [broadcast-cleanup-period](#broadcast-cleanup-period) are ignored.) **💡 TIP**\ We recommend that you configure a daily cleanup of broadcast objects. For example, set `broadcast-cleanup-time` to `0` and [broadcast-cleanup-period](#broadcast-cleanup-period) to `1440` mins (= 24 hrs), so that broadcast objects are cleaned up at midnight every day. Use [broadcast-cleanup-age](#broadcast-cleanup-age) to control how old the broadcast objects should be before they’re actually removed from the cache. ## cleanup-stale-timeout `cleanup-stale-timeout` specifies when to clean up stale objects (such as subscriptions). When all the DataSource peers in a data service have been down for `cleanup-stale-timeout` seconds, all the objects that can no longer be updated by the service are deleted from this DataSource application’s cache. **Use in:** C **Syntax:** `cleanup-stale-timeout ` **Type:** float **Default value:** `-1.0` (no timeout) ## service-request-timeout `service-request-timeout` specifies the global request timeout in seconds for all data services. This is the time that this DataSource application waits for a data service to respond to a request (for example, a subscription request). If the DataSource peers providing the service don’t respond within this time, the requested object is assumed to not be available and the application sends a discard message for the object to the peers. A value of `0` or `-1` means no timeout is set and this DataSource application then waits indefinitely for a data service to respond to a request. Use this setting in Liberator if you want to allow all of the Liberator’s subscription requests to be [open subscriptions](../liberator/liberator-open-subscriptions.md) (see How can I... [Keep subscriptions open when the peer is down](../liberator/liberator-keep-subscriptions-open-when-the-peer-is-down.md)). `service-request-timeout` is a global setting applied to all data services. You can override this timeout for a particular data service by setting the [request-timeout](#request-timeout) option of [add-data-service](#add-data-service). **Use in:** C **Syntax:** `service-request-timeout ` **Type:** float **Default value:** `10.0` seconds ## Data service default behaviour If you don’t define a data service for your DataSource application, it’ll act as if you defined the following configuration and will therefore send each subscription request to all of its active DataSource peers at once. **Default data service configuration** ``` add-data-service service-name default required-state down include-pattern ^/ add-source-group required FALSE add-priority remote-label source1 end-priority end-source-group add-source-group required FALSE add-priority remote-label source2 end-priority end-source-group ... add-source-group required FALSE add-priority remote-label sourceN end-priority end-source-group end-data-service ``` --- **See also:** * Reference: [DataSource peers configuration](datasource-datasource-peers-configuration-part-1.md) * How can I... [Use timers in DataSource applications](datasource-use-timers-in-datasource-applications.md) * How can I... [Enable heartbeats between DataSource applications](datasource-enable-heartbeats-between-datasource-applications.md)