# Migrating to scalable data services Enable Discovery’s scalable data services to allow [data services](../datasource/datasource-data-services.md) to recognise new DataSource instances added to a deployment at runtime. ## Requirements The instructions on this page assume that you have already completed the first four stages in the migration path below: * [x] [Deploy Discovery server](discovery-deploying-discovery.md) * [x] [Connect DataSources to Discovery](discovery-connecting-datasources-to-discovery.md) * [x] [Migrate to Discovery licensing](discovery-licensing-migrating-components.md) * [x] [Migrate to peer discovery](discovery-migrating-peer-discovery.md) * [ ] **Migrate to scalable data services** ## Overview In a traditional data service, providing peers are defined by lists of static labels ([`remote-label`](../datasource/datasource-data-services-configuration.md#remote-label)). Any peer added at runtime that is not listed in the data service’s configuration is not recognised by the data service as a provider. In a _scalable_ data service, providing peers are defined by regular expression ([`remote-label-regex`](../datasource/datasource-data-services-configuration.md#remote-label-regex)). The regular expression is evaluated at runtime for each subscription request, and any peer available at runtime with a label that matches the regular expression is recognised by the data service as a provider. In a data service’s configuration, providing peers are divided into one or more source groups, each further divided by priority (primary, secondary, ...). ```plantuml hide empty members entity "Data service" as dataservice entity "Source group" as sourcegroup entity "Priority" as priority entity "Providing peer" as peer dataservice -right-|{ sourcegroup sourcegroup -right-|{ priority priority -right-|{ peer note bottom of peer Requests are load-balanced across peers. end note note bottom of priority Priorities are defined in descending order of priority; primary priority first. Requests are routed to the highest priority that has at least one available peer. end note note bottom of sourcegroup Requests are routed to each source group. end note ``` Scalable data services support many different arrangements of providing peers (see [Scalable data services](discovery-scalable-data-services.md) for examples), but the simplest arrangement, and the one most suitable for deployments to container-orchestrated environments, uses a single-priority source group: ```plantuml component liberator1 { database MyDataService [ ""add-data-service"" "" service-name MyDataService"" "" include-pattern ^/MYSUBJECT"" "" add-source-group"" "" add-priority"" "" remote-label-regex ^transformer[0-9]+"" "" end-priority"" "" end-source-group"" ""end-data-service"" ] } component "transformer1" as transformer1 component "transformer2" as transformer2 component "transformer//N//" as transformern MyDataService --> transformer1 MyDataService --> transformer2 MyDataService --> transformern ``` For the sake of clarity, the instructions on this page assume that you are migrating data services to a single-priority source group, as illustrated in the example above. This arrangement is not suitable for all use cases. Please review the examples illustrated in [Scalable data services](discovery-scalable-data-services.md), and contact Caplin Support for guidance. The instructions on this page also assume that your data services don’t require source affinity or service rebalancing. Both features are supported in scalable data services. See sections [Source affinity](discovery-scalable-data-services.md#source-affinity) and [Service rebalancing](discovery-scalable-data-services.md#service-rebalancing) on the [Scalable data services](discovery-scalable-data-services.md) page for details. To migrate a deployment to scalable data services, you may need to make changes to the core configuration of components. How much configuration you need to change, and where you need to change it, depends on whether your deployment’s configuration is under manual control or under the management of the Deployment Framework. * If your deployment’s configuration is under manual control, follow instructions in [Manually configured deployments](#manually-configured-deployments). * If your deployment’s configuration is managed by the Deployment Framework, follow the instructions in [Deployment Framework deployments](#deployment-framework-deployments) ## Manually configured deployments Follow the steps below: 1. Locate the `add-data-service` configuration items in your deployment: **List configuration files containing `__add-data-service__`** ``` $ grep -RIl --include='*.conf' 'add-data-service' ``` 2. Within each data service (`add-data-service`) in your deployment, convert each source group (`add-source-group`) to use a single priority (`add-priority`) containing one `remote-label-regex` configuration item. ``` add-data-service … add-source-group … add-priority remote-label-regex __pattern__ end-priority end-source-group … end-data-service ``` **Converting a data service with failover** Consider the fictional data service below: ``` add-data-service service-name MyDataService include-pattern ^/MYSUBJECT add-source-group add-priority remote-label myadapter-1 remote-label myadapter-2 remote-label myadapter-3 end-priority add-priority remote-label myadapter-4 remote-label myadapter-5 remote-label myadapter-6 end-priority end-source-group end-data-service ``` To specify providing peers by regular expression, make the following changes: ```diff add-data-service service-name MyDataService include-pattern ^/MYSUBJECT add-source-group - add-priority - remote-label myadapter-1 - remote-label myadapter-2 - remote-label myadapter-3 - end-priority - add-priority - remote-label myadapter-4 - remote-label myadapter-5 - remote-label myadapter-6 - end-priority + add-priority + remote-label-regex ^myadapter-[0-9]+ + end-priority end-source-group end-data-service ``` ## Deployment Framework deployments Follow the steps below: 1. If your deployment uses any of the Transformer services below, activate both the service’s standard blade _and_ its complementary Discovery blade: | Standard blade | Complementary blade | Minimum version | | --- | --- | --- | | AlertsService | AlertsServiceDiscovery | 7.1.0 | | ChartingService | ChartingServiceDiscovery | 7.1.1 | | RefinerService | RefinerServiceDiscovery | 7.1.4 | | WatchlistService | WatchlistServiceDiscovery | 7.1.1 | 2. If your deployment uses any of the Transformer built-in services below, activate both the service’s standard blade _and_ its complementary Discovery blade: | Standard blade | Complementary blade | Minimum version | | --- | --- | --- | | PersistenceServiceClient | PersistenceServiceClientDiscovery | 7.1.15 | 3. If any other Transformer service blades and adapter blades in your deployment have complementary configuration blades for Discovery, activate them now. 4. For all remaining blades in your deployment, follow the steps below: 1. Locate `add-data-service` configuration items in the blade’s configuration: **List configuration files containing `__add-data-service__`** ``` $ grep -RIl --include='*.conf' 'add-data-service' active_kits/__blade_name__ global_config/overrides/__blade_name__ ``` 2. Within each data service (`add-data-service`), convert each source group (`add-source-group`) to use a single priority (`add-priority`) containing one `remote-label-regex` configuration item. ``` add-data-service … add-source-group … add-priority remote-label-regex __pattern__ end-priority end-source-group … end-data-service ``` **Converting a data service with failover** Consider the fictional data service below: ``` add-data-service service-name MyDataService include-pattern ^/MYSUBJECT add-source-group add-priority remote-label myadapter-1 remote-label myadapter-2 remote-label myadapter-3 end-priority add-priority remote-label myadapter-4 remote-label myadapter-5 remote-label myadapter-6 end-priority end-source-group end-data-service ``` To specify providing peers by regular expression, make the following changes: ```diff add-data-service service-name MyDataService include-pattern ^/MYSUBJECT add-source-group - add-priority - remote-label myadapter-1 - remote-label myadapter-2 - remote-label myadapter-3 - end-priority - add-priority - remote-label myadapter-4 - remote-label myadapter-5 - remote-label myadapter-6 - end-priority + add-priority + remote-label-regex ^myadapter-[0-9]+ + end-priority end-source-group end-data-service ``` ## Next steps You’ve reached the end of the migration path. * [x] Deploy Discovery server * [x] Connect DataSources to Discovery * [x] Migrate to Discovery licensing * [x] Migrate to peer discovery * [x] Migrate to scalable data services