# Writing a Transformer Pipeline Module Transformer is a real-time data transformation engine. The data transformation services are implemented in Transformer Modules. The Transformer API is used to create custom [Transformer modules](../../caplin-platform/transformer/transformer-transformer-modules.md) (in C, Java, Lua or JavaScript) which act on data messages being issued to client applications. Transformer houses the modules and manages the data coming in and out of them. The following sections take you through how to write a Transformer pipeline module in the Lua scripting language. Following that, there’s an optional exercise in which you can [write a pipeline module in JavaScript](#optional-exercise-writing-a-javascript-pipeline-module). icon:tv[] Download the [presentations](https://docs.caplin.com/developer/tutorial-resources/platform-adv/presentation-Advanced_Platform_IntegrationSuite.zip) for this course. ## Writing a Lua Pipeline Module The Pricing Integration Adapter is connected to Transformer. However, Transformer contains no modules apart from the Refiner module, and is not altering the data being sent/received by the adapter in any way. In this tutorial you will implement a Transformer module using the pipeline module (Lua API). This module will calculate the spread between the Bid and Ask prices sent by the Pricing Adapter adding a "Spread" field to the record before sending this on to Liberator and the client applications. ### Create a new blade The first step is to create a new blade, directly in the Deployment Framework. Create a directory with the name of the blade (e.g. "SpreadLuaBlade") and the following subdirectories and files within it. (We will fill these configuration and script files as we go along.) This directory should be created inside the `kits` directory of the Deployment Framework. ![tutorial-adv-lua1](../../images/tutorial-adv-lua1.jpg) ### Blade Fields This module is a new backend component which needs to have a fields configuration. It needs to know about all the fields configured in the Pricing Integration Adapter, and additionally, a new "Spread" field which it will be appending to the price update records. Fill in the fields.conf file: ``` # - Latency chaining fields add-field LATENCY_TIMESTAMP -10301 add-field LTY_INIT_TS -10302 add-field LTY_LIST_EVENT -10303 add-field LTY_LIST_TS -10304 # - Add further fields below add-field Time -10001 add-field Last -10002 add-field InstrumentName -45028 0 0 add-field BidPrice -45029 0 0 add-field AskPrice -45030 0 0 add-field BidPriceId -45031 0 0 add-field AskPriceId -45032 0 0 add-field BidGFA -45033 0 0 add-field AskGFA -45034 0 0 add-field Timestamp -45035 0 0 add-field Spread -45036 0 0 ``` ### Liberator and Transformer Configuration The new blade must include configuration for the core components that it interacts with. This configuration should typically consist of: * Data service definitions for the Liberator that allow the Liberator to request the data provided by the blade via the pipeline. * Transformer pipeline component configuration for the Lua module. Fill the Liberator configuration in `SpreadLuaBlade/Liberator/etc/rttpd.conf`: ``` # Liberator configuration for data from the # SpreadLuaBlade Lua module blade. add-data-service service-name SpreadLuaBlade${THIS_LEG} include-pattern ^/FX add-source-group required true add-priority label transformer${THIS_LEG} end-priority if "${FAILOVER}" == "ENABLED" add-priority label transformer${OTHER_LEG} end-priority endif end-source-group end-data-service ``` This is fairly standard configuration. The specific points to note are: * The blade name is used in [`service-name`](../../caplin-platform/datasource/datasource-data-services-configuration.md#service-name) configuration options. * The macros `THIS_LEG` and `OTHER_LEG` are used to differentiate the configuration of primary and secondary failover legs. Fill the Transformer configuration in `SpreadLuaBlade/Transformer/etc/pipeline.conf`: ``` # Location of the Lua module blade's pipeline components pipeline-paths ${ccd}/pipeline add-pipeline id SpreadLuaBlade${THIS_LEG} pipeline-file SpreadLuaBlade.lua listener-regex ^/FX/.* flags suppressupdate update-func update end-pipeline ``` The specific points to note are: * The blade name is used as the pipeline [`id`](../../caplin-platform/transformer/transformer-pipeline-configuration.md#id), and in the Lua module name that is defined by the [`pipeline-file`](../../caplin-platform/transformer/transformer-pipeline-configuration.md#pipeline-file) configuration option. * The `ccd` macro defines the current directory path of the file in which it is referenced. So in this example, because the `ccd` reference is in the file `SpreadLuaBlade/Transformer/etc/pipeline.conf`, the macro defines the path `SpreadLuaBlade/Transformer/etc`. * The line `pipeline-paths ${ccd}/pipeline/` instructs the Transformer to look for the blade’s pipeline files in the directory `SpreadLuaBlade/Transformer/etc/pipeline/` * The `flags suppressupdate` means all messages caught by your pipeline are not sent automatically, so you will have to do it explicitly. ### Creating the Lua Module Script Place the following code into `SpreadLuaBlade/Transformer/etc/pipeline/SpreadLuaBlade.lua`: ```lua require("calc") function update(packet) local bidPrice = packet:getfield("BidPrice") local askPrice = packet:getfield("AskPrice") packet:setfield("Spread", askPrice - bidPrice) packet:send() end ``` The specific points to note are: * The `require("calc")` instruction imports the calculation API. * An `update` function is defined, which obtains the `BidPrice` and `AskPrice` fields from each incoming record whose subject begins with `/FX/` (the Transformer’s pipeline module configuration for the SpreadJSBlade controls which records are selected - see the line `listener-regex ^/FX/.*`). Note that the name of this function must match that defined in the [update-func](../../caplin-platform/transformer/transformer-pipeline-configuration.md#update-func) option in the pipeline module configuration. * `local bidPrice = packet:getfield("BidPrice")` retrieves the value of the BidPrice field. * `packet:setfield("Spread", askPrice - bidPrice)` adds a "Spread" field to the record and sets its value. * `packet:send()` sends the new response message on to Liberator. ### Try it out! Follow the steps below: 1. Activate the blade: `./dfw activate SpreadLuaBlade` 2. Restart the Deployment Framework components, and start up the mock server and your Pricing Integration Adapter. 3. Use Liberator Explorer to request the container "/CONTAINER/FX/MAJOR". You should see that the "Spread" field has now been added among the others and is calculated with each new price update: ![tutorial-adv-lua2](../../images/tutorial-adv-lua2.JPG) ## Optional exercise: Writing a JavaScript Pipeline Module Besides implementing pipeline modules in the Lua scripting language, you can also code them in JavaScript. To do this, follow the same steps as in [Writing a Lua Pipeline Module](#writing-a-lua-pipeline-module) above, but with the following differences: ### Creating the JS blade directory in the Deployment Framework The blade is called **SpreadJSBlade**, so create a directory called `SpreadJSBlade` in the `kits` directory of the Deployment Framework. Add the subdirectories and files shown in Writing a Lua Pipeline Module, but with the name of the pipeline script (code) file set to `SpreadJSBlade.js` ### Liberator and Transformer Configuration Fill the Liberator configuration in `SpreadJSBlade/Liberator/etc/rttpd.conf`: ``` # Liberator configuration for data from the # SpreadJSBlade JavaScript module blade. add-data-service service-name SpreadJSBlade${THIS_LEG} include-pattern ^/FX add-source-group required true add-priority label transformer${THIS_LEG} end-priority if "${FAILOVER}" == "ENABLED" add-priority label transformer${OTHER_LEG} end-priority endif end-source-group end-data-service ``` Fill the Transformer configuration in `SpreadJSBlade/Transformer/etc/pipeline.conf`: ``` # Location of the JavaScript module blade's pipeline components pipeline-paths ${ccd}/pipeline add-pipeline id SpreadJSBlade${THIS_LEG} pipeline-file SpreadJSBlade.js listener-regex ^/FX/.* flags suppressupdate update-func update end-pipeline ``` ### Creating the JavaScript Module Script Place the following code into `SpreadJSBlade/Transformer/etc/pipeline/SpreadJSBlade.js`: ```js var log = require("log"); function update(packet) { var subject = packet.getSubject(); var bidPrice = packet.getField("BidPrice").value; var askPrice = packet.getField("AskPrice").value; var spread = askPrice - bidPrice; log.log(log.INFO, "Spread for <" + subject + "> is " + spread + "\n"); packet.setField("Spread", spread); packet.send(); } ``` The specific points to note are: * `var log = require("log")` imports the JavaScript pipeline’s Logging API. * An `update` function is defined, which obtains the `BidPrice` and `AskPrice` fields from each incoming record whose subject begins with `/FX/` (the Transformer’s pipeline module configuration for the SpreadJSBlade controls which records are selected - see the line `listener-regex ^/FX/.*`). Note that the name of this function must match that defined in the [`update-func`](../../caplin-platform/transformer/transformer-pipeline-configuration.md#update-func) option in the pipeline module configuration. * After obtaining the subject name of the incoming record, the function obtains the `bidPrice` and `askPrice` fields from the record and uses them to calculate the spread. **📌 NOTE**\ In JavaScript, the expression `adkPrice - bidPrice` subtracts a floating-point value from a floating-point value and may result in precision errors. In production code you would use a library such as [bignumber.js](https://github.com/MikeMcl/bignumber.js/) for floating-point arithmetic. * The logging API (`log.log()`) is called to log the spread calculated for the subject. * The new `Spread` field is then added to the record (`packet.setField()`) and then `packet.send()` sends the new response message on to the Liberator.