# Pipeline configuration These configuration items enable you to add a [Lua pipeline](transformer-transformer-modules.md#standard-modules) to Transformer and specify how Transformer is to use it. **📌 NOTE**\ Transformer’s pipeline configuration must be kept in a separate configuration file called `pipeline.conf`. If you’re using the [Deployment Framework](../deployment-framework/index.md), put changes and additions to this configuration in `/global_config/overrides/servers/Transformer/etc/pipeline.conf` where `` is the Deployment Framework’s topmost directory. ## add-pipeline `add-pipeline` specifies the configuration of a Lua pipeline that is to be loaded into Transformer. **Syntax:** ``` add-pipeline xref:id[id] [string] xref:pipeline-file[pipeline-file] [string] xref:add-info[add-info] [array of strings] xref:discard-func[discard-func] [string] xref:exclude-regex[exclude-regex] [array of strings] xref:flags[flags] [integer] xref:initialise-func[initialise-func] [string] xref:listener-regex[listener-regex] [array of strings] xref:log-flags[log-flags] [integer] xref:nodata-func[nodata-func] [string] xref:no-delay-load[no-delay-load] [boolean] xref:provider-exclude-regex[provider-exclude-regex] [array of strings] xref:provider-regex[provider-regex] [array of strings] xref:request-func[request-func] [string] xref:required-fields[required-fields] [array of strings] xref:status-func[status-func] [string] xref:type[type] [integer] xref:udp-command[udp-command] [array of strings] xref:udpsig-func[udpsig-func] [string] xref:update-func[update-func] [string] end-pipeline ``` | Option | Type | Default | Description | | --- | --- | --- | --- | | `add-info` | array of strings | `NULL` (no extra information added) | Specifies extra information that’s added to the pipeline module database (see the [Pipeline Database configuration](transformer-pipeline-database-configuration.md)). Each pair of strings in the supplied array is a key + value pair, where the key part constitutes a secondary key in the database. The secondary key + value pair is stored in the database under the primary key `/PIPELINE/` For example: `add-info foo bar` stores the value `"bar"` in the database under the primary key `/PIPELINE/` and the secondary key `"foo"`. | | `discard-func` | string | +[none]+ | The name of the Lua function to call when an update is received whose subject matches the [`provider-regex`](#provider-regex) pattern, and no modules or peers wish to receive updates for this subject. The signature of the function must be: `function (peer, subject)` The function must return `0` if the discard can’t be handled, or `1` if it can. This function is always called on the main thread. | | `exclude-regex` | array of strings | +[an empty list]+ | A space-separated list of one or more regular expressions used to match (select) the subject names of updates that are _not_ to be processed by this Lua pipeline. You can put multiple instances of this option in the `add-pipeline` specification. Also see [`listener-regex`](#listener-regex). The equivalent option for filtering out the subject names of _subscription requests and discard requests_ is [`provider-exclude-regex`](#provider-exclude-regex). | | `flags` | string or integer | `0` (no flags specified) | Flags that specify the behaviour of this Lua pipeline. You can specify a flag by name or numeric value: * `none` or `0`: No flags specified. * `manyprovider` or `1`: The pipeline can receive multiple requests and discards for the same [subject](../datasource/datasource-subjects-symbols-and-fields.md). * `initallatstart` or `2`: The pipeline’s initialisation function, as defined by the [`initialise-func`](#initialise-func) option, is to be called on all the pipeline’s threads as soon as the Transformer starts up and the pipeline has been loaded. This setting only works when the [`no-delay-load`](#no-delay-load) option is `TRUE`. * `clearupdate` or `4`: Updates coming into this pipeline will be cleared after processing, to prevent them from being processed by other pipelines or other Transformer modules. * `suppressupdate` or `8`: Incoming updates won’t be distributed to connected peers. * `accessory` or `16`: Subscriptions on this pipeline are accessory subscriptions. This means that the pipeline doesn’t itself subscribe to any of the subjects for which it listens for updates (see [`listener-regex`](#listener-regex)). Instead, it assumes that some other pipeline or Transformer module will subscribe to these subjects, and it just listens for any incoming updates relating to them. Use the `|` operator to specify compatible multiple flag values; for example: `flags manyprovider|initallatstart` | | `id` | string | +[automatically generated]+ | A unique identifier for the Lua pipeline. This is used by the [`add-info`](#add-info) option, and is also recorded against entries in the log file specified by the [`logfile`](#logfile) configuration item. If you don’t define an `id`, Transformer generates one internally when it loads the pipeline. Its name is `pipeline-`, where index is the ordinal number of this `add-pipeline` configuration item within the Transformer’s configuration, starting from zero. For example, the internal id of the twentieth `add-pipeline` item in the configuration is `pipeline-19`. | | `initialise-func` | string | +[none]+ | The name of the Lua function to call when the script named in [`pipeline-file`](#pipeline-file) is first loaded. The signature of the function must be: `function ()` The initialise function is called when the script is first loaded; it’s always called on the main thread. Subsequently, it may also be called multiple times, depending on the setting of the [Pipeline threads](transformer-pipeline-threads-configuration.md) configuration item [`num-threads`](transformer-pipeline-threads-configuration.md#num-threads) The script is normally loaded when a matching update is received, but if you also want the `initialise()` function to be called when the Transformer is started up, use the [`no-delay-load`](#no-delay-load) option. | | `listener-regex` | string array | [any subject matches, unless it’s excluded by [`exclude-regex`](#exclude-regex)] | A space-separated list of one or more regular expressions used to match (select) the subject names of updates that are to be passed to the Lua function specified in the [`update-func`](#update-func) option. This option’s also used to match the subject names of nodata messages that invoke the Lua function specified in the [`nodata-func`](#nodata-func) option. You can put multiple instances of this option in the `add-pipeline` specification. Also see [`exclude-regex`](#exclude-regex), [`update-func`](#update-func) and [`nodata-func`](#nodata-func). The equivalent option for selecting the subject names of _subscription requests and discard requests_ is [`provider-regex`](#provider-regex). | | `log-flags` | integer|string | `0` (no extra logging) | Flags that specify additional items that can be logged to the [`logfile`](#logfile) for this Lua pipeline. You can specify a flag by name or numeric value: * `none` or `0`: No additional logging. * `update` or `1`: Additionally log received updates. * `nodata` or `2`: Additionally log received nodata indications. * `filter` or `4`: Additionally log received filters. * `timer` or `8`: Additionally log timer interactions. * `all` or `-1`: All of `update`, `nodata`, `filter` and `timer`. You can only specify _one_ flag value. This means that you can either specify just one type of additional item to log, or you can specify `all` to include all of the additional items in the log. To specify additional logging for dynamically created pipelines, use [`pipeline-logflags`](#pipeline-logflags). | | `nodata-func` | string | +[none]+ | The name of the Lua function to call when a nodata message is received whose subject (symbol) matches one of the patterns specified by [`listener-regex`](#listener-regex) The signature of the function must be: `function (peer, symbolname, flags)` In the implementation of the function, don’t inspect the `flags` parameter, since it has no significance. This function can be called from any of the Lua pipeline’s worker threads (see [`num-threads`](transformer-pipeline-threads-configuration.md#num-threads) in the [Pipeline threads](transformer-pipeline-threads-configuration.md) configuration). | | `no-delay-load` | boolean | `FALSE` | When `TRUE`, this option forces the Lua script named in [`pipeline-file`](#pipeline-file) to be loaded as soon as the Transformer starts up, rather than being loaded only when first required. | | `pipeline-file` | string | +[none]+ | The name of the file containing the code for the script of the Lua pipeline that’s to be loaded. For example: `pipeline-file myscript.lua` | | `provider-exclude-regex` | array of strings | +[an empty list]+ | A space-separated list of one or more regular expressions used to match (select) the subject names of of subscription requests and discard requests that are _not_ to be processed by this Lua pipeline. You can put multiple instances of this option in the `add-pipeline` specification. Also see [`provider-regex`](#provider-regex). The equivalent option for filtering out _updates_ is [`exclude-regex`](#exclude-regex). | | `provider-regex` | array of strings | [any subject matches, unless it’s excluded by [`provider-exclude-regex`](#provider-exclude-regex)] | A space-separated list of one or more regular expressions used to match (select) the subject names of subscription requests and discard requests that are to be processed by this Lua pipeline. You can put multiple instances of this option in the `add-pipeline` specification. Also see [`provider-exclude-regex`](#provider-exclude-regex), [`request-func`](#request-func) and [`status-func`](#status-func). The equivalent option for selecting the subject names of _updates_ is [`listener-regex`](#listener-regex). | | `request-func` | string | +[none]+ | The name of the Lua function to call when a subscription request is received whose subject (symbol) matches one of the patterns specified by `[provider-regex](#provider-regex).` The signature of the function must be: `function (peer, symbolname)` The function must return `0` if the request can’t be handled, or `1` if it can. This function is always called on the main thread. | | `required-fields` | array of strings | +[an empty list]+ | A space-separated list of one or more field names. When the pipeline receives an update (filtered according to the settings of [`listener-regex`](#listener-regex) and [`exclude-regex`](#exclude-regex)), the update will only be processed if at least one of the fields listed in `required-felds` is present in the update. Each field name in the list must correspond to a field name in an [`add-field`](../datasource/datasource-field-definition-format.md#add-field) configuration item in the relevant `fields.conf` file. | | `status-func` | string | +[none]+ | The name of the Lua function to call when an update to the status of a subject is received, where the subject matches the [`provider-regex`](#provider-regex) pattern. The signature of the function must be: `function (feed, subject, status, code, status_message)` where: * `feed` identifies the DataSource peer the update came from. * `status` is the subject [status](../datasource/datasource-subject-status.md) (`OK`, `STALE` or `LIMITED`). * `code` is always 0 (not used). * `status-message` gives more detail about the subject status. This function is called on the pipeline thread for the subject being processed. | | `type` | integer | `0 (none)` | The type of this Lua pipeline, as a name or value: * `none` or `0`: This pipeline is not a default pipeline. * `default` or `1`: This pipeline is a default pipeline that will capture and process all updates not caught by other pipelines. Note that the default setting of this option is not `default`. | | `udp-command` | +[array of strings]+ | +[none]+ | A list of UDP commands that this Lua pipeline will respond to. When Transformer receives a UDP command that’s in this list, it calls the pipeline’s Lua function that’s specified by the [`udpsig-func`](#udpsig-func) option. Example: `udp-command source_up` | | `udpsig-func` | string | +[none]+ | The name of the Lua function to call when Transformer receives a UDP command. The signature of the function must be: `function (msg)` where `msg` is a string containing the UDP message, the first field of which is the UDP command. This function can be called from any of the Lua pipeline’s worker threads (see [`num-threads`](transformer-pipeline-threads-configuration.md#num-threads) in the [Pipeline threads](transformer-pipeline-threads-configuration.md) configuration). | | `update-func` | string | +[none]+ | The name of the Lua function to call when an update is received whose subject matches one of the patterns specified by [`listener-regex`](#listener-regex) The signature of the function must be: `function (update)` where the `update` parameter is of type `dsdata` (see the DSData Update Packets module in the _[Lua Pipeline Module API Documentation,opts="nofollow"](https://docs.caplin.com/developer/api/transformer_pipeline_sdk/6/)_) This function can be called from any of the Lua pipeline’s worker threads (see [`num-threads`](transformer-pipeline-threads-configuration.md#num-threads) in the [Pipeline threads](transformer-pipeline-threads-configuration.md) configuration). | ## logfile `logfile` specifies the filename of the pipeline activity log in which pipeline messages are recorded. All log messages generated using the Logging routines package of the Transformer (Lua) Pipeline Module API are also sent to this file (see the [_Lua Pipeline Module API Documentation_),opts="nofollow"](https://docs.caplin.com/developer/api/transformer_pipeline_sdk/6/). Pipeline database activity is also recorded in this log. (see [`database-log-options`](transformer-pipeline-database-configuration.md#database-log-options) in [Pipeline database configuration](transformer-pipeline-database-configuration.md).) The filename can contain the parameters `%a` and `%h` At run time, `%a` is replaced by the Transformer’s application name (see the DataSource configuration item [application-name](../datasource/datasource-run-time-environment-configuration.md#application-name)), and `%h` is replaced by the host name of the machine on which the Transformer is running. **Syntax:** `logfile ` **Type:** string **Default value:** `pipeline.log` ## log-level `log-level` specifies the severity of errors and events that are reported to the pipeline’s log file (see [`logfile`](#logfile)). **Syntax:** `log-level ` **Type:** string **Default value:** `INFO` **Values accepted:** | Log-level-name | Description | | --- | --- | | `DEBUG` | Reports all errors and events. | | `INFO` | Reports events and information regarding normal operation, and all events included in the `WARN`, `NOTIFY`, `ERROR` and `CRITICAL` debug levels. | | `WARN` | Reports minor errors and all events included in the `NOTIFY`, `ERROR` and `CRITICAL` log levels. | | `NOTIFY` | Reports errors and all events included in the `ERROR` and `CRITICAL` log levels. | | `ERROR` | Reports serious errors and all events included in the `CRITICAL` log level. | | `CRITICAL` | Reports critical errors that prevent the pipeline from running. | ## pipeline-logflags `pipeline-logflags` specifies flags that make dynamic Lua pipelines log additional items to the [`logfile`](#logfile). (A dynamic pipeline is a pipeline that’s created at run-time by another pipeline.) **Syntax:** `pipeline-logflags ...` **Type:** array of strings **Default value:** `none` (no additional logging for dynamic pipelines) **Values accepted:** | Flag name | Meaning | | --- | --- | | `none` | No additional logging | | `update` | Additionally log received updates | | `nodata` | Additionally log received nodata indications | | `filter` | Additionally log received filters | | `timer` | Additionally log timer interactions | | `all` | All of `update`, `nodata`, `filter` and `timer`. | **Example:** ``` pipeline-logflags update nodata ``` This configuration causes all dynamic pipelines to log both received updates and received nodata indications in addition to the standard log items. **💡 TIP**\ To specify additional logging for a non-dynamic pipelines, use the [`log-flags`](#log-flags) option of [`add-pipeline`](#add-pipeline). ## pipeline-paths `pipeline-paths` specifies a space-separated list of the directory paths where Transformer’s Lua pipeline scripts and libraries can be found. You can separate the scripts and libraries across directories for ease of maintenance. The directory path can contain the parameter `%r` At run time, `%r` is replaced by the root directory ([`application-root`](../datasource/datasource-run-time-environment-configuration.md#application-root)) under which the Transformer runs. **Syntax:** `pipeline-paths ` **Type:** array of strings **Default value:** `%r/lib/pipeline` (This is where the core Caplin-supplied packages reside.) ## Example Here’s some example pipeline configuration. ``` xref:add-pipeline[add-pipeline] # This is a default pipeline xref:type[type] default xref:pipeline-file[pipeline-file] mydefaultpipeline.lua xref:initialise-func[initialise-func] myinitfunc xref:listener-regex[listener-regex] ^/I/.*\.L xref:exclude-regex[exclude-regex] ^/I/VOD.L xref:update-func[update-func] myupdatehandler xref:nodata-func[nodata-func] mynodatahandler xref:provider-regex[provider-regex] ^/I/.*\.L xref:provider-exclude-regex[provider-exclude-regex] ^/I/VOD.L xref:request-func[request-func] myrequesthandler xref:status-func[status-func] mystatushandler end-pipeline ``` * The pipeline’s specified as a default pipeline, which means that it captures and processes all updates that aren’t caught by other pipelines. You have to explicitly declare that it’s a default pipeline (`[type](#type) default`), because otherwise the default configuration setting of `type none` would apply, which would mean the pipeline _wasn’t_ a default one. * Since there’s no [`id`](#id) option in the configuration, the pipeline’s identifier is automatically generated when the pipeline is loaded. * The pipeline script is loaded from the file `mydefaultpipeline.lua` in the (default) directory `%r/lib/pipeline` and it’s initialised by a call to the function `myinitfunc()`. * The `myupdatehandler()` function of the Lua script `mydefaultpipeline.lua` handles all updates whose subjects have the form `/I/.L` (specified by [`listener-regex`](#listener-regex)), _except_ the subject `/I/VOD.L/` (specified by [`exclude-regex`](#exclude-regex)). The function `mynodatahandler()` handles all nodata messages received for the same subjects. * The `myrequesthandler()` function handles all subscription requests for the same subjects, but in this case, the subject filters must be specified by [`provider-regex`](#provider-regex) and [`provider-exclude-regex`](#provider-exclude-regex). The `mystatushandler()` function handles all status change messages received for these subjects. --- **See also:** * Reference: [Pipeline threads configuration](transformer-pipeline-threads-configuration.md) * Reference: [Pipeline database configuration](transformer-pipeline-database-configuration.md) * Reference: [Transformer Pipeline SDK (API),opts="nofollow"](https://docs.caplin.com/developer/api/transformer_pipeline_sdk/6/) * How can I... [Write a Lua Pipeline for Transformer](transformer-write-a-lua-pipeline-for-transformer.md)