 |
Qorus Integration Engine® Enterprise Edition 6.0.16_prod
|
|
Qorus Integration Engine® Enterprise Edition 6.0.16_prod
|
Back to the Developer's Guide Table of Contents
Qorus Integration Engine™ introduced YAML metadata files for Qorus integration object metadata separate from code files in version 4.1, deprecating support for old-style file formats with mixed code and metadata as special comments in the code.
The Qorus IDE does not support the old mixed code and metadata file formats for Qorus development.
As Python support was introduced in Qorus 5, for example, Python is only supported in new objects with separate YAML metadata files.
The old mixed code and metadata formats will continue to be supported for backwards compatibility, however it is not recommended to use this format for any new developments.
In Qore-language workflow definition files, the actual workflow definitions are made by assigning values to the workflows variable in the workflow description file. The top-level keys represent the version(s) of the workflow(s) being described, and the member data of each version key defines that version of the workflow. For a description of this variable, see Workflow Definitions.
The format_version variable in the workflow definition file must be set to "2.6" to declare compatibility with the workflow definition file format documented in this manual.
In order to describe the metadata to the loader (oload), variables with pre-defined names and a pre-defined structure must be defined in this file.
Qore Workflow Definition File Variables
| Variable | Description |
format_version | Set to "2.6" to declare compatibility with the oload format defined in this documentation for Qorus 6.0.16_prod |
workflows | Defines workflow metadata |
queues | Defines asynchronous queues |
groups | Defines interface groups |
events | Defines workflow synchronization events |
Any other variables, functions, and classes defined in the file will be ignored unless explicitly executed in the script (for example, when building the workflows or queues data structures).
The "workflows" variable in a Qore workflow definition file must declare a hash defining workflow metadata.
"workflows" global variable represent workflow names; these keys must also be assigned to a hash, where each key is a version of the workflow being defined. The value assigned to each version key must be a workflow description hash, representing the workflow metadata definition.The workflow description hash has the following keys (optional keys are enclosed in square brackets []):
"my_attach:1.0"); cannot be supplied if "class" is used"my_detach:1.0"); cannot be supplied if "class" is used"my_workflow_init:1.0"); cannot be supplied if "class" is used"my_error_handler:1.0"); cannot be supplied if "class" is used"my_error_func:1.0"); cannot be supplied if "class" is usedHere step and function designators (except for in the "functions" tag above) define a specific version of the appropriate object. This is done by specifying a string in the following format: "name:version" or by specifying a hash with name and version keys set to the appropriate values.
Workflow Definition Example
See Deprecated Workflow Parameters for details on how to define workflow in the workflow definition file, and see $OMQ_DIR/examples/user/TEST-WORKFLOWS for some example workflow definition files.
Queues provide the storage and delivery mechanism by which the results of executing an asynchronous event for an asynchronous step are delivered to the right step instance. The queues variable in the workflow definition file must be set to a hash defining queue metadata. The top-level key gives the queue names, and the member data under the queue is a hash with a the desc key, giving a description of the queue.
queues."queuename" = ("desc": "description string" ));If a workflow has at least one asynchronous step, a queue must be defined. A single queue can be used for any number of asynchronous steps, however the keys in a queue must be unique. See the step queue attribute for more information.
The following are examples of queue definitions:
In order to allow workflows to declare membership to Interface Groups, the group may be declared in the workflow definition file by assigning a value to the global groups variable.
groups."groupname" = ("desc": "description string" ));The following is an example of a group definition:
Workflow synchronization events are used to synchronize processing from multiple workflow orders; each workflow synchronization event has a key that will be bound to the workflow synchronization event step by calling QorusEventStepBase::bindEvent() / Qorus DataProvider API: Bind Workflow Synchronization Event, or QorusEventStepBase::bindEventUnposted() / Qorus DataProvider API: Bind Workflow Synchronization Event Unposted in the step's primary step code. The workflow synchronization event key is unique within it's workflow synchronization event type; as such, the event type can be compared to a queue name.
The workflow synchronization event type is referenced in workflow synchronization event steps in the eventtype tag.
The actual event type definition is made in the workflow definition file by assigning a value to the global events variable:
events."eventname" = ("desc": "description string" ));The following are examples of workflow synchronization event type definitions:
In this hash, descriptions for single character keys of custom status codes available for the workflow can be defined.
See
The workflow hash description consists of the step dependencies and other attributes.
The following diagram illustrates a subset of the attributes of a workflow.
Workflow Parameter Descriptions
| Parameter | Description |
| author | The author of the workflow |
| steps | This is where steps and step dependencies are defined |
| desc | A string giving the description for the workflow |
| workflow-modules | lists modules providing new step base classes for step classes |
| remote | flag that indicates if a workflow can run in a remote qwf process or not (see qorus-client.remote for how the default value is assigned by oload) |
| autostart | the number of workflow execution instances to start when the system starts |
| [sla_threshold] | the amount of time as an integer in seconds in which each workflow order should get a final status; if not present the default value of 30 minutes is assumed (see DefaultWorkflowSlaThreshold) |
| max_instances | the maximum number of workflow execution instances that can be started |
| classes | If defined, this list of strings designates the Qore classes that will be loaded into the workflow's program object from the CLASSES table when the workflow is cached (represented by the "Shared Classes" object in the diagram above). Note that only the Qorus class object name should be given as it appears in the database without any version number, the latest version of the class will be loaded into the workflow's Program object. |
| constants | If defined, this list designates the Qore constant definitions that will be loaded into the workflow's program object from the CONSTANTS table when the workflow is cached (represented by the "Shared Constants" object in the diagram above). Note that only the Qorus constant object name should be given as it appears in the database without any version number, the latest version of the constant will be loaded into the workflow's Program object. |
| functions | If defined, this list designates the Qore function definitions that will be loaded into the workflow's program object from the FUNCTIONS table when the workflow is cached (represented by the "Shared Functions" object in the diagram above). Note that only the Qorus function object name should be given as it appears in the database without any version number, the latest version of the function will be loaded into the workflow's Program object |
| mappers | If defined, this list designates the mappers that will be registered with the workflow so they are available with the UserApi::getMapper() call in the workflow's code; the mapper name must be given with an explicit version number in each string (ex: "my-mapper-1:1.0") |
| vmaps | If defined, this list designates the value maps to register with the workflow so they can be used with the UserApi::getValueMap() call in the workflow's code; no version number can be given in the value map names |
| attach | (only if class) is not defined) If defined, this string identifies a function that will be executed every time Qorus caches and starts working on ("attaches to") a particular workflow order data instance (represented by the "Attach Logic" object in the diagram above). The function's version may be given by appending a colon and the version identifier to the name as follows: "detach_function_name:1.0" |
| detach | (only if class) is not defined) If defined, this string identifies a function that will be executed every time Qorus stops working on ("detaches from") a particular workflow order data instance (represented by the "Detach Logic" object in the diagram above). The function's version may be given by appending a colon and the version identifier to the name as follows: "detach_function_name:1.0". |
| onetimeinit | (only if class) is not defined) If defined, this string identifies a function that will be executed once when each workflow execution instance is started (represented by the "Init Logic" object in the diagram above). The function's version may be given by appending a colon and the version identifier to the name as follows: "onetimeinit_function_name:1.0" |
| error_handler | (only if class) is not defined) If defined, this string identifies a function that will be executed every time an error is raised by the workflow (represented by the "Error Handler" object in the diagram above). The function's version may be given by appending a colon and the version identifier to the name as follows: "error_handler_function_name:1.0" |
| errorfunction | This string identifies a function that is called by Qorus when the workflow definition is loaded by oload (represented by the "Error Definitions" object in the diagram above). It should return a hash of error information. See Workflow Error Handling and Recovery for more information. The function's version may be given by appending a colon and the version identifier to the name as follows: "error_function_name:1.0" |
| options | The options key should be assigned to a hash defining valid workflow options |
| keylist | This key should be assigned to a list giving valid workflow keys for fast lookups of workflow order instance data |
| groups | list of workflow and service groups the workflow will belong to |
| statuses | a hash defining extended custom status descriptions. |
If the options key is defined in the workflow definition, the workflow will advertise and accept only the options defined under this key as valid options for the workflow (aside from system options that can be overridden at the workflow level, which are always accepted as workflow options). Valid options are given as a hash assigned to this key, where the hash keys are the option names, and the values assigned to the keys are the descriptions of the options.
Here is an example of am options declaration in a YAML workflow definition file:
Here is an example of am options declaration in a deprecated Qore workflow definition file:
Options set on workflows are persistent; the option values are written to the WORKFLOW_OPTIONS table.
The workflow definition option keylist defines one or more "order keys" that can be used to quickly look up workflow order data instances. Any number of workflow keys may be given in the workflow definition, in contrast to the single external_order_instanceid that can be given at the workflow order data instance object level and can be used for the same purpose (quickly looking up workflow order data instances from a external key).
Valid keys are defined as per the following examples:
Keys can be set and retrieved by the WorkflowApi::setOrderKeys() and WorkflowApi::getOrderKeys() methods (in Qore) or the WorkflowApi.setOrderKeys() and WorkflowApi.getOrderKeys() methods (in Java), respectively. Key metadata is saved in the WORKFLOW_KEYS table, and workflow order data instance keys are saved in the ORDER_INSTANCE_KEYS table. Note that a single key value can be saved for more than one workflow order data instance; the indexes on the ORDER_INSTANCE_KEYS only enforce that values may not be repeated for the same workflow_instanceid and key name.
The REST API provides methods to lookup workflow orders by workflow keys; see the following API for more information:
The workflow definition option groups defines one or more Interface Groups that the workflow is a member of. Interface groups can also be declared in the workflow file using the global groups variable (see Deprecated Interface Group Definition).
Interface group membership is declared as per the following examples:
The inter-step dependencies are defined in the steps key in the workflow definition. This section describes how to define step dependencies and simple step definitions. For details on how to define complex steps, see Step Definitions.
The basic format of this data structure is a list. At the top level, a list indicates sequential dependencies, as in the following examples:
In this example; the workflow EXAMPLE-WORKFLOW 1.0 is created. The step dependencies are linear: MyStep1 has no dependencies and therefore is the starting step, MyStep2 is dependent only on MyStep1, and MyTestStep3 is the final step in the workflow and is dependent only on MyStep2.
In this simple example, 3 steps will be created (or updated if the steps already exist) with the same names and version specifiers as the class names above and placed in simple linear dependency in the steps list.
"MyStep1" version "1.0", "MyStep2" version "1.0", and "MyStep3" version "1.0" must also be defined; See array steps, subworkflow steps, asynchronous steps, validation code, function definition file.It is possible to create more complex step dependencies. For example, if one of the entries in the top-level steps list is itself a list, then all entries in the sublist have the same dependency and therefore will be executed in parallel threads. Furthermore, of one of the entries in the sublist is also a list, then they will be assigned sequential dependencies (i.e.: be executed sequentially) within this sublist. Using this simple syntax, it is possible to define a complex multithreaded workflow with minimum effort.
Here are more complex examples:
In this example, MyStep1 will be executed first, then 2 threads will start in parallel. In one thread, MyStep2 and MyStep3 will be executed in sequence. In the other thread, MyStep4 will be executed. Only when all of these steps reach a OMQ::StatComplete status, will MyStep5 be executed as the final step in the workflow.
The execution diagram will look like the following:
Using this simple rule of alternating list levels for sequential and parallel execution, it is possible to create very complex multithreaded workflows with minimum effort.
The "workflow-modules" option lists modules providing new base classes for step classes as in the following example.
Modules declared like this will be loaded into each workflow's Program object, and their classes can be used as base classes for step classes.
onetimeinit logic. Use the WorkflowApi::updateInstanceData() method (in Qore) or the WorkflowApi.updateInstanceData() method (in Java) to save resources acquired, and WorkflowApi::getInstanceData() (in Qore) or the WorkflowApi.getInstanceData() method (in Java) to retrieve the resources during the workflow's execution.nothing sub functionname() {}The recommended way to implement attach logic is to use a workflow class.
nothing sub functionname() {}nothing sub functionname(string status, *string external_order_instanceid) {}Detach Function Parameters
| Type and Name | Description |
string status | The status being set for the workflow order data instance, see Workflow, Segment, and Step Status Descriptions |
__7_ string external_order_instanceid | The external key for the workflow data being processed, if any exists, otherwise NOTHING |
nothing sub functionname(string errcode, *hash errinfo, any opt) {}Error Handler Function Parameters
| Type and Name | Description |
string errcode | The error code string (ex: "QORE-EXCEPTION") |
__7_ hash errinfo | This parameter will only be present if the error has been defined by the Deprecated Workflow Error Function; if so, the hash should have at least the following keys (key names in square brackets (i.e. [name]) are optional): - desc: description of the error- severity: Error Severity Codes - status: either OMQ::StatRetry (meaning that the step should be retried) or OMQ::StatError - [retry-delay]: the delay in seconds before the error should be retried (only valid when status is OMQ::StatRetry)- [business]: if True, this error represents a business error (rather than a technical error); usually implying that there is a problem with the consistency of the order data |
any opt | This is an optional parameter that can be supplied by the workflow when an error is raised. Normally it will be a string providing additional information about the error. |
hash sub functionname() {}"PAYMENT-MESSAGE-TIMEOUT"), and each value is a hash with error information as given in the following description of the keys:desc: the description of the errorseverity: the severity code; see Error Severity Codes, default: OMQ::ES_Majorstatus: the status code for the step due to this error; either OMQ::StatRetry or OMQ::StatError (default if not present)business: is this a business error? default: False retry-delay: the default retry time for the error as an integer giving seconds (ie 1200 for 20 minutes) or a relative date/time value (i.e. 2D + 12h or P2D12H, both meaning 2 days and 12 hours); the relative date/time value format is normally recommended as it's more readablelevel: the error level value; see Error Level Type Constants for possible values; default: OMQ::ErrLevelAuto ("AUTO")desc key is required. Note:"severity" key is not present, the error has a default severity of OMQ::ES_Major ("MAJOR")"ERROR")"AUTO")"retry-delay" key should only be given on errors with status set to OMQ::StatRetry ("RETRY")In order to define class steps, each step declaration in the workflow definition file must be defined as a hash containing the following attributes:
Step Definition Hash Key Descriptions
| Step Key | Type | Mand.? | Description |
| name | string | Y | The name and optionally the version of the step (i.e.: "step1:1.0") |
| version | string | N | The version of the step; this key is not required in the step definition if the step's version is given in the step's name (i.e.: "step1:1.0") |
| desc | string | N | The description of the step; if this key is not present, then the Qorus server will return the description of the primary step code when step information is requested |
| classname | string | N | The name and optional version (i.e. "MyStep1:1.0") of the step class, providing all the logic for a step. Note that if this key is not given, oload will assume that a function-based step is being defined |
| classversion | string | N | The version of the step class if not given in the "classname" parameter |
| queue | string | N | The name of the queue associated with this step (corresponding to the "Message Queue" element in the diagram above). This key is required for asynchronous steps |
| arraytype | string | N | For array steps, this valid must be "SERIES", in which case the step will be an array step |
| subworkflow | bool | N | If this is set to True, then the step type attribute will be set to OMQ::ExecSubWorkflow making the step a subworkflow step. Steps may not be simultaneously subworkflow and asynchronous or workflow synchronization event steps |
| eventtype | string | N | The name of the workflow event type for workflow synchronization event steps; in this case the step may not be a subworkflow or asynchronous step |
| user-interaction | bool | N | If this is set to True, then the asynchronous step will support APIs for user interaction; can only be set on asynchronous steps; if not present the default value is False |
Step configuration items are defined in YAML metadata; the programmatic definition of configuration items is deprecated.
Step configuration items in Qore are declared by overriding the QorusConfigurationItemProvider::getConfigItemsImpl() method in the step class. The step configuration items are registerted as part of the basic configuration of the step by oload when the workflow is loaded.
Each ConfigItemInfo hash in the return value of this method defines a configuration item for the step. The value of configuration items can then be retrieved and used in the workflow by calling one of the following APIs.
| API Type | API | Description |
| Qore | WorkflowApi::getConfigItemValue() | retrieves a single configuration item value |
| Qore | WorkflowApi::getConfigItemHash() | retrieves all configuration items |
Step configuration items are defined in YAML metadata; the programmatic definition of configuration items is deprecated.
Step configuration items in Java are declared by overriding the QorusStepBase.getConfigItemsImpl() method in the step class. The step configuration items are registerted as part of the basic configuration of the step by oload when the workflow is loaded.
Each ConfigItem object in the return value of this method defines a configuration item for the step. The value of the configuration item can then be retrieved and used in the workflow by calling one of the following APIs.
| API Type | API | Description |
| Java | WorkflowApi.getConfigItemValue() | retrieves a single configuration item value |
| Java | WorkflowApi.getConfigItemHash() | retrieves all configuration items |
Steps can also provide user-defined metadata which is returned as part of the step's description by overriding one of the following methods in the step class:
Note that step metadata is created by oload when the step is loaded, therefore this method should not rely on any runtime features of Qorus that are not available in oload.
For backwards compatibility, Qorus supports defining steps with functions defining each logic or code attribute of a step. In order to define complex function-based steps, each step declaration in the workflow definition file must be defined as a hash containing the following attributes:
Step Definition Hash Key Descriptions
| Step Key | Type | Mand.? | Description |
| name | string | Y | The name and optionally the version of the step (i.e.: "step1:1.0") |
| version | string | N | The version of the step; this key is not required in the step definition if the step's version is given in the step's name (i.e.: "step1:1.0" |
| desc | string | N | The description of the step; if this key is not present, then the Qorus server will return the description of the primary step code when step information is requested |
| funcname | string | N | The name and version of the primary step code (corresponding to the "Step Logic" element in the diagram above). Note that if this key is not given, the step function's name and version will be assumed to be the same as the step's (as defined in the name key) |
| valname | string | N | The name and version of the validation code for the step (corresponding to the "Validation Function" element in the diagram above). The validation code is run whenever the step is recovered. Note that validation code may not be defined for subworkflow or workflow synchronization event steps |
| endname | string | N | The name and version of the back-end code for asynchronous steps (corresponding to the "Async Logic" element in the diagram above). Note that if this key is defined, a queue must be defined, and the step type attribute will be set to OMQ::ExecAsync. The step may not also be a subworkflow step or a workflow synchronization event step |
| queue | string | N | The name of the queue associated with this step (corresponding to the "Message Queue" element in the diagram above). This key is required for asynchronous steps |
| arrayname | string | N | The name and version of the array function (corresponding to the "Array Logic" element in the diagram above). If this key is defined, then the step will be an array step |
| subworkflow | bool | N | If this is set to True, then the step type attribute will be set to OMQ::ExecSubWorkflow making the step a subworkflow step. Steps may not be simultaneously subworkflow and asynchronous or workflow synchronization event steps |
| eventtype | string | N | The name of the workflow event type for workflow synchronization event steps; in this case the step may not be a subworkflow or asynchronous step |
| user-interaction | bool | N | If this is set to True, then the asynchronous step will support APIs for user interaction; can only be set on asynchronous steps; if not present the default value is False |
The arraytype attribute may be given in a step definition to ensure that the step is defined as an array step. For class-based steps, this step attribute is mandatory for array steps, and will ensure that one of the above step classes is used as a base class for the step's class.
Example:
Qorus supports defining functions as step attributes instead of using a class for backwards-compatibility. This section describes defining a primary step function for a step.
nothing sub functionname() {}nothing sub functionname(auto arrayelement) {}NOTHING (any return value will be ignored)see $OMQ_DIR/examples/user/TEST-WORKFLOWS/test-functions-v1.0.qfd for further example function definitions for all function types in this section.
Qorus supports defining functions as step attributes instead of using a class for backwards-compatibility. This section describes defining a validation function for a step.
string sub functionname() {}string sub functionname(auto array_element) {}string sub functionname(*string async_key, auto array_element) {}string sub functionname(*string async_key) {}string: step status constantQorus supports defining functions as step attributes instead of using a class for backwards-compatibility. This section describes defining a asynchronous back-end function for a step.
nothing sub functionname(auto queue_data) {}nothing sub functionname(auto queue_data, auto array_element) {}array_element argument will be passed to the back end function only if the step is an array step. Otherwise only the queue_data argument will be passed to the function; the queue_data argument is the data that is posted on the queue when with the POST /api/latest/async-queues/{queue}?action=update call when the queue is updated.NOTHING (any return value will be ignored)queue tag must be set to a string giving the name of the queue for asynchronous steps. The queue will link the asynchronous step's primary step code with the back-end code.QUEUE_DATA) and pass it to the asynchronous back-end code in order to determine the step's status.Qorus supports defining functions as step attributes instead of using a class for backwards-compatibility. This section describes defining a asynchronous back-end function for a step.
any sub functionname() {}list: List of elements for array step; if the list is empty, then the array step is skippedany: A single element to execute a single substep in the array stepnothing: if NOTHING is returned, then the array step is skippedTrue, then the step will be a subworkflow step, and the step's status will be bound to a new workflow (the subworkflow) when the QorusSubworkflowStepBase::bindSubworkflow() method (for class-based steps) or Qorus DataProvider API: Bind Subworkflow is called. If neither this method nor QorusSubworkflowStepBase::skipSubworkflow() (for class-based steps) or Qorus DataProvider API: Skip Subworkflow is called, then the step will exit with an error. Otherwise, if a subworkflow is bound, the step will receive a WAITING status until the subworkflow is COMPLETE or ERROR, at which time the step will be updated with the subworkflow's status.True Qorus function definition files should have the extension: *.qfd
Function definition files are used to create function objects in the Qorus schema that can then be used by workflows (or loaded into service program objects as well for GENERIC functions).
Functions are defined by adding metadata tags in the form of special line comments to normal text files containing function definitions. The tags are parsed by the oload program, which will create function objects in the Qorus database according to the definitions in the file.
Function definitions begin with metadata tags as given in as specially-formatted comments as seen in the example below. After the header tags, one or more function definitions then follow, and the END tag terminates the function definition.
Function Definition File Tags
| Tag | Mand.? | Description |
type | Y | must be the first tag in a function definition, value must be one of the Step Function Types |
name | N | If the name tag is not defined, then the name of the function object in the database will be taken from the last function defined in the code block. If the name tag is defined, then a function with this name generally must be defined in the code block when used as a step function, however when used as a library function the name of the function object does not have to correspond to any function in the code block |
version | Y | Version string for the function object |
desc | Y | Description of the function object |
author | N | The optional author of the code object |
patch | N | Optional patch string – should be updated every time the same version of the function object is updated in the database |
TAG | N | Option tag definition in the format "key: value" |
END | Y | This tag must be included to terminate the function definition |
Qorus class definition files should have the extension: *.qclass or *.qclass.java (for Java-based classes).
Class definition files are used to create class objects in the Qorus schema that can then be used by workflows (or loaded into service program objects as well).
Like function objects, classes are defined by adding metadata tags in the form of special comments to normal text files containing class definitions. The tags are parsed by the oload program, which will create class objects in the Qorus database according to the definitions in the file.
Class definitions begin with metadata tags as given in as specially-formatted comments as seen in the examples below. After the header tags, one or more class definitions then follow, and the END tag terminates the class definition.
Class Definition File Tags
| Key | Mand.? | Description |
name | N | If the name tag is not defined, then the name of the class object will be taken from the last class defined in the code block. If the name tag is defined, then a class with this name should normally be defined in the code block, but the current version of Qorus does not enforce this |
version | Y | Version string for the class object |
desc | Y | Description of the class object |
author | N | The optional author of the code object |
patch | N | Optional patch string – should be updated every time the same version of the class object is updated in the database |
lang | N | the possible values are "qore" for Qore code (the default) and "java" for code targeting the Java 8 JVM (see Developing in Java) |
requires | N | Optional list of classes required by the current class; causes the required classes to be loaded when the current class is loaded |
TAG | N | Option tag definition in the format "key: value"; for Java classes, entries in the classpath can be added here by adding a colon-separated list of path names to the "classpath" tag; note also that $OMQ_DIR (or any other server-side environment variable) can be used in classpath entries and will be resolved to the Qorus directory |
END | Y | This tag must be included to terminate the class definition |
classpath tag can be used to add entries to the classpath for Java classes; $OMQ_DIR can be used in classpath entries as in the above example and will be resolved to the Qorus directory; see Java Classpath Handling in Qorus for more informationSee the classes listed in Step Source Class Definitions for example class definitions for class-based steps
Qorus constant definition files should have the extension: *.qconst
Constant definition files are used to create constant objects in the Qorus schema that can then be used by workflows (or loaded into service program objects).
Like function and class objects, constant objects are defined by adding metadata tags in the form of special comments to normal text files containing constant definitions. The tags are parsed by the oload program, which will create constant objects in the Qorus database according to the definitions in the file.
Constant definitions begin with metadata tags as given in as specially-formatted comments as seen in the example below. After the header tags, one or more constant definitions then follow, and the END tag terminates the constant definition.
Constant Definition File Tags
| Key | Mand.? | Description |
name | Y | The name tag must be defined for constant objects |
version | Y | Version string for the constant object |
desc | Y | Description of the constant object |
author | N | The optional author of the code object |
patch | N | Optional patch string – should be updated every time the same version of the constant object is updated in the database |
TAG | N | Option tag definition in the format "key: value" |
END | Y | This tag must be included to terminate the constant definition |
Qorus service definition files have the extension: *.qsd or *.qsd.java for Java services.
Service definition files are used to create service objects in the Qorus schema that can then be loaded and their methods can be called from any Qorus code, and, for external methods, from external applications through lightweight web-service protocols exported through the HTTP server as well.
Services are defined by adding metadata tags in the form of specially-formatted comments to normal text files containing at least one service definition and at least one method definition for the service. The tags are parsed by the oload program, which will create the service objects in the Qorus database according to the definitions in the file.
The first part of the file should contain the definitions for the service metadata. Service metadata definitions are entirely comprised of metadata tags as shown in the example below; the ENDSERVICE tag terminates the service metadata definition.
Service Metadata Definition File Tags
| Key | Mand.? | Description |
service | Y | The name of the service |
serviceversion | Y | Version string for the service object. |
servicedesc | Y | Description of the service object. |
serviceauthor | N | The optional author of the code object |
class-based | N | Set to true to define a class-based service (recommended); if not set or false, a function-based service will be assumed (only supported for backwards compatibility) |
class-name | N | Set this to the class name of the service's class in case it differs from the service name; this allows the service to have names that are not valid identifiers in the source language (Qore or Java); implies class-based: true |
| service-modules | N | lists modules providing base classes for class-based services |
| remote | N | The optional remote status of the service, indicating if the service will run in an independent qsvc process (always False for system services; see qorus-client.remote for how the default value is set by oload when loading user services) |
| autostart | N | The value of the autostart property of the service (default: False, managed by operations) |
lang | N | the possible values are "qore" for Qore code (the default) and "java" for code targeting the Java 8 JVM (see Developing in Java); note that lang: java implies class-based: true (see class-based services) |
classes | N | An optional list of class objects to load into the service program object |
constants | N | An optional list of constant objects to load into the service program object |
functions | N | An optional list of functions to load into the service program object |
mappers | N | An optional list of mappers to register with the service so they are available with the UserApi::getMapper() call |
vmaps | N | An optional list of value maps to register with the service so they can be used with the UserApi::getValueMap() call |
patch | N | Optional patch string; does not affect version compatibility; this value should be updated every time the same version of the service object is updated in the database. |
parse-options | N | Comma-separated parse options for the service: one of OMQ::ServiceParseOptionList; this is applied to all methods and also any service resource template programs |
define-auth-label | N | defines an authentication label with an associated value in the format label=value |
define-group | N | Allows Interface Groups to be defined; format is: name: description; see below for an example |
groups | N | one or more Interface Groups that the service is a member of |
resource | N | each resource line describes one or more files (if wildcards are used) that will be loaded into the database as service file resources for use in serivces that provide HTML UI services; using this tag the resource type will be derived from the file name (either "text", "binary", or "template") |
text-resource | N | defines text file resources (accepts wildcards) |
bin-resource | N | defines binary file resources (accepts wildcards) |
template | N | defines template file resources (accepts wildcards) |
TAG | N | Option tag definition in the format "key: value"; for Java services, entries in the classpath can be added here by adding a colon-separated list of path names to the "classpath" tag; note also that $OMQ_DIR (or any other server-side environment variable) can be used in classpath entries and will be resolved to the Qorus directory |
ENDSERVICE | Y | This tag must be included to terminate the service metadata definition. |
classpath tag can be used to add entries to the classpath for Java services; $OMQ_DIR can be used in classpath entries as in the above example and will be resolved to the Qorus directory; see Java Classpath Handling in Qorus for more informationService configuration items are declared in Qore by overriding the QorusConfigurationItemProvider::getConfigItemsImpl() method in the service class. The service configuration items are registerted as part of the basic configuration of the service by oload when the service is loaded.
Each ConfigItemInfo hash in the return value of this method defines a configuration item for the service. The value of configuration items can then be retrieved and used in the service by calling one of the following APIs.
| API Type | API | Description |
| Qore | ServiceApi::getConfigItemValue() | retrieves a single configuration item value |
| Qore | ServiceApi::getConfigItemHash() | retrieves all configuration items |
Step configuration items in Java are declared by overriding the QorusService.getConfigItemsImpl() method in the service class. Service configuration items are registerted as part of the basic configuration of the service by oload when the service is loaded.
Each ConfigItem object in the return value of this method defines a configuration item for the service. The value of the configuration item can then be retrieved and used in the service by calling one of the following APIs.
| API Type | API | Description |
| Java | ServiceApi.getConfigItemValue() | retrieves a single configuration item value |
| Java | ServiceApi.getConfigItemHash() | retrieves all configuration items |
Function-based services are supported for backwards compatibility; a function-based service's methods are defined by functions.
Method definitions following the service metadata definition, also using tags to define the method metadata for the oload program. The END tag terminates the method definition.
Service Method Definition Tags
| Key | Mand.? | Description |
lock | N | must be either none, read, or write (default none), to determine how the service's RWLock will be grabbed before the method is executed |
name | N | If the name tag is not defined, then the name of the method will be taken from the first function defined in the code block. If the name tag is defined, then a function with this name must be defined in the code block (name tags are optional for function-based service methods) |
desc | Y | Description of the method object |
author | N | The optional author of the code object |
internal | N | If this boolean flag is set to True, then the method will not be automatically exported through the any network interface |
write | N | If this boolean flag is set to True, then the method will be marked as a write method, meaning that external callers will have to have the CALL-USER-SERVICES-RW role to call the method if RBAC security is enabled (Enterprise Edition only) |
END | Y | This tag must be included with function-based services to terminate the method definition |
Service authentication labels allow for the authentication method for a service to be changed at runtime with the UI or the API by an authorized user.
Authentication labels are unique to a service; if the value of an authentication label is changed for a specific service, it does not affect any authentication label with the same name for another service.
Authentication labels are used by the QorusParametrizedAuthenticator class.
Authentication labels are defined with an authentication label name and one of the following values:
"permissive": allows all requests to succeed"default": uses default Qorus authentication"default-basic": uses default Qorus authentication and returns the WWW-Authenticate header with 401 Unauthorized responses to ensure that web browsers will request a username and password for basic authentication in the next request.Qorus job definition files should have the extension *.qjob or *.qjob.java for Java-based jobs to identify the file's contents properly to the schema loader.
Jobs are defined by adding metadata tags in the form of specially-formatted comments to normal text files containing at least one complete job definition. The tags are parsed by the oload program, which will create the jobs in the Qorus database according to the information in the file.
The file consists of tags defining the job and then the code that makes up the job. At the end of the source code, the END tag terminates the job definition.
Job Definition File Tags
| Key | Mand.? | Description |
name | Y | The name of the job |
version | Y | Version string for the job object; this version is informative; only one version of a job is stored in the database at a time |
desc | Y | Description of the job object |
author | N | The optional author of the code object |
job-modules | N | lists modules providing base classes for class-based jobs |
remote | N | The optional remote status of the job, indicating if the job will run in an independent qjob process (see qorus-client.remote for how the default value is assigned by oload) |
active | N | A boolean value giving the active status of the job; if not present, the default is True |
run-skipped | N | A boolean value telling the system if the job should be run immediately if the last scheduled run was missed due to system downtime; if not present, the default is True |
duration | * | a schedule duration in seconds; if present the job will be scheduled to run every n seconds where n is the value assigned to this tag; either this tag or the schedule tag must be present, but not both |
schedule | * | a cron specification, see Job Schedule for more information; either this tag or the duration tag must be present, but not both |
expiry-date | N | An optional expiration date; the job will not be run (and will be automatically set to inactive) when the expiration date is reached |
class-based | N | Set to true to define a class-based job (recommended); if not set or false, a function-based job will be assumed (only supported for backwards compatibility) |
class-name | N | Set this to the class name of the job's class in case it differs; this allows the job to have names that are not valid identifiers in the source language (Qore or Java) |
lang | N | the possible values are "qore" for Qore code (the default) and "java" for code targeting the Java 8 JVM (see Developing in Java) |
classes | N | An optional list of class objects to load into the job program object |
constants | N | An optional list of constant objects to load into the job program object |
functions | N | An optional list of functions to load into the job program object |
mappers | N | An optional list of mappers to register with the job so they are available with the UserApi::getMapper() call |
vmaps | N | An optional list of value maps to register with the job so they can be used with the UserApi::getValueMap() call |
define-group | N | Allows Interface Groups to be defined; format is: name: description; see below for an example |
groups | N | one or more Interface Groups separated by commas that the job is a member of |
TAG | N | Option tag definition in the format "key: value" |
END | Y | This tag must be present after the job's source code to terminate the job definition |
The job's code should follow the metadata tags given above.
Note that the logic for the job is defined by either a job class (recommended) or a function (supported for backwards compatibility).
The recommended way to define a job is by setting the class-based option to true and defining the job's logic as a subclass of one of the following classes depending on the source language:
class-based = true; see QorusJob for more information and an example Java job definitionclasspath tag can be used to add entries to the classpath for Java classes; $OMQ_DIR can be used in classpath entries as in the above example and will be resolved to the Qorus directory; see Java Classpath Handling in Qorus for more informationThe "job-modules" option lists modules providing base classes for Class-Based Jobs as in the following example.
Modules declared like this will be loaded into each jobs's Program object, and their classes can be used as base classes for a job class.
Job configuration items are declared in Qore by overriding the QorusConfigurationItemProvider::getConfigItemsImpl() method in the job class. The job configuration items are registerted as part of the basic configuration of the job by oload when the job is loaded.
Each ConfigItemInfo hash in the return value of this method defines a configuration item for the job. The value of the configuration item can then be retrieved and used in the job by calling one of the following APIs.
| API Type | API | Description |
| Qore | JobApi::getConfigItemValue() | retrieves a single configuration item value |
| Qore | JobApi::getConfigItemHash() | retrieves all configuration items |
Step configuration items in Java are declared by overriding the QorusJob.getConfigItemsImpl() method in the job class. Job configuration items are registerted as part of the basic configuration of the job by oload when the job is loaded.
Each ConfigItem object in the return value of this method defines a configuration item for the job. The value of the configuration item can then be retrieved and used in the job by calling one of the following APIs.
| API Type | API | Description |
| Java | JobApi.getConfigItemValue() | retrieves a single configuration item value |
| Java | JobApi.getConfigItemHash() | retrieves all configuration items |
Function-based jobs are supported for backwards compatibility; a function-based job must have a run() function. This function will be called when the job is scheduled. Note that only active jobs that are not members of any disabled Interface Groups will be scheduled and run.
The job's code also has the job api at its disposal.
run() must be present in function-based jobs.Whenever a job is executed, a record in the JOB_INSTANCE table is created. If an error is raised by calling JobApi::err() with a severity greater than OMQ::ES_Warning, the job instance will get a status of OMQ::JS_Error, otherwise the job instance will get a status of OMQ::JS_Complete. In the case that the Qorus system crashes while the job has a OMQ::JS_InProgress status, then, when the system is recovered, the JOB_INSTANCE row will get a OMQ::JS_Crash status.
Job instances can never be recovered; the status is saved for information purposes only.
To save information about processing status, call the JobApi::saveInfo() method.
The UserApi::log() method is used to save information in the job's log file; see System, Service, Workflow, and Job Logging for more information about log file locations and file formats.
Mappers are defined by adding metadata tags in the form of specially-formatted comments to normal text files containing a complete mapper definition. The tags are parsed by oload, which will create the mappers in the Qorus database according to the information in the file.
The file consists of tags defining the mapper and then option and field definitions. At the end of the mapper definition, the END tag terminates the mapper.
Mapper Definition File Tags
| Key | Mand.? | Description |
name | Y | The name of the mapper |
version | Y | Version string for the mapper object; multiple versions of the same mapper can exist in the database and interfaces (workflows, services, and jobs) must refer to the mapper using its explicit version |
patch | N | A descriptive patchlevel attribute that does not affect version compatibility |
desc | Y | Description of the mapper object |
author | N | The optional author of the mapper object |
parse_options | N | A comma separated list of local parse options for the mapper code; one or more of (options must be given as shown without any namespace prefixes): - PO_REQUIRE_TYPES - PO_REQUIRE_PROTOTYPES - PO_STRICT_ARGS - PO_ASSUME_LOCAL - PO_ALLOW_BARE_REFS |
type | Y | The type of mapper; must be a system type or a type provided by a mapper module |
classes | N | An optional list of class objects to load into the mapper program object (subject to %lockdown sandboxing restrictions; see Deprecated Mapper Library Objects for more information) |
constants | N | An optional list of constant objects to load into the mapper program object (subject to %lockdown sandboxing restrictions; see Deprecated Mapper Library Objects for more information) |
functions | N | An optional list of functions to load into the mapper program object (subject to %lockdown sandboxing restrictions; see Deprecated Mapper Library Objects for more information) |
define-group | N | Allows interface groups to be defined; format is: name: description; see below for an example |
groups | N | one or more interface groups that the mapper is a member of for access purposes (mappers cannot be enabled or disabled) |
Mapper example:
"output" option is not defined since it's defined automatically by the InboundTableMapper mapper typeThe following mapper types are provided by the system:
"Mapper": implemented by OMQ::MapperType; provides a Mapper::Mapper object at runtime"InboundTableMapper": implemented by OMQ::InboundTableMapperType; provides a OMQ::QorusInboundTableMapper object at runtime"QorusSqlStatementOutboundMapper": implemented by OMQ::QorusSqlStatementOutboundMapperType; provides a OMQ::QorusSqlStatementOutboundMapper object at runtime"QorusRawSqlStatementOutboundMapper": implemented by OMQ::QorusRawSqlStatementOutboundMapperType; provides a OMQ::QorusRawSqlStatementOutboundMapper object at runtimeAdditional mapper types can be provided by implementing a mapper module; see Deprecated Mapper Module Developent for more information.
Common options for all mappers are defined by the Mapper class here: Mapper Options, but each mapper type can extend this list with its own options, some of which may be required as well.
Mappers object must provide the input and output options by default, but these may be generated automatically by the particular mapper type (for example, the OMQ::InboundTableMapperType class automatically generates the output option based on the target table); see the mapper type documentation for more information.
Mapper input options are specified with the following syntax:
OPTION: option_name: expressionWhere:
See the example in the preceding section.
This option is a hash that describes the input data structures; see the "input" key of Mapper Options for more information.
Note that only input field names defined as keys in this hash can be used as input field names in mapper field definitions.
While all mapper ojects generated by a mapper type must define the "input" and "output" keys, mapper types may define these automatically (for example, the OMQ::InboundTableMapperType class automatically generates the output option based on the target table); see the mapper type documentation for more information.
This option is a hash that describes the output data structures; see the "output" key of Mapper Options for more information.
Note that only output field names defined as keys in this hash can be used as Deprecated Mapper Fields.
While all mapper ojects generated by a mapper type must define the "input" and "output" keys, mapper types may define these automatically (for example, the OMQ::InboundTableMapperType class automatically generates the output option based on the target table); see the mapper type documentation for more information.
Mapper field transformations are based on Mapper Specification Format and are specified with the following syntax:
FIELD: output_field: expressionWhere:
All Qorus mapper field descriptions support the following two additional keys which are added at runtime to all mapper:
"context": any string value assigned to this key will be used as the argument to UserApi::expandTemplatedValue(); the return value of this method call will be used as the value of the field"template": if assigned to True; the value of the field will be passed as the sole argument to UserApi::expandTemplatedValue(); the return value of this method call will be used as the value of the fieldMappers support library objects in way similar to workflows, services, and jobs, however mapper library objects are subject to %lockdown sandboxing restrictions, so every library object imported into a mapper Program object must meet this criterion or it cannot be used in a mapper.
%lockdown prohibts any I/O; also logging is not allowed from within a mapper, so generally library code used elsewhere in Qorus workflows, services, and jobs is not suitable for use in a mapper.
Mapper library objects can be used in workflows, services, and jobs however.
New mapper types can be added to the system by implementing mapper modules and placing them in $OMQ_DIR/user/modules/ and adding the modules' names to the qorus.mapper-modules system option.
Each module must call the map_register_mapper() function in the module's init closure or call reference.
Example module:
User connections, Datasource Connections, Qorus to Qorus Connections are defined in connection files with the suffix ".qconn".
Qorus user code should access external resources using user connections in all possible cases because such connections are monitored and the interace is managed (stopped and started) when the dependent connections become unavailable and again available (additionally, operational alerts are issued and cleared in these cases.
When connections are made directly (for example, by using a HTTPClient object directly instead of using a user connection), then the system does not monitor the connection's health, does not manage dependent interface status nor raise or clear alerts when the connection is unavailable or again available.
Deprecated user connection definition file syntax:
connection-name = ( key = value, ... )
The use of whitespace (spaces, newlines, tabs, etc) is allowed; empty lines are ignored. Text after the hash character ('#') indicates the start of a line comment when found outside a quoted string. Quoted strings are allowed, and certain characters must be included in quoted strings and escaped to be used in connection attributes; see Deprecated Connection File Parsing for more information.
Conditional parsing is also supported with the following parse directives:
%ifdef: if the given parse define is defined, then the definitions up to the next %else or %endif are parsed, otherwise the block is skipped%ifndef: if the given parse define is not defined, then the definitions up to the next %else or %endif are parsed, otherwise the block is skipped%else: toggles the parsing state from the last %ifdef or %ifndef %endif: closes a conditionally-parsed blockParse defines are set in the System Options File by system option qorus.defines.
Supported (case sensitive) keys are given in the following table.
User Connection Keys
| Key | Mandatory? | Description | Example |
url | Y | full URL for the resource; see connection object types for information on the URLs handled and the object class returned by UserApi::getUserConnection() | http://user:pass@localhost:8001 |
desc | N | public description of the connection | production server 2 |
tags | N | key-value pairs to be associated with the connection | "tag1=value1, tag2=2, tag3=\"3\"" |
Additional keys can (and depending on the connection type must) be defined according to supported or required options for the given connection type. See connection object types for information on supported options per connection type.
"qorus" connection name is provided by default by the QorusConnectionProvider module as a connnection to the local Qorus instance, therefore it's recommended not to use "qorus" for a user or remote connection nameThe following is an example of a connection file using conditional parsing:
%ifdef Production
# production connection definition
sftp-file1 = (
desc = FILE1 SFTP Polling / Delivery connection,
url = sftp://file1@sftp.example.com,
keyfile = /export/home/qorus-prod/.ssh/id_rsa-file1
)
%else
# test/devel connection definition
sftp-file1 = (
desc = FILE1 SFTP Polling / Delivery connection,
url = sftp://file1@sftp-test.example.com,
keyfile = HOME/.ssh/id_rsa-file1
)
%endifConnection files are parsed as free-form text where whitespace is generally ignored.
The following characters are special in connection files, and can only be used literally when included in quoted strings:
": indicates the start of a quoted string$: indicates that environment variable substitution should be performed"\r") characters are ignored in connection files; to include a CR character in an attribute, use it in a quoted string as described below") and parentheses (Quoted strings must be delimited with "; for forward compatibility, it's best to delimit any strings with special characters listed above with ".
Quoted strings accept escape characters; any character may be escaped by prefacing the character with a \ character, however the following escape characters have special values in connection files when escaped:
"\n": indicates a newline (ASCII 10)"\r": indicates a carriage return (ASCII 13)"\t": indicates a tab (ASCII 9)To use one of the special characters listed above, use the character in a quoted string and escape it as in the following example:
https-remote-1 = (
desc = remote HTTPS connection,
url = "https://user:'pass'\$\"word\"@example.com:11098",
)All attributes of connection files (remote and user connection files) are subject to environment variable substitution. Any text with the following format will be substituted with the value of the given environment variable:
$VAR${VAR}For example, in the following file:
fs-staging = (
desc = Staging filesystem connection,
url = file://$DIR_ROOT/staging,
)Assuming that the $DIR_ROOT environment variable is assigned to /project/qorus, then the value of the url option would be set to file:///project/qorus/staging.
To prohibit environment variable substitution and use a literal "$" character in a URL (for example, as a character in a password value in a URL), enclose the string in single or double quotes and place a backslash character before the "$" sign as follows:
https-remote-1 = (
desc = remote HTTPS connection,
url = "https://user:pass\$word@example.com:11098",
)In this way the url option would be set to https://user:pass$word@example.com:11098. Due to environment variable substitution, "$" characters are special characters that must be escaped as in the above examples in quoted strings in connection files to be used literally.