Qorus Integration Engine®
4.0.3.p2_git
|
User services are like named and versioned API sets that can be live-upgraded. User service methods are available to all workflows and jobs (and to other services) and can be automatically exported to other applications through lightweight web services (RPC protocols and REST) through the HTTP server (in fact, this is the default behavior, but can be inhibited by setting the internal flag on the service method).
Additionally, services can also have one or more background threads, and therefore do not have to wait for input data to perform some processing.
Services can also be called from the command-line using the ocmd program as follows:
unixprompt% ocmd omq.[system|user].service.<service>.<method> args...
Services are loaded from the database into their own program objects and have their own internal API. All user-defined services must be of the type "user"
. System services are delivered with Qorus and should not be modified, as this is likely to affect system stability.
The following diagram illustrates the a subset of the attributes of a service.
Service methods define the user-visible interface of the service. The logic in a service program is defined by the method code and any library objects (functions, classes, or constant definitions) loaded into the service's program object.
Besides the name, version, type (always "user"
for user-defined services), and description of the service, there are the "autostart"
and "remote"
flags. If the "autostart"
flag is set to True
, then the service will be started automatically whenever Qorus is started or whenever its RBAC access group is renabled; see Service Autostart Parameter for more information.
The "remote"
flag indicates if the service will run in an independent qsvc process or not; see Service Remote Parameter for more information.
Otherwise, a service is loaded and initialized any time a method of that service is called and the service has not already been loaded. In this case all the service's methods are loaded and parsed into the same program object, along with any library function, classes, and constants associated with the service. Any parse exceptions during service loading will prohibit the service from being loaded.
If a service has an init()
method, this method is called as soon as the service has been loaded and parsed, and before any call to a service method is made. For example, if a service with an init()
method is loaded because a call to another method is made, the init()
method will be called first, and then the called method will be called and the value returned to the caller.
An exception when running the init()
method will cause the service to be unloaded, and the exception will be returned to the caller.
If the service has a start()
method, then it must also have a stop()
method. The start()
method will be run in a separate thread and is expected to run until the stop()
method is called. The stop()
method should signal the routine running in the start()
method to exit.
When the start()
method returns, the service is automatically unloaded. To start threads in user code in a service, use the svc_start_thread() or svc_start_thread_args() functions (in which case, to use these functions, the service must also have a stop()
method).
init()
method is explicitly called, this call is ignored, in the sense that, if the service is already loaded, no action is taken, and if it is not, the service is loaded and initialized, so the init()
method is called as per the rules above. Therefore to simple ensure that a service is loaded and initialized, it is safe to call the init()
method of a service at any time. Also note that the init()
method may be safely called even if no init()
method is defined for the service.If the "autostart"
flag is set to True
, then the service will be started automatically whenever Qorus is started or whenever its RBAC access group is renabled.
"autostart"
value is considered to be managed by operations, which means that once a service has been loaded into Qorus, if its "autostart"
value is updated with the API, then those API-driven changes are persistent and will not be overwritten by subsequent loads of the service by oload.The "remote"
flag indicates if the service will run as an independent qsvc process communicating with other elements of Qorus Integration Engine with a distributed queue protocol rather than internally in the qorus-core process.
When workflows run in separate qsvc processes, it provides a higher level of stability and control to the integration platform as a whole, as a service with implementation problems cannot cause the integration platform to fail.
There is a performance cost to running in separate qsvc processes; service startup and shutdown is slightly slower, and communication with qorus-core also suffers a performance hit as all communication must be serialized and transmitted over the network.
Calls to service methods in remote processes may not leave thread resources active when returning from the method; doing so will cause an exception to be thrown and for the thread resource to be cleaned up automatically.
Additionally, service methods running in remote qsvc processes cannot return non-serializable data (non-serializable objects, references, callable data types), and also their methods cannot accept these values as arguments. Furthermore, all HTTP or other protocol support implemented in remote services will be subject to round-trip network serialization to and from qorus-core.
"remote"
flag must be set to False.The default for this option depends on the client option qorus-client.remote (if this client option is not set, then the default value is True).
The remote value can be changed at runtime by using the following REST API: PUT /api/latest/services/{id_or_name}?action=setRemote
remote
flag is considered to be managed by operations, which means that once an interface has been loaded into Qorus, if its remote
flag is updated with the API, then those API-driven changes are persistent and will not be overwritten by subsequent loads of the interface by oload.Service authentication labels allow for the authentication method for a service to be changed at runtime with the UI or the API by an authorized user.
Authentication labels are unique to a service; if the value of an authentication label is changed for a specific service, it does not affect any authentication label with the same name for another service.
Authentication labels are used by the QorusParametrizedAuthenticator class.
Authentication labels are defined with an authentication label name and one of the following values:
"permissive"
: allows all requests to succeed"default"
: uses default Qorus authentication"default-basic"
: uses default Qorus authentication and returns the WWW-Authenticate
header with 401 Unauthorized
responses to ensure that web browsers will request a username and password for basic authentication in the next request.Here is an example:
# define-auth-label: auth=permissive
Service methods are the user-visible entry-points to a service and as such contain the logic of the service.
The following diagram illustrates a subset of the attributes of a service method.
Each service method has several attributes associated with it.
The service's name is used when calling the method.
The description can be seen when listing services with the ostatus program or the Qorus GUI.
Each service has a read-write lock object (Qore class RWLock) associated with it. Methods can be tagged to acquire the lock for reading, writing, or not acquire the lock at all (default behavior).
This attribute can be used to provide safe multi-threaded access to internal resources. For example, if the service provides methods to read data and a method to reload the data from an external resource (such as a database), then the read methods can be tagged to acquire the lock in read mode, and the method to reload the data can be tagged to acquire the lock in write mode. This way, the reload method will block until all reads have finished, and the reads will block if any reload is in progress.
The read-write lock will be released safely when execution returns to the Qorus system, even if an exception is thrown in the method.
If the "internal"
flag is set to True
on a method, then that method can only be called internally. Any direct call to this method from an external source (such as through lightweight web-service protocols exported through the HTTP server) will result in the method not being called and an exception returned to the called instead.
This flag should be set on methods that return objects, as objects cannot be sent through network interfaces.
Qorus service definition files should have the extension: *.qsd
or *.qsd.java
for Java services.
Service definition files are used to create service objects in the Qorus schema that can then be loaded and their methods can be called from any Qorus code, and, for external methods, from external applications through lightweight web-service protocols exported through the HTTP server as well.
As with function, class, and constant objects, services are defined by adding metadata tags in the form of specially-formatted comments to normal text files containing at least one service definition and at least one method definition for the service. The tags are parsed by the oload program, which will create the service objects in the Qorus database according to the definitions in the file.
The first part of the file should contain the definitions for the service metadata. Service metadata definitions are entirely comprised of metadata tags as shown in the example below; the ENDSERVICE
tag terminates the service metadata definition.
Service Metadata Definition File Tags
Key | Mand.? | Description |
service | Y | The name of the service |
serviceversion | Y | Version string for the service object. |
servicedesc | Y | Description of the service object. |
serviceauthor | N | The optional author of the code object |
class-based | N | Set to true to define a class-based service (recommended); if not set or false , a function-based service will be assumed (only supported for backwards compatibility) |
class-name | N | Set this to the class name of the service's class in case it differs from the service name; this allows the service to have names that are not valid identifiers in the source language (Qore or Java); implies class-based: true |
service-modules | N | lists modules providing base classes for class-based services |
remote | N | The optional remote status of the service, indicating if the service will run in an independent qsvc process (always False for system services; see qorus-client.remote for how the default value is set by oload when loading user services) |
autostart | N | The value of the autostart property of the service (default: False , managed by operations) |
lang | N | the possible values are "qore" for Qore code (the default) and "java" for code targeting the Java 8 JVM (see Developing in Java); note that lang: java implies class-based: true (see class-based services) |
classes | N | An optional list of class objects to load into the service program object |
constants | N | An optional list of constant objects to load into the service program object |
functions | N | An optional list of functions to load into the service program object |
mappers | N | An optional list of mappers to register with the service so they are available with the UserApi::getMapper() call |
vmaps | N | An optional list of value maps to register with the service so they can be used with the UserApi::getValueMap() call |
patch | N | Optional patch string; does not affect version compatibility; this value should be updated every time the same version of the service object is updated in the database. |
parse-options | N | Comma-separated parse options for the service: one of OMQ::ServiceParseOptionList; this is applied to all methods and also any service resource template programs |
define-auth-label | N | defines an authentication label with an associated value in the format label=value |
define-group | N | Allows Interface Groups to be defined; format is: name: description ; see below for an example |
groups | N | one or more Interface Groups that the service is a member of |
resource | N | each resource line describes one or more files (if wildcards are used) that will be loaded into the database as service file resources for use in serivces that provide HTML UI services; using this tag the resource type will be derived from the file name (either "text" , "binary" , or "template" ) |
text-resource | N | defines text file resources (accepts wildcards) |
bin-resource | N | defines binary file resources (accepts wildcards) |
template | N | defines template file resources (accepts wildcards) |
TAG | N | Option tag definition in the format "key: value" ; for Java services, entries in the classpath can be added here by adding a colon-separated list of path names to the "classpath" tag; note also that $OMQ_DIR (or any other server-side environment variable) can be used in classpath entries and will be resolved to the Qorus directory |
ENDSERVICE | Y | This tag must be included to terminate the service metadata definition. |
classpath
tag can be used to add entries to the classpath for Java services; $OMQ_DIR
can be used in classpath entries as in the above example and will be resolved to the Qorus directory; see Java Classpath Handling in Qorus for more informationThe recommended way to define a service is by setting the class-based
option to true
or by setting the class-name
tag (which implies class-based: true
and defining the service's logic as a subclass of one of the following classes depending on the source language:
The service starts with a metadata definition, followed by the class defining the service itself. All public methods of the class will be exported (static and non-static), method tags should be used to define method metadata for the oload program. The END
tag is optional and, if used, terminates the method definition.
Service Method Definition Tags
Key | Mand.? | Description |
lock | N | must be either none , read , or write (default none ), to determine how the service's RWLock will be grabbed before the method is executed |
name | N | The name tag must be defined for class-based service methods; also a public method with this name must be defined in the code block as well |
desc | Y | Description of the method object |
author | N | The optional author of the method, in case it differs from the service's author |
internal | N | If this boolean flag is set to True , then the method will not be automatically exported through the any network interface |
write | N | If this boolean flag is set to True , then the method will be marked as a write method, meaning that external callers will have to have the CALL-USER-SERVICES-RW role to call the method if RBAC security is enabled |
END | Y | This tag may be included to terminate the method definition but is not required for class-based service methods |
Class-based services can have a constructor and classes can have static initialization, but please note that if the service has configuration items, it must be instantiated by oload in order to create the service's configuration in the system. In such a case, if the constructor or static class initialization requires features that are only available at runtime in Qorus itself, the errors raised will cause service class instantiation or static class instantiation to fail.
To work around this, put all initialization requiring runtime support in Qorus in the init method of the service. This method is only called when the service is initialized by Qorus at runtime.
The service constructor takes no arguments.
Class-based services (and only class-based services) can declare configuration items to allow for the behavior of the service to be modified by users at runtime using the operational web UI or the REST API.
Service configuration items are:
Service configuration items are designed to allow users to affect the execution of a service so that changes can be made by authorized users in the UI without requiring a change to development.
If the strictly_local
flag on a service configuration item is False
, then the service configuration item is not local and the value can also be set on global level.
If the strictly_local
flag is False
, then the service configuration item is local and hence the value for this item cannot be set on global level.
Service configuration items are declared in Qore by overriding the QorusConfigurationItemProvider::getConfigItemsImpl() method in the service class. The service configuration items are registerted as part of the basic configuration of the service by oload when the service is loaded.
Each ConfigItemInfo hash in the return value of this method defines a configuration item for the service. The value of configuration items can then be retrieved and used in the service by calling one of the following APIs.
API Type | API | Description |
Qore | ServiceApi::getConfigItemValue() | retrieves a single configuration item value |
Qore | ServiceApi::getConfigItemHash() | retrieves all configuration items |
Step configuration items in Java are declared by overriding the QorusService.getConfigItemsImpl() method in the service class. Service configuration items are registerted as part of the basic configuration of the service by oload when the service is loaded.
Each ConfigItem object in the return value of this method defines a configuration item for the service. The value of the configuration item can then be retrieved and used in the service by calling one of the following APIs.
API Type | API | Description |
Java | ServiceApi.getConfigItemValue() | retrieves a single configuration item value |
Java | ServiceApi.getConfigItemHash() | retrieves all configuration items |
Instead of using metadata tags in special comments, the Java annotation interface QorusMethod should be used as in the following example:
copy()
service method with a Qore class-based service, name the class method _copy()
(with a leading underscore), and give the method name copy()
. The translation to the internal name (to avoid a conflict with the special class method copy()
) is done automatically by oload and by the Qorus runtime. See the following example for more information.constructor()
destructor()
methodGate()
memberGate()
memberNotification()
Static class variables are initialized when the class is loaded which is also performed by oload, therefore if any static class variables have initialization code that requires Qorus functionality, this can cause oload to fail, therefore it's recommended to put such static initialization in the class's constructor()
method instead as in the preceding example.
The "service-modules"
option lists modules providing base classes for class-based services as in the following example.
Modules declared like this will be loaded into each service's Program object, and their classes can be used as base classes for class-based services.
Function-based services are supported for backwards compatibility; a function-based service's methods are defined by functions.
Method definitions following the service metadata definition, also using tags to define the method metadata for the oload program. The END
tag terminates the method definition.
Service Method Definition Tags
Key | Mand.? | Description |
lock | N | must be either none , read , or write (default none ), to determine how the service's RWLock will be grabbed before the method is executed |
name | N | If the name tag is not defined, then the name of the method will be taken from the first function defined in the code block. If the name tag is defined, then a function with this name must be defined in the code block (name tags are optional for function-based service methods) |
desc | Y | Description of the method object |
author | N | The optional author of the code object |
internal | N | If this boolean flag is set to True , then the method will not be automatically exported through the any network interface |
write | N | If this boolean flag is set to True , then the method will be marked as a write method, meaning that external callers will have to have the CALL-USER-SERVICES-RW role to call the method if RBAC security is enabled |
END | Y | This tag must be included with function-based services to terminate the method definition |
Qorus services can implement generic HTTP request handlers by subclassing OMQ::AbstractServiceHttpHandler and binding it either to the global Qorus listeners or service-specific listeners using svc_bind_http().
Using this mechanism, Qorus services can implement custom HTTP-based protocols or use one of the existing heler classes to export RPC services, REST APIs, web sockets server event sources, or even generic HTML user interfaces and Qorus web UI extensions.
Service file resources are resource files that are declared with the "resource"
tag in a service definition file and loaded into the database with oload.
These file resources can then be served automatically if the service binds a OMQ::AbstractServiceHttpHandler object with svc_bind_http() to serve HTTP requests.
Each resource name is its relative file name; for example if the following declaration is made in a service:
# resource: html/*
And the file html/extension.html
is loaded by oload as a service file resource, then "html/extension.html"
is a valid resource name, and could be set as the default resource (the resource that's served when no other resource is matched to a request) with AbstractServiceHttpHandler::setDefaultResource().
For example, if the following code is used to create the service HTTP handler object and bind the handler:
Then a request like the following is received:
GET /my-service/html/extension.html HTTP/1.1
The cx.resource_path value in the call to AbstractServiceHttpHandler::handleRequestImpl() will be set to "html/extension.html"
, which in this example would be a valid service file resource, therefore if the AbstractServiceHttpHandler::handleRequestImpl() returns NOTHING, this resource would be automatically served to the requestor.
Mixed text/html and Qore code can be automatically rendered and served if the resource has the file extension: "qhtml"
. In this case the final rendered text is rendered and served automatically according to the rules described in WebUtil::StaticTemplateManager::add().
Summary of unrendered text format:
{%%
qore statement %%}
{{
qore expression returning a string }}
Example of a raw, unrendered resource:
{% foreach hash h in (ctx.list) { %} <tr> <td class="name"><a href="?rpath={{ ctx.parent_url ? ctx.rpath : "" }}/{{ h.name }}" {{ h.type != "DIRECTORY" ? "class=\"file\"" : ""}}>{{ h.name }}</a></td> <td class="text-left" width="40">{{ h.type != "DIRECTORY" ? "File" : "Directory" }}</td> <td class="text-right">{{ h.type != "DIRECTORY" ? h.size : "--" }}</td> </tr> {% } %}
Qorus's service template manager sets up the embedded Program object for templates with additional code similar to a service Program.
The following are additions and enhancements available to template processing and rendering when used with Qorus services:
"Qorus"
(common to workflows, services, and jobs)"QorusServer"
(common to workflows, services, and jobs)"QorusHasUserConnections"
(common to workflows, services, and jobs)"QorusHasAlerts"
(common to workflows, services, and jobs)"QorusService"
(service-specific)"QorusHasHttpUserIndex"
(service-specific)parse-options
attribute of the service; it's not possible to set them with parse directives inside the template, as injected code (which is injected as source) would be injected with the wrong style.The web UI can be extended with Qorus services by registering the extension with the following function:
The only argument of this function must be a class descended from OMQ::QorusExtensionHandler as in the following example:
At least one service file resource is required: the bootstrap HTML page; this resource is set by calling OMQ::QorusExtensionHandler::setDefaultResource() as in the above example.
Resources must be declared in the service definition file with a service file resource declaration; in the above example, this declaration might look like the following:
# resource: templates/index.qhtml
The following is an example UI extension service with example resources.
<h1>Hello World!</h1>
REST APIs can be developed with Qorus services by subclassing the OMQ::AbstractServiceRestHandler class and binding it with ServiceApi::bindHttp().
Consider the following example: rest-example-v1.0.qsd
This is a simple service declaration that exports a (very simple) REST API at "/example" that returns the string "OK"
to the following requests:
GET /example HTTP/1.1
: handled by ExampleRestHandlerGET /example/subclass HTTP/1.1
: handled by ExampleRestClassQorus services can provide websocket server services by subclassing OMQ::AbstractServiceWebSocketHandler and binding it with ServiceApi::bindHttp().
Consider the following example websocket-example-v1.0.qsd
:
The above example service will listen for all system events with waitForEvents() (polling every second for a service stop event) and serialize any new system events with JSON using make_json() and send to all connected clients using WebSocketHandler::sendAll().
Qorus services can implement a custom FTP server for receiving data by subclassing OMQ::AbstractFtpHandler and calling ServiceApi::bindFtp() on the resulting object.
Consider the following example service ftp-example-v1.0.qsd
:
This service will create an FTP server on port 18002 by default and store any files received in the "/tmp"
directory by adding ".tmp"
on the filename.
The AbstractFtpHandler::fileReceived() callback could then be modified to create a workflow order instance for the file's data if necessary (for example).