Qorus Integration Engine®  5.1.19_git
System Architecture

Introduction

Qorus Integration Engine® provides a framework for the reliable operation of business-critical interfaces. From a high level, the Qorus Integration Engine® server is made up of a set of processes backed by metadata and status information stored in a database schema, communicating with other systems and clients through the network or the enterprise message bus as follows:

Qorus High Level External Architecture

System Architecture Overview

The following figure gives a high-level overview of the internal system architecture:

Qorus Integration Engine® System Architecture Overview

The Qorus system provides a workflow logic cache, a workflow API, and a workflow execution engine, a service loader, a service API, a job engine and API, many other APIs, and a multithreaded HTTP server servicing REST, YAML-RPC, XML-RPC, JSON-RPC, and HTTP requests, among others. Because the REST, YAML-RPC, XML-RPC, and JSON-RPC handlers export all service methods automatically as well as built-in system API methods, user services can be considered run-time extensible and run-time upgradeable API providers as well.

The APIs made available by the Qorus system are listed in the following table.

Qorus APIs

API Type Description
Workflow API Internal Set of functions, objects, classes, etc exported for use by workflow code
Service API Internal Set of functions, objects, classes, etc exported for use by services
Job API Internal Set of functions, objects, classes, etc or use by job code
Mapper API Internal Set of functions, objects, classes, etc for use by mapper code
System Service API Internal and External Set of services delivered with the system providing additional functionality to Qorus
REST API Qorus system network API

=

Note
Service methods are exported through the system API by default (see Service Method Internal Flag). Because these methods are available both internally and externally, they can be verified and tested separately from dependent interface components. Exported service methods can also be called from the network or from the command line (using the qrest program) or other programs using REST or other protocols.

Qorus relies on its system schema to store most of its configuration information, some system logic (in system services), and all user configuration.

The following table outlines some of the data that is stored in the system database.

Qorus Database Configuration

Data Description
Workflow Metadata The description of a workflow (name, version, steps, dependencies, etc) and all workflow logic written in Python, Java, or Qore using the workflow API
Service Metadata The description of a system or user service (method names, version, etc) and service method logic written in Python, Java, or Qore and using the service API
Job Metadata The descripion of time-based jobs and job logic using the job API
Workflow Order Data Workflow order data instances, the data and processing status

Qorus is a highly multithreaded integration platform. Due to the fact that much of the system code is stored in a database, and can be "refreshed" (i.e. old version deleted from the cache, new version read in from the database and started) while the system is running, not only can use logic be upgraded without shutting the system down, parts of the workflow system itself (system services) can also be upgraded without shutting the system down.

The majority of the additional programs included with Qorus are client programs to the integration engine. They communicate with the Qorus server either via the HTTP server using REST, YAML-RPC, XML-RPC, or JSON-RPC protocols or via ZeroMQ for clients using the high-performance encrypted network cluster API.

Qorus features a built-in multithreaded HTTP server which functions as the system's primary interface to the outside world and serves various protocols as well as HTTP/S.

All system activities happen in a set of processes that manage many concurrent threads to get their work done.

Qorus Directory Layout

For self-contained application installations; the Qorus Integration Engine® application directory has a structure as given in the following table.

Qorus Integration Engine® Self-Contained Application Directory Layout

Directory Description
$OMQ_DIR/bin System binaries are located here
$OMQ_DIR/etc System configuration files
$OMQ_DIR/examples/client-lib Example programs using the qore language client-library
$OMQ_DIR/examples/qore Example programs in the qore language
$OMQ_DIR/examples/user Example Qorus user code definitions
$OMQ_DIR/jar Qorus Java files
$OMQ_DIR/lib System and included third-party library files
$OMQ_DIR/qlib Qore-language include file directory
$OMQ_DIR/python Qorus Python modules
$OMQ_DIR/releases System load files
$OMQ_DIR/system System service definitions
$OMQ_DIR/templates Miscellaneous template files
$OMQ_DIR/user Qorus user definition files, code objects, interface descriptions, release source

Configuration Files

Contents of this section:

System Options File

The options file contains system and client options. The location of the file depends on the installation type:

  • $OMQ_DIR/etc/options for tar installations

Qorus System options have the format:

  • qorus.option = value

Example:

qorus.auto-recover = true 

Also a colon may be used instead of an = as in:

qorus.auto-recover: true 

Any text after a # character is assumed to be a comment.

One of the most important option that should be set before the system will start is instance-key. This must be a unique identifier for the instance that will enable the system to avoid accidentally starting the same instance more than once and will also allow the system to recover the application session gracefully errors should the system ever terminate abnormally. The default value for this key is "qorus-test-instance".

Example qorus.instance-key option setting:

qorus.instance-key: bss-uat-1 

Additionally, if the log files for Qorus should be written to a directory other than the default (the default for tar installations is $OMQ_DIR/log), the qorus.logdir option must be set to the directory where Qorus log files will be written. This directory must be writable by the user running the Qorus server:

Example qorus.logdir option setting:

qorus.logdir: /var/log/qorus

Another key of interest in the initial installation is http-secure-server:

Example qorus.http-secure-server option setting:

qorus.http-secure-server: 192.168.20.77:8085{cert=$OMQ_DIR/etc/cert1.pem,key=$OMQ_DIR/etc/key1.pem}

If no qorus.http-secure-server or qorus.http-server options are set, the default for the HTTP server value is 8001, meaning that Qorus will listen on all interfaces on port 8001 for unencrypted requests, and any APIs requiring encrypted client connections will be unavailable.

See also

Qorus client options have the format:

  • qorus-client.option = value

Example:

qorus-client.proxy-url = http://user:pass@192.168.25.7:8080 
See also
Qorus Client Domain Options for information on client options.

Obsolete System Files Migrated to the Database

Upgrade Migration Info: System Datasource (dbparams) File

As of Qorus 4.0 the dbparams file is no longer used by Qorus; all system datasources are handled as datasource connection objects instead.

When upgrading a per-4.0 version of Qorus, datasources defined in any existing dbparams file are automatically migrated to the system database and created as datasource connection objects.

See also
Datasource Connections

Upgrade Migration Info: Remote Qorus Instance Configuration File

As of Qorus 4.0 the remoteconnections file is no longer used by Qorus; all Qorus remote connections are handled as Qorus remote connection objects instead.

When upgrading a per-4.0 version of Qorus, Qorus remote connections defined in any existing remoteconnections file are automatically migrated to the system database and created as Qorus remote connection objects.

See also
Qorus to Qorus Connections

System Startup

Once launched, the system initializes itself with the following steps:


Step 1: Parse Options File

Parse System Options File; if any syntax errors are found, information is displayed and the program exits.

See also
System Options for detailed descriptions of all system options.

Step 2: Open System Datasources

Open System Datasource "omq". If the system "omq" datasource cannot be opened, then error information is displayed and the program exits.


Step 3: Open Log Files

If the log files cannot be opened, an error message is displayed and the program exits.


Step 4: Initialize RBAC Framework

users, roles, permissions, and interface groups are read in from the database. If RBAC security is enabled and no single user has both OMQ::QR_LOGIN and OMQ::QR_SHUTDOWN permissions, an error message is displayed and the system will refuse to start.


Step 5: Open Application Instance Session

If any errors are encountered, information is logged, and the program exits. The session is recovered here if system option qorus.auto-recover is True.

See also
Application Session Model for more information about opening an application session and session recovery.

Step 6: Initialize Services

Services with the autostart flag set are started automatically. Any errors auto-starting services are logged, but do not otherwise affect system startup.


Step 7: Start HTTP Server

If any errors are encountered, information is logged, any loaded services are deleted, and the program exits.


When the HTTP server is started, listeners for each entry in the System Options File (or as overridden on the command-line) will listen for requests on their respective interfaces and ports. To start and stop workflows and services, to shut down the system, to change options, etc, the appropriate system API calls must be made, normally as properly formatted REST, YAML-RPC, XML-RPC, or JSON-RPC commands sent to Qorus' HTTP server. For more details, please see HTTP Server.

System startup raises the following system event: SYSTEM_STARTUP

System Shutdown

If the shutdown message is received, the system performs the following steps:


Step 1: Stop All Workflows

All running workflow execution instances are stopped. While system shutdown is in progress, it is not possible to start new workflow execution instances.


Step 2: Stop all Jobs

All jobs are stopped.


Step 3: Stop and Unload All Services

All system and user services are stopped and unloaded from the service cache.


Step 4: Stop HTTP Server

Any requests in progress must complete before the server can be stopped. After this point, no external communication is possible with the Qorus system.


Step 5: Close Application Session

The application session is marked as closed in the system database.


Step 6: Close System Log Files

System log files are closed.


System shutdown is the last system event raised by the system; the event raised is SYSTEM_SHUTDOWN.

Role Based Access Control

Qorus Integration Engine® supports a security model based on Role Based Access Control (RBAC). In this model, users are assigned roles that have permissions associated with them that allow certain actions or types of actions to be performed in the system.

Qorus security is disabled by default, it is enabled by setting the qorus.rbac-security option to True in the Qorus System Options File. Also see the qorus.rbac-force-user option.

RBAC Concepts

RBAC Concept Description
user An authorized login to the system
role A security configuration that can be associated to users
permission Permissions are associated with roles that allow actions to be performed on the system
interface group An interface group contains lists of workflows, services, jobs, mappers, and value maps; associated interfaces can be enabled or disabled for operational reasons by enabling or disabling the interface group, and the group can be applied to roles to limit access to those objects through the system API.

When RBAC security is disabled, no authorization or authentication is necessary to call any functionality in Qorus. When RBAC is disabled, unauthenticated users have all possible user and system permissions.

When RBAC security is enabled, only users with a valid username and password may connect to the Qorus HTTP server; i.e. HTTP authentication is required; the HTTP username is the Qorus RBAC user, and the password is the user's password. Each user will have a set of assigned roles; the roles assigned determine what functionality is available to the user.

If a certain action requires one or more permissions, and the user does not have any roles with the required permission or permissions, an AUTHORIZATION-ERROR will be returned to the caller and the action will not be performed.

If a user does not have a role with the DEFAULT group, then access to workflow, services, jobs, mappers, and value maps (including associated configuration information and data) will be restricted to the objects in the interface groups that the user's roles have. In this case, calls to API methods will filter out any workflows, services, jobs, mappers, and value maps not in the interface group list, and any attempt to directly access an object (including configuration and data) for objects not in the interface group list will result in an exception being thrown.

See also

RBAC security only applies to communication through the Qorus HTTP server. The following components of Qorus are not subject to RBAC security because they use direct database access (and therefore can be used when the Qorus server is offline but the database is up):

Qorus Tools/Components Not Subject to RBAC Security

Tool/Component Description
oload Qorus Integration Engine® system database object manager and loader
user-tool Command-line, offline (database-only) RBAC manager (create, change, and delete users, roles, permissions, and interface groups)
schema-tool Qorus Integration Engine® schema manager
Note
RBAC security is normally not applied to internal API calls (unless explicitly applied with callNetworkAPIArgsWithAuthentication() for example)

RBAC Users

Each user has a name, password, and a list of roles. Users are identified by name and do not have an ID. A user's roles determine its permissions and interface groups.

A particular user's permissions and interface groups are calculated by combining all the permissions and interface groups for all of its roles.

If a user has at least one role with the DEFAULT interface group, then it will have unrestricted access to all workflows, services, and jobs.

RBAC Roles

Roles represent security configurations used to provide capabilities to users. They define a set of permissions and a list of interface groups to be used when creating new users. Roles may contain any number of system and/or user permissions and interface groups associated to them.

The following system roles are delivered with the system:

Note
Qorus is delivered with a set of default roles that are updated in each schema upgrade. System roles should not be changed since any changes will be overridden in the next schema upgrade or schema alignment operation.

RBAC Permissions

Permissions are associated to roles and enable actions or class of actions to be performed in Qorus. System permissions pertain to actions in Qorus, user permissions are only checked by Qorus user code. System permissions cannot be changed to deleted. Permission names are unique; a user permission cannot have the same name as a system permission and vice-versa.

See also
RBAC Permissions for a list of all system permissions supported by Qorus.

Users permissions can be created and access can be checked manually in user services by calling one of the following methods:

Interface Groups

An interface group is contains a list of workflows, user services, jobs, mappers, value maps, finite state machines, and data pipelines; groups can be enabled and disabled, and roles can have a list of groups they can access.

Interface groups service the following functions:

  • they allow larger Qorus installations the flexibility to restrict access only to data and services needed by particular operations or business personnel (ie permit a multi-tenant configuration of Qorus)
  • they allow interfaces to be grouped logically
  • they allow interfaces to be temporarily enabled or disabled as a group for operational reasons

If a group is disabled, then no workflows, services, or jobs in the group can be started; if a group is disabled while member workflow execution instances, services are running or loaded, or jobs are active, then they will be immediately stopped when the group is disabled.

When an interface group is enabled, then all workflows with a non-zero autostart value are started, all services with their autostart flag set are loaded, and all active jobs in the group are reactivated (assuming they are not also members of another disabled group or they do not have connection dependencies that prohib them to be started).

Note
  • Access to system services cannot be restricted through membership in interface groups; access to system services can only be controlled through granting or removing the permissions OMQ::QR_CALL_SYSTEM_SERVICES_RO and OMQ::QR_CALL_SYSTEM_SERVICES_RW from a user's role
  • The "DEFAULT" group cannot be disabled or deleted

DEFAULT Interface Group

The "DEFAULT" group (OMQ::DefaultGroupName) with ID 0 (OMQ::DefaultGroupID) is a special group in that it automatically grants access to all workflows, services, jobs, mappers, value maps, Finite State Machines, and data pipelines.

By default Qorus roles have access to the "DEFAULT" group, so that all roles provided with Qorus provide unrestricted access to all interfaces in Qorus.

LDAP Authentication and RBAC Integration

Qorus can authenticate and retrieve user and role information from an LDAP server using the qorus.rbac-external option with the "QorusLdapAuth" argument.

In this case additional options need to be set the options file as in the following table.

LDAP Options

Option Mandatory? Default Example Description
ldap.uri Yes n/a ldaps://localhost:1389 Sets the URI for the LDAP server (use "ldap" for non-encrypted connections, "ldaps" for encrypted connections)
ldap.base Yes n/a dc=example,dc=com Sets the base DN for searches
ldap.directbindprefix No n/a domain-name\ This string will be prefixed to any user name given for the bind DN
ldap.directusername No False True If True then the username given will be used for authentication and lookups against the LDAP server instead of creating a bind DN
ldap.roleattr No n/a orclmemberof The user attribute giving the roles the user has
ldap.roleregex No n/a ^cn=([-a-z0-9_]+) The regular expression extraction pattern for extracting the roles from the roleattr attribute; note that a caseless match is performed by using RE_Caseless in the internal call to regex_extract()
ldap.ubase No "ou=people" ou=staff Sets the user base for user information, this and the ldap.base argument will be used to create the bind DN
ldap.uentry No "uid" cn Sets the attribute to be used to identify users

Simple example option configuration for LDAP authentication:

qorus.rbac-security: true
qorus.rbac-external: QorusLdapAuth
ldap.uri: ldap://localhost:1389
ldap.base: dc=net,dc=internal

More complex example option configuration for LDAP authentication with Active Directory:

ldap.uri: ldap://192.168.231.18
ldap.base: dc=enterprise,dc=local
ldap.directusername: true
ldap.directbindprefix: enterpriseldap.roleattr: memberof
ldap.roleregex: ^cn=([-a-z0-9_\.]+)
ldap.uentry: samaccountname

System Auditing

Qorus supports writing out database records for system and user events; this solution is called system auditing; records are written to the AUDIT_EVENTS table.

System auditing is enabled by setting the qorus.audit system option. Audit events can be used to keep a security record of actions taken in the system, or to explicitly track KPIs (Key Process Indicators) for SLA tracking, for example (see the audit_user_event() function, for example).

The qorus.audit system option accepts the codes in the following table (corresponding to Audit Options).

Audit Option Codes

Code Description
alerts Setting this option will cause alert events to be written to the AUDIT_EVENTS table
api Setting this option will cause every api call that makes changes to be audited in the AUDIT_EVENTS table
groups Setting this option will cause group status change events to be written to the AUDIT_EVENTS table
jobs Setting this option will cause job start and stop events to be audited
job-data Setting this option will cause job instance start and stop events to be audited (each time a job is executed an event will be written to AUDIT_EVENTS when the job starts and then when it stops)
oload Setting this option will cause oload to write audit events when source code is loaded into the system schema
services Setting this option will audit service start and stop events
system Setting this option will audit system startup, shutdown, and recovery events
user-events Setting this option will cause every call to audit_user_event() to be written to the AUDIT_EVENTS table; if this audit option is not set, then calls to audit_user_event() will be discarded
workflow-data Setting this option will audit workflow status changes; note that this will have a performance impact as each change to a workflow order's status will cause a row to be written in the AUDIT_EVENTS table
workflows Setting this option will audit workflow start and stop events
Note
If an event is not audited, then it is neither written to the database nor logged in the audit log file.

Audit Events

Audit events are written to the AUDIT_EVENTS table; each audit event has attributes as in the following table:

Audit Event Attributes

Column Type Description
audit_eventid number a unique number assigned to the event (created from a database sequence)
[related_audit_eventid] number an optional event id that this event is related to
[workflowid] number if the event is related to a workflow, the workflowid
[workflow_instanceid] number if the event is related to a workflow instance, the workflow instance id
[stepid] number if the event is related to a workflow step, the stepid
[ind] number if the event is related to a step, the step index number (always 0 for non array steps)
[jobid] number if the event is related to a job, the jobid
[job_instanceid] number if the event is related to a job instance, the job instance id
[serviceid] number if the event is related to a service, the serviceid
audit_event_code number a code giving the audit event type; see Audit Event Codes for possible values
[audit_user_event] string for user events, the user event code
[reason] string a reason for the event (ex: if due to an API call, then the API name, ex "REST PUT api/latest/services/my-example-service/method?action=call", if due to normal system processing, then the string "internal call")
who string the RBAC user who initiated the action that led to the event, or "Qorus" if due to normal system processing or no username is available
source string the source of the call that led to the event (ex: "source: ipv6[::1]:50653 listener: ipv6[::]:8001") or "system" if due to normal system processing
[info1] string an informational string related to the event
[info2] string an informational string related to the event
created date/time the date and time the event was created in the database

System Events

Qorus includes an event sink that maintains a list of system events. System events can be read internally and externally through the system API.

The system option qorus.max-events controls the maximum number of system events the system will store.

System events are identified by an event class and an event code, and each event also contains addtional information describing details of the event. Note that there is a special system event, USER_EVENT, that is used for all user events; user events are differentiated by their custom content as supplied by the user code when the event is raised.

The following give some links to additional information about events:

Database Transaction Model

State Machine

Basically the Qorus Integration Engine® system can be considered a workflow state machine that uses the Oracle database to store the workflow state data.

After every atomic transaction any changes to the database are immediately committed. By design there are no cases where state information relevant to state recovery after a crash is internally queued in the system and committed later. While this can have a negative performance impact, it reduces the impact of system crashes and makes complete system/workflow recovery possible (as long as the database remains intact).

Note
Never update the Qorus system schema directly using SQL; always system APIs. Any inconsistencies in the system schema can lead to data corruption, deadlocks, or other errors. Additionally any row, table, or other locks created manually when Qorus is running can lead to deadlocks; never lock any database object using SQL in the system schema while the system is running.

Access to the system schema where transaction management is required is made through a connection pool, where connections are automatically assigned to threads as needed. Connections are acquired when the first statements that make changes to the database are executed and released to the pool when each thread does a commit or rollback on the connection.

Recovery Model

Due to the system's design, the worst-case scenario for a system crash is that workflow order data instances (i.e. also segment and step instances) will incorrectly have an OMQ::StatInProgress status in the database after the crash. That means that the information loss is well defined and limited to the bottom level of the workflow order data instance; the steps with an OMQ::StatInProgress status may have been subject to information loss.

When a crashed session is recovered, all workflow order data instances, segment instances, and step instances with an OMQ::StatInProgress are modified to OMQ::StatRetry and the workflow order data instance rows (the parent object) are cleared of the crashed session's ID.

A well-defined workflow will have recovery logic where necessary that will verify the status of any errored step before repeating the step's actions (by defining validation logic in the step). In this case, the workflow developer can ensure the automatic recovery of the step with no data loss after a system crash.

HTTP Server

Qorus includes a built-in HTTP server providing a gateway to internal protocol handlers. To better support compression with large messages, the HTTP server understands and can generate "deflate" (RFC1951), "gzip" (RFC1952), and "bzip" content-encodings and will respect client preferences as transmitted in the "Accept-Encodings" HTTP header.

When Role Based Access Control is enabled, every request to the HTTP server must include HTTP authentication; the HTTP username is the Qorus RBAC user, and the password is the user's password (however unauthentication connections can be made from certain hosts if the qorus.rbac-force-user option is set).

A new thread is launched to handle each incoming request so the system can handle many requests simultaneously.

Qorus supports the following top-level paths by default (note that requests are also matched by Content-Type if applicable):

Qorus HTTP Server Protocol Handlers

Path Content-Type Handler Description
/YAML application/x-yaml YamlRpcHandler Handles YAML-RPC requests; acts as a YAML-RPC gateway to the system API
/RPC2 text/xml XmlRpcHandler Handles XML-RPC requests; acts as an XML-RPC gateway to the system API
/JSON application/json JsonRpcHandler Handles JSON-RPC requests; acts as a JSON-RPC gateway to the system API
/SOAP text/xml or application/soap+xml SoapHandler Handles SOAP requests for user services exported with ServiceApi::registerSoapHandler()
/api application/x-yaml, text/xml, application/json REST Implements the system REST API
/apievents n/a WebSocketHandler Handles requests for system events over the RFC-6455 WebSocket protocol
/creator n/a WebSocketHandler Handles requests for the creator API with the RFC-6455 WebSocket protocol
/log n/a WebSocketHandler Handles requests for log file events over the RFC-6455 WebSocket protocol
/remote-command n/a WebSocketHandler Handles requests for remote commands over the RFC-6455 WebSocket protocol
/webdav n/a FsWebDavHandler Handles requests for WebDAV clients

Any requests not matched with a protocol handler are handled by the default handler, which serves the system web UI.

WebDAV Server

Remote clients can access the $OMQ_DIR/user filesystem using the WebDAV protocol.

In order for a user to access the $OMQ_DIR/user filesystem using the WebDAV protocol, users must have one of the following permissions (in addition to the LOGIN permission needed to login to Qorus):

  • WEBDAV-CONTROL: provides all possible WebDAV permissions for filesystem access and operational permissions; allows reading, writing and deleting of all files and directories under $OMQ_DIR/user
  • WEBDAV-RWD: allows reading, writing, and deleting of accessible files and directories
  • WEBDAV-ACCESS-ALL: allows access to all files and directories under $OMQ_DIR/user but does not provide any operational permissions
  • WEBDAV-READ: allows reading accessible files and directories
  • WEBDAV-WRITE: allows writing (creating and updating) accessible files and directories
  • WEBDAV-DELETE: allows deleting accessible files and directories

If a user does not have either WEBDAV-CONTROL or WEBDAV-ACCESS-ALL, then accessible files and directories depend on the users interface groups; additionally the directory under $OMQ_DIR/user having the same name as the user is always accessible as well.

Directories with the same name as interface groups that the user has associated with their account are also accessible.