Qorus Integration Engine®  4.0.2_git
Release Notes

Qorus Release Notes History

Latest version: Qorus 4.0.2

Qorus 4.0

Release Overview

This is a major release of Qorus with the following major changes:

  • An Application Cluster Implementation
    Separating Qorus interfaces into separate processes brings much improved system reliability and resiliency. Communication between cluster elements is provided by the ZeroMQ library, which provides a high-performance distributed queue layer for reliable information exchange in Qorus. Private network communication is encrypted with an elliptic curve encryption scheme which provides high performance and high security. By isolating interfaces into separate processes and by monitoring and restarting processes when they terminate unexpectedly, the overall stability of the system has been dramatically increased.
  • Support for Reusable Solutions Controlled by Users
    Interface module support allows for new high level integration objects to be defined based on Qorus's fundamental integration objects; these objects can be developed, packaged, and delivered as generic reusable solutions for specific integration challenges or as products on top of the Qorus Integration Engine platform. Interface object configuration item support enables end users to control the runtime behavior of Qorus interfaces and to eliminate the need for development for such changes.
  • Native Java Integration
    Qorus 4.0 introduces complete support for Java, allowing Qorus interface code to be developed directly in Java. See Developing in Java for more information.
  • Updated System Requirements Support for operating systems other than Linux is now only provided through Qorus in Docker containers. See Minimum System Requirements for updated system requirements.
    Note
    Qorus 4.0 will not start interfaces if runtime values of tunable system parameters exceed a threshold of system-defined soft limits; see Minimum System Requirements for more information

Important Upgrade and Backwards Compatibility Information

  • Cluster Functionality
    • Interfaces can now be configured to run in remote mode for overall increased platform reliability at the expense of higher memory usage and a performance penalty regarding intracluster communication. To update interfaces to run internally in qorus-core as with previous versions of Qorus; the following new REST APIs can be called:
    • Running interfaces in remote mode has the following advantages:
      • Bugs in interfaces that could lead to a crash limit the impact of that bug to the interface process and do not affect other interfaces.
      • Memory leaks that cause an interface process to exceed the maximum process memory threshold will cause the errant process to be killed by the Qorus master process, limiting the affect of the memory leak to only the specific interface (see the max-process-memory-percent option).
      • Badly-written interfaces that deadlocks or will not exit or shut down can be killed using the REST API from the command line or the system UI.
      Note that oload will set the remote value to True for any interface that is loaded after the upgrade without an explicit remote option. This behavior can be changed by setting the qorus-client.remote option in the System Options File
    • New programs:
      • qctl: cluster control program
      • qdsp: distributed datasource pool processes
      • qjob: distributed job interface processes
      • qorus-core: central Qorus cluster process handling global services, interface shared state, and non-clustered interfaces
      • qsvc: distributed service interface processes
      • qwf: distributed workflow interface processes
    • Updated programs:
      • qorus: cluster master process; the primary supervisor process for the Qorus cluster on the node where it runs; this process provides the same interface for starting / recovering a Qorus application instance as in previous versions for backwards compatibility
    • New system options:
    • Workflow behavior changes:
      • when resetting a workflow running in a workflow process, the process is stopped and restarted, and after the restart, all running workflow execution instances have new execution IDs
    • Logging updates:
      • log levels are controlled by logger objects and level settings on each logger; the qorus.verbose option has been deprecated and is only used for migrating an indication of the log level to each logger
      • cluster process log files are found in the "cluster" subdirectory of the log directory
      • all stdout and stderr logs from all cluster processes are also redirected to their cluster log files (this includes qorus which no longer prints any output to stdout or stderr while running)
    • Interface definition changes
      • job parameters in job definition files now support the "remote" attribute to define if a job should run in a remote qjob process or not
      • service parameters in service definition files now support the "remote" attribute to define if a service should run in a remote qsvc process or not
      • workflow parameters in workflow definition files now support the "remote" attribute to define if a workflow should run in a remote qwf process or not
    • New system event class and events:
    • System schema table additions:
      • CLASS_DEPENDENCIES
      • CLUSTER_PROCESSES
      • CONNECTION_TYPES
      • GLOBAL_CONFIG_ITEMS
      • JOB_CONFIG_ITEMS
      • JOB_GLOBAL_CONFIG_ITEMS
      • JOB_INSTANCE_STATS
      • JOB_INSTANCE_STATS_STAGE
      • SERVICE_AUTH_LABELS
      • SERVICE_CONFIG_ITEMS
      • SERVICE_GLOBAL_CONFIG_ITEMS
      • STEP_CONFIG_ITEMS
      • WORKFLOW_GLOBAL_CONFIG_ITEMS
    • Only one Qorus instance can be running on a Qorus schema at any one time
  • Class-Based Development From Qorus 4.0 onwards, it is not recommended to use function-based interface development, as all new features and functionality will be implemented in the new class-based APIs; see:
  • Connection Updates
  • Workflow Changes
  • System Option Changes
    • max-events: the default value for this option has changed from 100,000 to 10,000 to reduce memory consumption
    • compat-ignore-empty-order-key: this option will cause Qorus to ignore empty workflow order keys instead of throwing an exception
    • compat-jni-types: this option will disable the more complete and robust automatic data type conversions supported in the updated version of the jni module which was introduced to provide much improved Java integration in Qorus 4.0. This option is recommended to be set for compatibility reasons when upgrading Qorus instances with existing interfaces that make use of the jni module. If this global option is set, it can be disabled in individual interfaces by including %module-cmd(jni) set-compat-types false in Qore code to reenable full automatic type conversions; see JNI Module Compatibility Options for more information.
    • Deprecated the qorus.flush-status option: the only functional effect of enabling this option was a loss of performance, so it has been made non-functional (its value is ignored) and has been deprecated
    • Deprecated the qorus.verbose option: log levels are controlled by logger objects and level settings on each logger; the qorus.verbose option is only used for migrating an indication of the log level to each logger
    • Deprecated the qorus.workflow-params option: this option was complex, generally unused, and better covered by custom error configuration, so it has been made non-functional (its value is ignored) and has been deprecated
    • The info-snapshot option has been removed as workflow order and job result information is always kept up to date in Qorus and the snapshotting mechanism is no longer necessary
    • The dbparams-file option has been removed. DB connections are stored in the system DB.
    • The remoteconnections-file option has been removed. Qorus remote connections are stored in the system DB.
    • The qorus.systemdb option is a new and mandatory option containing a connection string to the system DB. It is migrated automatically from an existing dbparams file when Qorus 4.0 is installed over an existing Qorus installation.
  • System Service Changes
  • REST API v3
    • /api/v3/...: the REST API has been extended and a new version is now available under this URI path prefix; see the previous link for changes in REST API v3
    • the REST API has been updated to return 403 Forbidden responses for authentication and related errors where appropriate
  • Snapshots for workflow and job instances are integral part of Qorus server
    • The info-snapshot mechanism is now fully integrated into the Qorus process and it does not need periodical refreshing. The option is obsolete and has been removed.
  • User Connections
    • All types of connections (user, Qorus remote, and datasource) are stored in the system database. Connections are migrated automatically from existing configuration files when Qorus 4.0 is installed over an existing Qorus installation.
  • RPC API Changes
  • TAR Installation Changes
    • The qore binary delivered with TAR packages no longer sets the module path automatically; to ensure correct operations after upgrading with a TAR package, make sure and set your QORE_MODULE_DIR environment variable accordingly; see Configure Environment Variables for more information

Additional New Features and Changes

Bug Fixes in Qorus 4.0

Issue ID Severity Description
2926 Normal fixed bugs in qctl: Qorus Cluster Control Process killing and removing processes with the qctl killall command
2920 Normal fixed a bug in creating workflow orders with multiple unique keys
2811 Normal fixed a bug in oload where the active attribute of jobs was not updated when existing jobs were reloaded
2649 Normal fixed a bug in the REST API where it was not possible to create a workflow-specific error without first creating a global error with the same name
2638 Normal fixed a bug in the REST API where arguments given in the URI path were sometimes ignored
2540 Normal fixed a bug where session recovery could deadlock if the connection to the server could not be established and did not time out
2500 Normal the active key of the PUT /api/v3/jobs/{id_or_name}?action=setActive response now reflects the actual job active state
2448 Normal fixed a bug clearing user storage with the REST API
2434 Normal fixed an error in the wf_get_step_info() function documentation regarding the desc key which was previously incorrectly documented to be the description key
2305 Normal workflow errors and warnings with a non-string info argument no longer result in an internal error being raised
2300 Normal fixed a bug where the the error_count and warning_count values were missing on systems using a PostgreSQL DB for the system schema
2225 Normal fixed a bug where the onetimeinit function was executed twice for synchronous workflow orders

Bug Fixes in Qore

See Bug Fixes in Qore

See also

Qorus 4.0.1

Release Overview

This is a bugfix release of Qorus and also introduces APIs for writing interface tests in Java.

Important Changes for Developers

To run tests, the qorus-client.allow-test-execution option must be set to true in the System Options File or tests based on the QorusInterfaceTest module will no longer run; ex:

echo qorus-client.allow-test-execution: true >> $OMQ_DIR/etc/options 

New Features

Bug Fixes in Qorus

Issue ID Severity Description
2975 Normal fixed a race condition in qsvc process recovery with active HTTP handlers where HTTP requests received while the processes was recovering could result in spurious errors
2968 Normal fixed a bug removing all roles from a Qorus user
2962 Normal fixed a bug where global configuration items were not deleted by oload when there were no more references to it
2959 Normal fixed a bug rotating log files
2954 Normal fixed a bug in the UI where global config items were displayed for interfaces that do not access them
2948 Normal fixed a bug where the UI would not allow BLOCKED orders to be canceled and for CANCELED orders to be updated to BLOCKED
2947 Normal fixed a bug displaying workflows in the UI loaded at runtime
2946 Normal fixed a bug where external RBAC auth modules that required QorusClientCore could not be loaded
2941 Normal fixed a bug handling the CANCELED status for errors
2940 Normal fixed a bug where workflow action synchronization when resetting new workflows that were not previously cached was missing
2937 Normal fixed a bug where workflow status change events were not emitted for all order changes
2935 Normal fixed a bug in the UI handling sensitive data updates
2929 Normal added APIs to allow system loggers to be modified
2927 Normal fixed a bug where oload did not refresh the map when global config items were updated
2926 Normal fixed a bug in qctl killall where the the operation would fail if there was a program starting
2920 Normal fixed bugs in workflow order creation APIs when multiple unique keys were passed
2919 Normal fixed a bug in the UI where workflow actions were hidden when the UI sidebar was expanded
2918 Normal fixed qdsp process termination recoveries in client processes
2917 Normal fixed bugs in the UI handling service breadcrumbs
2915 Normal fixed a bug where the workflow order list page showed incorrect workflow execution instance counts
2913 Normal fixed issues related to quick search boxes in the UI
2906 Normal fixed a bug in the UI where the config tab was not available in the workflow order page
2905 Normal fixed setting config items values with an empty string to be handled as no value
2903 Normal fixed a server-side type error in workflow order step information in a REST API that would cause the UI to go blank after making a workflow order retry in some cases
2899 Normal fixed service authentication label support and documentation
2886 Normal fixed a bug with the remote deployment APIs deploying Java-based interfaces
2878 Normal fixed a bug where interfaces were deleted from the DB even if the network request to disable the interface first is rejected due to authorization errors
2877 Normal added new APIs and UI elements to make it easier to update loggers and appenders
2873 Normal fixed a bug where logger appender updated would not take effect before the interface was reset
2869 Normal fixed a race condition in qorus-core starting qorus
2866 Normal fixed a bug sending websocket events for system logs
2859 Normal fixed a bug changing the log level of loggers with the REST API
2858 Normal fixed a bug where API commands to stop workflows would appear to fail with a type error when the action actually succeeded internally
2836 Normal fixed a bug in cluster restarts with qorus-core failures
2790 Normal fixed a bug where loopback connections were being returned in the response to GET /api/v3/system/health in the remote key
2758 Normal fixed a bug when there is no 'next run' value displayed in the admin UI
2617 Normal added a Java API for testing
2302 Normal fixed a bug where the cluster would be left in an inconsistent and non-responsive state if the qorus-core process terminated prematurely when shutting down the cluster

Bug Fixes in Qore

See Bug Fixes in Qore

Qorus 4.0.2

Release Overview

This is a bugfix release; for details, see the issues addressed below.

Important Upgrade and Backwards Compatibility Information

  • A bug fix to log handling required any existing logger information to be reset to the default configuration; instances upgrading from Qorus 4.0 or Qorus 4.0.1 will have their logger information reset by this update.
  • All logging file name template options have been deprecated as they no longer have any effect; log file names for appenders that generate log files are generated based on the template string in the specific appender and not based on any system option. Appender configurations can be updated by authorized users with the REST API or the system UI.

Bug Fixes in Qorus

Issue ID Severity Description
3008 Normal fixed a bug in oload deleting workflow order instances with sensitive data
3004 Normal fixed a bug in oload: Data Loading Tool and Schema Manager where global configuration items won't be deleted from the database (the workflow_global_config_items table) if they've been removed from a step previously loaded to Qorus
3002 Normal added the DefaultQorusRBACAuthenticator and DefaultQorusRBACBasicAuthenticator classes to the service API; added docs
3001 Normal fixed logging issues where logging was not enabled until later in process initialization time and log file name patterns were not respected
3000 Normal fixed a bug in ServiceApi::registerSoapListeners() in qsvc processes
2999 Critical worked around a bug in libzmq where socket descriptors would leak at runtime causing Qorus to stop responding under heavy load and resulting in data corruption in the system DB schema
2997 Normal fixed a bug where log file names did not reflect the correct instance name
2994 Major fixed a bug where interface logging did not respect the log level in all cases
2988 Normal fixed a bug where the UI would not display boolean values in system options correctly
2987 Critical fixed a bug where qdsp processes could hang for long periods of time due to an error where potentially expensive I/O operations were performed while holding a critical lock
2981 Normal fixed a bug where updating a global config item would cause a new entry to appear in the UI
2980 Major fixed a bug where delays in connection monitoring actions caused excessive contention for updating connections
2541 Normal fixed a bug in oload to disallow creating objects with non-alphanumeric characters in their names (dashes and underscores are allowed); also fixed in the REST API