Qorus Integration Engine®  5.1.22_git
Release Notes

Qorus Release Notes History

Latest version: Qorus 5.1.22

Qorus 5.1

Release Overview

This is a major release of Qorus with the following major changes:

  • Dynamic Java API Support
    • All Qore and Qorus APIs are now available in Java thanks to dynamic Java API support; bytecode for wrapper classes allowing Qore and Qorus APIs to be called from Java are automatically created with appropriate import statements; oload has been updated to use a custom Java compiler that supports compile-time generation of bytecode as well as runtime support in the custom classloader solution used in Qorus; see Developing in Java fore more information.
      As of Qorus 5.1, all Qorus APIs are equally available in all supported programming languages; Java can be used to create HTTP/REST/FTP/WebSocket handlers and any other functionality that was only available to Qore and Python in previous releases.
  • Autoscaling for Qorus
    • Qorus (qorus master) and stateless services (qsvc processes) now support autoscaling when Qorus is run under Kubernetes; new system and service options support autoscaling controls from the system API and the Qorus operational UI. This enables fault-tolerance, load-sharing, and unlimited horizontal scalability for CPU- and memory-bound solutions, as Qorus and stateless services can scale to any limits in a public or private cloud environment under Kubernetes.
  • Stateless Services
    • Stateless services enable true scalable microservice functionality for Qorus, allowing a single Qorus service to run in an arbitrary number of processes / containers for load balancing and fault tolerance; stateless services can implement any functionality of non-stateless services including methods as well as protocol handlers; if a call to a stateless service fails because the service is unavailable, a transparent failover is made to another service to ensure reliability and scalability in public and private cloud environments. Furthermore, when Qorus is run under Kubernetes, stateful sets are created and destroyed automatically for Qorus stateless services; limits and autoscaling paratemeters can be set indepedently for each stateless service
  • WebDAV Server Support
    • The $OMQ_DIR/user filesystem is exported to WebDAV clients automatically under the webdav path using the Qorus URL; see WebDAV Server for more information
  • Support for Alpine Linux and ARM Processors
    • Qorus 5.1 adds support for Alpine Linux to provide the smallest possible images for microservice deployments
    • Support for 32- and 64-bit ARM processors was added as well to support environments requiring power-efficiency and high performance on these CPUs as well as for supporting embedded and edge computing scenarios

Important Upgrade and Backwards Compatibility Information

  • Connection Names Must Be Unique
    • From Qorus 5.1 onwards, all connection names must be unique across datasources, remote Qorus connections, and user connections. When the schema is upgraded for Qorus 5.1, any conflicting connection names are renamed automatically.
    • All Qorus loggers now use the logger layout pattern to generate all log messages; in previous versions logger layout patterns were respected, however a fixed prefix was always generated for each log message. The new default logger layout pattern is %d{YYYY-MM-DD HH:mm:SS.xx} %h:%P T%t [%p]: %m%n. See Logger::LoggerLayoutPattern for more information on logger layout patterns
  • Qorus Core URL Considerations
    • Qorus APIs that return the qorus-core system URL now attempt to return a non-local URL to facilitate distributed deployments. Functionality that depends on the hostname localhost (such as forcing RBAC source IP-based authentication with qorus.rbac-force-user) may be affected; see qorus-client.client-url for An option that allows the client URL to be set explicitly

New Features and Changes

Bug Fixes in Qorus

Issue ID Severity Description
3671 High fixed a bug handling ZeroMQ message cancellations that could result in a deadlock
3670 Normal fixed a bug where user storage was not initialized properly from users provided by external RBAC providers
3669 Normal fixed a bug where services bound on wildcard listeners would get a LISTENER-SHARING-ERROR if another service listener was already bound on the same wildcard listener
3668 Normal fixed a deadlock in workflow handling related to a race condition handling workflow stop actions and qwf process aborts
3667 Normal fixed a race condition handling qwf terminations during initialization that could cause a deadlock shutting down qorus
3666 Low Qorus URL is returned incorrectly from system APIs when HTTP certificate and private key info is given in the qorus.http-secure-server option
3648 Low fixed a race condition handling simultaneous qsvc shutdowns and abnormal terminations that could result in spurious log messages and an incorrect error reported as a response to the service stop command
3642 Normal fixed a bug where terminating a program may appear to fail on Alpine Linux which will cause a deadlock when shutting down Qorus
3635 Normal fixed a bug when running on Oracle DBs where transactions were not restarted with ORA-00060 errors
3632 Normal fixed a bug where the SQL table cache was not caching tables properly

Changes and Bugfixes in Qore

See Qore 1.0

Qorus 5.1.1

Release Overview

This is a bugfix release that addresses important issues with features introduced in Qorus 5.1:

  • This release addresses internal authentication issues with cluster deployments or Qorus; as of this release, requesting a Qorus REST connection object without specifying authentication information will automatically set an authentication header with an internal token that is used in each external request to authenticate connections with system privileges, eliminating the need to set explicit authentication information in the client-url in the System Options File, which can represent a security hole. Instead internal code that relies on external REST communication with Qorus can count on authenticated access, and client programs with access to the Qorus network key information will automatically retrieve an appropriate internal token that is used for communication with the server. If a username and password is already included in the qorus-client.client-url option value, then nothing changes with this release; the existing client username and password is respected, and no internal system token is acquired. See the issues addressed below for more information.
  • This release includes many fixes and improvemenets for Python and Java interoperability

Important Changes

  • Removed Restrictions on Public Service Methods
    • Java and Qore services are no longer required to declare all public methods in the main service class as exported service methods; this aligns Java and Qore services with Python and also allows for the main service class to serve as an event source in all supported languages

Bug Fixes in Qorus

Issue ID Severity Description
3682 Normal fixed a bug where oload would fail with zero-length resource files
3681 High fixed a bug where oload would fail to load Java classes that depended on Qore or Python classes and therefore required dynamic code generation in oload
3678 High fixed a bug where remote services could not start if they registered service resources in the service class's constructor
3677 High fixed bugs removing UI extensions from internal data structures when services terminate that could cause UI extensions to be unusable afterwards
3676 Low filtered out additional utility threads from thread stack output in qctl
3675 Normal removed restrictions on declaring all public methods as service methods for Java and Qore
3674 Normal fixed a race contions in logging to services and jobs that could cause spurious errors
3673 High fixed a bug where passive masters would fail to start in Kubernetes if client HTTP connections to the REST API provided by qorus-core required authentication
3672 High fixed a bug where REST clients in a cluster were not able to authenticate properly when RBAC was enabled
3642 High fixed a deadlock handling stateful service aborts and HTTP resource handler requests

Changes and Bugfixes in Qore

Qorus 5.1.2

Release Overview

This is a bugfix release; see below for more information

Bug Fixes in Qorus

Issue ID Severity Description
3694 High fixed a bug where it was not possible to create OAuth2 client configurations for external users (ex: LDAP / AD users)
3693 High fixed a bug where oload: Data Loading Tool and Schema Manager could not load Java services without old imports for the service annotation class
3692 High fixed non-functional MySQL / MariaDB functionality in the qorus-alpine and qorus-alpine-minimal images
3691 High fixed non-functional MS SQL Server and Sybase connectivity in the qorus-alpine and qorus-alpine-minimal images
3690 High fixed non-functional Oracle connectivity in the qorus-alpine image
3689 High fixed non-functional Java in the qorus-alpine image
3688 High fixed a deadlock shutting down workflows from internal interface code caused by an internal data serialization error
3687 Normal fixed a bug where workflow orders could result in an invalid status in the DB where they would have to be recovered manually in cases where the connection to the DB server becomes unreliable or sporadically unavailable for a period of two minutes or more
3686 Normal updated the qorus.system-pool-maximum default value from 10 to 40; updated the qorus.db-max-threads value from 10 to 30
3685 High fixed a bug where internal system transactions could fail with spurious errors due to a race condition handling the thread-local transaction status
3684 Normal fixed internal race conditions with external API authentication that could cause spurious errors, particularly with external authentication providers such as LDAP and OAuth2

Qorus 5.1.3

Release Overview

This is a bugfix release; see below for more information

Bug Fixes in Qorus

Issue ID Severity Description
3704 High fixed a bug in oload: Data Loading Tool and Schema Manager where it was not possible to load Java classes using Qorus APIs outside the direct context of an interface
3702 Normal fixed a bug where it was not possible to implement loopback qorus rem4ote connections without using the qorus.rbac-force-user option
3701 Normal fixed a bug where WebSocket events for connection changes were not emitted in all cases, causing the UI to be out of date with the server
3700 Normal fixed a bug where connections with tags could not be deleted through the API
3699 High fixed a regression that caused a 409 Conflict error to be returned to unauthenticated HTTP/S connections when RBAC was disabled
3698 Normal fixed a bug where REST APIs that update config items could fail when the DB was not aligned with the metadata cache
3697 Normal fixed a race condition handling service unload calls at the same time as a remote service abort event
3696 High fixed a bug where GET /api/latest/system?action=wstoken returned a token with a very short TTL which would cause the remote deployment API to fail with complex releases

Qorus 5.1.4

Release Overview

This is a bugfix release; in particular cluster stability has been enhanced particularly regarding error and crash recovery scenarios; see below for more information

Bug Fixes in Qorus

Issue ID Severity Description
3713 High fixed a bug handling qorus-core: Qorus Core Cluster Process crash recoveries
3712 High fixed bugs handling complex crash recover scenarios with stateless services
3711 High fixed a bug in Ubuntu Docker images where a missing library made it impossible to communicate with MS SQL Server DBs
3710 Normal fixed a bug detecting loopback Qorus remote connections
3709 Normal fixed a lock error with loopback Qorus connection management
3708 Normal fixed a bug where workflow log messages sometimes appeared in the qorus-core log file
3707 Normal fixed a bug where oload would fail with Java code with Python dependencies with modules that could only be loaded once per procress (ex: tensorflow)
3706 High fixed a bug where qwf: Qorus Workflow Cluster Process processes could not be started in nodes with passive masters in Kubernetes
3705 High fixed a bug in oload where config item value changes were not processed correctly for Java-based jobs

Qorus 5.1.5

Release Overview

This is a bugfix release; see below for more information

Bug Fixes in Qorus

Issue ID Severity Description
3719 High fixed errors starting remote qsvc and qjob processes in non-master notes under Kubernetes
3718 Normal fixed an error in the UI where info tabs were not rendered correctly on space-constrained windows with long object names in some cases
3716 High fixed errors handling internal messaging on very slow or memory constrained systems that could lead to a full cluster deadlock
3715 Normal fixed errors in the WebSocket remote deployment API and the REST raw remote file API handling windows paths from clients
3714 Normal fixed a race condition executing inactive remote jobs with multiple simultaneous requests
3699 Normal fixed handling external REST / WebSocket requests for authentication tokens when RBAC is disabled

Qorus 5.1.6

Release Overview

This is a bugfix release; see below for more information

New Features

Bug Fixes in Qorus

Qorus 5.1.7

Release Overview

This is a bugfix release; see below for more information

Bug Fixes in Qorus

Issue ID Severity Description
3724 Normal fixed errors in UserApi::updateOutputData() where subfields of the output data could not be written to the output location

Qorus 5.1.8

Release Overview

This is a bugfix release; see below for more information

Bug Fixes in Qorus

Issue ID Severity Description
3725 High fixed a bug where Qorus failed to process valid mappers with user-defined fields under a field of type auto

Qorus 5.1.9

Release Overview

This is a bugfix release; see below for more information

Bug Fixes in Qorus

Issue ID Severity Description
3734 Normal fixed a bug updating connections when the URL scheme changes and the options of the new connection type are not compatible with the old options; in this case incompatible options are discarded
3732 Normal only mark connections as down if they fail at least two consecutive pings
3730 Normal fixed a bug in the web UI where valid pipelines could not be viewed
3729 High fixed a bug in Kubernetes cluster recovery that could cause Qorus to refuse to start
3727 High fixed a bug where the web UI did not mask sensitive connections option values
3726 Normal fixed a bug in the web UI where all input fields in mappers would be displayed as having type undefined

Qorus 5.1.10

Release Overview

This is a bugfix release; see below for more information

New Features

Bug Fixes in Qorus

Issue ID Severity Description
3738 High fixed a bug where oload: Data Loading Tool and Schema Manager would not create all config items for pipelines consistently, leaving them unable to be used
3737 High fixed a bug where pipeline processor elements could not reliably access their configuration item data
3736 High fixed a bug where APIs that replace workflow dynamic data would replace static data instead if the order was currently cached
3735 Normal added new APIs to allow workflow order dynamic data to be updated atomically

Qorus 5.1.11

Release Overview

This is a bugfix release; see below for more information

Bug Fixes in Qorus

Issue ID Severity Description
3748 High fixed a bug recovering failed restarts of qjob: Qorus Server Cluster Process processes after a master failover
3747 High fixed a bug where the qdsp: Cluster Datasource Pool Process process for the system schema (qdsp-omq) would not always be restarted when there is a node failover
3746 High fixed a potential deadlock in qorus-core: Qorus Core Cluster Process when recovering a failed session
3745 High fixed a bug where qwf: Qorus Workflow Cluster Process processes that died and could not be restarted were not marked as terminated in qorus-core: Qorus Core Cluster Process
3744 High fixed a bug in handling process restart errors after active master failovers when there are not enough cluster resources to restart processes
3743 High fixed bug in handling active master failovers that could cause a deadlock
3742 High fixed a bug in qorus-core restarts under Kubernetes that could lead to platform instability
3741 Normal timeouts in stateless qsvc registration messages were not handled correctly, leading to instability and outages with stateless services when the cluster was under high load
3740 Normal fixed bugs in oload: Data Loading Tool and Schema Manager handling mappers with field names with embedded "." characters
3739 Normal fixed a bug in POST /api/latest/system/listeners
2459 High fixed a bug where recovering large instances from terminated Docker or Kubernetes instances took excessively long to complete

Qorus 5.1.12

Release Overview

This is a bugfix release; see below for more information

Bug Fixes in Qorus

Issue ID Severity Description
3750 High fixed a bug where MS SQL Server connections with the freetds module were not usable from Docker and Kubernetes

Qorus 5.1.13

Release Overview

This is a bugfix release; see below for more information

Bug Fixes in Qorus

Issue ID Severity Description
3752 Normal connections cannot be updated with the REST API unless the description is also included in the call

Qorus 5.1.14

Release Overview

This is a bugfix release; see below for more information

Bug Fixes in Qorus

Issue ID Severity Description
3753 High fixed a bug where a race condition in processing canceled ZeroMQ messages could lead to a deadlock
3751 High fixed a bug where oload: Data Loading Tool and Schema Manager would fail when loading more than one service with a Python module dependency that can only be loaded once per process

Qorus 5.1.15

Release Overview

This is a bugfix release; see below for more information

Qorus 5.1.16

Release Overview

This is a bugfix release; see below for more information

Bug Fixes in Qorus

Issue ID Severity Description
3761 High fixed a bug where oload failed to validate interfaces with mixed Java and Qore dependencies in some cases
3760 Normal fixed a bug where QorusClient::addConnectionModuleOption() and QorusClient::addDataProviderModuleOption() only added the final value when adding multiple values to the option
3759 High fixed a race condition with stateless services where they were not always recovered correctly after a qorus-core restart
3758 High fixed a bug where it was not possible to use make-release in Docker images due to a missing template file
3757 Normal fixed a bug where it was not possible to use non-Qore scripts as qscripts in Qorus releases
3756 High fixed a bug where a long outage of the primary master and the main datasource server due to Kubernetes infrastructure problems would prohibit Qorus from restarting automatically when the problems were resolved
3755 Normal fixed a bug where the Qorus web UI would show an error screen after logging in when the Qorus server was restarted during a UI session
3754 High fixed a bug where a Kubernetes cluster could fail to recover automatically in conditions where the qorus-core container is not responding to ZeroMQ messages, but the HTTP server is up

Qorus 5.1.17

Release Overview

This is a bugfix release; see below for more information

Bug Fixes in Qorus

Issue ID Severity Description
3766 Normal fixed a bug in oload: Data Loading Tool and Schema Manager where workflows with a Java-based workflow class would fail to load if they implemented a detach method
3765 High fixed a bug where it was not possible to reliably implement destructors in Qore interface classes as the Program context was deleted first in some cases
3764 High fixed a bug where API authentication would fail for some operations when RBAC was disabled; affected primarily Web UI operations
3763 High fixed a bug where oload would not reset config items for interfaces where the value was updated in all cases
3762 Normal implemented missing template for value map substitution: $value-map:{}
3751 High fixed a bug where oload: Data Loading Tool and Schema Manager would fail when loading more than one job with a Python module dependency that can only be loaded once per process
See also
jni Module Version 2.0.4

Qorus 5.1.18

Release Overview

This is a bugfix release; see below for more information

Bug Fixes in Qorus

Issue ID Severity Description
3767 Normal fixed a regression in oload: Data Loading Tool and Schema Manager related to handling interfaces with Python dependencies that must be validated in separate processes

Qorus 5.1.19

Release Overview

This is a bugfix release; see below for more information

Bug Fixes in Qorus

Issue ID Severity Description
3768 High fixed a bug where step classes with constructors that required config item information failed at runtime

Qorus 5.1.20

Release Overview

This is a bugfix release; see below for more information

New Features

Bug Fixes in Qorus

Issue ID Severity Description
3781 Normal fixed a bug where it was not possible to check or test for permissions from a named user
3778 High fixed a bug where config items in FSMs could not be updated with the web UI
3776 High fixed a bug where a job could fail to execute due to a race condition in the qorus-core process
3775 High fixed a bug where FSMs as substates in an FSM could not be opened in the web UI
3774 High fixed a bug where updating a job instance's status at runtime could fail on Oracle with an ORA-00060 error due to missing indices on foreign constraints on the JOBS table causing excessive runtime locking
3773 Normal fixed a bug where it was difficult to implement service-based token authentication and user handling
3771 Normal fixed a bug where Qorus would not accept an email address as a valid username
3770 High fixed a bug where Qorus would not start in some cases under Kubernetes when the qorus-core deployment would start before the qorus master statefulset
3769 Normal fixed a bug in the Qorus server's websocket communications with the UI when creating or editing new connections that caused incorrect option information to be displayed in the UI
See also

Qorus 5.1.21

Release Overview

This is a bugfix release; see below for more information

Note
Documentation for Mapper Output Field Options was added in this release

Important Upgrade and Backwards Compatibility Information

The jni module has undergone fixes for dynamic imports of Qore and Python modules; the result is that all symbols imported from qore modules must be referenced using the qoremod package, and all symbols imported from Python modules must be referenced using the pythonmod package. Previous releases of Qorus allowed for ambiguity which also could result in runtime errors and furthermore made it impossible to use dynamic imports of symbols from Qore or Python modules from native Java. Some simple logic to detect references to builtin Qore modules with the qore package path has been added, however if mixed package usage (i.e. both qore and qoremod packages) are used to reference symbols from the same Qore module, compile and runtime errors will result. Always use the qoremod and pythonmod packages to refer to symbols provided by Qore and Python modules, respectively.

Example
// **do not** import qore.Mapper.Mapper for example
import qoremod.Mapper.Mapper;
See also
Java Dynamic Import Statements for more information

Bug Fixes in Qorus

Issue ID Severity Description
3796 High fixed a bug where schema-tool verify / align actions would not run when the system was down
3795 High fixed a bug where restartable transactions in the server would run in an infinite loop with no delay if the DB server is down; now, after 2 retries, there is a 10 second delay between retries and more verbose logging
3794 Normal fixed a bug in the remote deployment API where Java tests could not be used with dynamic APIs
3793 Normal added missing qjavac script to supporting compiling Java programs using dynamic Python and Qore APIs
3792 High fixed API performance degradation in long-running Qorus instances in Kubernetes by capping memory history list sizes in qsvc and qorus-core processes and thereby eliminating excessively large/infinitely growing internal API messages
3791 Normal confusing error message with errors with UserApi::expandTemplatedValue()
3790 Normal fixed a bug where the Qorus client module would take a long time to initialize without reporting any errors if a valid Qorus process entry existed in the process table but the process did not respond
3789 Normal fixed a bug where a deadlock would result from stopping a Qorus listener by a request handled by the same listener
3788 Normal fixed bugs handling state data in config item resolution in finite state machines and in pipeline factory option resolution; added documentation
3787 High fixed a bug handling service initialization errors that could result in detached service resources that could prohibit a valid service from starting subsequently
3786 High fixed a race condition starting remote services providing global API services that could result in a deadlock
3785 Normal fixed a bug where template resolution with the $qore-expr: template would sometimes fail needlessly when non-string data was resolved in intermediate template resolution
3784 Normal fixed a bug where oload would cause external RBAC users (ex LDAP / AD users) to be purged when reloading groups
3783 Normal fixed a bug where the finite state machine name was not provided in error messages while executing an FSM
3782 Normal fixed a bug where the QorusApiDataProvider module was not usable
3764 High fixed a bug where certain APIs failed when Qorus was run with RBAC disabled (affected default Docker instances, for example)
See also

Qorus 5.1.22

Release Overview

This is a bugfix release; see below for more information

Bug Fixes in Qorus

Issue ID Severity Description
3803 Normal fixed a bug where error messages in finite state machines do not always contains the FSM name or the state name, making it harder to find the source of problems
3801 Normal fixed a bug where log messages in remote workflow and job processes were sometimes missing context info
3800 High fixed a bug handling error transitions in finite state machines; if there was more than one error transition and the last transition did not match the transition criteria, then the entire state will be treated as if no error transition matched, regardless of how other transitions were handled
3799 Normal fixed documentation errors regarding input data in local context data in mappers and in finite state machines
3797 Normal fixed a bug in the UI where the configuration for finite state machine blocks was not visible
3798 Normal fixed a bug where template expressions using the Qore language were not evaluated in a Qore context in finite state machine execution
3728 Normal fixed a bug in the UI where data type information was not visible
See also