Release Notes

Table of Contents

• Qorus 4.1
  • Release Overview
  • Important Upgrade and Backwards Compatibility Information
  • New Features
  • Bug Fixes in Qorus 4.1
  • Bug Fixes in Qore
• Qorus 4.1.1
  • Release Overview
  • New Features
  • Bug Fixes in Qorus
• Qorus 4.1.2
  • Release Overview
  • New Features
  • Bug Fixes in Qorus
• Qorus 4.1.3
  • Release Overview
  • New Features
  • Bug Fixes in Qorus
• Qorus 4.1.4
  • Release Overview
  • New Features
  • Bug Fixes in Qorus
• Qorus 4.1.4.p1
  • Release Overview
  • Bug Fixes in Qorus
• Qorus 4.1.4.p2
  • Release Overview
  • Bug Fixes in Qorus
• Qorus 4.1.4.p3
  • Release Overview
  • Bug Fixes in Qorus
• Qorus 4.1.4.p4
  • Release Overview

Qorus Release Notes History

Latest version: Qorus 4.1.4.p4

Qorus 4.1

Release Overview

This is a major release of Qorus with the following major changes:

• Data Provider API
  • The Data Provider API allows any system to be introspected and communicated with, through record-based or request/response APIs; it supports high-performance volume operations as well as efficient search, creation, update, upsert, and deletion of data.
  • The mapper infrastructure has been updated to fully support the Data Provider API as well as now supporting autonomous mode where data transformations are automatically performed on data retrieved from the input provider and stored in the output provider after data transformations. For data providers supporting bulk operations, this results in extremely high performance data integration flows based entirely on configuration.
  • The DataProvider module was added and many modules were updated to support its API
  • New system option: qorus.dataprovider-modules
• Qorus Interface Metadata in YAML Format
  Defining Qorus interface metadata in YAML format has been introduced to facilitate the creation, editing and management of Qorus interfaces in the new Qorus IDE (see Implementing Qorus Objects Using YAML format).
  The new IDE supports creating, editing, deleting, and packaging interface code along with their metadata as well as ensuring the alignment of code and metadata in interface objects.
• Java Updated to Java 11
  Qorus's Java support has been updated to OpenJDK 11; see Java Requirements for more information
• Enhanced Unicode Support
  All interface objects in Qorus can now be made up of any Unicode characters including those outside the Latin or European character sets (ex: "ワークフローの例" is a valid workflow name as of this release)
• Qorus Cluster Startup Improvements
  Qorus starts up faster and is more responsive during startup; enabled workflows, services, and jobs now start in parallel, additionally, excess serialization in accessing internal data structures during startup was removed.

Important Upgrade and Backwards Compatibility Information

• New Environment Variables
  Set "QORE_DATA_PROVIDERS=QorusDataProviders" in your system environment to Qorus data providers with the qdp program for command-line introspection and manipulation of data providers
• Qore Parser Fixes
  The Qore library has undergone the following parser fixes:
  • corrected the cast<> operator to no longer silently accept NOTHING; to get the old behavior, use the %broken-cast parse directive in individual code objects, or set qorus.compat-broken-cast in the System Options File.
  • the deprecated returns keyword is no longer accepted by default; to get the old behavior, use the %allow-returns parse directive in individual code objects, or set qorus.compat-allow-returns in the System Options File.
• Event Code Changes
  logger and appender event codes were reassigned to the correct range starting with 5200: see LOGGER_CREATED for new values
• Abstract Method Signature Change
  • AbstractDatasource::getDBEncoding() now has a return type of *string as it can return no value; in previous versions of Qore this method could return NOTHING despite the return type; this fix will break any classes derived from AbstractDatasource, so adjust your source code accordingly
• Metadata-Based Interface Objects
  oload will load YAML-metadata-based integration objects first and does not handle dependencies between old-style objects with embedded metadata and YAML-metadata-based objects. Errors will result if a YAML-metadata-based object created from the IDE depends on older code when oload loads the objects for the first time. It is recommended to migrate all objects to YAML-metadata-based objects and not to mix the two types of objects in the same release.

New Features

• Service Config Item Updates
  Services can now react actively to configuration item changes; see:
  • Qore: ServiceApi::setConfigItemChangeCallback()
  • Java: ServiceApi.setConfigItemChangeCallback()
  for more information
• New Common Interface APIs
  • Qore:
    • DbRemoteRawReceive class
    • QorusLocalRestHelper class
    • UserApi::checkObjectCache()
    • UserApi::clearObjectCache()
    • UserApi::clearObjectFromObjectCache()
    • UserApi::getContextUserName()
    • UserApi::getDataProvider()
    • UserApi::getObjectCacheSize()
    • UserApi::getThreadCallStack()
    • UserApi::logWithLevel()
    • UserApi::logWithLevelArgs()
    • UserApi::saveObjectInObjectCache()
    • UserApi::startCapturingObjectsFromJava()
    • UserApi::stopCapturingObjectsFromJava()
  • Java:
    • ConfigItemValueCallback class
    • QorusLocalRestHelper class
    • DbRemoteRawReceive class
    • UserApi.checkObjectCache()
    • UserApi.clearObjectCache()
    • UserApi.clearObjectFromObjectCache()
    • UserApi.getContextUserName()
    • UserApi.getDataProvider()
    • UserApi.getObjectCacheSize()
    • UserApi.saveObjectInObjectCache()
    • UserApi.startCapturingObjects()
    • UserApi.stopCapturingObjects()
• New Service APIs
  • Qore: ServiceApi::setConfigItemChangeCallback()
  • Java: ServiceApi.setConfigItemChangeCallback()
• New Class-Based Mapper API
  • Qore: MapperApi
  • Java: MapperApi
• New Template Substitution Type
  • parse-value: parses the given string argument as a Qore value after recursive template substitution
  • temp: retrieves or writes to workflow temporary data
  See Config Item / Building Block Template Substitution Strings for more information
• New Constants For New Permissions
  • DATA-PROVIDER-CONTROL
  • DATA-PROVIDER-CREATE
  • DATA-PROVIDER-READ
  • DATA-PROVIDER-UPDATE
  • DATA-PROVIDER-DELETE
  • DATA-PROVIDER-REQUEST
• REST API v4
  • /api/v4/...: the REST API has been extended and a new version is now available under this URI path prefix; see the previous link for changes in REST API v4
• Connection Socket Debugging
  • new REST APIs to enable and disable socket debugging; when enabled, connections will log all socket data transferred in the interface log file:
    • PUT /api/v4/remote/qorus/{name}?action=enableDebugData
    • PUT /api/v4/remote/qorus/{name}?action=disableDebugData
    • PUT /api/v4/remote/user/{name}?action=enableDebugData
    • PUT /api/v4/remote/user/{name}?action=disableDebugData
• New System Events
  • CONNECTION_DEBUG_DATA_CHANGE
    • Qore constant: OMQ::QEC_CONNECTION_DEBUG_DATA_CHANGE
    • Java constant: com.qoretechnologies.qorus.OMQ.QEC_CONNECTION_DEBUG_DATA_CHANGE
• Qorus Client Updates
  • QorusClientBase module added for faster initialization of common client code, furthermore connection initialization is only done on demand which provides much faster responses to client actions not requiring database access
  • QorusClient::addModuleOption(): allows for qorus module options to be updated in client or release scripts
  • QorusClient::configureInterfaces(): allows configuration item values to be set in client or release scripts
  • QorusClient::createGlobalError(): allows global workflow error configurations to be created in client or release scripts
  • QorusClient::createRole(): allows roles to be created in client or release scripts
  • QorusClient::createUser(): allows users to be created in client or release scripts
  • QorusClient::createUserConnection(): allows user connections to be created in client or release scripts
  • QorusClient::createOrUpdateWorkflowError(): allows workflow-specific error configurations to be created or updated in client or release scripts
• Qorus Test Updates
  • the QorusInterfaceTest class was enhanced to include support for starting and stopping Qorus in the base class
  • the QorusInterfaceTest module now includes the WorkflowLogHelper class to assist in outputting log messages while running workflow tests
• Interface Creation Updates
  • Added support for oload to load Qorus objects defined in YAML-format (feature 3102)
  • Added support for class-based workflows (feature 3274)
  • Added support for oload to load Qorus workflow error definitions in YAML-format (feature 3292)
  • Added support for library objects for steps in YAML-format (feature 3325)

Bug Fixes in Qorus 4.1

Issue ID	Severity	Description
3396	Normal	fixed bugs recovering processes after qorus-core and or qdsp-omq along with interface processes crash
3395	Normal	fixed a bug recovering qorus-master crashes with uninitalized processes in the cluster process table
3332	Normal	improved client library initialization time

Bug Fixes in Qore

See Bug Fixes in Qore

Qorus 4.1.1

Release Overview

This is a bugfix release with minor feature updates; for details, see the issues addressed below.

New Features

• Template Substitution Updates
  
  • UserApi::expandTemplatedValue() will expand string values in lists and hashes recursively by default to make it easier to manage complex values in config items
  • the $transient: template type was implemented for thread-local temporary storage for all interface types, meant to provide temporary storage for IDE building-block connectors; see Config Item / Building Block Template Substitution Strings for more information
• File Location Updates
  Qorus adds the new FileLocationHandler module; support for connections requiring file data now can refer to file data as service resources in addition to other schemes based on standard protocols; ex:

  resource://service-name:resource-name.yaml 

• New APIs
  
  • Qore: UserApi::getTypeFromPath()
  • Java: UserApi.getTypeFromPath()
• Java API Updates
  
  • The Java com.qoretechnologies.qorus.UserApi.getUserConnection() API now returns native Java objects for user connections whenever possible

Bug Fixes in Qorus

Issue ID	Severity	Description
3435	Normal	fixed a bug in oload where deleting objects not cached in a running server would fail
3434	Normal	suppressed excess connection-handling logging in the Qorus client; only errors are displayed
3430	Normal	fixed a bug in UserApi::expandComplexValue() with an empty hash
3429	Normal	fixed a bug in the return type for the /api/latest/dataprovider/basetypes REST API for "or nothing" types
3426	Normal	fixed a bug folding values into strings in template substitution (ex: UserApi::expandTemplatedValue())
3425	Normal	fixed a bug in oload handling config items when updating or deleting objects

See Bug Fixes in Qore

Qorus 4.1.2

Release Overview

This is a bugfix release with new features; for details, see the issues addressed below.

New Features

• User-Defined Type Support
  This release of Qorus adds support for user-defined types:
  • oload now supports user-defined type in YAML format
  • in the REST API allows types to be created, updated, and deleted on demand by authorized users:
    • POST /api/latest/dataprovider/types
    • DELETE /api/latest/dataprovider/types
• Workflow Static Data Type Enforcement
  
  • Static Data Type
  • new REST API to retrieve interfaces with a data context: GET /api/latest/system?action=interfacesWithDataContext
• Template Substitution Enhancements
  Alternate values can be given for expressions that evaluate to no value; additional value types are supported; see Config Item / Building Block Template Substitution Strings for more information
• New Constants For New Permissions
  • TYPE-CACHE-CONTROL
  • TYPE-CACHE-RELOAD
  • TYPE-CACHE-CREATE
  • TYPE-CACHE-DELETE
• New APIs
  • Qore: UserApi::getQorusType()
  • Qore: UserApi::clearPersistentStateData()
  • Qore: UserApi::getPersistentStateData()
  • Qore: UserApi::savePersistentStateData()
  • Java: UserApi.getQorusType()
  • Java: JobApi.clearPersistentStateData()
  • Java: JobApi.getPersistentStateData()
  • Java: JobApi.savePersistentStateData()
• Mapper and Data Provider Enhancements
  • Mappers now support the ignore_missing_input option to allow for efficient delta/update/upsert operations with variable output keys depending on the presence of input keys
  • Request-response data providers now allow for runtime resolution of URI paths in the data provider path based on template string substitution
• New Modules
  • BillwerkRestClient user module for REST HTTP connections to billwerk.com's REST API
• New Connection Types
  • BillwerkRestConnection user connection type ("billwerkrest" and "billwerkrests")

Bug Fixes in Qorus

Issue ID	Severity	Description
3470	High	fixed a bug in oload "oload" where deleting a mapper with step references would fail
3469	High	fixed a bug in the REST API where datasource and Qorus connection options could not be removed
3468	High	fixed a deadlock in handling process-related errors in qorus-master
3463	High	fixed a bug where oload would overwrite config item values updated with the API when objects were reloaded
3461	High	fixed a race condition in internal zmq client destruction that could cause to a cluster deadlock
3459	High	fixed a bug where the debug-system option was not respected in remote processes
3454	High	fixed a bug handling invalid mappers that could lead to a deadlock after a mapper reset
3453	High	fixed encryption in cluster communication; new cluster API version is 0.3
3447	Normal	fixed a bug in UserApi::getTypeFromPath() when the path had a leading "/" character
3444	Normal	fixed a bug in the REST API returning mappers with custom fields
3443	Normal	fixed handling the -D option in oload
3440	High	fixed a deadlock when a qdsp process cannot be initialized after restarting; reverted the process start ACK message to be issued only after the new pool is correctly initialized to avoid a serious platform deadlock in network datasource pool handling
3438	Normal	fixed a bug handling "SIZE" messages in AbstractFtpHandler
3429	Normal	fixed a bug in oload where an error was raised when deleting an interface with a non-local config item value

See Bug Fixes in Qore

Qorus 4.1.3

Release Overview

This is a bugfix release with new features; for details, see the issues addressed below.

Important Upgrade and Backwards Compatibility Information

• Disabling Interfaces
  APIs that disable interfaces now run synchronously; that is; when the API returns, the interface has already been stopped and disabled. In previous versions of Qorus, API calls that disabled interfaces returned immediately, and the disable action was performed asynchronously in the background. This sometimes led to errors and race conditions, so the behavior of these interfaces have changed. Affected REST APIs:
  • PUT /api/latest/jobs/{id_or_name}?action=disable
  • PUT /api/latest/jobs?action=disable
  • PUT /api/latest/services/{id_or_name}?action=disable
  • PUT /api/latest/services?action=disable
  • PUT /api/latest/workflows/{id_or_name}?action=disable
  • PUT /api/latest/workflows?action=disable

New Features

• Template Substitution Enhancements
  If template substitution values are resolved through intermediate lists of data structures, the final resolved value will be a list of all values resolved in each list element; see Config Item / Building Block Template Substitution Strings for more information
• New REST API
  • POST /api/latest/workflows/{id_or_name}?action=execSynchronousAsync

Bug Fixes in Qorus

Issue ID	Severity	Description
3490	Normal	fixed an error in workflow synchronization event steps in loosely-coupled subworkflow orders
3487	Normal	fixed ServiceFileHandler in services; handling resources failed due to a type error
3486	Normal	fixed a bug where workflow orders being executed synchronously could not be canceled or set to ERROR
3484	Normal	fixed UserApi::callRestApi() when called from HTTP handlers in qsvc processes
3480	Normal	fixed bugs in Java API com.qoretechnologies.qorus.test.CallService
3479	High	fixed a deadlock terminating qdsp processes related to ZeroMQ context management
3478	High	fixed job cron execution for trigger dates during DST changes when the clock moves forward
3477	High	fixed a deadlock in qsvc crash recovery when service resources are acquired in the init method
3475	High	fixed an error with services with resource (ex: HTTP, REST, and SOAP server services) in qsvc crash recovery
3474	High	fixed a race condition with qorus-core crash recovery with running interfaces when auditing is enabled
3473	High	fixed a deadlock caused by a race condition handling qwf process early terminations or aborts when stopping the affected workflow
3472	High	fixed a deadlock caused by a race condition in cluster client object handling
3358	High	fixed race conditions deleting interfaces with oload

See Bug Fixes in Qore

Qorus 4.1.4

Release Overview

This is a bugfix release; for details, see the issues addressed below.

New Features

• Template Substitution Enhancements
  • implemented the $qore-expr:{} template;
  • implemented the $qore-expr-value:{} template
  see Config Item / Building Block Template Substitution Strings for more information
• Array Element Data
  • added the array_element key to the WorkflowApi::getStepInfo() return value
• Nested Mapper Support
  • Qorus now supports nested mappers when the submapper and related options are used in mappers

Bug Fixes in Qorus

Issue ID	Severity	Description
3507	High	fixed RPC API commands shutdown and shutdown-wait
3506	High	fixed bugs in active master recovery
3504	High	fixed a deadlock when an interface reset command is processed at the same time as a disable interface command
3503	High	fixed a deadlock in remote program initialization
3502	High	fixed a race condition with qctl restart
3499	High	fixed bugs handling merging non-string values when generating template strings
3498	High	fixed a bug where date-time values were not merged to template strings with a reasonable format
3497	High	fixed a bug where oload would not delete local values from interfaces
3493	High	fixed a race condition causing Qorus startups to fail unnecessarily if executed right after a shutdown
3492	High	fixed a deadlock in qorus-core shutdown caused by incorrect error handling for process shutdown errors
3491	High	fixed a deadlock caused by a race condition in qwf process recovery with running execution instances
3478	High	fixed a regression in job cron trigger dates calculation around midnight introduced in the last release
3358	High	fixed race conditions deleting interfaces with oload; finalizes the fix partially addressed in 4.1.3

See Bug Fixes in Qore

Qorus 4.1.4.p1

Release Overview

This is a bugfix release; for details, see the issues addressed below.

Bug Fixes in Qorus

Issue ID	Severity	Description
3511	High	fixed a deadlock when enabling / disbaling multiple interfaces
3508	High	fixed reporting RAM in use for qorus master nodes

See Bug Fixes in Qore

Qorus 4.1.4.p2

Release Overview

This is a bugfix release; for details, see the issues addressed below.

Bug Fixes in Qorus

Issue ID	Severity	Description
3525	Normal	fixed oload to ignore non-Qorus *.yaml files in deployments so that YAML service resources can be submitted from the IDE using the remote deployment REST API
3523	Normal	fixed qorus-core to start even if the UNIX domain socket listener cannot be started
3522	Normal	Qorus will not start as the root user anymore to avoid problems related to files with root permissions being created
3520	Normal	fixed an error where the "desc" key was missing in the response to GET /api/latest/services for loaded services
3519	Normal	fixed an error in GET /api/latest/mappers

Qorus 4.1.4.p3

Release Overview

This is a bugfix release; for details, see the issues addressed below.

Bug Fixes in Qorus

Qorus's ability to handle low-memory situation has been greatly improved with the following changes:

• qorus-master will refuse to start processes if they would overcommit physical RAM on the node; overcommit tolerance is controlled with the following new system option:
  • qorus.allow-node-overcommit-percent
• The default value for the qorus.max-process-memory-percent option is now 99
• The default value for the qorus.warning-process-memory-percent option is now 50
• Critical processes (qorus-core and qdsp-omq, the datasource pool process for the system schema) are never killed and are started onconditionally regardless of the amount of free memory

Issue ID	Severity	Description
3531	Normal	fixed deadlocks caused by race conditions when interfaces terminate during a qorus-core shutdown
3530	Normal	fixed a deadlock caused by a race condition and synchronization error recovering qorus-core crashes and terminating qwf processes
3529	Normal	fixed a race condition that could lead to a deadlock when deleting and recreating interfaces with oload in a single operation
3527	Normal	fixed a cosmetic bug in the web UI regarding an incorrect position of the close dialog control in configuration item windows
3526	Normal	fixed a bug in the web UI where the UI would show a blank page with some invalid mappers
3363	Normal	fixed a bug in the web UI where the code window would reset to blank when websicket events would arrive for certain workflows

Qorus 4.1.4.p4

Release Overview

This is a bugfix release; for details, see the issues addressed below.

Issue ID	Severity	Description
3535	Normal	fixed a confusing error message that was produced when the system couldn't start due to an unavailable system DB connection
3534	High	fixed bugs handling HTTP/REST/WebSocket logging, config item handling, and thread-local context in service request handlers
3533	Normal	fixed a deadlock that can occur with remote services that require service resources during service initialization
3479	High	fixed a final bug in qdsp reconnect handling that could cause Qorus to hang on shutdown due to connection threads being left in place

See Bug Fixes in Qore