Qorus Integration Engine®  4.0.3.p2_git
Qorus 2.0 Series

Table of Contents

back to Release Notes

Qorus 2.0.3.p5

Issues Fixed

Bug ID Severity Description Change
167 MAJOR if there is a DB or other crash when a workflow order goes from READY to IN-PROGRESS, it could reach an unrecoverable state fixed transaction model to commit workflow status together with initial segment to ensure recoverability
168 MAJOR static data is not returned by the system.info service in most cases when the data is cached updated cache logic to correct bug
169 MAJOR when reading in large amounts of workflow order data events, sometimes waiting threads are not notified that data is available on their queues updated queue logic to notify waiting threads also during queue initialization

Qorus 2.0.3.p4.1

Issues Fixed

Bug ID Severity Description Change
164 MAJOR bug in socket read cases no data to be returned if the remote end shuts down the socket in an orderly fashion fixed in Qore library
165 CRITICAL member handling bug in the context statement can cause a crash fixed in Qore library
166 CRITICAL binary += operator can cause a crash if the operand is the lvalue (i.e. $bin += $bin) fixed in Qore library

Qorus 2.0.3.p4

Issues Fixed

Bug ID Severity Description Change
159 BLOCKING system API omq.system.reset-all-workflows silently fails fixed bug in reset logic
160 BLOCKING a race condition in handling detached segment statuses can cause a workflow to get a COMPLETE status when the detached segment has executed at all fixed bug in segment and workflow order data instance status handling so that detached segments are always executed correctly
161 MAJOR detach function fails when a workflow cache reset is in progress fixed detach logic to wait until the reset is complete and then continue
162 MINOR starting workflows fails when a workflow reset is in progress fixed start logic to wait until the reset is complete and then continue starting
163 MINOR temporary workflow execution instances, created for handling detach functions, are reported by the API these execution instances have been made private and are no longer visible by the GUI and command-line tools

Qorus 2.0.3.p3

Issues Fixed

Bug ID Severity Description Change
158 MAJOR getDynamicData() and getInstanceData() no longer return a single value when a single value is requested; instead they return a one-element list fixed to revert back to documented behavior; a single value is returned in these cases

Qorus 2.0.3.p2

Issues Fixed

Bug ID Severity Description Change
155 MAJOR cannot access dynamic data or other workflow order information in the detach function implementation fixed so that workflow data can be accessed in the detach function
156 MINOR business error internal status not reset on array steps in some conditions bug in logic fixed so that the business error internal status is consistent with the database status
157 BLOCKING calling omq.system.replace-static-data() causes the static data to be erased, internally and in the DB bug in implementation fixed

Qorus 2.0.3.p1

Issues Fixed

Bug ID Severity Description Change
151 CRITICAL a bug in the qore library can cause the process to crash if a file is parsed that ends while still in a block comment bug in qore library fixed; this situation is handled correctly by reporting an error back to the caller
152 BLOCKING the user install script install.sh was not usable due to missing functions and syntax errors in the shell script code missing functions added and syntax errors fixed
154 MINOR -c "schema-tool -C" tries to create the schema even if it already exists logic fixed to output a user-friendly error message if the schema already exists

Qorus 2.0.3

Bug Fixes

Issues Fixed

Bug ID Severity Description Change
112 MINOR When workflows are stopped, an in-memory temporary workflow execution instance was created for each workflow order data instance if it had a detach function causing the system to execute detaches much slower than necessary. The fix is to reuse the same workflow execution instance for detach functions.
140 MAJOR If an array function raises an error the first time it is executed, then it cannot be skipped by returning NOTHING the next time it's executed. If the array function returns NOTHING and only a single step exists, then the entire step is skipped, even if it's an ansynchronous or subworkflow step.
141 CRITICAL Workflow order keys are not cached correctly, causing setOrderKeys() to fail when already-existing order keys are included in the API call. Furthermore the order keys are not available in the getOrderKeys() call unless they have been saved with setOrderKeys() while the workflow data is still cached. Workflow order keys are cached and handled correctly with this release.
142 MAJOR An exception results in some cases when a user tries to stop a workflow while it's still initializing, causing it to be impossible to stop the workflow Appropriate checking was implemented to avoid this race condition.
143 MINOR schema-tool does not recognize the start of a multi-line block with Oracle code if the name of the object does not appear on the same line as the create [or replace] text Block recognition for Oracle SQL was improved.
144 MAJOR The undo segment can become full while recovering sessions with large numbers of workflow instance rows Transactions are now broken into smaller blocks and parallized, improving performance and solving the too-large-transaction problems. db-max-threads system parameter implemented to control parallelization.
145 MAJOR When caching large amounts of workflow data, the undo segment can become full and furthermore the system can remain unresponsive for long periods of time see above
146 NORMAL When reading in queue entries, data is queued based on the order it was created and not based on the date/time the queue entry was finalized Queue entries are now queued in order of their modified timestamp, not their created timestamp.
147 MAJOR The install script would not install new system services if the schema was not upgraded schema-tool updated with new option –new-release; install script updated to call schema-tool with this new option when necessary
148 NORMAL The install script would not accept tablespace name overrides on the command-line install script updated to accept –data-ts=arg and –index-ts=arg options and pass them to schema-tool
149 MINOR schema-tool did not pass datasource parameters to oload; loading system services would fail if datasource parameters given to install.sh and a dbparams file already existed; services would be loaded in the wrong database schema-tool updated to pass datasource parameters on to oload
150 MAJOR the datasource.flushToFile() method not tagged as a "write" method, so any user with CALL-SYSTEM-SERVICES-RO role could change datasource parameters permanently method tagged as a "write" method

New Features and Changes

  • Performance and Scalability Improvements
    The database interface to the event queueing system has been overhauled; the queries have been streamlined and the number of of queries to load complex workflows has been reduced. Additionally the Oracle database API has been significantly expanded to improve performance and scalability with large databases. More detail is provided below about breaking down and parallelizing large transactions that dramatically improves responsiveness and performance in large databases.
  • Staged Session Recovery
    Session recovery is now performed in stages and can be parallelized. This increases performance and ensures that rollback space is not exhausted when recovering sessions with a large number of workflow data rows. Session recovery is performed in a number of threads controlled by the new qorus.db-max-threads system option.
  • Staged Workflow Data Queue Population
    When the first instance of a particular workflow is started, the order data instances and associated statuses are read in from the database and put into event queues. In previous releases of Qorus, this was done in single monolithic transactions. In this release, the data is read in blocks from the database and put in the appropriate event queues in background threads (in a thread pool with a number of threads equal to the value of qorus.db-max-threads, a new system option introduced in this release), which increases system responsiveness and also reduces the size of the DB transactions used so that rollback space is not exhausted when very large amounts of workflow order instance data is read in.
  • New System Option: qorus.db-max-threads
    This new system option controls how many threads are launched in order to parallelize database operations such as session recovery and workflow data queue population. This option is related to and must be strictly less than qorus.system-pool-maximum, which gives the maximum number of parallel connections in the system schema DatasourcePool.
  • New Internal API Function: omqusleep() This new API function allows services or workflows to sleep for a number of microseconds, instead of a resolution of seconds as was previously available with the omqsleep() function.
  • oload Update
    oload will now print warnings if error definitions (as returned by a workflow's error function) have a RETRY or ERROR status with a MINOR or less severity code.
  • oview Update
    oview now uses a faster API when showing workflow instance data information without workflow instance data details.

System Schema Data Model 2.0.3 Changes

The Oracle API was updated to support the new segmented session recovery and workflow queue population solution introduced in this release. Furthermore, queries were tuned to better support large databases to improve system performance and scalability.

Qorus 2.0.2.p8

Issues Fixed in 2.0.2.p8

Bug ID Severity Description
136 MAJOR If an exception is raised initializing a service with the auto-start flag set, then the entire qorus system will fail to start.
137 MAJOR an error in the regular expression for "create package" in schema-tool causes single statements to be incorrectly grouped into statement blocks
138 BLOCKING oracle driver 1.0.9 and earlier has a memory error when binding clob and blobs in placeholders using the SQL::CLOB and SQL::BLOB placeholder specifications that can lead to a Qorus server crash
139 MAJOR Bug in the oracle driver causes binding by value to fail when a character string's length exceeds 32K bytes

Qorus 2.0.2.p7

Issues Fixed in 2.0.2.p7

Bug ID Severity Description
132 NORMAL schema-tool does not check minimum DB driver versions
133 MINOR a bug in the compareVersion() library function causes version comparisons to fail and therefore system schema driver version checks to fail
134 CRITICAL workflow resets do not work properly
135 CRITICAL resetting workflows does not work properly when flush-status is disabled

Qorus 2.0.2.p6

Issues Fixed in 2.0.2.p6

Bug ID Severity Description
128 CRITICAL FtpClient connections never complete due to this bug
129 MAJOR the function callObjectMethodArgs() has a memory error that can cause a crash
130 BLOCKING builtin functions are executed even if an exception is thrown while evaluating their arguments
131 MAJOR a bug in the WSDL class causes "int" validation to fail when processing SOAP messages

Qorus 2.0.2.p5

Issues Fixed in 2.0.2.p5

Bug ID Severity Description
125 NORMAL Array steps are not internally marked as IN-PROGRESS in the internal step cache when created; however this can only have an affect if the workflow is shut down before all elements of the array step have a chance to execute
126 MAJOR oload does not load service descriptions
127 BLOCKING cannot set any workflow options due to a bug in the workflow option code

Qorus 2.0.2.p4

Release Overview

Major bugfixes and new APIs, minor schema changes; see below for details.

Fixes

Issues Fixed in 2.0.2.p4

Bug ID Severity Description
116 MAJOR quoted text in block comments was not ignored - i.e. if an end block comment string (*/) appeared in quoted text in a block comment, the scanner assumed the comment was closed, even though such text should be ignored
117 MAJOR internal socket and file code did not retry if interrupted by a signal - affects reads, writes, select, accept, file locking code, etc
118 CRITICAL under some conditions, access to private members of a class was possible from outside the class
119 BLOCKING if the user supplied a file mode (3rd argument) to File::open2() it would cause a crash due to a typo in the method's internal implementation
120 NORMAL it was not possible to implement a static and non-static method of a class with the same name, contrary to the documentation
121 NORMAL makeXMLFragment() and makeFormattedXMLFragment() were not tagging strings with the correct encoding if the default encoding was not UTF-8
122 NORMAL split() and string concatenation did not work properly on strings containing binary data with embedded nulls
123 MAJOR if a class with no parent classes referenced a base class method of an existing class, a crash would result while parsing

Qorus Change List

  • new system API: omq.system.exec-synchronous-existing()
    This new system API allows operators to execute a synchronous workflow on an existing workflow order data instance; this can be useful to test a new workflow configuration on existing data
  • new system API: omq.system.skip-step-without-retry()
    This allows for more than one step to be skipped without triggering an immediate retry until the last step is updated
  • oload now supports the import-code directive for loading in code from an external file when defining services, for example:
    # name: shared_method
# import-code: ../shared/functions.qfd
# END
  • dynamic order data can be set when orders are initially created
    affected code: system APIs: omq.system.submit-data.[workflow].[version](), omq.system.exec-synchronous-workflow() and internal APIs createOrder(), execSyncronousWorkflow(). All the preceding now accept a new optional key: dynamicdata to allow for dynamic data to be saved against an initial order when the workflow order data is created. This can be useful if the order data is very large and can be purged when the workflow is complete; this can now be done in the last step of a workflow, for example.

System Schema Data Model 2.0.2.2 Changes

A new column was added to the WORKFLOWS table: "deprecated number(1) null". This will allow workflows to be tagged as deprecated to allow old workflows to be filtered from lists of active workflows.

Qore Change List

This release is based on the final release of Qore 0.7.6, and includes many bug fixes over Qore 0.7.5 as well as the following features:

  • in the switch statement, case expressions now support the == operator for soft comparisons (types are automatically converted)
  • added the rename() function to rename or move files
  • added the HTTPClient::isConnected() method
  • the -= operator now operates on objects (with string and list operands) consistently like it does with hashes
  • new classes: XmlDoc, XmlReader, XmlNode for more robust parsing and XML document iteration, including XPath support
  • added support for RelaxNG schema validation: parseXMLWithRelaxNG(), parseXMLAsDataWithRelaxNG(), XmlDoc::validateRelaxNG() added binary_to_string() function to convert a binary object directly to a string

Qorus 2.0.2.p3

Fixes

Issues Fixed in 2.0.2.p3

Bug ID Severity Description
113 NORMAL if multiple HTTP headers are received with the same name, only the last one is reported; with this fix the header will be returned as a list with all values
114 BLOCKING all Qorus system options are unlocked in Qorus 2.0.2 - 2.0.2.p2; changing a system option that should be locked could cause system instability or a deadlock
115 NORMAL multiple workflows defined in the same workflow file will fail to load; only the steps in the last workflow defined will be created in the database, causing other workflows to fail to load

System Schema Data Model 2.0.2.1 Changes

  • The oracle and pgsql database APIs were updated; the create_workflow_instance stored procedure was updated to accept dynamic data for workflow order data instance creation.
  • All databases including mysql were updated to reflect schema version 2.0.2.1 in the system_properties table; rows updated: "omq.schema-version" and "omq.schema-compatibility".

Qorus 2.0.2.p2

Issues Fixed in 2.0.2.p2

Bug ID Severity Description
108 NORMAL Exceptions in workflow code do not cause the error position to be recorded in log files unless OMQ::LL_DEBUG_1 or higher is set
109 NORMAL system service method info.getWorkflowOverview() ignores the workflow ID list passed
110 MINOR order key keys and values are not checked for excessive length when creating or changing key values
111 NORMAL when importing XSD definitions with the SoapServer (when binding Qorus services to a WSDL) or with the SoapClient, namespace definitions in the imported XSD are ignored

Qorus 2.0.2.p1

Issues Fixed in 2.0.2.p1

Bug ID Severity Description
106 CRITICAL schema upgrade from 1.8.0 to 2.0.0 fails with Oracle 9i as Oracle 9i drops the primary key index when the primary key constraint is dropped
107 MAJOR system api omq.system.reset-workflow() fails is a workflow version is given, saying the workflow is not cached even though it is cached

Qorus 2.0.2

Release Overview

Major bugfixes and major new features including an event system and APIs, persistent workflow parameters including the ability to set independent retry parameters per workflow segment; business and technical error differentiation; minor schema updates.

Issues Fixed in 2.0.2

Bug ID Severity Description
96 NORMAL an error in "schema-tool -C" or "schema-tool -U" would cause incorrect corrective text to appear
97 NORMAL error definitions with missing severity codes are not assigned MAJOR by default as per the documentation
98 NORMAL in case of recoveries with option flush-status = False (default), some steps could be left with status IN-PROGRESS after workflow status is flushed to the DB
99 NORMAL session IDs not set to 0 when workflow is COMPLETE on Oracle and PostgreSQL
100 BLOCKING system.datasource service ignores the qorus.dbparams-file system option and looks in $OMQ_DIR/etc by default
101 NORMAL system install.sh script not respecting qorus.dbparams-file option
102 CRITICAL reusing a Datasource (or DatasourcePool connection) after auto-reconnect fails causes a crash with oracle, sybase, and freetds drivers
103 MAJOR timeouts on Socket::connect() were not implemented correctly
104 NORMAL memory leaks can occur in the tibae module when exceptions are raised
105 NORMAL info.getWorkflowOverview() does not respect the time portion of the date/time value passed

Change List

  • System Event Infrastructure
    The Qorus application server has been updated to track and make available system and user events. The following new system APIs are now available: Additionally, the following new functions are available in Qorus service code: The following new function is available in both workflow and service code:
  • Business and Technical Error Differentiation
    Beginning with Qorus 2.0.2, it's possible to differentiate business and technical errors on the workflow status level, on the step status level, and tracked in each logged error. Whenever a step gets an ERROR status and the error's definition hash has the new key "business" : True, the step will be saved with a business error status (by setting "step_instance.retries = -1") and the workflow order data instance will be marked with "business_error = 1". New DB fields are workflow_instance.business_error and error_instance.busines_eError.
    The business error status is reported by ostatus and by the system.info service as well.
  • Workflow Parameters
    Workflow options can now be set by default per workflow type using the new system option qorus.workflow-params. Additionally, a special syntax can be used to set per-segment retry and asynchronous retry delays for workflow types (i.e. a specific workflow configuration identified by a workflow name and version).
  • Installation and Upgrade Improvements
    The Qorus system install script, install.sh, has been updated to allow the initial system schema Datasource connection parameters to be given on the command-line, making initial installs easier; in this case the connection parameters will be used immediately to install the system schema, additionally a line will be written to the dbparams file as well. Additionally, install.sh and schema-tool have been updated so that schema-tool has taken over more responsibility for executing the install or upgrade process, so that in case of interrupted installs or upgrades, the entire process can be recovered with schema-tool.

System Schema Data Model 2.0.2 Changes

Two changes were made in the system schema as follows:

  • A business_error field was added to the workflow_instance and error_instance tables
  • The DB API was updated to accommodate setting the business_error fields in the workflow_instance and error_instance tables (oracle and pgsql only).

Qorus 2.0.1

Release Overview

Major bugfixes, support for port numbers in datasource parameters, new API omq.system.skip-step(), system.status service deprecated (will be maintained for backwards compatibility)

Issues Fixed in 2.0.1

Note that bug IDs in bold type describe bugs that were present in Qorus before version 2.0.0, otherwise the bugs were introduced in 2.0.0 and fixed in 2.0.1.

Bug ID Severity Description
83 CRITICAL not possible to load the 2.0.0 system schema for PostgreSQL using schema-tool (bug introduced in 2.0.0)
84 CRITICAL mysql SQL upgrade script for upgrading from schema 1.8.0->2.0.0 does not work due to bugs (bug introduced in 2.0.0)
85 CRITICAL upgrading postgresql schemas from v1.7.2 -> 2.0.0 fails due to bugs (bug introduced in 2.0.0)
86 MAJOR Qorus fails to start if cache size set in options file (bug present since v1.7+)
87 CRITICAL not possible to load or upgrade to the 2.0.0 system schema for Oracle using schema-tool (bug introduced in 2.0.0)
88 CRITICAL the status.skipStep() method is broken in 2.0.0 (bug introduced in 2.0.0)
89 MAJOR qorus does not check minimum required service versions when loading system services (design issue)
90 MAJOR qorus and schema-tool do not check schema DB driver versions, although minimum versions are required (design issue)
91 MINOR system option dbparams-file is not being set when the filename is derived by the system (bug present in 1.8.* series)
92 CRITICAL ERROR statuses are not propagated to parent workflows from subworkflows (bug introduced in 2.0.0)
93 NORMAL workflows can be released with an INCOMPLETE status when no SEGMENT_INSTANCE rows have been created (bug introduced in 2.0.0)
94 NORMAL qorus client programs do not detect the server's URL correctly in all cases (bug present in 1.8.* series)
95 NORMAL cannot trigger an immediate retry of a workflow with status RETRY (bug present in 1.8.* series)

Change List

  • Datasource Definitions Accept an Optional Port Number
    Oracle, MySQL, PostgreSQL, Sybase, and FreeTDS datasource definitions now accept an optional port number (Oracle, Sybase, and FreeTDS also accept hostnames as well) so that direct connections can be made to databases without TNS (for Oralcle) or on non-standard ports (for MySQL and PostgreSQL). The client and server APIs (including the system.datasource service ) have been updated to accept optional ports in the datasource definitions and report them if present.
  • Minor Schema Update
    In order to fix bug 88, the schema was updated; see System Schema Data Model 2.0.1 Changes for more information
  • New System API Function: omq.system.skip-step()
    In order to properly skip execution of a step, the functionality had to be integrated with the internal workflow data cache manager, so a system API was created to make calling this functionality as efficient as possible.
  • system.status Service Completely Deprecated
    The status.skipStep() method was replaced by an internal version integrated with the internal workflow data cache manager; as this was the last method in the service that had native functionality; the entire service is now deprecated. The new version of this method calls the new system API omq.system.skip-step(). The service and all methods will remain for backwards compatibility, but will be marked as deprecated.

System Schema Data Model 2.0.1 Changes

  • In order to fix bug 88 (broken system.status.skipStep() method), fixes in the DB API were necessary, making it necessary to upgrade the schema's version to 2.0.1
  • All DBs were updated to reflect schema version 2.0.1 (MySQL was updated in order to remain consistent with the other DBs, even though it does not yet contain any API)

Qorus 2.0.0

Release Overview

This is a major new release of Qorus, target for improved performance, scalability and providing additional facilities for schema and database maintenance. The release is based on the Qore 0.7.4 library, which primarily consists of bug fixes since the Qore 0.7.3 release.

The system schema is updated to version 2.0.0 in this release. This is a major update to the system schema that could break compatibility with any code accessing the Qorus system schema directly. Issues Fixed in 2.0.0

Bug ID Severity Description
42 NORMAL a rare race condition updating workflow status could leave COMPLETE workflows with incorrect status (see here for fix)
71 MAJOR uncaught DB errors in schema upgrades could cause the DB to become corrupt and data to be lost (see here for fix)
72 MINOR omq.system.get-option-info() returns type "string" for all options, even options with other types (internal API fix)
73 NORMAL giving a single value for the workflow key "keylist" in the workflow definition file can cause spurious errors to be thrown (fixed in oload)
74 CRITICAL performance degrades quickly with asynchronous steps due to corrupted indexes on QUEUE_DATA on oracle (see here for fix)
75 CRITICAL all triggers to set the updated date/time are broken on all data servers (Oracle, PostgreSQL and MySQL; see here for fix)
76 CRITICAL ocmd does not parse string argument correctly (ex "00050-20" will be parsed and evaluated as an arithmetic expression and sent as "30"; fixed in ocmd; see here for fix)
77 NORMAL INCOMPLETE workflows cannot be set to CANCELED or ERROR (internal fix)
78 CRITICAL when a workflow order data instance's status is changed from CANCELED to ERROR, internal events are not queued (internal fix)
79 MAJOR signal handling is not functional when qorus is run in daemon mode (fixed in qore library 0.7.4)
80 MAJOR the Qorus HTTP server does not wait until data is received before closing the connection when returning an error status (fixed in qore library 0.7.4 and by setting TCP_NOWAIT in the Qorus HTTP server)
81 MAJOR Not possible to cancel workflow order data instances with status READY (internal fix)
82 MAJOR oload exits with non-user-friendly exception text when descriptions are longer than 240 characters instead of truncating and issuing a warning (fixed in oload)

Change List

  • Major Schema Changes
  • schema-tool Changes
    • schema-tool assumes the responsibility for running SQL scripts in all datasources, including Qorus schema upgrades and schema creation scripts
    • runs SQL scripts by using the Qore DB API, statement by statement; if an error occurs, by default execution stops and can be restarted from the point where the error occurred, supported for any datasource for Oracle, MySQL and PostgreSQL dataservers (fix for bug 71)
    • because SQL is now executed only through the API, the following client options in the options file are ignored:
      • qorus-client.sqlplus-command
      • qorus-client.psql-command
      • qorus-client.mysql-command
    • supports schema upgrades from any schema version as long as an upgrade path exists with schema upgrade scripts; the program now tries to find the shortest path from the initial version to the target version, and then executes the schema upgrade scripts in sequence using the new, safe, restartable DB API method as documented above.
    • includes options for scanning index health and rebuilding indexes and showing the block buffer cache hit ratio (on Oracle) and analyzing table and schema statistics on any defined datasource (Oracle, MySQL, and PostgreSQL)
    • includes new logic for tablespace name substitution to intelligently determine which tablespace names refer to data and index tablespaces, so that tablespace name substitution will work with any tablespace name on any datasource (in previous versions of Qorus, only Qorus system tablespace names could be substituted)
    • the following new command-line options support the new features:
      Option Description
      -A,–analyze[=ARG] analyze indexes and report status (Oracle only)
      ARG=datasource-name to use another schema
      -R,–rebuild[=ARG] analyze and rebuild indexes (Oracle only)
      ARG=datasource-name to use another schema
      -F,–force-rebuild[=ARG] force rebuilding of all indexes in system schema
      ARG=datasource-name to use another schema
      -S,–schema-stats[=ARG] gather statistics for system schema
      ARG=datasource-name to use another schema
      -T,–table-stats=ARG gather statistics/analyze table ARG
      -b,–block-buffer[=ARG] show block buffer hit ratio (Oracle only)
      ARG=datasource-name to use another schema
      -i,–ignore-errors ignore errors when loading schema files

  • Improved Workflow Execution
    Comprehensive workflow status is now maintained and managed within the Qorus workflow data cache, reducing the SQL load on the DB server and improving performance (also fixing bug 42); this is related to the following feature as well (fix for bug 42).

  • Aggressive Workflow Status Caching
    When this new feature is enabled, workflow, segment, and step status changes are not synchronized to the database until the workflow order data is released; meaning that in the ideal case (no errors occur, and the workflow order data instance remains in the cache), the workflow's data entries go from IN-PROGRESS to COMPLETE without any intermediate status changes in the DB.

    This drastically reduces the SQL load on the DB server and dramatically increases performance, particularly on workflows with very many of array step iterations. Recovery logic is not affected; after a system outage, when recovering, the step dependencies are used to find out which steps have a COMPLETE status and which need recovery, just as if the step statuses had been synchronized to the database. When the workflow data is released, all step statuses are synchronized normally to the database.

    Additionally, the system.info service has been enhanced to take workflow order data status information from the internal cache if available, so that workflow order data status details will show the actual workflow status, as summary information taken from the DB will show IN-PROGRESS for all segments and steps when the workflow order data is cached.

    Due to the large performance benefits of this feature, it is enabled by default in this version of Qorus. It can be disabled via the new boolean system option: qorus.flush-status
    • Stored Procedure API for Internal Use
      Delivered for most common actions on the system schema; this reduces SQL I/O load and dataserver CPU load required with dynamic SQL, thereby improving performance (for Oracle and PostgreSQL dataservers only).
    • Improved Qore-language Qorus Client Library
      The new Qorus client library provides $omqservice and $omqapi objects for easily calling Qorus API functions and service methods from client code. All of the command-line tools except ocmd have been updated to use these new objects for communication to the Qorus server. The $omqapi object supports both XML-RPC and JSON-RPC for communication to the Qorus server, so the qorus-client.client-url option in the options file may now be set to a jsonrpc:// URL, however, in this case, be aware that the JSON-RPC protocol does not have a native type for date/time values, so calls returning date/time values are generally returned as strings. This particular affects the ostatus program. For this reason, it is recommended not to use a client URL specifying the JSON-RPC protocol.
    • system.arch Service Updates
      The system.arch service was updated to work with the new schema version 2.0.0; the arch.stage1(), arch.stage2(), and arch.restore() methods were removed, creating the temporary table with a list of workflows to purge now includes the additional step of ensuring that only complete hierarchies are included by removing all workflows that have parent or child workflows in their hierarchies that are not included in the purge list; the arch.purge() method was updated to call arch.analyze_rebuild_indexes() after purging on Oracle as well, the service was updated to work with PostgreSQL and MySQL dataservers, and the following methods were added:
      • arch.purge_count(): returns the number of workflows that will be purged according to the current parameters
      • arch.info(): returns the current parameters in use by the service
      • arch.analyze_rebuild_indexes(): Oracle-only: analyzes indexes and rebuilds them if either the ora-idx-max-height (maximum allowable height for indexes, default is 3) or ora-idx-max-pct-del-leaf (maximum percentage of deleted leaf nodes, default 20) are exceeded. This method will only analyze indexes if the system property arch.rebuild-indexes is set to 0, otherwise it will also rebuild the indexes if necessary.

  • QUEUE_DATA Table Usage Changes
    The use of the QUEUE_DATA table changed; instead of deleting the row for the asynchronous data when it is no longer needed, the status is changed to USED in order to prevent the indexes on this table from quickly deteriorating on Oracle as happened with all Qorus versions up to this one. QUEUE_DATA rows are deleted from the database when arch.purge() is run on the workflow data (fix for bug 74)
  • system.info Service Updates
    The system.info service was updated to provide backwards-compatible data structures in existing service methods to maintain compatibility with user scripts, the old application GUI, and any external applications calling those methods; additionally, the following new methods were added giving streamlined information according to the new schema:
    • info.getWorkflowStatus2()
    • info.getWorkflowMetadata2()
    • info.getWorkflowMetadataFromName2()
    • info.getServiceMethod2()
    • info.getSimpleWorkflowMetadata2()
      The following methods were removed:
    • info.getFlowMetadata()
    Other methods taking the flow ID or flow instance ID now take the workflow ID or workflow instance ID, respectively.

  • ocmd Changes
    The ocmd program was enhanced to provide better command-line argument parsing for arguments to Qorus API calls (fix for bug 76)
  • oload Changes
    The oload program was updated in the following ways:
    • supports schema version 2.0.0
    • supports new workflow definition file format 2.0: differences from version 1.2 are:
      • no flows may be defined
      • the workflow data structure now must have a steps entry giving the step dependencies (previously under the flow object)
      • and no initflow element may be defined.
    • Workflow definition formats 1.0, 1.1, and 1.2 are still supported for backwards compatibility
    • if descriptions are longer than 240 characters, the description is truncated and a warning is issued (fix for bug 82)
    • no longer is responsible for running schema scripts through Qorus datasources, oload options to do this still work, but oload calls schema-tool to execute the SQL
  • system.omqmap Service Updates
    system.omqmap now caches complete workflow metadata information
  • system.queue Service Updates
    The queue.submitWaiting() and queue.submitCorrected() methods were removed from the service (these methods were documented as not to be called from user code in previous releases). The following new methods were added:
    • queue.correctData()
    • queue.updateKey()
    • queue.getInfo()
    • queue.getInfoFromQueueKey()
  • Log File Entry Changes
    Because flows have been removed from this version of Qorus, flow names and IDs were removed from log file entries as well
  • Changes to Default Values For System Options For the Workflow Cache
    The following options for the workflow order data cache have new default values, reflecting the larger role the cache plays in this version of Qorus: The default values will only be used if no other value has been set for these options in the Qorus system options file.
  • Set Order Keys At Workflow Creation
    It is now possible to set workflow order key-value pairs at order creation by passing a hash under the order-keys key of the workflow parameters hash in the createOrder() function and in the omq.system.submit-data.[workflow].[version]() API call.
  • Flow Information Removed from API Calls
  • Step Description Enhancements
    An optional desc key may be given in complex step definitions (when defined with a hash) to give the step description If the step description is not present, then the Qorus server will return the primary step function's description (if present) as the description of the step

System Schema Data Model 2.0.0 Changes

System schema version 2.0.0 makes major changes to table relationships and contents.

Note
Before upgrading to version 2.0.0, make sure to make a complete backup of the Qorus system schema, additionally, make sure enough space is available in rollback segments so that the schema migration can complete successfully.
  • FLOW and FLOW_INSTANCE Tables Removed
    All references to flows have been removed from the data model. There is no more FLOWS table for flow metadata, and no more FLOW_INSTANCE table for flow instances, relations between other tables to FLOWS.FLOWID and FLOW_INSTANCE.FLOW_INSTANCEID have been flattened to point directly to WORKFLOWS.WORKFLOWID and WORKFLOW_INSTANCE.WORKFLOW_INSTANCEID. This change was made because workflow and flow instances were always limited to a strict 1:1 relationship in all versions of Qorus up to this one. Removing flows makes the DB smaller, faster, and more efficient.
  • Workflow, Segment, Step and Queue Data Statuses Abbreviated
    In order to save space and improve performance in the database, workflow, segment, step, and queue data statuses are migrated to a single character as follows:
    Old Status New Status
    IN-PROGRESS I
    READY Y
    COMPLETE C
    RETRY R
    ERROR E
    WAITING W
    ASYNC-WAITING A
    INCOMPLETE N
    CANCELED X
    Note
    The unused NEW and PENDING statuses are mapped automatically to ERROR if found in the database.
  • Queue Data Statuses
    For the same reason, queue data statuses were also updated:
    Old Status New Status
    WAITING W
    RECEIVED R
    ERROR E
    USED U

USED (U) is a new status in schema version 2.0.0, indicating that the row is no longer active (see fix for bug 74). The changes are only visible in the data model; the system API and services always return and expect the old status strings for backwards-compatibility.

  • All CREATEDBY and MODIFIEDBY Columns Dropped
    All CREATEDBY and MODIFIEDBY columns were dropped from all tables containing them as they were nearly all universally filled with the schema owners' name anyway (only the oload tool would set the current UNIX username, which, in many installations, was the application user).
  • All Triggers Updated to Unconditionally Set the MODIFIED Column
    Triggers in previous versions did not set the MODIFIED column correctly. In this version, the MODIFIED columns are all updated unconditionally when a row is updated. All triggers on all tables with MODIFIED columns were updated with this fix (fix for bug 75).
  • Stored Procedure API for Oracle and PostgreSQL
    A package of stored procedures and functions for Oracle (package name: qorus_api) and a collection of functions for PostgreSQL have been developed and added to the system schema to improve performance and reduce the CPU load on the server by replacing the need to use dynamic SQL for common SQL tasks with stored procedure and/or function calls.
    Note
    These new database APIs are not for external use; the Qorus system API should be used instead to manipulate the system.
    See also
    Qorus 1.8 Series