Qorus Integration Engine®  4.0.3_git
arch-v4.0.3.qsd File Reference

Qorus archive service. More...

Classes

class  QorusSystemArchService
 the main arch service class More...
 

Detailed Description

Qorus archive service.

This service provides a data management and purging (deleting) service for Qorus system database. This service archives Qorus metadata and live/production data into an archiving database/datasource, and those data are removed from the live instance as well. It also supports handling DB table space management.

The only supported way to setup an archiving schema is to use schema-tool as follows:

prompt% schema-tool -V --arch=omqarch

In the above example, omqarch is a special system datasource defined in Datasource Connections - the above command will prepare the datasource for use as an archiving datasource by installing the system schema and setting internal system configuration parameters.

The service's main method is QorusSystemArchService::archive().

The arch service handles all live data related to workflow instances (order data) in COMPLETE and CANCELED statuses.

Errors raised by this service are prefixed with "ARCH" string; see List of Errors

Configuration

The service supports the following config items:

Key Mand. Default Description
datasource Y - The archive datasource name. The datasource has to be specified in Datasource Connections
day-interval Y - A number of days to be kept in current Qorus instance DB. For example, set it to 365 to keep one year history in the live/production schema - older completed/canceled orders will be archived
day-interval-workflows N - A number of days to be kept in current Qorus instance DB for workflow instances. For example, set it to 365 to keep one year history in the live/production schema - older completed/canceled orders will be archived. If there is no value for day-interval-workflows set, the day-interval is used.
day-interval-jobs N - A number of days to be kept in current Qorus instance DB for job instances. For example, set it to 365 to keep one year history in the live/production schema - older completed jobs will be archived. If there is no value for day-interval-workflows set, the day-interval is used.
batch-size N 1000 Size of single-batch transaction block. Its value can be sized to fit database UNDO/TEMP segment handling
batch-sleep N 0 A sleep/delay between batch loops (if configured in max-loops or in direct calls) in seconds
max-loops N 0 Set maximum loops (batch-size) related for archive/purge method. 0 means run until there is something to process.
no-metadata N False Do not archive metadata if it's set to True. Only live orders will be archived
archive N True Switch on/off archiving. The arch service does not do anything if it's set to False and the QorusSystemArchService::archive() method is called.
purge N False Switch on/off standalone purging. The arch service does not do anything if it's set to False and the QorusSystemArchService::purge() method is called. Standalone purge cannot be run if archive config is set to True.
space-management N False It can free database tables space if it is set to True
filter N - Optional data filter. See Archiving Filters for description
statistics N False Allow to compute database statistics in the source schema
statistics-options N - A hash with optional settings for statistics computation. See Database Statistics
shadow-datasources N - Optional list of datasource names to be kept on the same DB schema version. This option is used only for upgrading via schema-tool: Schema Manipulation Helper Tool. Potential usage: rotating arch.datasource schema etc. Option's value datasource can be listed here too.
Note
the "active" configuration item has been removed; to ensure that the service does not run, disable it with qrest put services/arch/disable

List of Errors

Error Error Text Action Suggested
ARCH-MISSING-FEATURE This feature is not implemented for this DB server This action is not available for currently used database platform. Contact Qore Technologies with new feature request
ARCH-ERROR-CONFIG Config item "datasource" is not set Ensure that an archiving datasource is defined in the Datasource Connections and set it in the "datasource" config item; see Configuration
ARCH-ERROR-CONFIG Qorus schema versions are different. Source: HASH Target: HASH Source (live/production) schema version is different from target (archive) schema version. Update archive schema version to requested version with standard Qorus tools
ARCH-ERROR-CONFIG Config item "day-interval" is 0 but must be greater than 0 Requested time interval is not configured correctly; see Configuration
ARCH-ERROR-CONFIG Archiving is disabled by config Archiving is disabled by user request. Enable it in the config item; see Configuration ("archive")
ARCH-ERROR-CONFIG Purging is disabled by config Purging is disabled by user request. Enable it with the "purge" config item; see Configuration ("purge")
ARCH-ERROR-CONFIG Unsupported cfg type: STRING An unexpected error occurred. Contact Qore Technologies with a bug report at suppo.nosp@m.rt@q.nosp@m.orete.nosp@m.chno.nosp@m.logie.nosp@m.s.co.nosp@m.m
ARCH-ERROR-CONFIG Cannot run plain-purge when archive mode is configured It's potentialy unsafe to run "purge" when the instance is configured to use "archive" (data cannot be removed without archiving)
ARCH-ERROR-CONFIG Space management is disabled by config Tablespace management is disabled by user request; enable it with the "space-management" config item; see Configuration ("space-management")
ARCH-ERROR-CONFIG Compute statistics disabled by config Statistics management is disabled by user request; enable it in with the "statistics" config item; see Configuration ("statistics")
ARCH-ERROR-INTERRUPT Run interrupted by user request User can interrupt the currently running archive/purge action by calling the interrupt() method
ARCH-ERROR-FILTER Filter error Some values provided to archive or purge filters are invalid. Fix the filter regarding the error message

Archiving Filters

The archiving process can be set up with additional filters, for example, to allow handling of specified workflows only.

Filter setup

Note
All workflow instances are checked as a full tree. It means that filter can be skipped if the filtered out workflow instance is a sub-workflow for example.

Filter structure

Filters are set in a hash. Filters in the top-level hash are logically AND-ed; filter values are logically OR-ed. See example below.

The allowed keys are:

Key Type Description
workflow_instanceid list a list with exact workflow instance IDs
workflows list a list of hashes where the hash is created with workflow name : workflow version pair
day-interval integer If set it overrides base configuration day-interval value
day-interval-jobs integer If set it overrides base configuration day-interval-jobs value
day-interval-workflows integer If set it overrides base configuration day-interval-workflows value
batch-size integer If set it overrides base configuration batch-size value
batch-sleep integer If set it overrides base configuration batch-sleep value
no-metadata c boolean Do not archive metadata if it's set to True. Only live orders will be archived

Filter example:

# select WFIID 1 or 345 or 1200 but only when they are instances of
# SIMPLETEST:1.0 or ARRAYTEST:2.0
hash<auto> filter = {
"workflow_instanceid": (1, 345, 1200),
"workflows": {
"SIMPLETEST": "1.0",
"ARRAYTEST": "2.0"
},
"day-interval": 30,
);

Database Related Information

The arch service uses its own datasources for production and archive DB instances. There are no Qorus system datasources used at all. So for example brute force session killing will affect only current arch runs.

Database Statistics

Warning
Statistics are critical part of any modern database platform. Please be extreme careful when you play with it. In any doubt consult your database administrator.

Qorus arch service uses Qore's SqlUtil DBA Tools for this area internally. Please consult the documentation for various options - chapter: Database Statistics

Oracle Session Instrumentation

Archiving DB sessions are specially marked in the DB layer if the Oracle Database Server is used and if the omq datasource has privileges to DBMS_APPLICATION_INFO package (usually available by default).

Sessions are marked in Oracle system catalogue views like v$sessions:

Column Value or Example Description
MODULE 'Qorus ARCH' Always string 'Qorus ARCH'. It can be used as a quick reference for archiving
ACTION 'space management' A string describing currently running step. For example: 'metadata archiving', 'instance archiving', etc.
CLIENT_INFO 'QORUS-ARCH: rimmer-1' A string formed by 'QORUS-ARCH:' and Qorus option qorus.instance-key