Qorus Integration Engine®  5.1.39_git
arch-v5.1.qsd File Reference

More...

Classes

class  QorusSystemArchService
 system arch service More...
 

Detailed Description

Qorus archive service

See also
QorusSystemArchService for method information This service provides a data management and purging (deleting) service for Qorus system database. This service archives Qorus metadata and live/production data into an archiving database/datasource, and those data are removed from the live instance as well. It also supports handling DB table space management. The only supported way to setup an archiving schema is to use schema-tool as follows: @verbatimprompt% schema-tool -V –arch=omqarch In the above example, omqarch is a special system datasource defined in Datasource Connections - the above command will prepare the datasource for use as an archiving datasource by installing the system schema and setting internal system configuration parameters. The service's main method is QorusSystemArchService::archive(). The arch service handles all live data related to workflow instances (order data) in COMPLETE and CANCELED statuses. Errors raised by this service are prefixed with "ARCH" string; see archerrors archconfig Configuration The service supports the following config items:
Key Mand. Default Description
datasource Y - The archive datasource name. The datasource has to be specified in Datasource Connections
day-interval Y - A number of days to be kept in current Qorus instance DB. For example, set it to 365 to keep one year history in the live/production schema - older completed/canceled orders will be archived
day-interval-workflows N - A number of days to be kept in current Qorus instance DB for workflow instances. For example, set it to 365 to keep one year history in the live/production schema - older completed/canceled orders will be archived. If there is no value for day-interval-workflows set, the day-interval is used.
day-interval-jobs N - A number of days to be kept in current Qorus instance DB for job instances. For example, set it to 365 to keep one year history in the live/production schema - older completed jobs will be archived. If there is no value for day-interval-workflows set, the day-interval is used.
batch-size N 1000 Size of single-batch transaction block. Its value can be sized to fit database UNDO/TEMP segment handling
batch-sleep N 0 A sleep/delay between batch loops (if configured in max-loops or in direct calls) in seconds
max-loops N 0 Set maximum loops (batch-size) related for archive/purge method. 0 means run until there is something to process.
no-metadata N False Do not archive metadata if it's set to True. Only live orders will be archived
archive N True Switch on/off archiving. The arch service does not do anything if it's set to False and the QorusSystemArchService::archive() method is called.
purge N False Switch on/off standalone purging. The arch service does not do anything if it's set to False and the QorusSystemArchService::purge() method is called. Standalone purge cannot be run if archive config is set to True.
space-management N False It can free database tables space if it is set to True
filter N - Optional data filter. See Archiving Filters The archiving process can be set up with additional filters, for example, to allow handling of specified workflows only. Filter setup - globally for all archiving runs: use method QorusSystemArchService::set_filter() - for the current run: use the $filter argument for QorusSystemArchService::archive() or QorusSystemArchService::purge(). Note that setting the filter in the method overrides any existing global filter set by QorusSystemArchService::set_filter(). for description
statistics N False Allow to compute database statistics in the source schema
statistics-options N - A hash with optional settings for statistics computation. See archdatabases-statistics
shadow-datasources N - Optional list of datasource names to be kept on the same DB schema version. This option is used only for upgrading via schema-tool: Schema Manipulation Helper Tool. Potential usage: rotating arch.datasource schema etc. Option's value datasource can be listed here too.
Note
the "active" configuration item has been removed; to ensure that the service does not run, disable it with qrest put services/arch/disable

List of Errors <table>

Error

Error Text

Action Suggested

ARCH-MISSING-FEATURE

This feature is not implemented for this DB server

This action is not available for currently used database platform. Contact Qore Technologies with new feature request

ARCH-ERROR-CONFIG

Config item "datasource" is not set

Ensure that an archiving datasource is defined in the Datasource Connections and set it in the "datasource" config item; see Configuration The service supports the following config items:

ARCH-ERROR-CONFIG

Qorus schema versions are different. Source: HASH Target: HASH

Source (live/production) schema version is different from target (archive) schema version. Update archive schema version to requested version with standard Qorus tools

ARCH-ERROR-CONFIG

Config item "day-interval" is 0 but must be greater than 0

Requested time interval is not configured correctly; see Configuration The service supports the following config items:

ARCH-ERROR-CONFIG

Archiving is disabled by config

Archiving is disabled by user request. Enable it in the config item; see Configuration The service supports the following config items: ("archive")

ARCH-ERROR-CONFIG

Purging is disabled by config

Purging is disabled by user request. Enable it with the "purge" config item; see Configuration The service supports the following config items: ("purge")

ARCH-ERROR-CONFIG

Unsupported cfg type: STRING

An unexpected error occurred. Contact Qore Technologies with a bug report at suppo.nosp@m.rt@q.nosp@m.orete.nosp@m.chno.nosp@m.logie.nosp@m.s.co.nosp@m.m

ARCH-ERROR-CONFIG

Cannot run plain-purge when archive mode is configured

It's potentialy unsafe to run "purge" when the instance is configured to use "archive" (data cannot be removed without archiving)

ARCH-ERROR-CONFIG

Space management is disabled by config

Tablespace management is disabled by user request; enable it with the "space-management" config item; see Configuration The service supports the following config items: ("space-management")

ARCH-ERROR-CONFIG

Compute statistics disabled by config

Statistics management is disabled by user request; enable it in with the "statistics" config item; see Configuration The service supports the following config items: ("statistics")

ARCH-ERROR-INTERRUPT

Run interrupted by user request

User can interrupt the currently running archive/purge action by calling the interrupt() method

ARCH-ERROR-FILTER

Filter error

Some values provided to archive or purge filters are invalid. Fix the filter regarding the error message

Archiving Filters The archiving process can be set up with additional filters, for example, to allow handling of specified workflows only. <b>Filter setup</b> - globally for all archiving runs: use method QorusSystemArchService::set_filter() - for the current run: use the \c $filter argument for QorusSystemArchService::archive() or QorusSystemArchService::purge(). Note that setting the filter in the method overrides any existing global filter set by QorusSystemArchService::set_filter(). @note All workflow instances are checked as a full tree. It means that filter can be skipped if the filtered out workflow instance is a sub-workflow for example. <b>Filter structure</b> Filters are set in a hash. Filters in the top-level hash are logically AND-ed; filter values are logically OR-ed. See example below. The allowed keys are: <table>

Key

Type

Description

workflow_instanceid

list

a list with exact workflow instance IDs

workflows

list

a list of hashes where the hash is created with workflow name : workflow version pair

day-interval

integer

If set it overrides base configuration day-interval value

day-interval-jobs

integer

If set it overrides base configuration day-interval-jobs value

day-interval-workflows

integer

If set it overrides base configuration day-interval-workflows value

batch-size

integer

If set it overrides base configuration batch-size value

batch-sleep

integer

If set it overrides base configuration batch-sleep value

no-metadata

c boolean

Do not archive metadata if it's set to True. Only live orders will be archived

Filter example:

# select WFIID 1 or 345 or 1200 but only when they are instances of# SIMPLETEST:1.0 or ARRAYTEST:2.0hash<auto> filter = { "workflow_instanceid": (1, 345, 1200), "workflows": { "SIMPLETEST": "1.0", "ARRAYTEST": "2.0" }, "day-interval": 30,);

Database Related Information The arch service uses its own datasources for production and archive DB instances. There are no Qorus system datasources used at all. So for example brute force session killing will affect only current \c arch runs. @subsection archdatabases-statistics Database Statistics @warning Statistics are critical part of any modern database platform. Please be extreme careful when you play with it. In any doubt consult your database administrator. Qorus arch service uses Qore's SqlUtil DBA Tools for this area internally. Please consult the documentation for various options - chapter: Database Statistics @subsection archdatabases-oracle Oracle Session Instrumentation Archiving DB sessions are specially marked in the DB layer if the Oracle Database Server is used and if the \c omq datasource has privileges to \c DBMS_APPLICATION_INFO package (usually available by default). Sessions are marked in Oracle system catalogue views like \c v$sessions: <table>

Column

Value or Example

Description

MODULE

'Qorus ARCH'

Always string 'Qorus ARCH'. It can be used as a quick reference for archiving

ACTION

'space management'

A string describing currently running step. For example: 'metadata archiving', 'instance archiving', etc.

CLIENT_INFO

'QORUS-ARCH: rimmer-1'

A string formed by 'QORUS-ARCH:' and Qorus option qorus.instance-key