Qorus archive service. More...

Classes
class	QorusSystemArchService
	the main arch service class More...

Detailed Description

Qorus archive service.

This service provides a data management and purging (deleting) service for Qorus system database. This service archives Qorus metadata and live/production data into an archiving database/datasource, and those data are removed from the live instance as well. It also supports handling DB table space management.

The only supported way to setup an archiving schema is to use schema-tool as follows:

prompt% schema-tool -V --arch=omqarch

In the above example, omqarch is a special system datasource defined in Datasource Connections - the above command will prepare the datasource for use as an archiving datasource by installing the system schema and setting internal system configuration parameters.

The service's main method is QorusSystemArchService::archive().

The arch service handles all live data related to workflow instances (order data) in COMPLETE and CANCELED statuses.

Errors raised by this service are prefixed with "ARCH" string; see List of Errors

Configuration

The service supports the following config items:

Key	Mand.	Default	Description
`datasource`	Y	-	The archive datasource name. The datasource has to be specified in Datasource Connections
`day-interval`	Y	-	A number of days to be kept in current Qorus instance DB. For example, set it to 365 to keep one year history in the live/production schema - older completed/canceled orders will be archived
`day-interval-workflows`	N	-	A number of days to be kept in current Qorus instance DB for workflow instances. For example, set it to 365 to keep one year history in the live/production schema - older completed/canceled orders will be archived. If there is no value for `day-interval-workflows` set, the `day-interval` is used.
`day-interval-jobs`	N	-	A number of days to be kept in current Qorus instance DB for job instances. For example, set it to 365 to keep one year history in the live/production schema - older completed jobs will be archived. If there is no value for `day-interval-workflows` set, the `day-interval` is used.
`batch-size`	N	`1000`	Size of single-batch transaction block. Its value can be sized to fit database UNDO/TEMP segment handling
`batch-sleep`	N	`0`	A sleep/delay between batch loops (if configured in `max-loops` or in direct calls) in seconds
`max-loops`	N	`0`	Set maximum loops (batch-size) related for archive/purge method. 0 means run until there is something to process.
`no-metadata`	N	`False`	Do not archive metadata if it's set to `True`. Only live orders will be archived
`archive`	N	`True`	Switch on/off archiving. The arch service does not do anything if it's set to `False` and the QorusSystemArchService::archive() method is called.
`purge`	N	`False`	Switch on/off standalone purging. The arch service does not do anything if it's set to `False` and the QorusSystemArchService::purge() method is called. Standalone purge cannot be run if `archive` config is set to True.
`space-management`	N	`False`	It can free database tables space if it is set to `True`
`filter`	N	-	Optional data filter. See Archiving Filters for description
`statistics`	N	`False`	Allow to compute database statistics in the source schema
`statistics-options`	N	-	A hash with optional settings for statistics computation. See Database Statistics
`shadow-datasources`	N	-	Optional list of datasource names to be kept on the same DB schema version. This option is used only for upgrading via schema-tool: Schema Manipulation Helper Tool. Potential usage: rotating `arch.datasource` schema etc. Option's value `datasource` can be listed here too.

Note: the "active" configuration item has been removed; to ensure that the service does not run, disable it with qrest put services/arch/disable

List of Errors

Error	Error Text	Action Suggested
`ARCH-MISSING-FEATURE`	This feature is not implemented for this DB server	This action is not available for currently used database platform. Contact Qore Technologies with new feature request
`ARCH-ERROR-CONFIG`	Config item `"datasource"` is not set	Ensure that an archiving datasource is defined in the Datasource Connections and set it in the `"datasource"` config item; see Configuration
`ARCH-ERROR-CONFIG`	Qorus schema versions are different. Source: HASH Target: HASH	Source (live/production) schema version is different from target (archive) schema version. Update archive schema version to requested version with standard Qorus tools
`ARCH-ERROR-CONFIG`	Config item `"day-interval"` is 0 but must be greater than 0	Requested time interval is not configured correctly; see Configuration
`ARCH-ERROR-CONFIG`	Archiving is disabled by config	Archiving is disabled by user request. Enable it in the config item; see Configuration (`"archive"`)
`ARCH-ERROR-CONFIG`	Purging is disabled by config	Purging is disabled by user request. Enable it with the `"purge"` config item; see Configuration (`"purge"`)
`ARCH-ERROR-CONFIG`	Unsupported cfg type: STRING	An unexpected error occurred. Contact Qore Technologies with a bug report at suppo.nosp@m.rt@q.nosp@m.orete.nosp@m.chno.nosp@m.logie.nosp@m.s.co.nosp@m.m
`ARCH-ERROR-CONFIG`	Cannot run plain-purge when archive mode is configured	It's potentialy unsafe to run `"purge"` when the instance is configured to use `"archive"` (data cannot be removed without archiving)
`ARCH-ERROR-CONFIG`	Space management is disabled by config	Tablespace management is disabled by user request; enable it with the `"space-management"` config item; see Configuration (`"space-management"`)
`ARCH-ERROR-CONFIG`	Compute statistics disabled by config	Statistics management is disabled by user request; enable it in with the `"statistics"` config item; see Configuration (`"statistics"`)
`ARCH-ERROR-INTERRUPT`	Run interrupted by user request	User can interrupt the currently running archive/purge action by calling the interrupt() method
`ARCH-ERROR-FILTER`	Filter error	Some values provided to `archive` or `purge` filters are invalid. Fix the filter regarding the error message

Archiving Filters

The archiving process can be set up with additional filters, for example, to allow handling of specified workflows only.

Filter setup

globally for all archiving runs: use method QorusSystemArchService::set_filter()
for the current run: use the $filter argument for QorusSystemArchService::archive() or QorusSystemArchService::purge(). Note that setting the filter in the method overrides any existing global filter set by QorusSystemArchService::set_filter().

Note: All workflow instances are checked as a full tree. It means that filter can be skipped if the filtered out workflow instance is a sub-workflow for example.

Filter structure

Filters are set in a hash. Filters in the top-level hash are logically AND-ed; filter values are logically OR-ed. See example below.

The allowed keys are:

Key	Type	Description
`workflow_instanceid`	`list`	a list with exact workflow instance IDs
`workflows`	`list`	a list of hashes where the hash is created with workflow name : workflow version pair
`day-interval`	`integer`	If set it overrides base configuration `day-interval` value
`day-interval-jobs`	`integer`	If set it overrides base configuration `day-interval-jobs` value
`day-interval-workflows`	`integer`	If set it overrides base configuration `day-interval-workflows` value
`batch-size`	`integer`	If set it overrides base configuration `batch-size` value
`batch-sleep`	`integer`	If set it overrides base configuration `batch-sleep` value
`no-metadata`	c boolean	Do not archive metadata if it's set to `True`. Only live orders will be archived

Filter example:

# select WFIID 1 or 345 or 1200 but only when they are instances of
# SIMPLETEST:1.0 or ARRAYTEST:2.0
hash<auto> filter = {
    "workflow_instanceid": (1, 345, 1200),
    "workflows": {
        "SIMPLETEST": "1.0",
        "ARRAYTEST": "2.0"
    },
    "day-interval": 30,
);

Database Related Information

The arch service uses its own datasources for production and archive DB instances. There are no Qorus system datasources used at all. So for example brute force session killing will affect only current arch runs.

Database Statistics

Warning: Statistics are critical part of any modern database platform. Please be extreme careful when you play with it. In any doubt consult your database administrator.

Qorus arch service uses Qore's SqlUtil DBA Tools for this area internally. Please consult the documentation for various options - chapter: Database Statistics

Oracle Session Instrumentation

Archiving DB sessions are specially marked in the DB layer if the Oracle Database Server is used and if the omq datasource has privileges to DBMS_APPLICATION_INFO package (usually available by default).

Sessions are marked in Oracle system catalogue views like v$sessions:

Column	Value or Example	Description
`MODULE`	`'Qorus` `ARCH'`	Always string 'Qorus ARCH'. It can be used as a quick reference for archiving
`ACTION`	`'space` `management'`	A string describing currently running step. For example: 'metadata archiving', 'instance archiving', etc.
`CLIENT_INFO`	`'QORUS-ARCH`: `rimmer-1'`	A string formed by 'QORUS-ARCH:' and Qorus option qorus.instance-key