Qorus Integration Engine®  4.0.3_git
Installation Guide
See also
Open-Source References and Components

Installation Overview

Qorus Integration Engine® is a self-contained application that is relatively easy to install. The most complex part of the installation is the preparation of the database (covered in Database Configuration Overview).

Minimum System Requirements

Qorus Installation Engine® can be installed on a wide variety of UNIX/Linux systems as well as other operating systems when using Docker containers.

Minimum System Requirements:

  • CPU: x86_64 (64-bit) CPU
  • RAM: 2 GB RAM
  • Disk: 250 MB
  • Operating System: Linux or any operating system supporting Docker

The above minimum requirements are for a small installation with limited capabilities. Large installations of Qorus Integration Engine® require significantly more RAM and disk space. For example, a very large Qorus Integration Engine® installation might have 80GB of RAM, 10 GB of disk space for logs, and 500 GB of disk space for the database.

Java Requirements

For Java support, Qorus requires a OpenJDK 8, with the JDK installation where javac is available in the PATH.

This is because Qorus interface code using Java is always delivered as source and is compiled when loaded into the system. This ensures that Qorus can provide full source code transparency while also taking advantage of efficient Java byte code execution (as well as JIT).

Jar and class files used by Qorus interfaces can be delivered in compiled form and do not have to be delived in source code form.

See also
Developing in Java for more information

Linux Operating System Requirements

Qorus Integration Engine® 4.0.3_git has a distributed architecture and requires a large number of descriptors for network communication and process entries for processes and threads.

Linux Kernel Settings

Ensure that the following kernel parameter is set to at least the following value (larger numbers are also OK):

  • fs.file-max = 5242880

For Red Hat Enterprise Linux, CentOS, Oracle Linux, and other Red Hat-based distributions, create or edit an appropriate file in /etc/sysctl.d with a line as above, and then run

/sbin/sysctl -p 

as root.

For other Linuxes, create or edit /etc/sysctl.conf with a line as above, and then run

/sbin/sysctl -p 

as root; if this file does not exist, see your distribution's documentation about how to set kernel parameters.

The recommended value for the kernel's maximum PID value is as follows (larger numbers are also OK):

  • kernel.pid_max = 131072

If this value is too low, it can be changed as per the above instructions.

Note
  • For Linux kernel 3+ the limit on the number of descriptors can be checked by typing:
    cat /proc/sys/fs/file-max 
    and the current number of descriptors in use can be checked by typing:
    cat /proc/sys/fs/file-nr 
    this command returns three numbers: allocated file descriptors, free allocated file descriptors, and maximum file descriptors
  • For Linux kernel 3+ the limit on the number of processes can be checked by typing:
    cat /proc/sys/kernel/pid_max 

Linux User Limit Settings

The user running Qorus Integration Engine® 4.0.3_git must have minimum hard and soft limits for the number of descriptors available (nofile) and the number of processes and threads (nproc).

The minimum recommended soft limit values for each parameter are as follows:

  • nofile: 8192
  • nproc: 8192

For Red Hat Enterprise Linux, CentOS, Oracle Linux, and other Red Hat-based distributions, create or edit an appropriate file in /etc/security/limits.d as in the following examples:

/etc/security/limits.d/20-nproc.conf
* soft nproc 8192
root soft nproc unlimited
/etc/security/limits.d/25-nofile.conf
* soft nofile 8192
* hard nofile 16384

Substitute "*" in the lines above with the application user name to set these limits for the Qorus user only.

Note
  • nproc applies to processes and threads together, and each Qorus Integration Engine® 4.0.3_git process is multithreaded, therefore the nproc number must be significantly higher than the maximum number of Qorus processes running at any one time.
  • To check the hard limits for the number of descriptors (nofile) and the number of processes and threads (nproc) for your user, type:
    ulimit -Hn -Hu 
  • If multiple installations of Qorus Integration Engine® 4.0.3_git will be used on a single machine with a single Linux user, the above per-user limits should be raised accordingly
  • Qorus will start with lower limits, but when runtime limits exceed 80% of the system-defined limits, ongoing alerts are raised, and after this as the limit approaches the maximum value, Qorus will refuse to start interfaces.

Database Requirements

Qorus Integration Engine® 4.0.3_git requires a database for persistent storage. The database server must be installed and configured separately to Qorus Integration Engine® installation; this manual does not explain how to install and configure the database server in detail, but rather explains how to install and configure the Qorus system schema in an already-running database server.

Qorus Integration Engine® can use an Oracle, MySQL/MariaDB, or PostgreSQL database to store its system schema containing all user object (workflow, service, class definitions, etc) metadata and workflow state information. Instructions on how to install the database schema (once the database server software itself has been installed and configured) for Qorus on each supported database type are given in Database and Schema Preparation.

See the following sections for detailed requirements: Oracle Requirements, MySQL/MariaDB Requirements, PostgreSQL Requirements.

Oracle Requirements

Oracle Server Requirements
The Oracle database server must be 11.2 or higher. Please note that Oracle Express Edition, available as a free of charge download from Oracle, and freely licensed for commercial use, is also supported for the database server.

Oracle Client Libraries
Oracle 11.2 or newer client shared libraries are required for Oracle functionality. Qorus Integration Engine®’s oracle module is dynamically linked with the Oracle client libraries, therefore earlier unsupported versions will not work.

Oracle Instant Client (available as a free of charge for download from Oracle), Oracle client libraries from a 11.2 or newer server installation, or Oracle client libraries from an Oracle Express Edition installation can be used.

If Oracle Instant Client libraries are to be used, the following Oracle Instant Client package is required at a minimum:

  • instantclient-basic
Note
You must have the client installed for the same architecture as the Qorus server; for example, using Qorus x86_64 binaries requires x86_64 Oracle client libraries. In general, 64-bit versions of Qorus cannot function with 32-bit Oracle client libraries.

MySQL/MariaDB Requirements

MariaDB Server Requirements
The minimum supported version for the MySQL/MariaDB server is 5.1. You cannot install Qorus 4.0.3_git on earlier MySQL/MariaDB versions.

Note
While you can install Qorus on any MySQL/MariaDB version from 5.1 onwards, using a MySQL/MariaDB database in a commercial setting requires a MariaDB Network license from MariaDB AB; no such license is included with Qorus Integration Engine®. In general, no third-party software licenses are included with Qorus Integration Engine®.

MariaDB Client Libraries
Support for MySQL/MariaDB is statically linked using MariaDB v5 client libraries to the mysql module provided with Qorus Integration Engine®. MariaDB client support is provided under an open source license; no additional files are required.

PostgreSQL Requirements

PostgreSQL Server Requirements
The minimum supported version for the PostgreSQL server is 9.0.0. You cannot install Qorus 4.0.3_git on earlier PostgreSQL versions.

PostgreSQL Client Libraries
PostgreSQL client libraries are required to communicate with PostgreSQL servers. Version 9 client libraries should be used; normally the same or newer version than the server should be used.

SAP / Sybase ASE Requirements

SAP / Sybase Adaptive Server Enterprise is not currently supported for the Qorus Integration Engine® system schema. You cannot install the Qorus Integration Engine® system schema on a SAP ASE database; you can only communicate with SAP ASE databases from Qorus user code.

SAP ASE Open Client 15 client libraries are required to communicate with SAP ASE servers using the "sybase" driver. Qorus Integration Engine®’s sybase module is dynamically linked with the SAP ASE Open Client 15 client libraries, therefore no previous version will work.

Note
Sybase / SAP ASE databases can also be accessed using the freetds driver included with Qorus (which also has the advantage or providing access to MS SQL Server DBs as well).

Database and Schema Preparation

Database Configuration Overview

Regardless of the installation type (LSB or tar), a database must be prepared to hold the Qorus system schema before the Qorus server can be started.

Qorus Integration Engine® requires two database schemas to operate as in the following table.

Qorus Integration Engine® System Schemas (Datasources)

Schema Type Description
omq Required System schema, system and user metadata, workflow data will be installed here
omquser Recommended User-modifiable schema containing custom user objects; for initial installs, this schema will be empty

The system and user schemas can be created in different databases; they can even be of different database types.

Each schema will be accessed through a datasource (see Datasource Connections), which represents a connection to a schema on a particular database server, using a particular username and password, as well as other optional information.

The omq datasource must point to an Oracle, MySQL/MariaDB/Percona, or PostgreSQL database, but the omquser Datasource has no restrictions on the database driver.

Note that the database server does not necessarily have to be located on the same system as the Qorus Integration Engine® server, however, for performance reasons, a fast network connection is required between the two systems at the minimum.

Note
Note that it is highly recommended that any Qorus Integration Engine® database instance destined for production use should be in a clustered configuration to ensure the highest possible reliability. Additionally, regular backups/dumps of the database should be made in case of a catastrophic database error. All Qorus data, metadata, configuration and states, order data, etc, is stored in the system schema. If the system schema becomes corrupted, Qorus will not function.

Note that no changes can be made by the user to the Qorus system schema or Qorus may no longer function. The optional user schema may be used in any way desired.

The following sections will describe how to set up the system and user schemas for Oracle, PostgreSQL, and MySQL/MariaDB/Percona databases.

Database Sizing General Information

Qorus database/tablespace sizing is highly dependent on the application architecture, the amount of workflow order instance data stored, the type and volume of data stored workflows and jobs, and the archiving frequency.

Best Practices for Archiving

  • Setup the system.arch service and the qorus-archiver job to keep the live DB schema as small as possible.
  • Proactively resolve all workflow order instances with ERROR statuses to ensure efficient archiving runs
  • Execute standard DBA tasks regularly (update statistics, vacuum on PostgreSQL, etc.)
  • Monitor usage of table segments and assign appropriate storage parameters to table(s) as necessary
Note
An archiving plan is a must for any Qorus instance; Qorus performance will suffer with very large databases caused by lack of archiving.

Datbase Sizing Formula

The following is a rough formula for determining the tablespace sizing for an initial install, assuming a conservatively large workflow order instance data size and data to index ratio:

omq_data tablespace size = max. workflow order instances * 25 KB + max. job history records * 10 KB
omq_index tablespace size = omq_data tablespace size / 3

The minumum recommended tablespace sizes for a test or evaluation instance are:

omq_data: 256 MB
omq_index: 128 MB

The space used by system tables is different for each table depending on its function. See Data Model

  • List of values (lookup) tables.
    • Static storage
    • Very small
  • Metadata tables
    • Tables containing metadata and source code for Qorus objects (workflows, jobs, instances)
    • Static storage most of the time
    • Small to medium
  • Instance tables
    • Tables containing order/instance data for each processing
    • OLTP
    • Large

Database Planning for Special Tables

This is a list if tables which may require special care by DBAs in very large installations with the most important tables listed first.

The most critical table is the table holding workflow order data with columns staticdata and dynamicdata stored as large character objects (CLOB, TEXT depending on the DB server for the system schema):

  • ORDER_INSTANCE

The following tables grow quickly in some circumstances as well:

  • WORKFLOW_INSTANCE
  • JOB_INSTANCE
  • ERROR_INSTANCE
  • SENSITIVE_ORDER_DATA (LOB)
  • ORDER_INSTANCE_KEYS
  • SENSITIVE_ORDER_DATA_KEYS

Depending on the processing volume, the system may benefit from a custom storage setup for particular tables and data; while monitoring the system database, to optimize application performance, DBAs can consider establishing the following:

  • dedicated tablespace(s) for tables listed above
  • dedicated dedicated tablespace(s) for large objects
  • dedicated tablespace(s) for indexes

Database Sizing Example

Consider a real life example of a system where there are many workflows used with large staticdata, dynamicdata, and sensitivedata:

  • workflow count: 360
  • overall DB size: 1 TB
  • ORDER_INSTANCE table size: 9 GB
    • dynamicdata LOB storage: 481 GB
    • staticdata LOB storage: 201 GB
  • archive DB size: 12 TB
  • retention period in the live schema: 1 month
  • retention period for the archive schema: 2 years
Note
Keeping data archived for 2 years as in the above example will have an adverse impact on any queries that also search archived data due to the very large DB size required to store the data. Additionally, every organization should review its data retention policies regarding sensitive data, as legal and regulatory requirements may impact the data archiving plan. It is generally recommended to keep data in the live and archive schemas for as little time as possible for performance reasons.

Oracle Configuration

This step describes how to set up Oracle database schemas and tablespaces for use with Qorus Integration Engine® on an Oracle 11.2 database server or later. The creation of an Oracle instance is outside the scope of this documentation. Use the dbca tool or see your Oracle documentation for more information. However Qore Technologies highly recommends that any Qorus Integration Engine® database instance destined for production use should be in a clustered configuration to ensure the highest possible reliability.

Note
Note that even with clustered configurations, load-sharing must be disabled in the tnsnames.ora file used by the Qorus server; the Qorus server must make all connections to the same physical server as the cluster synchronization delay will cause problems with synchronized transaction management within the Qorus server.

The Qorus system datasources to be defined in the next step that will point to the schemas described here are omq (for the Qorus system schema) and optionally omquser (for the optional user schema). The omq system schema is necessary for the operation of the system.

Oracle Privileges

The priviledges described here apply to the system schema user (omq). Example scripts for creating tablespaces and schema users follow in Example Tablespace and User Creation Scripts.

Name Mandatory Qorus Module Description
CREATE SESSION Yes Core allows the user to connect to the database
CREATE PROCEDURE Yes Core allows the user to create procedures
CREATE SEQUENCE Yes Core allows the user to create sequences
CREATE TABLE Yes Core allows the user to create tables
CREATE TRIGGER Yes Core allows the user to create triggers
CREATE TYPE Yes Core allows the user to create types
CREATE VIEW Yes Core allows the user to create views
EXECUTE DBMS_STATS No schema-tool: Schema Manipulation Helper Tool, Deleting Old Data From the Database This package is used for gathering statistics, but it can also be performed manually by a DBA
EXECUTE DBMS_OUTPUT No Debug Used in the debugging code of the system info service
EXECUTE DBMS_APPLICATION_INFO No Deleting Old Data From the Database Used for logging while archiving
SELECT V$BUFFER_POOL_STATISTICS No schema-tool: Schema Manipulation Helper Tool schema-tool: Schema Manipulation Helper Tool uses it for index tree analysis, but it can also be handled by manually by a DBA
Note
  • Oracle suggests using roles rather than raw GRANTs in its documentation; please consult your DBA when setting up new Oracle schemas
  • The names and passwords of the schemas can be set per installation; any valid Oracle name can be used. In this document, the names omq and omquser will be used to refer to these schemas. In the next step, the datasources omq and omquser will be set up to point to these schemas as well.

Oracle Tablespaces

Each schema requires two tablespaces, one for data and one for indexes. Tablespace names are flexible and do not necessarily have to be as described in the table below; tablespace names can be overridden with the –-data-ts and –index-ts options to the oload and schema-tool programs and by default in the options file with the qorus-client.omq-data-tablespace and qorus-client.omq-index-tablespace options. However, for the rest of this document, the names will appear as given below.

Schema Tablespace Type Description
omq omq_data Required All system data is stored in the tablespace; this should be the default tablespace for the omq user/schema, and the user must have permissions to write to this tablespace (ie UNLIMITED TABLESPACE or a quota on this tablespace)
omq omq_index Required All system index are stored here.
omquser omquser_data Optional All data in the user schema is stored in this tablespace; this should be the default tablespace for the omquser user/schema, and the user must have permissions to write to this tablespace (ie UNLIMITED TABLESPACE or a quota on this tablespace)
omquser omquser_index Optional All user indexes are stored here.
Note
If you use alternate tablespace names, you must define the tablespace names in the options file with the qorus-client.omq-data-tablespace and qorus-client.omq-index-tablespace options so that automatic schema management will work properly

Oracle Tablespace Sizing

See Database Sizing General Information as an initial reference

The optional omquser schema will be solely populated with user objects therefore the storage characteristics are installation-dependent and therefore no sizing information can be given here. The Qorus development team and Oracle DBAs should confer and agree on sizing during the design and development process of workflows and services.

To create the two required Oracle schemas and the tablespaces, SQL similar to that found in the $OMQ_DIR/templates/omq-db.sql and $OMQ_DIR/templates/omquser-db.sql can be used. Note that the directory paths and file names should match your installation, and of course the tablespace sizes should match your planning. These files are only given as examples and should not be used without modification for your environment and ideally should be verified by a DBA prior to execution.

Oracle cluster configuration is outside the scope of this document (see your Oracle documentation instead), but a clustered database is recommended for production instances.

Once the schemas and tablespaces have been created, verify the connectivity using $ORACLE_HOME/bin/sqlplus.

Note
As mentioned before, ensure that load balancing is not enabled in the tnsnames.ora file for the Qorus system schema. The Qorus server must make all connections to the same physical database server as the Oracle cluster synchronization delay will cause problems with synchronized transaction management within the Qorus server.

Example Tablespace and User Creation Scripts

These files are provided as examples only, paths and passwords at least should be modified to correspond to your Oracle installation. This example file can be found in $OMQ_DIR/templates/oracle-omq-db.sql

create tablespace omq_data
        datafile '/opt/oradata/XBOX_MASTER/datafile/omq_data01.dbf' size 512M reuse
        autoextend on next 256M
        default storage (initial 1M next 1M minextents 1 maxextents 1000
        pctincrease 0);

create tablespace omq_index
        datafile '/opt/oradata/XBOX_MASTER/datafile/omq_index01.dbf' size 256M reuse
        autoextend on next 128M
        default storage (initial 1M next 1M minextents 1 maxextents 1000
        pctincrease 0);

create user omq identified by omq default tablespace omq_data temporary tablespace temp;

grant create session, create procedure, create sequence, create table, create trigger, create type, create view, unlimited tablespace to omq;

This example file can be found in $OMQ_DIR/templates/oracle-omquser-db.sql

create tablespace omquser_data
        datafile '/opt/oradata/XBOX_MASTER/datafile/omquser_data01.dbf' size 128M reuse
        autoextend on next 64M
        default storage (initial 1M next 1M minextents 1 maxextents 1000
        pctincrease 0);

create tablespace omquser_index
        datafile '/opt/oradata/XBOX_MASTER/datafile/omquser_index01.dbf' size 64M reuse
        autoextend on next 32M
        default storage (initial 500k next 500k minextents 1 maxextents 1000
        pctincrease 0);

create user omquser identified by omquser default tablespace omquser_data temporary tablespace temp;

grant create session, create procedure, create sequence, create table, create trigger, create type, create view, create synonym, unlimited tablespace to omquser;
Note
  • Do not forget to change the passwords to match your organization's security requirements.
  • It's not recommended to use "grant unlimited tablespace" in real deployments; assign specific tablespace permissions according to the users' needs

PostgreSQL Configuration

This step describes how to set up PostgreSQL database schemas for use with Qorus Integration Engine® on PostgreSQL server 9.0.0 or later.

PostgreSQL installation and connectivity configuration (postgresql.conf, pg_hba.conf, etc configuration) is outside the scope of this documentation (however some basic information and examples are provided in the following section). See your PostgreSQL documentation for detailed instructions on how to install the PostgreSQL database server.

If the Qorus server and PostgreSQL servers are not located on the same machine, PostgreSQL must be configured to accept connections on a network interface so Qorus can communicate with the server.

Once PostgreSQL has been installed and configured on the target system, Qorus databases, tablespaces, and users can be created.

It is recommended to use UTF-8 character encoding in all PostgreSQL databases where international characters may be used with Qorus.

Each schema requires two tablespaces, one for data and one for indexes.

See Database Sizing General Information

Tablespace names are flexible and do not necessarily have to be as described in the table below; tablespace names can be overridden with the –-data-ts and –index-ts options to the oload and schema-tool programs and by default in the options file with the qorus-client.omq-data-tablespace and qorus-client.omq-index-tablespace options. However, for the rest of this document, the names will appear as given below.

Note
If you use alternate tablespace names, you must define the tablespace names in the options file with the qorus-client.omq-data-tablespace and qorus-client.omq-index-tablespace options so that automatic schema management will work properly

Required PostgreSQL Tablespaces

Schema Tablespace Type Description
omq omq_data Mandatory All system data is stored in the tablespace
omq omq_index Mandatory All system index are stored here
omquser omquser_data Optional All data in the user schema is stored in this tablespace
omquser omquser_index Optional All user indexes are stored here

Example Database, Tablespace and User Creation Scripts

These files are provided as examples only; paths and passwords at least should be changed for security reasons. This example file can be found in $OMQ_DIR/templates/pgsql-omq-db.sql

create database omq encoding = 'utf8';
\connect omq;
create tablespace omq_data location '/opt/postgresql90/db/omq_data';
create tablespace omq_index location '/opt/postgresql90/db/omq_index';
create user omq password 'omq';
grant create, connect, temp on database omq to omq;
grant create on tablespace omq_data to omq;
grant create on tablespace omq_index to omq;
grant select on all tables in schema pg_catalog to omq;

This example file can be found in $OMQ_DIR/templates/pgsql-omquser-db.sql

create database omquser encoding = 'utf8';
\connect omquser;
create tablespace omquser_data location '/opt/postgresql90/db/omquser_data';
create tablespace omquser_index location '/opt/postgresql90/db/omquser_index';
create user omquser password 'omquser';
grant create, connect, temp on database omquser to omquser;
grant create on tablespace omquser_data to omquser;
grant create on tablespace omquser_index to omquser;
grant select on all tables in schema pg_catalog to omquser;
Note
Do not forget to change the passwords to match your organization's security requirements.

These files can be installed with the psql command as follows (use the appropriate user and password for your installation, the following are only examples):

prompt$ psql –Upostgres < $OMQ_DIR/templates/pgsql-omq-db.sql
prompt$ psql –Upostgres < $OMQ_DIR/templates/pgsql-omquser-db.sql 

Basic PostgreSQL Connectivity Configuration

Example pg_hba.conf File

The following file allows connections to all databases and all users without a password for local users, and requires a username and password from all network users. Do not use this file if local security is a concern:

# CAUTION: Configuring the system for local "trust" authentication allows
# any local user to connect as any PostgreSQL user, including the database
# superuser. If you do not trust all your local users, use another
# authentication method.

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
# "local" is for Unix domain socket connections only
local   all         all                               trust
# IPv4 local connections:
host    all         all         127.0.0.1/32          trust
# IPv6 local connections:
host    all         all         ::1/128               trust
# require a password from network users
host    all         all         0.0.0.0/0             md5
host	all         all         ::/0                  md5 

If you need to connect to your PostgreSQL databases from other machines on the network, set the following in your postgresql.conf file to configure the server to listen on all network interfaces:

listen_addresses = '*' 

For more information on these files, or on PostgreSQL security or configuration, see your PostgreSQL documentation.

MySQL/MariaDB/Percona Configuration

Because MariaDB and Percona are a drop-in replacements for MySQL, references to MariaDB below are also valid for MySQL (which is an Oracle property; Qore Technologies is not affiliated with Oracle) and Percona. Any differences between MySQL, MariaDB, and Percona will be highlighted below.

This step describes how to set up MariaDB database schemas for use with Qorus Integration Engine® on MariaDB server 5.1 or later. MariaDB installation is outside the scope of this documentation. See your MariaDB documentation for detailed instructions on how to install the MariaDB database server. Once MariaDB has been installed on the target system, Qorus databases and users can be created.

It is recommended to use UTF-8 character encoding in all MariaDB databases where international characters may be used with Qorus.

Note
All MariaDB tables used in the Qorus Integration Engine® system schema use the XtraDB (InnoDB for MySQL) engine for transaction management and row-level locking.

Qorus imposes no restrictions on the database or user names used for either the system or user schemas when using MariaDB as a database server.

Example Database and User Creation Scripts

These files are provided as examples only, at least the passwords should be changed for security reasons. This example file can be found in $OMQ_DIR/templates/mysql-omq-db.sql

create user 'omq'@'%' identified by 'omq';
grant super on *.* to 'omq'@'%';
create user 'omq'@'localhost' identified by 'omq';
grant super on *.* to 'omq'@'localhost';

create database omq character set 'utf8';
grant all on omq.* to 'omq'@'%';
grant all on omq.* to 'omq'@'localhost'; 

This example file can be found in $OMQ_DIR/templates/mysql-omquser-db.sql

create user 'omquser'@'%' identified by 'omquser';
grant super on *.* to 'omquser'@'%';
create user 'omquser'@'localhost' identified by 'omquser';
grant super on *.* to 'omquser'@'localhost';

create database omquser character set 'utf8';
grant all on omquser.* to 'omquser'@'%';
grant all on omquser.* to 'omquser'@'localhost'; 
Note
Do not forget to change the passwords to match your organization's security requirements

These files can be installed with the mysql command as follows (use the appropriate user and password, the following are only examples):

prompt$ mysql –uroot < $OMQ_DIR/templates/mysql-omq-db.sql
prompt$ mysql –uroot < $OMQ_DIR/templates/mysql-omquser-db.sql 

Installing or Upgrading from TAR.GZ Packages

Installing From TAR.GZ Packages

A first-time installation of Qorus Integration Engine® from a tar.gz package involves preparing the file system and installing the system files. Either preceding or following this, the system database schema must be prepared and populated. This last topic is covered in Database and Schema Preparation.

From a high-level, the installation goes as follows:

  • create the application user (for UNIX/Linux systems)
  • create the application filesystem including log file directory
  • set up the user environment
  • do a partial install of the application to install templates for the configuration files
  • create/edit application configuration files

Then the system database schema will be prepared and populated as follows:

Note
Qorus Integration Engine® as of version 4.0.3_git does not support a load-balancing configuration. Each running instance of Qorus is independent, and there are limitations to running more than one Qorus Integration Engine® instance on the same Oracle schema. See Limitations of Multiple Instances Sharing the Same Schema for more information.

Qorus Integration Engine® 4.0.3_git can support a fault-tolerant cluster configuration, however.

Overview

Before starting the system installation, ensure that you have the appropriate file from the following list containing binary support for your platform for Qorus Integration Engine®. Examples of compatible Linux distributions are: Fedora, Red Hat Enterprise Linux, OpenSUSE, Novell SUSE Linux Enterprise Server.

The release file, when unpacked, will create the files described in the following table in the subdirectory:

  • qorus-4.0.3_git/

Qorus Integration Engine® TAR.GZ Release Manifest

File Description
install.sh Installation script
qorus-4.0.3_git-Linux2.6-x86_64-binary.tar.gz x86_64 Linux (kernel 2.6+) binaries
qorus-4.0.3_git-noarch.tar.gz Common files
releases/qorus-4.0.3_git.qrf System service load file

Note that depending on the release package, only one of the above binary packages may be present in the release file.

On UNIX/Linux systems, it is highly recommended that a new user be created for this installation. The name for the user used in this documentation is qorus.

System Parameters

Qorus can use many file descriptors, depending on the size of the installation.

Linux User Limits
Make sure that the number of file descriptors for the Qorus application user is at least set to the following values in /etc/security/limits.conf (substitute qorus for the actual application user name if necessary):

qorus soft nofile 4096
qorus hard nofile 8192

Solaris File Handle Limits
Make sure that the number of file descriptors per process is at least set to the following values in /etc/system:

set rlim_fd_cur=4096
set rlim_fd_max=8192

Qorus generally does not require special kernel parameters, however on some architectures it is advised to check the values of some kernel parameters. See the following tables for more information.

HP-UX Kernel Parameters

Parameter Description
max_thread_proc The default value of 256 is too low for a busy Qorus Integration Engine® installation. The recommended setting is 1024 or higher. This is a dynamically tunable parameter (does not require a reboot).

Use kmtune on HP-UX 11.11 (v1), kctune on HP-UX 11.23 (v2) and later to check and update kernel parameters.

Prepare the System Schema


Step 1: Prepare Database User and Empty System Schema
Prepare the system schema as described in Database and Schema Preparation.

Make sure and test the connection and have the connection parameters ready for use in Step 3: Install Qorus Application and Populate System Schema.


Step 2: Setup User and File System
The Qorus application itself takes less than 20 MB of disk space.

Space must be planned for log directories that can fill up fast with busy systems, and for user code (workflow, service definitions, other Qorus user configuration files).

50 MB is recommended for the Qorus Integration Engine® application directory, and a minimum of 500 MB for the log directory, larger sizes can be necessary for larger installations, depending on the frequency of log rotation and the amount of logging done by the workflows and services running.

The log directory can appear anywhere on the file system and does not have to be a subdirectory of the Qorus application directory, although the default log directory location is $OMQ_DIR/log.

On UNIX/Linux system, a dedicated user is not required for the system, but it is highly recommended. The username can be any valid user name, for the purposes of this document, the username qorus will be assumed. The application directory could be the Qorus user’s home directory or any other directory (for example: the mount point of a mounted filesystem) on the system. In the examples in this document, the Qorus user qorus will have an application directory located in /opt/qorus.

Note
On all platforms the user running Qorus must have read and write access to the application directory and log directory trees.

Configure Environment Variables

The following environment variables have to be set for the system to function properly.

Qorus System Environment Variables

Variable Description
OMQ_DIR The main application directory
PATH $OMQ_DIR/bin must appear in the path
QORE_CONNECTION_PROVIDERS Must be set to "QorusConnectionProvider" in your system environment to take advantage of the QorusConnectionProvider module as a connection provider to command-line programs so that Qorus user connection or remote connection identifiers can be used instead of URL strings in Qore programs like rest and sfrest, etc
QORE_DATASOURCE_PROVIDERS Must be set to "QorusDatasourceProvider" in your system environment to take advantage of the QorusDatasourceProvider module as a datasource provider to command-line programs so that Qorus datasource names can be used instead of full datasource connection strings in Qore programs like sqlutil, schema, and schema-reverse
QORE_LIBRARY Must be set to the absolute path to the native Qore library on the system for Java programs to use Qore functionality (ex: $OMQ_DIR/lib/libqore.so)
QORE_MODULE_DIR For non-LSB, non-RPM-based installs, must be set to $OMQ_DIR/qlib:$OMQ_DIR/lib/modules:$OMQ_DIR/user/modules for the proper operation of the system so that system user modules can be found; this is needed by the running server and for client programs. Multiple directories can be given separated by colons (':')
QORUS_RELEASE_DIR The directory for creating release packages with make-release; defaults to $HOME/releases if not set

If using the oracle driver for connectivity to Oracle databases, one of the following environment variables should be set depending on the Oracle client library used.

Environment Variables For Oracle Connectivity

Variable Description
ORACLE_HOME For full Oracle installations, this variable must be set to the Oracle installation’s home directory. This variable is not required for Oracle Instant Client installations.
TNS_ADMIN With Oracle Instant Client only: set to the directory where the tnsnames.ora file is located.

Environment Variables MS SQL Server and SAP / Sybase ASE Connectivity Through FreeTDS (freetds)

Variable Description
FREETDSCONF On UNIX/Linux: must point to the freetds configuration file (for example: $OMQ_DIR/etc/freetds.conf)

The following environment variable must be set to use the sybase driver with SAP/Sybase databases.

Qorus Environment Variables For Native SAP / Sybase ASE Connectivity

Variable Description
SYBASE Must point to the directory containing the interfaces file, where connection information for SAP ASE databases is specified.

Qorus Environment Variables on All Other UNIX/Linux Platforms

Variable Description
LD_LIBRARY_PATH Must have $OMQ_DIR/lib as well as optionally Oracle, MariaDB, PostgreSQL, SAP ASE, or other third-party library directories.

Set up the environment so that it is configured automatically at login time using .profile (for Bourne-type shells), .bash_profile for bash, the Bourne-Again Shell, or .cshrc (for csh-type shells).

UNIX/Linux Environment Variable Examples

Bourne-Compatible Shell Example
Replace the application directories in the following example with the correct directories in your installation.

# set OMQ_DIR to Qorus Integration Engine&reg; application directory
OMQ_DIR=/opt/qorus

# add Qorus library directory to LD_LIBRARY_PATH
LD_LIBRARY_PATH="$OMQ_DIR/lib:$LD_LIBRARY_PATH"

# add Qorus bin directories to path
PATH="$OMQ_DIR/bin:$PATH"

# add Qorus qore language client library module dir to QORE_MODULE_DIR
QORE_MODULE_DIR="$OMQ_DIR/qlib:$QORE_MODULE_DIR"

# export all the changes made above
export OMQ_DIR LD_LIBRARY_PATH PATH QORE_MODULE_DIR
Note
Make sure that your Qorus’ user’s environment is configured to use the qore libraries shipped with or recommended for use with this version of Qorus Integration Engine®. This is particularly important if you have the open-source version of qore installed elsewhere on your system. This can be accomplished by ensuring that $OMQ_DIR/lib appears first in your LD_LIBRARY_PATH. Be sure to use the version of Qore either shipped with Qorus or recommended for use with this version of Qorus, otherwise problems may result.

CSH-Compatible Example
Replace the application directories in the following example with the correct directories in your installation.

# set OMQ_DIR to Qorus Integration Engine&reg; application directory
setenv OMQ_DIR /opt/qorus

# add Qorus library directory to LD_LIBRARY_PATH
setenv LD_LIBRARY_PATH "$OMQ_DIR/lib:$LD_LIBRARY_PATH"

# add Qorus and Oracle bin directories to path
setenv PATH "$OMQ_DIR/bin:$PATH"

# add Qorus qore language client module include dir to QORE_MODULE_DIR
setenv QORE_MODULE_DIR "$OMQ_DIR/qlib:$QORE_MODULE_DIR"

Install Qorus Application and Populate System Schema


Step 3: Install Qorus Application and Populate System Schema
Once the Qorus system database user and empty schema have been prepared, the application user (if applicable) and application and log directories have been created and the environment has been properly set up for the application user, the application files can be installed and the system schema populated as described in this section.

Log in as the Qorus user and make sure the environment is properly configured. For example, execute:

UNIX/Linux Shell

prompt% echo $OMQ_DIR 
Note
environment variables are case-sensitive on UNIX/Linux

The application directory should be output to your console. If not, check your environment settings and try again.

Note
If the environment is not set up properly the installation may fail. The application directory $OMQ_DIR must exist and be writable by the Qorus user. Normally the Qorus user will be the owner of this directory, but regardless of ownership, the user running Qorus must have read/write access to the application and log directories.

If all the prerequisites documented above are met, change directory to the location of the Qorus installation file, and uncompress the archive as follows:

UNIX/Linux Shell

prompt% gunzip –dc qorus-4.0.3_git.tar.gz | tar xvf - 

This will create a subdirectory called qorus-4.0.3_git with contents as described in the manifest in the introduction to this manual.

Change the current directory to the new directory extracted in the previous command:

UNIX/Linux Shell

prompt% cd qorus-4.0.3_git 

In order to install the application file system in $OMQ_DIR and populate the system schema, the install script must be run with an option describing the initial connection to the system database. Use the connection parameters for the system schema to substitute in the following command; note that arguments enclosed by square brackets are optional. See the following table for the meaning of the required parameters in angle brackets.

Qorus Integration Engine® Datasource Parameters

Parameter Description
driver The driver to use for the connection, i.e. oracle, mysql, or pgsql
user The username for the connection
pass The password corresponding to the username
db The database name
enc The optional encoding to use for the connection in the format expected by the driver (i.e. AL32UTF8 for UTF-8 with the oracle driver)
host The optional hostname to use for the connection
port The optional port number to use for the connection

Note that if the data and index tablespace names for the system schema are not named omq_data and omq_index respectively, then the –data-ts=data_tablespace_name and –index-ts=index_tablespace_name options must be given as well.

Install the files and populate the initial database schema by executing the following command:

Install.sh Arguments ./install.sh -D<driver>:<user>/<pass<i><db>[(enc)][host][:port] [–data-ts=data-tablespace-name] [–index-ts=index-tablespace-name]

UNIX/Linux Shell Example

prompt% ./install.sh -Dpgsql:omq/omq@omq 

If the environment is properly configured, output will appear as follows:

checks are OK, installing from /Users/david/src/Qorus/releases/qorus-4.0.3_git
installing Qorus architecture-independent files from: qorus-4.0.3_git-noarch.tar.gz
installing release files: done
schema status is EMPTY
system schema database driver is pgsql
pgsql:omq@omq%localhost:5432: creating Qorus 4.0.3_git schema 4.0.3.7
CCCCCCCCCCCCCCCCCCCCCAACACACACAAAACACACACAAAAACACAAAACACAAACACAAAACACAAACACAAAACACACACACACACAAAACACAAAACACAAAAACACAAAACACACAAAAACACAAAAACACAAAACACAAACACAAAACACACACACACAAAAAAAAAAAAAAAAAACACACAAAACACAAAAAAAAAACACAAAACACACACACAAAAACACAAAAAAAAAACACAAACACAACACAAACAAAAAAAACACAACACAAAAAACACAAAAAACACAAAACAACACAAACACACACAAAAAACACACAAAAAAAACACAACACAAAAAACACAAAAAACACAAAACACAAAAAAAAAAAAAAACACAAAACACAAACAAACAAAAACACAAAAAAACACAAAAACACACACAAAAAAAAAAAAAAAAAACACACACAAACACAACACACACAAACACACACAAAAACACAAAAACACAAACACAAAAACACAAAAACACAAAAACACAAAAACACACACCCCCCCCCCCCCCCCCCCCCCCCC
560 changes made; schema is now at version 4.0.3.7
verifying reference data: IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIVIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
reference data: inserted: 232, verified: 1
running system release load script
pgsql:omq@omq%localhost: system schema "omq" is at version 4.0.3.7
executing schema release loader files
done executing release files
installation complete!

Errors in the user's environment will be flagged in the output of the script and must be repaired by hand after the installation script executed. For example, if the following text appears:

**********************************************************
ATTENTION: MANUAL STEPS NECESSARY TO COMPLETE INSTALLATION
**********************************************************

*** this script made changes to your environment in order to install
*** Qorus Integration Engine&reg;.  You must make these changes permanent
*** in order to run Qorus Integration Engine&reg;.  List of changes:

   +++ put /opt/qorus/lib first in the LD_LIBRARY_PATH 

Follow the instructions given to ensure that the environment is properly configured each time the Qorus user logs in. In this case, the LD_LIBRARY_PATH variable must be updated.

After successfully running install.sh, the Qorus Integration Engine® directory layout is installed as described in tar.

Upgrading a TAR.GZ Installation

This section describes upgrading a Qorus Integration Engine® installation from tar.gz files.

Prerequisites

Upgrading a Qorus Integration Engine® installation to version 4.0.3_git is supported from any publicly released version of Qorus. Furthermore, it is assumed that the Qorus user’s environment is already properly set up, as it should be for an existing installation.

Note
When upgrading the system must be shut down prior to the upgrade because the system schema will be upgraded to v4.0.3.7, and this cannot be performed while the system is running.

Perform Upgrade

To upgrade Qorus Integration Engine® to 4.0.3_git, change directory to the location of the Qorus Integration Engine® 4.0.3_git installation file, and uncompress the archive as follows:

UNIX/Linux Shell

prompt% gunzip –dc qorus-4.0.3_git.tar.gz | tar xvf - 

This will create a subdirectory called qorus-4.0.3_git with contents as described in the manifest in the introduction to this manual.

To create the application file system in $OMQ_DIR, execute the following commands:

UNIX/Linux Shell Example

prompt% cd qorus-4.0.3_git
prompt% ./install.sh

If the environment is properly configured, output will appear as follows (exact messages will differ depending on your system, system database server type, users in the options file, starting version for upgrade, etc - this example shows an upgrade from Qorus 3.1.0:

checks are OK, installing from /Users/david/src/Qorus/releases/qorus-4.0.3_git
installing Qorus architecture-independent files from: qorus-4.0.3_git-noarch.tar.gz
installing release files: done
schema status is 3.1.0
system schema database driver is pgsql
pgsql:omq@omq%localhost:5432: schema is at version 3.1.0 (targeting 4.0.3.7): running schema alignment
............................N........................N.....................N..................................N........................N.........................CACAAAAACAN....................N...........................N...................................................AAA.....M...............................................................................................................................................................................................................................................................N..................A..................................................................................MMM...............................................................................N.............................................CACA.........................
32 changes made; schema is now at version 4.0.3.7
Uverifying reference data: VVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVIIIIIIIIVVVVVVVVVVIIIIIIIIIIIIIIIIVVVIIIIIVVVVVIIIIIIIIIIVVVVIIIIIIIVIIIIVVIIIIIIVIIIIIIIII
reference data: verified: 101, inserted: 65
running system release load script
pgsql:omq@omq%localhost: system schema "omq" is at version 4.0.3.7
executing schema release loader files
done executing release files
installation complete!

Verify the Upgrade

After the successful upgrade, restart the Qorus Integration Engine® server. Then execute ostatus as follows:

UNIX/Linux Shell

prompt% ostatus 

which should then output:

Qorus 4.0.3_git <your-instance-key> sessionid 1 

If so, the installation is complete.