Qorus Integration Engine®  4.0.3.p2_git
Common Tasks and Examples

HOWTO Table

This section is meant to provide references to and supplement the Network System API and System Service Reference and the Qore Programming Language Reference Manual. Nearly all Qore Programming Language functionality is available to Qorus user code, for more details on Qore functionality, please refer to the Qore Programming Language Reference Manual.

Qorus "HowTo" Table

Task References
Create a Workflow Designing and Implementing Workflows
Example files: $OMQ_DIR/examples/user/INVENTORY-EXAMPLE-1
Create a Service Implementing Services
Example files: $OMQ_DIR/examples/user/services
Create Reusable Class Definitions for Qorus Interfaces Class Definition File
Example files: $OMQ_DIR/examples/user/classes
Create Reusable Constant Definitions for Qorus Interfaces Constant Definition File
Example files: $OMQ_DIR/examples/user/constants
Working with Databases see UserApi::getDatasourcePool() and Working With Databases
Communicate with a Remote Qorus Server OMQ::QorusSystemAPIHelper
Working with HTTP Servers see Working With the HTTP Protocol
HTTPClient class
Working with JSON and JSON-RPC see Working With JSON-RPC
JSON Module
Working with XML and XML-RPC see Working With XML
XML Module
Example program: $OMQ_DIR/examples/qore/xml-rpc-client.q
Working with YAML-RPC see YAML Module, YamlRpcClient class
Work with Sockets Socket class
Store Persistent Configuration Information Use OMQ::UserApi::prop_get() and OMQ::UserApi::prop_update()
Work with Asynchronous Queue Messages Use the REST API /api/latest/async-queues

Working With Databases

To work with a database, a datasource must be defined and known by the system.

For example, to acquire a shared DatasourcePool object for transaction management on the "billing" datasource, use the UserApi::getDatasourcePool() method as in the following example:

# acquire a DatasourcePool object for datasource "billing"
DatasourcePool billing = UserApi::getDatasourcePool()("billing");

If Qore user code enters a transaction with either Datasource or DatasourcePool objects and the thread is terminated (such as a workflow step or even a service call called from the network API) without closing the transaction, an exception will automatically be raised and the transaction will be rolled back.

Executing Store Procedures and Functions

The following is an example of executing a stored procedure on an Oracle database and retrieving the results:

# execute stored procedure "pkg.get_imsi" with iccid as argument,
# retrieving a hash as a result.
# The resulting hash will contain the following keys:
# imsi, status_code, error_code, error_desc
hash h = ds.exec("begin pkg.get_imsi(‘CRM', %v, :imsi, :status_code, :error_code, :error_desc); end;", iccid);

The following is an example of executing a function in an Oracle database and retrieving the result (note that you have to declare the variable type in the SQL):

# execute function create_blacklist_entry() with msisdn as argument,
# retrieving a hash with a single key "code" as a result.
# note that the hash is dereferenced with ".code" after the exec()
string code = ds.exec("declare code varchar2(30) := null; begin :code = create_blacklist_entry(%v); end;", Type::String, msisdn).code;

The Qore Oracle driver needs to know the buffer type for all placeholder bind operations in advance. The placeholders can be found in the SQL strings in the two examples above as the text prefixed by ":". When no buffer type is given, the Oracle driver assumes a string buffer. To declare another buffer type, use the Qore Type constants as arguments in the position corresponding to the placeholder position in the SQL string.

For example, here is an example similar to the first example above using different buffer types for the return values:

# execute stored procedure "pkg.get_imsi" with iccid as argument,
# retrieving a hash as a result.
# The resulting hash will contain the following keys:
# imsi, status_code, error_code, error_desc
hash h = ds.exec("begin pkg.get_imsi(‘CRM', %v, :date :blob, :clob, :integer); end;", iccid, Type::Date, Type::Binary, SQL::CLOB, Type::Integer);

Note that the MySQL, PostgreSQL, Sybase, and FreeTDS drivers do not require placeholder buffer specifications when executing stored procedures. For more examples of stored procedure calls and more information about connecting to other databases, see the Qore Programmer's Reference Manual in your Qorus Integration Engine installation directory.

Work with Data Streaming

The data streaming APIs provide high-level interfaces for the DataStream Protocol without need of the low level protocol interaction. DataStream socket I/O always takes place in a background thread, which allows the main thread to handle data operations in parallel with network I/O.

Streaming APIs are available for Qorus server objects (workflows, services, jobs) and also in the Qorus client.

The following classes provide APIs to stream data to/from remote Qorus instances:

Streaming API Constructor Options

Key Default Description
block 1000 (DB rows) or 16384 (FS bytes) a block size for DataStream transmission chunks giving the row count for DB streams or the bytes count for filesystem streams
encoding "UTF-8" the encoding of the target file; used in OMQ::FsRemoteSend
loglevel LL_INFO the default logging level used in Qorus user code
mode 0644 the file's creation mode as used in Qore::File::open2(); used in OMQ::FsRemoteSend
queue_block_size 2 the number of blocks to queue for sending before the main data thread will block; used in OMQ::DbRemoteSend and OMQ::FsRemoteSend
queue_size block * 2 the number of rows to queue before the main data thread will block; used in OMQ::DbRemoteReceive
select NOTHING Complex Select Criteria structure used in OMQ::DbRemoteReceive
timeout 60s a timeout in milliseconds for HTTP operations (ex: 120s)

Streaming API Transaction Management

A remote transaction can be performed in an external DB connected to a remote Qorus instance by using the system.sqlutil service to begin the transaction and then commit or abort it.

The OMQ::DbRemoteSend and OMQ::DbRemote classes will automatically start or continue a remote transaction by sending the "Qorus-Connection: Continue-Persistent" header when opening the stream. The OMQ::DbRemoteReceive class will do the same if the "transaction" option is set in the constructor() call.

Additionally, a call to OMQ::AbstractParallelStream::beginTransaction() can be made to explicitly start or continue a remote transaction in a remote database independently of any remote stream operations.

Remote transactions must be explicitly committed with a call to the sqlutil commit stream or to OMQ::DbRemoteSend::commit() or OMQ::DbRemoteReceive::commit() or OMQ::DbRemote::commit()

To abort a remote transaction, it's recommended to simply close the socket connection, particuarly because the HTTP connection could be in the middle of a stream action which would make HTTP messages impossible to send until the stream is completely sent or received. When the connection is closed, any streams in progress are immediately terminated and the remote transaction is automatically rolled back.

See also
Transaction Management with the sqlutil Service for more information

Database: Single DML Statements

DbRemote db("my-remote", "omquser");
on_succes db.commit();
on_error db.rollback()
# get list of available tables
*list all_tables = db.list_tables();
# single insert
db.insert("my_table", ("id" : 11, "foo" : "bar"));

Database: Simple Data Sending

DbRemoteSend send("my-remote", "omquser", "insert", "my_table");
on_success send.commit();
on_error send.disconnect();
send.append( ("id" : 1, "column1" : "foo") );
# type: STEP
# version: 1.0
# desc: stream sending example
# author: Petr Vanek (Qore Technologies, sro)
%new-style
%require-types
%strict-args
%enable-all-warnings
%requires SqlUtil
sub rem_stream_send() {
DatasourcePool dsstage = UserApi::getDatasourcePool("isepl");
on_success dsstage.commit();
on_error dsstage.rollback();
# just an sql logging
string sql;
on_exit log(LL_INFO, "sql: %s", sql);
# select source data
SqlUtil::AbstractTable t = UserApi::getSqlCache(dsstage, "h3g_je_idt");
hash sc_select = ( "columns" : ( "id", "entered_dr", "entered_cr" ), );
SQLStatement stmt = t.getRowIterator(sc_select, \sql);
DbRemoteSend out("test", "omquser", "insert", "h3g_je_idt_test");
on_success out.commit();
on_error out.disconnect();
out.append(stmt);
}
# END

Database: Simple Data Receiving

DbRemoteReceive recv("my-remote", "omquser", "select", "my_table");
while (auto d = recv.getData()) log(LL_INFO, "row: %y", d);
on_success send.commit();
on_error send.disconnect();
# rows will be logged out
# type: STEP
# version: 1.0
# desc: example of simple select
# author: Petr Vanek (Qore Technologies, sro)
%new-style
%require-types
%strict-args
%enable-all-warnings
sub rem_stream_recv() {
DbRemoteReceive recv("test", "omquser", "select", "h3g_je_idt_test");
while (auto d = recv.getData()) log(LL_INFO, "row: %y", d);
}
# END

Database: Complex Transactions (mixed mode)

# type: STEP
# version: 1.0
# desc: more complex transaction handling using remote streams
# author: Petr Vanek (Qore Technologies, sro)
%new-style
%require-types
%strict-args
%enable-all-warnings
class SelectInsert {
private {
DbRemoteSend m_send;
DbRemoteReceive m_recv;
}
constructor(string r) {
m_recv = new DbRemoteReceive(r, "omquser", "select", "h3g_je_idt_test", \received());
m_send = new DbRemoteSend(r, "omquser", "insert", "h3g_je_idt_test1");
m_recv.receive();
}
commit() { m_send.commit(); }
disconnect() { m_send.disconnect(); }
nothing received(auto row) {
log(LL_INFO, "received: %y", row);
m_send.append(row);
}
}
sub rem_stream_comb() {
SelectInsert si("test");
on_success si.commit();
on_error si.disconnect();
}
# END

Filesystem: Qorus client in action

%new-style
%require-types
%strict-args
%enable-all-warnings
%requires QorusClientCore
qorus_client_init2();
FsRemoteSend fs("test", "/tmp/qorus-client-stream.txt");
fs.append("lorem ipsum\n");
fs.commit();

Working With the HTTP Protocol

To communicate with HTTP servers, use the HTTPClient class provided by Qore.

The following is an example of connecting to an HTTP server and retrieving information:

# makes a secure connection to a remote HTTP server on port 8080
# and retrieves some data
HTTPClient hc(("url": "https://host.com:8080"));
string data = hc.get("/path/service?param=value&param2=value2");

The following is an example of connecting to an HTTP server and sending information:

# makes an unencrypted connection to a remote HTTP server using the
# default port (80), POSTs some data and retrieves the response
HTTPClient hc(("url": "http://host.com"));
string data = hc.post("/path/postSvc?param=value&param2=value2", msg_data);

Working With XML

The Qore programming language provides tight integration with XML by allowing easy serialization and deserialization between XML strings and Qore data structures. This means that manipulating XML data can be as easy as manipulating Qore data structures.

XML Functions

Function Description
parse_xml() Parses an XML string and returns a Qore data structure representing the string
parse_xml_with_schema() Same as parse_xml() but also allows for validation against an XSD schema
make_xml() Serializes a Qore data structure into an XML string without any formatting and with an XML header
make_xml_fragment() Serializes a Qore data structure into an XML fragment without formatting and without an XML header

For example, the following XML string can be parsed to a Qore data structure as follows:

xmlstr = ‘<?xml version="1.0" encoding="UTF-8"?>
<ORDER NAME="CHANGE_MSISDN" SEQ="1">
<KEYS BS="VOICE" MSISDN="436601234567" IMSI="232101234567890"
NEW-MSISDN="436994567890"/>
<SERVICES><PORTED-P OPERATOR="AT000002472"/></SERVICES>
</ORDER>‘;
hash order = parse_xml(xmlstr);

The local variable order would then be assigned to a hash with the following structure as viewed if output with the following:

printf("%N\n", order);
hash: (1 member)
ORDER : hash: (3 members)
^attributes^ : hash: (2 members)
NAME : "CHANGE_MSISDN"
SEQ : "1"
KEYS : hash: (1 member)
^attributes^ : hash: (4 members)
BS : "VOICE"
MSISDN : "436601234567"
IMSI : "232101234567890"
NEW-MSISDN : "436994567890"
SERVICES : hash: (1 member)
PORTED-P : hash: (1 member)
^attributes^ : hash: (1 member)
OPERATOR : "AT000002472"

Conversely, calling make_xml(order) would produce an XML string with the same contents of the original XML string (without formatting).

Note
Qore hashes also maintain hash key creation order, so key output order when serializing to XML strings is predicable and can be controlled when the parse_xml() function is used. That also means that hash keys retrieved with the keys operator will also be in creation order.
Qore XML integration is covered in detail in the Qore Programming Language Reference Manual. The following text should serve as an introduction to Qore's XML support.

Working With XML-RPC

XML-RPC is an internet web service standard and a simple and effective means of point-to-point messaging. XML-RPC sends information in a specific XML-based format over an HTTP connection. The Qorus HTTP server includes an XML-RPC handler that exports the entire Qorus system API as XML-RPC methods.

Information about the XML-RPC protocol, including the specification and implementation details can be found on: http://xmlrpc.org

Building upon Qorus's facility for easily serializing and deserializing to and from Qore data structures and XML strings, the same approach is applied to the XML-RPC protocol, allowing Qorus and external applications to communicate seamlessly with very little effort by the programmer. The following sections explain how to work with XML-RPC clients and servers with Qorus.

Note
Qore objects (instantiated classes) cannot be serialized to XML strings. Also the XML-RPC protocol limits integer data to 32 bit integers, while Qorus's integers are 64-bit signed integers, and additionally the XML-RPC protocol does not allow for time zone information to be represented in date/time values, and no fractional section information may be represented either. Keep in mind these limitations when using XML-RPC to communicate. For this reason, the YAML-RPC procotol is the preferred protocol to use when communicating with Qorus servers.

Communicating With XML-RPC Servers

The XmlRpcClient class can be used to communicate with XML-RPC servers. To connect to an XML-RPC server, instantiate the constructor with a hash having at least the "url" key with a URL to the XML-RPC server and then call the "call" method on the object as in the following example:

string url = "xmlrpc://hostname:port/path";
XmlRpcClient xrc(("url": url));
auto answer = xrc.call("methodname", arg1, arg2);

A successful call will return an answer in a hash under the key "params". If the hash key "fault" exists, then an error has occurred and the following keys will be present:

XML-RPC Fault Hash Description

Key Description
fault.faultCode An integer giving the error code.
fault.faultString A string giving a description for the error.

The value of the "params" key will depend on the method being called. Communication errors will result in exceptions being thrown.

Communicating with XML-RPC Clients

An XML-RPC server is required to communicate with XML-RPC clients. This is made easy by Qorus' architecture; by developing service methods and ensuring that the internal flag is not set to True, service methods will automatically be exported as lightweight web services through the HTTP server's XML-RPC handler (as well as the JSON-RPC handler and YAML-RPC handler) and made available to the network in general.

External systems can then call the new methods using the following method name format:

omq.user.service.servicename.methodname

Any arguments passed in the message will be passed along to the service method, and the return value will be serialized into XML-RPC format and sent back to the caller.

Working With JSON-RPC

JSON-RPC is an internet web service standard similar to XML-RPC and is the easiest way to communicate with JavaScript programs. JSON-RPC involves sending information in a specific JSON-based format over an HTTP connection. The Qorus HTTP server includes a JSON-RPC handler and makes the entire network API available through this handler.

Building upon Qorus's facility for easily serializing and deserializing to and from Qore data structures and JSON strings, the same approach is applied to the JSON-RPC protocol, allowing Qorus and external applications to communicate seamlessly with very little effort by the programmer.

The following sections explain how to work with JSON-RPC clients and servers with Qorus.

Note
Qore objects (instantiated classes) cannot be serialized to JSON strings. Also the JSON's number type is mapped either to a Qore float or integer type, depending on the value, and JSON-RPC has no facility for representing date/time values or binary data natively either.

Communicating With JSON-RPC Servers

The JsonRpcClient class can be used to communicate with JSON-RPC servers

To connect to a JSON-RPC server, instantiate the constructor with a hash having at least the "url" key with a URL to the JSON-RPC server and then call the "call" method on the object as in the following example:

string url = "jsonrpc://hostname:port/path";
JsonRpcClient jrc(("url": url));
auto answer = jrc.call("methodname", arg1, arg2);

Communication errors will result in exceptions being thrown.

Communicating with JSON-RPC Clients

A JSON-RPC server is required to communicate with JSON-RPC clients. This is made easy by Qorus' architecture; by developing service methods and ensuring that the internal flag is not set to True, service methods will automatically be exported as web services through the Qorus HTTP server's JSON-RPC handler (as well as the XML-RPC handler and YAML-RPC handler) and made available to the network in general.

External systems can then call the new methods using the following method name format:

omq.user.service.servicename.methodname

Any arguments passed in the message will be passed along to the service method, and the return value will be serialized into JSON-RPC format and sent back to the caller.