Implementation Technical How-Tos
How-To: Implement a REST Server Service
Introduction
Implementing a REST service in Qorus is straightforward; the service should have the REST implementation either as library code or initialized directly in the init method.
In the following example, we'll implement a service that handles the /rest-example
URI path with the following URI paths, methods, and actions:
PUT /rest-example/caller?action=updateStatus
orPUT /rest-example/caller/updateStatus
POST /rest-example/data
GET /rest-example/data?action=check;filename=filename
orGET /rest-example/data/check?filename=filename
Note: Qore and Qorus implement REST as an API; the HTTP message body for REST messages is assumed to be serialized data (and not HTML, although the REST server implementation does support HTML rendering as a response to HTTP requests. See REST Data Serialization Support for more information.
1. Implement the Root REST Handler Class
The first step will be to implement the root REST handler class, which must be a subclass of AbstractServiceRestHandler; this will be the main entry point for external REST requests in the service. For example:
# the root REST handler object
class ExampleRestHandler inherits AbstractServiceRestHandler {
constructor() : AbstractServiceRestHandler(rest-example) {
}
}
The string argument in the [constructor()] (https://www.qoretechnologies.com/manual/qorus/current/qorus/classOMQ_1_1AbstractServiceRestHandler.html#ae168909c9d873bd097c6dfaabace2616) above gives the root URI path component that this handler will service; see the documentation in the preceding link for more information.
2. Implement URI Path Handler Classes
Implement subclasses of AbstractRestClass for each URI path component to be handled. For example:
# class for handling REST data operations
class DataRestClass inherits AbstractRestClass {
string name() {
return data;
}
}
# class for handling REST caller operations
class CallerRestClass inherits AbstractRestClass {
string name() {
return caller;
}
}
2.1. Implement Action Method Handlers
Implement action method handlers for each URI path and each HTTP method and action that you want to handle in your AbstractRestClass URI path handling classes.
Method action handlers are implemented with the following naming convention:
method[Action]
Wheremethod
is an HTTP method in lower case, and[Action]
is an optional action.
See examples in the following table:
Request | Example Method |
---|---|
GET /path (no action) | hash get(hash cx, *hash ah) {} |
PUT /path (no action) | hash put(hash cx, *hash ah) {} |
POST /path (no action) | hash post(hash cx, *hash ah) {} |
DELETE /path (no action) | hash del(hash cx, *hash ah) {} |
OPTIONS /path (no action) | hash options(hash cx, *hash ah) {} |
GET /path?action=check | hash getCheck(hash cx, *hash ah) {} |
PUT /path?action=updateStatus | hash putUpdateStatus(hash cx, *hash ah) {} |
For example:
# class for handling REST data operations
class DataRestClass inherits AbstractRestClass {
string name() {
return data;
}
hash post(hash cx, *hash ah) {
UserApi::logInfo("POST received with args: %y", ah);
return RestHandler::makeResponse(200, OK); }
hash getCheck(hash cx, *hash ah) {
if (!exists ah.filename)
throw "DATA-ERROR", "missing 'filename' argument";
UserApi::logInfo("checking filename: %y (OK)", ah.filename);
# fake the response here
return RestHandler::makeResponse(200, OK);
}
}
class CallerRestClass inherits AbstractRestClass {
string name() {
return caller;
}
hash putUpdateStatus(hash cx, *hash ah) {
UserApi::logInfo(PUT received with args: %y", ah);
return RestHandler::makeResponse(200, OK);
}
}
See Implementing REST Services for more information.
3. Add URI Path Handlers
Add the URI path handlers that you've created in the constructor of the class subclassing AbstractServiceRestHandler by calling AbstractServiceRestHandler::addClass() with AbstractRestClass objects.
For example:
# the root REST handler object
class ExampleRestHandler inherits AbstractServiceRestHandler {
constructor() : AbstractServiceRestHandler(rest-example) {
addClass(new DataRestClass());
addClass(new CallerRestClass());
}
}
To add URI path handlers for further path components, reimplement the AbstractRestClass::subClass() method and return the appropriate URI path handler object (AbstractRestClass object) corresponding to the path component in the request as given by the name
argument to AbstractRestClass::subClass().
4. Instantiate the REST Handler Class and Bind the HTTP Handler In the service's init method function, instantiate the REST handler class and call svc_bind_http() to bind the handler to the HTTP server. If the AbstractServiceHttpHandler::addListener() call is made, then the REST handler will be bound to all global HTTP listeners for Qorus Integration Engine. For example:
class MyService inherits QorusServiice {
init() {
# create the REST handler object
ExampleRestHandler lh();
# bind the handler to all global Qorus listeners
bindHttp(lh);
}
}
5. Optional: Implement Custom User Authentication and Authorization (Enterprise Edition Only)
By default, a DefaultQorusRBACAuthenticator object is passed to the constructor, which results in standard Qorus RBAC security being applied (which is only enforced if rbac-security is enabled, in which case users must have at least the LOGIN role to connect to the REST handler).
Custom user authentication can be implemented for REST handlers by passing an object from a user-defined class that inherits directly from AbstractAuthenticator as the auth argument to the AbstractServiceRestHandler::constructor() method.
To allow any user to connect to the REST service, even if rbac-security is enabled, use a PermissiveAuthenticator object instead as in the following example:
class ExampleRestHandler inherits AbstractServiceRestHandler {
constructor() : AbstractServiceRestHandler(rest-example, False, new PermissiveAuthenticator()) {
addClass(new DataRestClass());
addClass(new CallerRestClass());
}
}
6. Example Service Source
The following code is in the rest-example-v1.0.qsd
example file.
# -*- mode: qore; indent-tabs-mode: nil -*-
# service: rest-example # serviceversion: 1.0
# servicedesc: REST API example service
# serviceauthor: Qore Technologies, s.r.o.
# parse-options: PO_NEW_STYLE, PO_REQUIRE_TYPES, PO_STRICT_ARGS
# autostart: true
# classname: RestExample
# class-based: true
# define-group: EXAMPLES: example interface objects
# define-group: REST-SERVICE-EXAMPLE-1: REST service example 1 interface objects
# groups: EXAMPLES, REST-SERVICE-EXAMPLE-1
# ENDSERVICE
# class for handling REST data operations
class DataRestClass inherits AbstractRestClass {
string name() {
return data;
}
hash post(hash cx, *hash ah) {
UserApi::logInfo("POST received with args: %y", ah);
return RestHandler::makeResponse(200, OK);
}
hash getCheck(hash cx, *hash ah) {
if (!ah.filename)
throw "DATA-ERROR", "missing 'filename' argument";
UserApi::logInfo("checking filename: %y (OK)", ah.filename);
# fake the response here
return RestHandler::makeResponse(200, OK);
}
}
# class for handling REST caller operations
class CallerRestClass inherits AbstractRestClass {
string name() {
return caller;
}
hash putUpdateStatus(hash cx, *hash ah) {
UserApi::logInfo("PUT received with args: %y", ah);
return RestHandler::makeResponse(200, OK);
}
}
# the root REST handler object
class ExampleRestHandler inherits AbstractServiceRestHandler {
constructor() : AbstractServiceRestHandler(rest-example) {
addClass(new DataRestClass());
addClass(new CallerRestClass());
}
}
class RestExample inherits QorusService {
# name: init
# desc: initializes the service and sets up the REST handler
init() {
# create the REST handler object
ExampleRestHandler lh();
# bind the handler to all global Qorus listeners
bindHttp(lh);
}
}
# END
How-To: Implement a Schema Module
Qorus provides support for automated database schema management through schema modules. Automated schema management means that the creation, upgrade, downgrade, and drop actions for all schema objects are completely automated. The base functionality for schema management in Qorus is provided by the Schema module.
The Schema module's approach is based on schema alignment and works as follows:
- Write a schema module file with a class that inherits AbstractSchema providing a description of the schema (and optionally reference data) as Qore data.
- Implement the following two
public
functions in the module: *get_datasource_name()
: returns the name of the Qorus datasource where the schema should reside - get_user_schema(): creates and returns the schema object itself
For example:
public namespace ExampleSchema {
public string sub get_datasource_name() {
return omquser;
}
public ExampleSchema sub get_user_schema(AbstractDatasource ds, *string dts, *string its) {
return new ExampleSchema(ds, dts, its);
}
class ExampleSchema inherits AbstractSchema {
# the actual schema implementation goes here
}
}
Schema modules end in .qsm
and are processed by oload, which will automatically run schema alignment in the database identified by the Qorus datasource provided by get_datasource_name()
in the schema module. If the schema implements the AbstractVersionedSchema interface, then oload will only execute schema alignment if the schema does not exist in the database or if the declared schema version is greater than the version in the database (or if the --force
option is given to oload). For more information on automatic schema management, see:
- User Schema Management: Qorus documentation with a complete schema module file example
- HowTo: Implement Automatic Schema Management: Qore Programming Language Wiki * Schema: Qore Programming Language Schema module documentation
- SqlUtil Schema Management: Qore Programming Language SqlUtil module providing the underlying API support for schema management
How-To: Synchronize Database Data Between Databases
Introduction
This HowTo explains how to implement inter-database data exchange interfaces in Qorus; by using Bulk DML, SqlUtil, and the DataStream protocol, Qorus provides infrastructure capable of transferring large amounts of data with high performance and a small memory footprint between databases from disparate vendors even in geographically-separated networking environments.
1. Local Source Database
With a locally connected source databases, you will need a datasource in the local system. You can create an SQLStatement object manually by acquiring a DatasourcePool object from a datasource with get_datasource_pool() as in the following example:
DatasourcePool dsp = UserApi::getDatasourcePool(source-db); SQLStatement stmt(dsp);
stmt.prepare(select id, name, amount from source_table where type = %v and amount >= 100, current_type);
Alternatively, you can use SqlUtil to acquire an input iterator by acquiring an AbstractTable object by calling get_sql_table() and then calling AbstractTable::getRowIterator() to create the SQLStatement object. Once you have an input iterator for the input data, you can use Bulk DML (Qore wiki) to retrieve the input data using SQLStatement::fetchColumns as in the following example:
# get the source table object AbstractTable source_table = UserApi::getSqlTable(source_datasource, source_table);
# get a select iterator from the source table
hash<auto> sh = { "columns": ("id", "name", "amount"), "where": {"type": "current_type", "amount": op_ge(100)}, };
string sql;
SQLStatement stmt = source_table.getRowIterator(sh, \sql);
if (opt.verbose)
UserApi::logInfo("input SQL: %yn", sql);
# even though we only select from the SQLStatement, we have to release the transaction lock when we're done
on_error source_table.rollback();
on_success source_table.commit();
# use a block size of 1000 to select the source data
while (*hash<auto> h = stmt.fetchColumns(1000)) {
# ... process the data
map process_row($1), h.contextIterator();
}
2. Remote Source Database
If the database is connected to a remote Qorus instance, you can use the DbRemoteReceive class to stream the remote data to the local system from the remote datasource.
# get the remote connection object QorusSystemRestHelper remote = UserApi::getRemoteRestConnection(remote);
# setup the select hash arguments for the remote source table
hash<auto> sh = { "columns": ("id", "name", "amount"), "where": {"type": "current_type", "amount": op_ge(100)}, };
# create the remote select stream object DbRemoteReceive recv(remote, remote_source_datasource_name, select, table_name, (select: sh));
# DbRemoteReceive::getData() returns data in a hash of lists format (column format)
while (*hash<auto> h = recv.getData()) {
# ... process the data
map process_row($1), h.contextIterator();
}
The DbRemoteReceive class uses the DataStream protocol to transfer the data, which is selected from the remote database using the sqlutil service in the remote Qorus instance.
The DbRemoteReceive class uses a block size of 1000
by default; this and other options can be set in the DbRemoteReceive::constructor() call (in the above example, only the select
option is used).
Note: The DataStream protocol requires a direct point-to-point connection or HTTP infrastructure that allows HTTP chunked transfers to pass through any intermediate nodes without modifying the HTTP traffic. If there is an HTTP proxy between the two Qorus instances that modifies HTTP chunked data sent through it, the DataStream protocol may not be usable.
3. Local Target Database
With a locally connected target database, you can use the BulkUpsertOperation or BulkInsertOperation classes to perform upserts (SQL merging or data alignment) or inserts in the target database, respectively.
These classes require an AbstractTable object for the target table, which can be acquired by calling get_sql_table().
Then the data to be upserted/merged or inserted can be passed to the object with the AbstractBulkOperation::queueData() method.
Note that this method will take raw data in column format (hash of lists), as returned from SQLStatement::fetchColumns or DbRemoteRecv::getData().
3.1. Local Source Target Database
Example with local source:
# get the local source datasource object AbstractTable source_table = UserApi::getSqlTable(source_datasource, source_table);
# get a select iterator from the source table hash hash<auto> sh = { "columns": ("id", "name", "amount"), "where": {"type": "current_type", "amount": op_ge(100)}, };
string sql;
SQLStatement stmt = source_table.getRowIterator(sh, \sql);
if (opt.verbose)
UserApi::logInfo("input SQL: %y", sql);
# even though we only select from the SQLStatement, we have to release the transaction lock when we're done
on_error source_table.rollback();
on_success source_table.commit();
# get the target table object
AbstractTable target_table = UserApi::getSqlTable(target_datasource, target_table);
# create the bulk upsert operation object BulkUpsertOperation upsert(target_table);
# perform bulk API and transaction handling on exit
on_error {
upsert.discard();
source_table.rollback();
}
on_success {
upsert.flush();
source_table.commit();
}
# use a block size of 1000 to select the source data
while (*hash<auto> h = stmt.fetchColumns(1000)) {
# BulkUpsertOperation::queueData() accepts data in column format (a hash of lists)
upsert.queueData(h);
}
3.2. Remote Source Target Database
Example with a remote source:
# get the remote connection object QorusSystemRestHelper remote = UserApi::getRemoteRestConnection(remote);
hash<auto> sh = { "columns": ("id", "name", "amount"), "where": {"type": "current_type", "amount": op_ge(100)}, };
# open the remote DB select stream
DbRemoteReceive recv(remote, remote_source_datasource_name, select, source_table_name, {"select": sh});
# get the target table object
AbstractTable target_table = UserApi::getSqlTable(target_datasource, target_table);
# create the bulk upsert operation object BulkUpsertOperation upsert(target_table);
# handle the bulk API and transaction handling on_error {
upsert.discard();
source_table.rollback();
}
on_success {
upsert.flush();
source_table.commit();
}
# use a block size of 1000 to select the source data
while (*hash<auto> h = recv.getData()) {
# BulkUpsertOperation::queueData() accepts data in column format (a hash of lists)
upsert.queueData(h);
}
Note: upserting/merging can only work if the target table has a unique key that can be used to perform the merge; see upserting or merging data for more information
4. Remote Target Database
With a remote target database, you can use the DbRemoteSend classes to stream upserts (SQL merge statements) or inserts in the target database.
The DbRemoteSend class uses the DataStream protocol to transfer the data, which is then upserted/merged or inserted into the remote database using the sqlutil service in the remote Qorus instance.
The DbRemoteSend class uses a block size of 1000
by default; this and other options can be set in the DbRemoteSend::constructor() call (in the example below no options are used).
Example with local source:
# get the source table object AbstractTable source_table = get_sql_table(source_datasource, source_table); # get a select iterator from the source table hash sh = ( columns: (id, name, amount), where: (type: current_type, amount: op_ge(100)), ); string sql; SQLStatement stmt = source_table.getRowIterator(sh, sql); if (opt.verbose) log(LL_DETAIL_2, input SQL: %yn, sql); # even though we only select from the SQLStatement, we have to release the transaction lock when we\\\\\\\\\\\\\\\'re done on_error source_table.rollback(); on_success source_table.commit(); # get the remote connection object QorusSystemRestHelper remote = get_remote_rest_connection(remote); # start the remote upsert stream DbRemoteSend out(qrest, remote_target_datasource_name, upsert, target_table); # in case of error, disconnect which will cause a rollback on the remote end # due to the fact that the DataStream protocol relies on HTTP chunked transfer # encoding, the socket could be in the middle of a chunked transfer when an # error occurs, therefore it\\\\\\\\\\\\\\\'s better to simply disconnect than to try to # execute a manual rollback when errors occur on_error out.disconnect(); on_success out.commit(); # use a block size of 1000 to select the source data while (*hash h = stmt.fetchColumns(1000)) { # DbRemoteSend::append(hash) accepts data in column format (a hash of lists) out.append(h); }
Example with a remote source:
# get the remote source connection object QorusSystemRestHelper remote_source = get_remote_rest_connection(remote_source); hash sh = ( columns: (id, name, amount), where: (type: current_type, amount: op_ge(100)), ); # open the remote DB select stream DbRemoteReceive recv(remote_source, remote_source_datasource_name, select, source_table_name, (select: sh)); # get the remote target connection object QorusSystemRestHelper remote_target = get_remote_rest_connection(remote_target); # start the remote upsert stream DbRemoteSend out(remote_target, remote_target_datasource_name, upsert, target_table); # in case of error, disconnect which will cause a rollback on the remote end # due to the fact that the DataStream protocol relies on HTTP chunked transfer # encoding, the socket could be in the middle of a chunked transfer when an # error occurs, therefore it\\\\\\\\\\\\\\\'s better to simply disconnect than to try to # execute a manual rollback when errors occur on_error out.disconnect(); on_success out.commit(); # use a block size of 1000 to select the source data while (*hash h = recv.getData()) { # DbRemoteSend::append(hash) accepts data in column format (a hash of lists) out.append(h); }
Note:
- If the source and target are both remote as in the above example, then the data will be transferred through the instance executing the code; this is only recommended if no direct connection between the source and target is possible.
- Upserting/merging can only work if the target table has a unique key that can be used to perform the merge; see upserting or merging data for more information
See Also: