Implementation Technical How-Tos

How-To: Implement a REST Server Service

Introduction

Implementing a REST service in Qorus is straightforward; the service should have the REST implementation either as library code or implemented directly in the init method.

In the following example, we'll implement a service that handles the /rest-example URI path with the following URI paths, methods, and actions:

• PUT /rest-example/caller?action=updateStatus or PUT /rest-example/caller/updateStatus
• POST /rest-example/data
• GET /rest-example/data?action=check;filename=filename or GET /rest-example/data/check?filename=filename

Note: Qore and Qorus implement REST as an API; the HTTP message body for REST messages is assumed to be serialized data (and not HTML, although the REST server implementation does support HTML rendering as a response to HTTP requests. See REST Data Serialization Support for more information.

1. Implement the Root REST Handler Class

The first step will be to implement the root REST handler class, which must be a subclass of AbstractServiceRestHandler; this will be the main entry point for external REST requests in the service.

For example:

# the root REST handler object
class ExampleRestHandler inherits AbstractServiceRestHandler {
    constructor() : AbstractServiceRestHandler("rest-example") {
    }
}

The string argument in the constructor() above gives the root URI path component that this handler will service; see the documentation in the preceding link for more information.

2. Implement URI Path Handler Classes

Implement subclasses of AbstractRestClass for each URI path component to be handled.

For example:

# class for handling REST data operations
class DataRestClass inherits AbstractRestClass {
    string name() {
        return "data";
    }
}

# class for handling REST caller operations
class CallerRestClass inherits AbstractRestClass {
    string name() {
        return "caller";
    }
}

2.1. Implement Action Method Handlers

Implement action method handlers for each URI path and each HTTP method and action that you want to handle in your AbstractRestClass URI path handling classes.

Method action handlers are implemented with the following naming convention:

• method[Action]

Where method is an HTTP method in lower case, and [Action] is an optional action.

See examples in the following table:

For example:

# class for handling REST data operations
class DataRestClass inherits AbstractRestClass {
    string name() {
        return "data";
    }

    hash<HttpHandlerResponseInfo> post(hash<auto> cx, *hash<auto> ah) {
        log(LL_INFO, "POST received with args: %y", ah);
        return RestHandler::makeResponse(200, "OK");
    }

    hash<HttpHandlerResponseInfo> getCheck(hash<auto> cx, *hash<auto> ah) {
        if (!exists ah.filename)
            throw "DATA-ERROR", "missing &#039;filename&#039; argument";

        log(LL_INFO, "checking filename: %y (OK)", ah.filename);
        # fake the response here
        return RestHandler::makeResponse(200, "OK");
    }
}

class CallerRestClass inherits AbstractRestClass {
    string name() {
        return "caller";
    }

    hash<HttpHandlerResponseInfo> putUpdateStatus(hash<auto> cx, *hash<auto> ah) {
        log(LL_INFO, "PUT received with args: %y", ah);
        return RestHandler::makeResponse(200, "OK");
    }
}

See Implementing REST Services for more information.

3. Add URI Path Handlers

Add the URI path handlers that you've created in the constructor of the class subclassing AbstractServiceRestHandler by calling AbstractServiceRestHandler::addClass() with AbstractRestClass objects.

For example:

# the root REST handler object
class ExampleRestHandler inherits AbstractServiceRestHandler {
    constructor() : AbstractServiceRestHandler("rest-example") {
        addClass(new DataRestClass());
        addClass(new CallerRestClass());
    }
}

To add URI path handlers for further path components, reimplement the AbstractRestClass::subClass() method and return the appropriate URI path handler object (AbstractRestClass object) corresponding to the path component in the request as given by the name argument to AbstractRestClass::subClass().

4. Instantiate the REST Handler Class and Bind the HTTP Handler

In the service's init method function, instantiate the REST handler class and call svc_bind_http() to bind the handler to the HTTP server.

If the AbstractServiceHttpHandler::addListener() call is made, then the REST handler will be bound to all global HTTP listeners for Qorus Integration Engine.

For example:

class MyService inherits QorusServiice {
    init() {
        # create the REST handler object
        ExampleRestHandler lh();
        # bind the handler to all global Qorus listeners
        bindHttp(lh);
    }
}

5. Optional: Implement Custom User Authentication and Authorization

By default, a DefaultQorusRBACAuthenticator object is passed to the constructor, which results in standard Qorus RBAC security being applied (which is only enforced if rbac-security is enabled, in which case users must have at least the LOGIN role to connect to the REST handler).

Custom user authentication can be implemented for REST handlers by passing an object from a user-defined class that inherits directly from AbstractAuthenticator as the auth argument to the AbstractServiceRestHandler::constructor() method.

To allow any user to connect to the REST service, even if rbac-security is enabled, use a PermissiveAuthenticator object instead as in the following example:

class ExampleRestHandler inherits AbstractServiceRestHandler {
    constructor() : AbstractServiceRestHandler("rest-example", False, new PermissiveAuthenticator()) {
        addClass(new DataRestClass());
        addClass(new CallerRestClass());
    }
}

6. Example Service Source

The following code is in the rest-example-v1.0.qsd example file.

# -*- mode: qore; indent-tabs-mode: nil -*-
# service: rest-example
# serviceversion: 1.0
# servicedesc: REST API example service
# serviceauthor: Qore Technologies, s.r.o.
# parse-options: PO_NEW_STYLE, PO_REQUIRE_TYPES, PO_STRICT_ARGS
# autostart: true
# classname: RestExample
# class-based: true
# define-group: EXAMPLES: example interface objects
# define-group: REST-SERVICE-EXAMPLE-1: REST service example 1 interface objects
# groups: EXAMPLES, REST-SERVICE-EXAMPLE-1
# ENDSERVICE

# class for handling REST data operations
class DataRestClass inherits AbstractRestClass {
    string name() {
        return "data";
    }

    hash<HttpHandlerResponseInfo> post(hash<auto> cx, *hash<auto> ah) {
        log(LL_INFO, "POST received with args: %y", ah);
        return RestHandler::makeResponse(200, "OK");
    }

    hash<HttpHandlerResponseInfo> getCheck(hash<auto> cx, *hash<auto> ah) {
        if (!ah.filename)
            throw "DATA-ERROR", "missing &#039;filename&#039; argument";

        log(LL_INFO, "checking filename: %y (OK)", ah.filename);
        # fake the response here
        return RestHandler::makeResponse(200, "OK");
    }
}

# class for handling REST caller operations
class CallerRestClass inherits AbstractRestClass {
    string name() {
        return "caller";
    }

    hash<HttpHandlerResponseInfo> putUpdateStatus(hash<auto> cx, *hash<auto> ah) {
        log(LL_INFO, "PUT received with args: %y", ah);
        return RestHandler::makeResponse(200, "OK");
    }
}

# the root REST handler object
class ExampleRestHandler inherits AbstractServiceRestHandler {
    constructor() : AbstractServiceRestHandler("rest-example") {
        addClass(new DataRestClass());
        addClass(new CallerRestClass());
    }
}

class RestExample inherits QorusService {
    # name: init
    # desc: initializes the service and sets up the REST handler
    init() {
        # create the REST handler object
        ExampleRestHandler lh();
        # bind the handler to all global Qorus listeners
        bindHttp(lh);
    }
}
# END

How-To: Implement a Schema Module

Qorus provides support for automated database schema management through schema modules.

Automated schema management means that the creation, upgrade, downgrade, and drop actions for all schema objects are completely automated.

The base functionality for schema management in Qorus is provided by the Schema module. The Schema module's approach is based on schema alignment and works as follows:

• Write a schema module file with a class that inherits AbstractSchema providing a description of the schema (and optionally reference data) as Qore data.
• Implement the following two public functions in the module:
  • get_datasource_name(): returns the name of the Qorus datasource where the schema should reside
  • get_user_schema(): creates and returns the schema object itself

For example:

public namespace ExampleSchema {
    public string sub get_datasource_name() {
        return "omquser";
    }

    public ExampleSchema sub get_user_schema(AbstractDatasource ds, *string dts, *string its) {
        return new ExampleSchema(ds, dts, its);
    }

    class ExampleSchema inherits AbstractSchema {
        # the actual schema implementation goes here
    }
}

Schema modules end in .qsm and are processed by oload, which will automatically run schema alignment in the database identified by the Qorus datasource provided by get_datasource_name() in the schema module.

If the schema implements the AbstractVersionedSchema interface, then oload will only execute schema alignment if the schema does not exist in the database or if the declared schema version is greater than the version in the database (or if the --force option is given to oload).

For more information on automatic schema management, see:

• User Schema Management: Qorus documentation with a complete schema module file example
• HowTo: Implement Automatic Schema Management: Qore Programming Language Wiki
• Schema: Qore Programming Language Schema module documentation
• SqlUtil Schema Management: Qore Programming Language SqlUtil module providing the underlying API support for schema management

How-To: Synchronize Database Data Between Databases

Introduction

This HowTo explains how to implement inter-database data exchange interfaces in Qorus; by using Bulk DML, SqlUtil, and the DataStream protocol, Qorus provides infrastructure capable of transferring large amounts of data with high performance and a small memory footprint between databases from disparate vendors even in geographically-separated networking environments.

1. Local Source Database

With a locally connected source databases, you will need a datasource in the local system.

You can create an SQLStatement object manually by acquiring a DatasourcePool object from a datasource with UserApi::getDatasourcePool() as in the following example:

DatasourcePool dsp = UserApi::getDatasourcePool("source-db");
SQLStatement stmt(dsp);
stmt.prepare("select id, name, amount from source_table where type = %v and amount >= 100", current_type);

Alternatively, you can use SqlUtil to acquire an input iterator by acquiring an AbstractTable object by calling UserApi::getSqlTable() and then calling AbstractTable::getRowIterator() to create the SQLStatement object. Once you have an input iterator for the input data, you can use Bulk DML (Qore wiki) to retrieve the input data using SQLStatement::fetchColumns() as in the following example:

# get the source table object
AbstractTable source_table = UserApi::getSqlTable("source_datasource", "source_table");

# get a select iterator from the source table
hash<auto> sh = {
    "columns": ("id", "name", "amount"),
    "where": ("type": current_type, "amount": op_ge(100)),
};
string sql;
SQLStatement stmt = source_table.getRowIterator(sh, sql);
if (opt.verbose)
    UserApi::logInfo("input SQL: %yn", sql);

# even though we only select from the SQLStatement, we have to release the transaction lock when we&#039;re done
on_error source_table.rollback();
on_success source_table.commit();

# use a block size of 1000 to select the source data
while (*hash<auto> h = stmt.fetchColumns(1000)) {
    # ... process the data
    map process_row($1), h.contextIterator();
}

2. Remote Source Database

If the database is connected to a remote Qorus instance, you can use the DbRemoteReceive class to stream the remote data to the local system from the remote datasource.

# get the remote connection object
QorusSystemRestHelper remote = UserApi::getRemoteRestConnection("remote");

# setup the select hash arguments for the remote source table
hash sh = (
    "columns": ("id", "name", "amount"),
    "where": ("type": current_type, "amount": op_ge(100)),
);

# create the remote select stream object
DbRemoteReceive recv(remote, "remote_source_datasource_name", "select", table_name, ("select": sh));

# DbRemoteReceive::getData() returns data in a "hash of lists" format (column format)
while (*hash h = recv.getData()) {
    # ... process the data
    map process_row($1), h.contextIterator();
}

The DbRemoteReceive class uses the DataStream protocol to transfer the data, which is selected from the remote database using the sqlutil service in the remote Qorus instance. The DbRemoteReceive class uses a block size of 1000 by default; this and other options can be set in the DbRemoteReceive::constructor() call (in the above example, only the select option is used).

Note: The DataStream protocol requires a direct point-to-point connection or HTTP infrastructure that allows HTTP chunked transfers to pass through any intermediate nodes without modifying the HTTP traffic. If there is an HTTP proxy between the two Qorus instances that modifies HTTP chunked data sent through it, the DataStream protocol may not be usable.

3. Local Target Database

With a locally connected target database, you can use the BulkUpsertOperation or BulkInsertOperation classes to perform upserts (SQL merging or data alignment) or inserts in the target database, respectively.

These classes require an AbstractTable object for the target table, which can be acquired by calling UserApi::getSqlTable(). Then the data to be upserted/merged or inserted can be passed to the object with the AbstractBulkOperation::queueData() method. Note that this method will take raw data in column format (hash of lists), as returned from SQLStatement::fetchColumns or DbRemoteRecv::getData().

3.1. Local Source Target Database

Example with local source:

# get the local source datasource object
AbstractTable source_table = UserApi::getSqlTable("source_datasource", "source_table");

# get a select iterator from the source table
hash<auto> sh = {
    "columns": ("id", "name", "amount"),
    "where": {"type": current_type, "amount": op_ge(100)},
};
string sql;
SQLStatement stmt = source_table.getRowIterator(sh, sql);
if (opt.verbose)
    UserApi::logInfo("input SQL: %yn", sql);

# even though we only select from the SQLStatement, we have to release the transaction lock when we&#039;re done
on_error source_table.rollback();
on_success source_table.commit();

# get the target table object
AbstractTable target_table = UserApi::getSqlTable("target_datasource", "target_table");

# create the bulk upsert operation object
BulkUpsertOperation upsert(target_table);

# perform bulk API and transaction handling on exit
on_error {
    upsert.discard();
    source_table.rollback();
}
on_success {
    upsert.flush();
    source_table.commit();
}

# use a block size of 1000 to select the source data
while (*hash h = stmt.fetchColumns(1000)) {
    # BulkUpsertOperation::queueData() accepts data in "column format" (a hash of lists)
    upsert.queueData(h);
}

3.2. Remote Source Target Database

Example with a remote source:

# get the remote connection object
QorusSystemRestHelper remote = UserApi::getRemoteRestConnection("remote");

hash<auto> sh = {
    "columns": ("id", "name", "amount"),
    "where": {"type": current_type, "amount": op_ge(100)},
};

# open the remote DB select stream
DbRemoteReceive recv(remote, "remote_source_datasource_name", "select", source_table_name, ("select": sh));

# get the target table object
AbstractTable target_table = UserApi::getSqlTable("target_datasource", "target_table");

# create the bulk upsert operation object
BulkUpsertOperation upsert(target_table);

# handle the bulk API and transaction handling
on_error {
    upsert.discard();
    source_table.rollback();
}
on_success {
    upsert.flush();
    source_table.commit();
}

# use a block size of 1000 to select the source data
while (*hash h = recv.getData()) {
    # BulkUpsertOperation::queueData() accepts data in "column format" (a hash of lists)
    upsert.queueData(h);
}

Note: upserting/merging can only work if the target table has a unique key that can be used to perform the merge; see upserting or merging data for more information

4. Remote Target Database

With a remote target database, you can use the DbRemoteSend classes to stream upserts (SQL merge statements) or inserts in the target database.

The DbRemoteSend class uses the DataStream protocol to transfer the data, which is then upserted/merged or inserted into the remote database using the sqlutil service in the remote Qorus instance. The DbRemoteSend class uses a block size of 1000 by default; this and other options can be set in the DbRemoteSend::constructor() call (in the example below no options are used).

Example with local source:

# get the source table object
AbstractTable source_table = UserApi::getSqlTable("source_datasource", "source_table");

# get a select iterator from the source table
hash sh = (
    "columns": ("id", "name", "amount"),
    "where": ("type": current_type, "amount": op_ge(100)),
);
string sql;
SQLStatement stmt = source_table.getRowIterator(sh, sql);
if (opt.verbose)
    log(LL_DETAIL_2, "input SQL: %yn", sql);

# even though we only select from the SQLStatement, we have to release the transaction lock when we're done
on_error source_table.rollback();
on_success source_table.commit();

# get the remote connection object
QorusSystemRestHelper remote = UserApi::getRemoteRestConnection("remote");

# start the remote "upsert" stream
DbRemoteSend out(qrest, "remote_target_datasource_name", "upsert", "target_table");

# in case of error, disconnect which will cause a rollback on the remote end
# due to the fact that the DataStream protocol relies on HTTP chunked transfer
# encoding, the socket could be in the middle of a chunked transfer when an
# error occurs, therefore it's better to simply disconnect than to try to
# execute a manual rollback when errors occur
on_error out.disconnect();
on_success out.commit();

# use a block size of 1000 to select the source data
while (*hash h = stmt.fetchColumns(1000)) {
    # DbRemoteSend::append(hash) accepts data in "column format" (a hash of lists)
    out.append(h);
}

Example with a remote source:

# get the remote source connection object
QorusSystemRestHelper remote_source = UserApi::getRemoteRestConnection("remote_source");

hash sh = (
    "columns": ("id", "name", "amount"),
    "where": ("type": current_type, "amount": op_ge(100)),
);

# open the remote DB select stream
DbRemoteReceive recv(remote_source, "remote_source_datasource_name", "select", source_table_name, ("select": sh));

# get the remote target connection object
QorusSystemRestHelper remote_target = UserApi::getRemoteRestConnection("remote_target");

# start the remote "upsert" stream
DbRemoteSend out(remote_target, "remote_target_datasource_name", "upsert", "target_table");

# in case of error, disconnect which will cause a rollback on the remote end
# due to the fact that the DataStream protocol relies on HTTP chunked transfer
# encoding, the socket could be in the middle of a chunked transfer when an
# error occurs, therefore it's better to simply disconnect than to try to
# execute a manual rollback when errors occur
on_error out.disconnect();
on_success out.commit();

# use a block size of 1000 to select the source data
while (*hash h = recv.getData()) {
    # DbRemoteSend::append(hash) accepts data in "column format" (a hash of lists)
    out.append(h);
}

Note:

• If the source and target are both remote as in the above example, then the data will be transferred through the instance executing the code; this is only recommended if no direct connection between the source and target is possible.
• Upserting/merging can only work if the target table has a unique key that can be used to perform the merge; see upserting or merging data for more information

See Also:

• DbRemote
• DbRemoteReceive
• DbRemoteSend
• sqlutil streams
• DataStream protocol
• BulkInsertOperation
• BulkUpsertOperation
• HowTo: Use Bulk DML (Qore wiki)