Implementing Mappers

Table of Contents

• Mapper Development Overview
• Mapper Files
  • Mapper Types
  • Mapper Options
    • Mapper Input Option
    • Mapper Output Option
  • Mapper Fields
  • Mapper Library Objects
• Mapper Module Developent

Mapper Development Overview

See also

Qorus Mappers for more information

Mappers are developed in text files, loaded into the system schema with oload and participate in releases built with make-release.

Each mapper has its own sandboxed Program container that contains the mapping logic and any library objects imported into it. Mapper Program objects have the following sandboxing restrictions set by default:

• %no-top-level
• %no-thread-control
• %no-process-control
• %require-our

For mapper library code, the following parse option is also set to restrict code from performing I/O or logging:

• %lockdown

Note

The following system options can cause additional options to be set by default:

• qorus.compat-broken-list-parsing
• qorus.compat-strict-bool-eval

See also

• Mapper API for more information on mapper API including mapper Program objects
• Qorus Mappers for more information

Note that workflows, services, and jobs must declare mappers to be able to access them. The following api call should be used to acquire a mapper in your interface code:

• UserApi::getMapper()

The return type of the above function will be compatible with Mapper::Mapper, but the exact type of object depends on the mapper type.

See also

Mapper Library Objects for more information.

Mapper Files

Mapper files defining new system mappers take the filename suffix: ".qmapper".

As with function, class, constant objects, services, jobs, and value maps, mappers are defined by adding metadata tags in the form of specially-formatted comments to normal text files containing a complete mapper definition. The tags are parsed by oload, which will create the mappers in the Qorus database according to the information in the file.

The file consists of tags defining the mapper and then option and field definitions. At the end of the mapper definition, the END tag terminates the mapper.

Mapper Definition File Tags

Key	Mand.?	Description
name	Y	The name of the mapper
version	Y	Version string for the mapper object; multiple versions of the same mapper can exist in the database and interfaces (workflows, services, and jobs) must refer to the mapper using its explicit version
patch	N	A descriptive patchlevel attribute that does not affect version compatibility
desc	Y	Description of the mapper object
author	N	The optional author of the mapper object
parse_options	N	A comma separated list of local parse options for the mapper code; one or more of (options must be given as shown without any namespace prefixes): - PO_REQUIRE_TYPES - PO_REQUIRE_PROTOTYPES - PO_STRICT_ARGS - PO_ASSUME_LOCAL - PO_ALLOW_BARE_REFS
type	Y	The type of mapper; must be a system type or a type provided by a mapper module
classes	N	An optional list of class objects to load into the mapper program object (subject to %lockdown sandboxing restrictions; see Mapper Library Objects for more information)
constants	N	An optional list of constant objects to load into the mapper program object (subject to %lockdown sandboxing restrictions; see Mapper Library Objects for more information)
functions	N	An optional list of functions to load into the mapper program object (subject to %lockdown sandboxing restrictions; see Mapper Library Objects for more information)
define-group	N	Allows interface groups to be defined; format is: name: description; see below for an example
groups	N	one or more interface groups that the mapper is a member of for access purposes (mappers cannot be enabled or disabled)

Mapper example:

# name: my-example-table-mapper
# version: 1.0
# desc: Example table mapper for documentation
# mappertype: InboundTableMapper
# author: Joesephine Programmer (Qore Technologies, s.r.o.)
# functions: my-mapper-lib
# parse-options: PO_NEW_STYLE, PO_STRICT_ARGS, PO_REQUIRE_TYPES
# define-group: EXAMPLES: example interface objects
# groups: EXAMPLES

OPTION: datasource: "omquser"
OPTION: table: "example_table"
OPTION: input: (
    "order_id": "unique order ID",
    "cust_id": "customer ID",
    "dealer_id": "dealer ID",
    "ordered_quantity": "product quantity",
    "product_id": "product ID",
    "order_date": "the date and time the order was made",
)

# id is populated from the sequence "example_table_s" in the target datasource
FIELD: id: ("sequence": "example_table_s")
FIELD: source_system: ("constant": "ORDERS")
# note that comments are preserved but ignored
FIELD: order_number: ("name": "order_id")
# order_code is populated according to a closure combining the cust_id and order_id values into a single string
FIELD: order_code: ("code": string sub (auto ignored, hash rec) {
        return sprintf("%s-%s", rec.cust_id, rec.order_id);
    },
)
FIELD: customer_number: ("name": "customer_id")
FIELD: dealer_code: ("name": "dealer_id")
FIELD: product_code: ("name": "product_id")
FIELD: qty: ("name": "ordered_quantity")
FIELD: ordered_date: ("name": "order_date")
# created_date is populated with the current date and time when the row is inserted
FIELD: created_date: ("code": date sub (auto ignored, hash rec) { return now_us(); })

Note

in the above example, the "output" option is not defined since it's defined automatically by the InboundTableMapper mapper type

Mapper Types

The following mapper types are provided by the system:

• "Mapper": implemented by OMQ::MapperType; provides a Mapper::Mapper object at runtime
• "InboundTableMapper": implemented by OMQ::InboundTableMapperType; provides a OMQ::QorusInboundTableMapper object at runtime
• "QorusSqlStatementOutboundMapper": implemented by OMQ::QorusSqlStatementOutboundMapperType; provides a OMQ::QorusSqlStatementOutboundMapper object at runtime
• "QorusRawSqlStatementOutboundMapper": implemented by OMQ::QorusRawSqlStatementOutboundMapperType; provides a OMQ::QorusRawSqlStatementOutboundMapper object at runtime

Additional mapper types can be provided by implementing a mapper module; see Mapper Module Developent for more information.

Mapper Options

Common options for all mappers are defined by the Mapper class here: Mapper Options, but each mapper type can extend this list with its own options, some of which may be required as well.

Mappers object must provide the input and output options by default, but these may be generated automatically by the particular mapper type (for example, the OMQ::InboundTableMapperType class automatically generates the output option based on the target table); see the mapper type documentation for more information.

Mapper input options are specified with the following syntax:

• OPTION: option_name: expression

Where:

• option_name: is the name of the option; this must be an option supported by the mapper. Note that all mappers inherit Mapper::Mapper, therefore all mappers support at least the options defined here: Mapper Options. Additionally, mappers must define the input and output options, but these may be defined automatically by the mapper type itself; see the documentation for the particular mapper type for more information
• expression: is a Qore expression that will be executed to give the value of the mapper option

See the example in the preceding section.

Mapper Input Option

This option is a hash that describes the input data structures; see the "input" key of Mapper Options for more information.

Note that only input field names defined as keys in this hash can be used as input field names in mapper field definitions.

While all mapper ojects generated by a mapper type must define the "input" and "output" keys, mapper types may define these automatically (for example, the OMQ::InboundTableMapperType class automatically generates the output option based on the target table); see the mapper type documentation for more information.

Mapper Output Option

This option is a hash that describes the output data structures; see the "output" key of Mapper Options for more information.

Note that only output field names defined as keys in this hash can be used as Mapper Fields.

While all mapper ojects generated by a mapper type must define the "input" and "output" keys, mapper types may define these automatically (for example, the OMQ::InboundTableMapperType class automatically generates the output option based on the target table); see the mapper type documentation for more information.

Mapper Fields

Mapper field transformations are based on Mapper Specification Format and are specified with the following syntax:

• FIELD: output_field: expression

Where:

• output_field: is the name of the output field; this must be also defined by the output option (either explicitly in the mapper definition or implicitly by the mapper type)
• expression: is a Qore expression describing how the output is generated; for possible values, see Mapper Specification Format

All Qorus mapper field descriptions support the following two additional keys which are added at runtime to all mapper:

• "context": any string value assigned to this key will be used as the argument to UserApi::expandTemplatedValue(); the return value of this method call will be used as the value of the field
• "template": if assigned to True; the value of the field will be passed as the sole argument to UserApi::expandTemplatedValue(); the return value of this method call will be used as the value of the field

Mapper Library Objects

Mappers support library objects in way similar to workflows, services, and jobs, however mapper library objects are subject to %lockdown sandboxing restrictions, so every library object imported into a mapper Program object must meet this criterion or it cannot be used in a mapper.

%lockdown prohibts any I/O; also logging is not allowed from within a mapper, so generally library code used elsewhere in Qorus workflows, services, and jobs is not suitable for use in a mapper.

Mapper library objects can be used in workflows, services, and jobs however.

Mapper Module Developent

New mapper types can be added to the system by implementing mapper modules and placing them in $OMQ_DIR/user/modules/ and adding the modules' names to the qorus.mapper-modules system option.

Each module must call the map_register_mapper() function in the module's init closure or call reference.

Example module:

# -*- mode: qore; indent-tabs-mode: nil -*-

%new-style
%enable-all-warnings
%strict-args
%require-types

module ExampleInboundTableMapper {
    version = "1.0";
    desc = "example inbound mapper provider module";
    author = "Qore Technologies, s.r.o.";
    url = "http://qoretechnologies.com";
    license = "proprietary";
    init = \init();
}

%requires TableMapper
%requires SqlUtil

class ExampleInboundTableMapperType inherits OMQ::InboundTableMapperType {
    string getName() {
        return "ExampleInboundTableMapper";
    }

    ExampleInboundTableMapper get(hash mapv, *hash opts) {
        return new ExampleInboundTableMapper(mapv, opts);
    }
}

sub init() {
    # register mappers
    map_register_mapper(new ExampleInboundTableMapperType());
}

#! returns the current date
const CurrentDateFilter = date sub (auto v, hash<auto> rec) { return now_us(); };

const DefaultMap = {
    "status": {"constant": "I"},
    "created_date": {"code": CurrentDateFilter},
};

hash<auto> sub get_wf_map() {
    code get_wfiid = softint sub (auto ignored, hash<auto> rec) {
        return UserApi::getUserContextInfo().workflow_instanceid;
    };
    return {
        "created_order_id": get_wfiid,
    };
}

public namespace ExampleInboundTableMapper;

#! this class provides an easy way to map row data to a single table and adds some default mappings
/** the following default mappings are provided (if the target table has the columns):
    - \c "status": constant \c "I"
    - \c "created_date": current date/time
    - \c "created_order_id": the current workflow_instanceid, if the mapper is created from a workflow
*/
public class ExampleInboundTableMapper::ExampleInboundTableMapper inherits public QorusInboundTableMapper {
    private {
        #! execution context
        hash<auto> cxh = UserApi::getUserContextInfo();
    }

    public {
    }

    #! builds the object based on a hash providing field mappings, data constraints, and optionally logic; this variant takes a Datasource/DatasourcePool or other wrapper class as an argument directly
    /** throws an exception if a field name in the map is not recognized or if a NOT NULL field is not mapped

        @param map the mapper description; see @ref mapperkeys for more info
        @param opts see @ref mapperoptions for more info; the following additional keys are required for this class:
        - \c "datasource": the name of the Qorus datasource
        - \c "table": the name of the target table for inserting

        @throw MAP-ERROR the map hash has a logical error (ex: 'trunc' key given without 'maxlen', invalid map key)
        @throw UNKNOWN-TABLE table name not recognized (not present in the TableDescriptions hash)
        @throw UNKNOWN-COLUMN field name not recognized
        @throw MISSING-COLUMN map is missing a mapping for a column with a NOT NULL constraint
    */
    constructor(hash mapv, *hash opts) : QorusInboundTableMapper(mapv, ExampleInboundTableMapper::getOpts(opts.table) + opts) {
    }

    static hash getOpts(string table_name) {
        return {
            "timezone": "Europe/Prague",
        };
    }

    #! returns a map hash with table properties added from the TableDescriptions hash; throws an exception if the table is not recognized or if a field name in the map is not recognized
    /** make sure the table name and all column names are in lower-case
        @param mapv the map hash

        @return the map hash with table attributes added (numeric fields and varchar2 field widths)

        @throw UNKNOWN-TABLE table name not recognized (not present in the TableDescriptions hash)
    */
    private setup(hash mapv, *hash opts) {
        # get static data if possible
        *hash sd = map_try_get_wf_static_data();

        # add default mappings as constants
        Columns cols = table.describe();

        QorusInboundTableMapper::setup(mapv, opts);

        hash mymap = DefaultMap;
        if (cxh.type == "workflow")
            mymap += get_wf_map();

        # add default mappings if the mapping doesn't already exist
        map mapc.$1 = mymap.$1, keys mymap, (!mapc.$1 && cols.hasKey($1));
    }

    #! identical to mapData()
    hash mapDataHash(hash rec) {
        return mapData(rec);
    }
}