Loading...
 
See also:

Designing Shared Interface Components

Rules for Shared Library Code Objects

Motivation: Code Quality, Eliminate Regressions

Shared code is any library object (function, class, or constant object) that is used by more than one interface (workflows, services, job, mappers).

When code is shared in multiple interfaces, there is a chance that any change to the shared code could have unintended consequences for other interfaces not meant to be affected by the change. The more complex the solution, the greater the chance that unintended regression errors could be caused when modifying the shared code.

Shared code is only acceptable if the following criteria are met:
  • the shared code is completely new
  • the shared code is part of interfacing fundamental infrastructure where there is a clear advantage to sharing the code
  • the shared code will only be used by interfaces currently part of the change or new development and therefore all affected code will be tested together

When changing existing shared code, particularly when the change or new development only should affect a portion of the interfaces that use the shared code, the shared code should be copied and the copy should be modified for the change to eliminate the chance of unintended regressions.
Note: Shared code should not be directly referenced as step function attributes; in this case the library function object should be referenced in the workflow and the new step function should call the library function to ensure that workflow recoverability compatibility is not affected when changes are made to workflows that share this library object (in addition, the above rules must be applied to any new development); see the next guideline

Shared Function Use in Workflows

Motivation: Code Quality, Eliminate Regressions

Shared functions used in steps should be always wrapped. Do not use a shared function as a real step function.

There are two possibilities here:
  1. provide a custom step name in the workflow definition file to ensure that your step has its own unique stepid and therefore is not shared (the last amount of code to write)
  2. use a code wrapper that calls the library function

Both options allow the step's definition to change without having to change the current workflow version and without affecting other workflows. These options are described in the following subsections.

Shared Functions With Unique Step Name

Unique step names can be provided in the step definition hash, and the shared library function can be used as the primary step function definition. Changing the primary step function can be made at any time, and the step can maintain its stepid, and therefore the workflow can change but still keep recovery compatibility.

Example:
const update_ls = (
    "name": "it_34_update_staging_dhl_028:1.0",
    "funcname": "sepl_inbound_update_staging_after_enrichment:1.0",
);

This has the advantage of the least effort compared to creating a wrapper function.

Shared Functions With Wrapper Functions

With this approach, shared functions are listed as library objects and then called from workflow-specific wrapper functions as in the following examples:
# type: STEP
# version: 1.0
# desc: lorem ipsum
# author: Petr Vanek (Qore Technologies, sro)
sub wf1_step1() {
    shared_function_implementation();
}
# END

# type: STEP
# version: 1.0
# desc: lorem ipsum
# author: Petr Vanek (Qore Technologies, sro)
sub wf2_step1() {
    shared_function_implementation();
}
# END


This involves slightly more effort, since the wrapper function objects must be developed and the shared code must be listed as a library object in the workflow as well.

Parse Options in Qorus User Code

Motivation: Code Maintainability, Performance

the following parse options should be used in Qorus code:
%new-style
%require-types
%strict-args
%enable-all-warnings


  • %new-style: allows for more a compact syntax and therefore less typing; note that the old2new script in the qore source repository https://github.com/qorelanguage/qore/tree/develop/examples:%22examples/ directory can be used to convert code from old-style to new-style
  • %require-types: serves for more maintainable code; old-style typeless code is much harder to understand and maintain, additionally, code with type declarations executes faster and more efficiently due to optimizations that can be made by the runtime engine when types are restricted in advance. Furthermore more programming errors can be caught at parse time which allows for faster development.
  • %strict-args: eliminates "noop" function variants from being used and also causes errors with argument passing to be raised instead of being silently ignored which can hide errors
  • %enable-all-warnings: allows for more errors to be caught at parse time, allowing for faster development at a higher quality
No warnings should appear when loading Qorus user code.

For services and mappers the following top-level attribute should be defined instead of the above:
# parse-options: PO_NEW_STYLE, PO_REQUIRE_TYPES, PO_STRICT_ARGS


This is particularly useful with services, because otherwise parse directives must be included separately in every method definition.

Connections

Motivation: Operational Control and Transparency

Connections should be used for all external connections; a connection-specific module should be developed for connections that don't have an existing connection class.

Why? Because connections allow for configurational transparency (connection configuration and status is displayed in the UI) as well automatic dependency tracking and monitoring, which allows Qorus to pro-actively alert operations to connection problems (or even missing filesystems, which are also implemented as connections). Therefore this applies equally to filesystem connections (ex: file://appl/data/mseplftp, for filesystem polling or sending; using a filesystem as an interface) as well as more standard network-based connection types (ex: sftp://partnerp@sftp.example.com). Note that filesystem connections are monitored not only for their presence (mount status) but also for when they exceed pre-defined usage thresholds; see alert-fs-full for more information.

See the following link for pre-defined connection object types: http://www.qoretechnologies.com/manual/qorus/latest/qorus/connmon.html#userconntypes

example (-+sftp-partner.qconn+-):
sftp-partner = (
    desc = Partner SFTP Polling / Delivery connection,
    url = sftp://parterp@sftp.example.com,
    keyfile = /opt/qorus/.ssh/id_rsa-partner
)


See the following for more information on defining user connections in Qorus: http://www.qoretechnologies.com/manual/qorus/latest/qorus/definingconnections.html

To define new connection types, implement a user module defining the connection type as a concrete implementation of the AbstractConnection class, and add the user module's name to the connection-modules option. Each connection type is associated to one unique scheme; schemes and connection factories are exported from the module using a special function in the connection module (-+public AbstractIterator sub get_schemes() {}+-); this is documented in the preceding system option documentation.

Author Labels

Motivation: Code Maintainability

All code should contain author labels or an author attribute. When a new author is added, add the name to the front if taking over ownership of the object, otherwise if just performing some minor changes (for example, the primary developer is on vacation), add the name to the end.

For example, in workflows:
$workflows."EXAMPLE-WORKFLOW"."1.0" = (
    "desc": "example workflow",
    "author": "Johnny Appleseed (Qore Technologies, s.r.o.)",
    ...
);


In functions, classes, or constants:
# type: GENERIC
# version: 1.0
# name: example_function
# desc: example function
# author: Johnny Appleseed (Qore Technologies, s.r.o.)


The same applies to jobs, services, etc.

Do Not Use Deprecated APIs

Motivation: Maintainability

Do not use any APIs considered deprecated; among others, this includes the old camel-case API names. Note that the camel-case APIs were deprecated in Qorus 3.0.2, so they have been deprecated for quite some time.

correct
hash h = wf_get_dynamic_data();


incorrect
hash h = getDynamicData();

Use get_sql_table() instead of Table::constructor()

Motivation: Performance

get_sql_table() returns an AbstractTable object from the table cache and therefore provides higher-performance and more efficient memory usage, since AbstractTable (and therefore Table) objects are large objects subject to heavy I/O to create in the form of a series of complex queries in the underlying database's data dictionary.

By using the table cache, these objects are created once on demand and then can be returned quickly on request to clients requiring the use of the table object.

correct
AbstractTable my_table = get_sql_table("my-datasource", "my_table_name");


incorrect
Table my_table(get_datasource_pool("my-datasource"), "my_table_name");

Workflows

Autostart

Motivation: Operational Control

All new workflows should include the following line in the workflow definition hash:
"autostart": 1,

in order to ensure that the workflow is autostarted when installed. Operations can change it themselves if necessary; note that any operational changes are then taken as the "master value" for the workflow autostart value; oloading the workflow subsequently will not change a value edited by operations.

Workflow OneTimeInit Function

Motivation: Code Maintainability

The Qorus onetimeinit function (workflow initialization) should initialize any objects that have a large initialization cost and set them in workflow execution instance data.

For options set in the onetimeinit function, only those options used by the workflow should be logged.

System Preferences/Options Usage and Access

Motivation: Code Maintainability

The following variables / parameters should be set in the onetimeinit function :
  • objects that are expensive to initialize and therefore would adversely affect performance if initialized for every step or every order

Normally workflow runtime options should not be set in the init function, because in such a case, any change to these parameters requires a workflow reload.

System preference values that define workflow runtime options should be acquired at runtime when needed, providing the following advantages:
  • much better readability, variables are in local scope
  • workflows/services/jobs do not need reload to put the new value in effect

Workflow and Step Function Naming Convention

Motivation: Code Maintainability

Workflow function files should have the same name as the function, and step files should have the same name as the primary step function. If a step has multiple functions (primary, validation, array, back-end, etc), then all functions can appear in the same file. Please refer to Recommended Naming Conventions.

Bulk DML Processing and Data Streaming

Use SQLStatement::fetchColumns() to Select Data

Motivation: Performance

This allows for the row block size to be used to fetch all the required rows in one round trip (for example, in the Oracle driver) which greatly improves performance, additionally when streaming data with DbRemoteSend (which uses column format by default), the data format used does not need any translation to the format used in the serialized messages.

Disconnect Instead of Calling Rollback in Case of Stream Errors

Motivation: Clear Error-Handling

In case of a network error it will be impossible to rollback anyway. Disconnecting without an explicit commit causes a rollback on the remote sqlutil service side in any case. By calling disconnect instead of rollback, extraneous error messages are avoided in local log files.
QorusSystemRestHelper mseplit = get_remote_rest_connection(sepl_getconf("msepl-it", "msepl-it"));
DbRemoteSend stream(mseplit, "omquser", "ps_etlit_doa_upd", "insert");
on_success stream.commit();
on_error stream.disconnect();
...

Note: DbRemoteSend::rollback() and DbRemoteReceive::rollback() disconnect by default in newer version fo Qorus in any case.

Use the Bulk Mapper API with InboundTableMapper

Motivation: Performance

Note that the bulk mapper API (InboundTableMapper::queueData(), InboundTableMapper::flush(), and InboundTableMapper::discard()) can also be used with child rows even if it's not possible with parent rows as in the following example:
on_success {
    mapper_snr.flush();
    mapper_brep_trans.commit();
}
on_error {
    mapper_snr.discard();
    mapper_brep_trans.rollback();
}

tid = mapper_brep_trans.insertRowNoCommit(h.record).transaction_id;
mapper_snr.queueData(h.record + ("transaction_id": tid));

Use BulkSqlUtil When Possible

Motivation: Performance

When inserting or upserting data, use the BulkInsertOperation and BulkUpsertOperation classes from the BulkSqlUtil module to perform bulk inserts or upserts; these classes use the bulk DML APIs to reach maximum performance.

Service Resources

Motivation: Code Maintainability

Services should use service file resources for file data; this particularly applies to services serving HTML pages (ie: UI extensions, etc) but also applies to services providing SOAP services (ex: WSDL and optionally XSD files).

Exceptions to this rule:
  • the files must be on the filesystem anyway; for example, the files are used by both services and connection files, for example for the same WSDLs or supporting XSDs used by SOAP server services and SOAP client connections

Interface Testing

Motivation: Code Correctness

  • Tests for each interface should be written using the QUnit-based QorusInterfaceTest module
  • Test scripts should have the same name as the object being tested
  • Ensure you have good test coverage:
    • "happy day" case (when everything works properly) must be implemented
    • error-handling (negative) tests must also be implemented
  • Test scripts should be designed to run on any valid environment; in particular:
    • configuration-dependent values in the interface should be derived in the same way in the test
      • no hardcoded paths - use file / directory connection objects (or generally the same logic used to derive the configuration information as the interface uses)
      • no hardcoded datasource names or other configuration information (same as above - use the same logic in the test as in the interface)
    • review all queries for index usage - when tests are run on the customer's systems, DBs may be very large and a query requiring a full table scan that runs fast on a small development database may take a very long time on a large shared dev or test system
  • test scripts should be executable and (as with all other executable Qore scripts) should use the following hash-bang line: #!/usr/bin/env qore; also parse options must be used as per Parse Options in Qorus User Code

Unit Testing

  • you should write unit tests while developing any code
  • unit tests can be written using the QMock module
  • test scripts should have the same name as the object being tested ended with .unit.qtest
  • Ensure you have good test coverage:
    • "happy day" case (when everything works properly) must be implemented
    • error-handling (negative) tests must also be implemented
  • unit test must NOT be dependent on any environment, datasource, filesystem, etc. Use the QMock module to mock all necessary APIs
  • test scripts should be executable and (as with all other executable Qore scripts) should use the following hash-bang line: #!/usr/bin/env qore; also parse options must be used as per Parse Options in Qorus User Code