Qorus Integration Engine®  4.0.3.p2_git
Implementing Jobs

Table of Contents

Implementing Qorus Jobs

Jobs are simple tasks executed on a schedule, similar to a cron job. The information and status of processing is stored in the database and is reported through system APIs. An internal API is also provided to jobs; see Job API Reference for more information.

Job Program objects are created with the following sandboxing restrictions to protect the system from dangerous operations (and in the case of %require-our, to require all variables to be declared before use to allow typographical errors in variable names to be reported immediately):

Note
The following system options can cause additional options to be set by default:
See also

Job Definition Files

Qorus job definition files should have the extension *.qjob or *.qjob.java for Java-based jobs to identify the file's contents properly to the schema loader.

Job definition files are used to create job definitions in the Qorus schema that will be run acording to their schedule.

As with function, class, constant objects, services, mappers, and value maps, jobs are defined by adding metadata tags in the form of specially-formatted comments to normal text files containing at least one complete job definition. The tags are parsed by the oload program, which will create the jobs in the Qorus database according to the information in the file.

The file consists of tags defining the job and then the Qore-language code that makes up the job. At the end of the source code, the END tag terminates the job definition.

Job Definition File Tags

Key Mand.? Description
name Y The name of the job
version Y Version string for the job object; this version is informative; only one version of a job is stored in the database at a time
desc Y Description of the job object
author N The optional author of the code object
job-modules N lists modules providing base classes for class-based jobs
remote N The optional remote status of the job, indicating if the job will run in an independent qjob process (see qorus-client.remote for how the default value is assigned by oload)
active N A boolean value giving the active status of the job; if not present, the default is True
run-skipped N A boolean value telling the system if the job should be run immediately if the last scheduled run was missed due to system downtime; if not present, the default is True
single-instance N A boolean value telling the system if the job can only be run on one Qorus instance at a time; if not present, the default is False
duration * a schedule duration in seconds; if present the job will be scheduled to run every n seconds where n is the value assigned to this tag; either this tag or the schedule tag must be present, but not both
schedule * a cron specification, see Job Cron Schedule for more information; either this tag or the duration tag must be present, but not both
expiry-date N An optional expiration date; the job will not be run (and will be automatically set to inactive) when the expiration date is reached
class-based N Set to true to define a class-based job (recommended); if not set or false, a function-based job will be assumed (only supported for backwards compatibility)
class-name N Set this to the class name of the job's class in case it differs; this allows the job to have names that are not valid identifiers in the source language (Qore or Java)
lang N the possible values are "qore" for Qore code (the default) and "java" for code targeting the Java 8 JVM (see Developing in Java)
classes N An optional list of class objects to load into the job program object
constants N An optional list of constant objects to load into the job program object
functions N An optional list of functions to load into the job program object
mappers N An optional list of mappers to register with the job so they are available with the UserApi::getMapper() call
vmaps N An optional list of value maps to register with the job so they can be used with the UserApi::getValueMap() call
define-group N Allows Interface Groups to be defined; format is: name: description; see below for an example
groups N one or more Interface Groups separated by commas that the job is a member of
TAG N Option tag definition in the format "key: value"
END Y This tag must be present after the job's source code to terminate the job definition

The job's code should follow the metadata tags given above.

Note that the logic for the job is defined by either a job class (recommended) or a function (supported for backwards compatibility).

Class-Based Jobs

The recommended way to define a job is by setting the class-based option to true and defining the job's logic as a subclass of one of the following classes depending on the source language:

Example Qore Class-Based Job Definition: example-job-v1.0.qjob:
# name: example-job
# version: 1.0
# desc: test job, run every 10 minutes every hour, every weekday
# schedule: */10 * * * mon-fri
# active: true
# run-skipped: true
# define-group: EXAMPLES: example interface objects
# groups: EXAMPLES
# classname: MyExampleJob
# class-based: true
# TAG: example-code: true
%new-style
%require-types
%strict-args
%enable-all-warnings
class MyExampleJob inherits QorusJob {
# the run() method implements the job's logic
run() {
log(LL_INFO, "job info: %y", getInfo());
}
}
# END
Example Java Class-Based Job Definition: java-example-job-v1.0.qjob.java
// name: java-example-job
// version: 1.0
// desc: test job, run every 10 minutes every hour, every weekday
// author: Qore Technologies, s.r.o.
// class-name: MyJavaExampleJob
// active: true
// run-skipped: true
// schedule: */10 * * * mon-fri
// define-group: EXAMPLES: example interface objects
// groups: EXAMPLES
// lang: java
// TAG: classpath: $OMQ_DIR/user/jar/my-jar-1.jar:$OMQ_DIR/user/jar/my-jar-2.jar
package com.qoretechnologies.qorus.example;
import com.qoretechnologies.qorus.job.QorusJob;
class MyJavaExampleJob extends QorusJob {
// the run method implments the job's logic
public void run() throws Throwable {
log(LL_INFO, "test job info: %y", getInfo());
}
}
// END
Note
  • Jobs implemented in Java default to class-based = true; see QorusJob for more information and an example Java job definition
  • The classpath tag can be used to add entries to the classpath for Java classes; $OMQ_DIR can be used in classpath entries as in the above example and will be resolved to the Qorus directory; see Java Classpath Handling in Qorus for more information

Class-Based Job Constructors and Static Initialization

Class-based jobs can have a constructor and classes can have static initialization, but please note that if the job has configuration items, it must be instantiated by oload in order to create the job's configuration in the system. In such a case, if the constructor or static class initialization requires features that are only available at runtime in Qorus itself, the errors raised will cause job class instantiation or static class instantiation to fail.

The job constructor takes no arguments.

Class-Based Job Configuration Items

Class-based jobs (and only class-based jobs) can declare configuration items to allow for the behavior of the job to be modified by users at runtime using the operational web UI or the REST API.

Job configuration items are:

  1. Created by oload when loading the job; despite the fact that they are retured with an API call, the configuration is not changed at runtime
  2. Read at runtime with job APIs to affect the functionality of the job
  3. Updated using the web UI using the REST API with PUT /api/latest/jobs/{id_or_name}/config/{name}

Job configuration items are designed to allow users to affect the execution of a job so that changes can be made by authorized users in the UI without requiring a change to development.

Note
All configuration item values are considered to be managed by operations by design, which means that once a configuration item has been loaded into Qorus, any changes to its value with the API are persistent and will not be overwritten by subsequent loads of the configuration item by oload.

Strictly Local Job Config Item

If the strictly_local flag on a job configuration item is False, then the job configuration item is not local and the value can also be set on global level.

If the strictly_local flag is False, then the job configuration item is local and hence the value for this item cannot be set on global level.

Note
All configuration item values are considered to be managed by operations by design, which means that once a configuration item has been loaded into Qorus, any changes to its value with the API are persistent and will not be overwritten by subsequent loads of the configuration item by oload.

Qore Class-Based Job Configuration Items

Job configuration items are declared in Qore by overriding the QorusConfigurationItemProvider::getConfigItemsImpl() method in the job class. The job configuration items are registerted as part of the basic configuration of the job by oload when the job is loaded.

Each ConfigItemInfo hash in the return value of this method defines a configuration item for the job. The value of the configuration item can then be retrieved and used in the job by calling one of the following APIs.

API Type API Description
Qore JobApi::getConfigItemValue() retrieves a single configuration item value
Qore JobApi::getConfigItemHash() retrieves all configuration items
Qore Job Config Item Declaration Example:
public class TestJob inherits QorusJob {
run() {
log(LL_INFO, "run method here");
}
private *hash<string, hash<ConfigItemInfo>> getConfigItemsImpl() {
return {
"example-string": <ConfigItemInfo>{
"default_value": "example",
"description": "This is an example configuration item",
},
"example-int": <ConfigItemInfo>{
"type": "int",
"default_value": 1,
"description": "This is an example configuration item",
},
};
}
}
Note
If the Qore-based job depends on a module that defines the configuration items, the job must be loaded with oload for the configuration items or changes to the configuration items to take effect; only oload reads the configuration items; changes to configuration items are not detected at runtime.

Java Class-Based Job Configuration Items

Step configuration items in Java are declared by overriding the QorusJob.getConfigItemsImpl() method in the job class. Job configuration items are registerted as part of the basic configuration of the job by oload when the job is loaded.

Each ConfigItem object in the return value of this method defines a configuration item for the job. The value of the configuration item can then be retrieved and used in the job by calling one of the following APIs.

API Type API Description
Java JobApi.getConfigItemValue() retrieves a single configuration item value
Java JobApi.getConfigItemHash() retrieves all configuration items
Java Job Config Item Declaration Example:
class JavaExampleJob extends QorusJob {
public void run() throws Throwable {
log(LL_INFO, "example job info: %y", getInfo());
}
protected HashMap<String, ConfigItem> getConfigItemsImpl() {
return new HashMap<String, ConfigItem>() {
{
put("example-string", new ConfigItem("This is an example configuration item").withDefaultValue("test"));
put("example-int", new ConfigItem("This is an example configuration item").withType("int"));
}
};
}
}
Note
If the Java-based service depends on a compiled object that defines the configuration items, the service must be loaded with oload for the configuration items or changes to the configuration items to take effect; only oload reads the configuration items; changes to configuration items are not detected at runtime.

Job Modules Parameter

The "job-modules" option lists modules providing base classes for class-based jobs as in the following example.

Example:
# name: example-job
# ...
# job-modules: MyJobModule1, MyJobModule2
# ...

Modules declared like this will be loaded into each jobs's Program object, and their classes can be used as base classes for class-based jobs.

See also
Job Extension Modules for more information.

Function-Based Jobs

Function-based jobs are supported for backwards compatibility; a function-based job must have a run() function. This function will be called when the job is scheduled. Note that only active jobs that are not members of any disabled Interface Groups will be scheduled and run.

The job's code also has the job api at its disposal.

Example Function-Based Job Definition
# name: test
# version: 1.0
# desc: test job, run every 10 minutes every hour, every weekday
# schedule: */10 * * * mon-fri
# active: true
# run-skipped: true
# define-group: EXAMPLES: example interface objects
# groups: EXAMPLES
# TAG: example-code: true
# functions: test_job_1
%new-style
%require-types
%strict-args
%enable-all-warnings
sub run() {
log(LL_INFO, "job info: %y", JobApi::getInfo());
test_job_1();
}
# END
Note
  • A job definition can any kind of code that can be parsed with the job's sandboxing restrictions, however at least one function called run() must be present in function-based jobs.
  • Function-based jobs are supported for backwards compatibility, it is recommended to implement class-based jobs instead.

Whenever a job is executed, a record in the JOB_INSTANCE table is created. If an error is raised by calling JobApi::err() with a severity greater than OMQ::ES_Warning, the job instance will get a status of OMQ::JS_Error, otherwise the job instance will get a status of OMQ::JS_Complete. In the case that the Qorus system crashes while the job has a OMQ::JS_InProgress status, then, when the system is recovered, the JOB_INSTANCE row will get a OMQ::JS_Crash status.

Job instances can never be recovered; the status is saved for information purposes only.

To save information about processing status, call the JobApi::saveInfo() method.

The UserApi::log() method is used to save information in the job's log file; see System, Service, Workflow, and Job Logging for more information about log file locations and file formats.

Job Remote Parameter

The "remote" flag indicates if the job will run as an independent qjob process communicating with other elements of Qorus Integration Engine with a distributed queue protocol rather than internally in the qorus-core process.

When workflows run in separate qjob processes, it provides a higher level of stability and control to the integration platform as a whole, as a job with implementation problems cannot cause the integration platform to fail.

There is a performance cost to running in separate qjob processes; job startup and shutdown is slightly slower, and communication with qorus-core also suffers a performance hit as all communication must be serialized and transmitted over the network.

The default for this option depends on the client option qorus-client.remote (if this client option is not set, then the default value is True).

The remote value can be changed at runtime by using the following REST API: PUT /api/latest/jobs/{id_or_name}?action=setRemote

Note
The remote flag is considered to be managed by operations, which means that once an interface has been loaded into Qorus, if its remote flag is updated with the API, then those API-driven changes are persistent and will not be overwritten by subsequent loads of the interface by oload.

Job Cron Schedule

This section describes the format of the schedule tag in a job definition.

It basically follows the format of a cron job time specification; it is made up of 5 fields separated by spaces as follows:

Field Values
minutes 0-59
hours 0-23
days of month 1-31
months 1-12 (or 3-letter English month abbreviations: Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec)
days of week 0-7 (0 and 7 are Sunday, or 3-letter English day abbreviations: Sun, Mon, Tue, Wed, Thu, Fri, Sat)

Case is ignored when giving month or day of week abbreviations. A field may be an asterisk (*), which stands for all possible values in that field.

Lists and ranges are allowed (also ranges and lists of abbreviated names are allowed); examples:

  • 1,2,5,9
  • 0-4,8-12,15
  • mon-wed,thu,sat

A skip/repeat specification can be appended with an asterisk (*) or a range in the format "/number", which means that the job should run every number units (ie minutes, hours, days, etc) in the range. For example:

  • */2: means every 2 minutes (or hours, days, etc, depending on the field)
  • 0-12/3: means every 3 hours from 0 to 12 (if given in an hour field)

Because fields are separated by spaces, no spaces are allowed within a field.

Note
The day of a command's execution can be specified by two fields, days of month, and days of week. If both fields are restricted (ie, aren't *), the command will be run when either field matches the current time. For example, "30 4 1,15 * fri" would cause a command to be run at 4:30 am on the 1st and 15th of every month, plus every Friday.

Some examples:

  • run ten minutes after midnight, every day
    "10 0 * * *"
  • run at 2:35pm on the first of every month
    "35 14 1 * *"
  • run at 11 pm on weekdays
    "0 23 * * mon-fri"
  • run 34 minutes after the hour, every other hour, every day
    "34 */2 * * *"
  • run at 5 after 4am every Sunday
    "5 4 * * sun"
Note
The job's schedule value is considered to be managed by operations, which means that once a job has been loaded into Qorus, if its schedule is updated with the API, then those API-driven changes are persistent and will not be overwritten by subsequent loads of the job by oload.
Since
Qorus 2.7.0.p2 added support for the skip/repeat specification ("/number")