Developing in Python

Table of Contents

• Introduction to Python Development in Qorus
• Python Module Handling in Qorus
  • Python Default Module Path
  • Python Module Dependency Packaging
  • Python Module Conflicts
  • Accessing Qore APIs from Python
• Python Thread Stacks
• Using Python in Qorus Workflows
• Using Python in Qorus Services
• Using Python in Qorus Jobs
• Restrictions on Python APIs
• Qore Objects Imported Into Python Programs
• Using the Qorus Qore-Language Client API in Python
• Writing Tests in Python for Qorus Interfaces

Back to the Developer's Guide Table of Contents

Introduction to Python Development in Qorus

Qorus supports native Python development where code attributes of objects can be defined with code in Python in addition to Java and Qore.

Python integration with Qore is implemented using the python module, which provides tight integration with both languages, including bi-directional dynamic and transparent imports of code and data between Qore and Java and Python as well.

All Qorus APIs are available in Python code in Qorus, however some APIs are automatically imported into Python programs. The main Qorus Python APIs imported into Python program containers as given in the following table.

Python Module Names for Qorus APIs

Type	Python Module	Description
module	job	The Qorus job module as imported from the main Qorus Qore-language job namespace
module	OMQ	The main OMQ namespace for Qorus
module	svc	The Qorus service module as imported from the main Qorus Qore-language service namespace
module	wf	The Qorus workflow module as imported from the main Qorus Qore-language workflow namespace

Python Interface API Class Names

Type	Python Class	Description
class	jobapi	The Python job API class based on the Qore API
class	svcapi	The Python service API class based on the Qore API
class	UserApi	The main Python user API class based on the Qore API
class	wfapi	The Python workflow API class based on the Qore API

See also

• Importing Qore APIs into Python
• Importing Java APIs Indirectly into Python Through Qore

The Python module path can be specified using the module_path tag in a class object; see Python Module Handling in Qorus for more information and examples.

Note

Because Qorus's Python integration uses the cpython implementation which employs a Global Interpreter Lock (GIL) to serialize access to the Python runtime, it's recommended to deploy CPU-intensive Python-based interface code in interfaces running in separate processes; to do this, ensure that the remote flag in your interface is True; see:

• Workflow Remote Parameter
• Service Remote Flag
• Job Remote Flag

for more information

Python Module Handling in Qorus

Generally, the Python module path is set with the tag module_path when defining Python code in Qorus. Furthermore, the recommended storage location for Python modules is in $OMQ_DIR/user/python.

The module_path tag accepts environment variables, and each element in the module path should be separated by a colon (":"); see the following examples for more information.

In class definitions or any other object with Python code, the Python module_path is set with the YAML metadata tag module_path as in the following example:

Example Module Path Tag Value

module_path: $OMQ_DIR/user/my-project/python:$OMQ_DIR/user/my-other-project/python 

Python Default Module Path

The following directories are automatically added to the Python module path in Qorus:

• $OMQ_DIR/user/python/site-packages
• $OMQ_DIR/user/python/lib/python<ver>/site-packages

In the last path above, <ver> is the Python major and minor version; ex: $OMQ_DIR/user/python/lib/python3.8/site-packages

Note

Some python modules can only be initialized once in a process (ex: tensorflow); solutions depending on such modules must always run a separate process,

Python Module Dependency Packaging

If a release should also package one or more Python modules, the following options to make-release will be needed:

• -a=ARG: this provides the source Python module directory that you want in the release
• -b=ARG: this provides the target directory name the files packaged with -a to be installed under $OMQ_DIR on the target server; the default location without this option is $OMQ_DIR/user/python/lib/python<version>/site-packages where <version> is the major and minor version of Python used (ex: 3.8)

Note

Python modules often contain OS- and CPU-specific shared libraries; take care that such Qorus packages are only installed on compatible target systems.

Python Module Conflicts

Some Python modules do not support multiple interpreters in the same process; Python code requiring these modules should always run in a separate process (see Workflow Remote Parameter, Service Remote Flag, and Job Remote Flag).

The Python error looks as follows:

builtins.ImportError: Interpreter change detected - this module can only be loaded into one interpreter per process.

Accessing Qore APIs from Python

Any Qore module can be imported into Python code using the special qore package provided by the qoreloader module, which is used to dynamically load Qore modules and make them available to the Python import system on demand.

An equivalent Python module is created based on the main namespace detected for the loaded Qore module.

Aside from the Qore modules and classses listed in the above section, any Qore code available in an interface or imported from a Qore module outside the main namespace can be automatically imported and accessed in Python code using the special qore.__root__ module.

The qore.__root__ module provides access to the root Qore namespace and therefore all accessible declarations in Qore.

For example, if a Qore class declared outside of any namespace (therefore finding itself in the root namespace) named MyClass is imported in a Python interface, it can be accessed as follows:

Example:

from qore.__root__ import MyClass
 
myclass = MyClass()

The following is an example from a Python test:

Test Example:

from job import QorusJob
from qore.__root__ import QoreTest as qt
from qore.reflection import Type
 
class PythonObjectCacheTest(QorusJob):
    def run(self):
        try:
            UserApi.stopCapturingObjectsFromPython()
            UserApi.clearObjectCache()
            type0 = Type("string")
            qt.assertEq(0, UserApi.getObjectCacheSize())
            UserApi.startCapturingObjectsFromPython()
            type1 = Type("string")
            qt.assertEq(1, UserApi.getObjectCacheSize())
        finally:
            UserApi.stopCapturingObjectsFromPython()

Python Thread Stacks

The python module implements support for multiple Python interpreters in Qorus programs along with multithreading by managing the Python interpreter thread state manually.

Additionally, the Python recursion limit is also set automatically as a function of the current thread stack size; the default stack size for all Qorus programs can be set with the qorus.stack-size option, and this option can furthermore be overridden in each interface by setting a value for the option on the interface directly.

Stack size and Python recursion limits are managed to avoid crashes due to stack exhaustion in Qore and Python code.

Note that the default stack size for Qorus is 8MB; since stack memory is allocated from overcommitted memory and starts small and is allocated real memory only on demand, large stack sizes do not have a affect on the process memory size as long as the program does not use the extra stack space. For programs that require more than 8MB of stack, you can set higher stack sizes by setting the appropriate value of the qorus.stack-size option on each interface directly while also setting the remote flag to True on the interface, ensuring that it will be run in a dedicated process. That way the higher stack size will be enforced only for the program that needs it.

Note

For processes running in qorus-core (i.e. where the remote flag is False), any locally-set qorus.stack-size option value is ignored; in qorus-core, only the global qorus.stack-size option is used.

See also

• qorus.stack-size
• Workflow Remote Parameter
• Service Remote Flag
• Job Remote Flag

Using Python in Qorus Workflows

Qorus workflows can be defined in Python by implementing a step class by subclassing one of the following step classes for the step:

• wf.QorusAsyncStep
• wf.QorusEventStep
• wf.QorusNormalStep
• wf.QorusSubworkflowStep
• wf.QorusAsyncArrayStep
• wf.QorusEventArrayStep
• wf.QorusNormalArrayStep
• wf.QorusSubworkflowArrayStep

Example (python-simple-test-step1-v1.0.qstep.py):

from wf import QorusNormalStep
 
class PythonSimpleTestStep1(QorusNormalStep):
    def primary(self):
        UserApi.logInfo("hello from Python")

Using Python in Qorus Services

Qorus services can be defined in Python by subclassing the svc.QorusService class as in the following example:

Example (python-simple-test-v1.0.qsd.py):

from svc import QorusService
 
class PythonSimpleTest(QorusService):
    def test(self, arg):
        UserApi.logInfo("Hello from Python")
        return arg

Using Python in Qorus Jobs

Qorus jobs can be defined in Python by implementing a job class and subclassing the job.QorusJob class as in the following example.

Example (python-simple-test-v1.0.qjob.py):

from job import QorusJob
 
class PythonSimpleTest(QorusJob):
    def run(self):
        UserApi.logInfo("Hello from Python")

Restrictions on Python APIs

Python is not subject to sandboxing controls like Qore code, so it's easier to do dangerous things with Python.

All normal Python programming best practices should be followed when programming in Python; make sure all resources are freed in finally blocks and so forth; Qorus can only manage Qore resources, the Python runtime manages all Python resources normally.

Do not do any of the following:

• Generally you should not create Python threads; any background functionality should be handled in a Qorus service; most Qorus APIs require the thread context to be set up properly before calling; only services support background threads; to start a thread in a service, use the svcapi.startThread() API
• Do not make any calls that affect the current running process (ex: sys.exit(), _exit(), os._exit() and similar)

Qore Objects Imported Into Python Programs

Qore objects imported into Python are subject to special reference handling. Strong references to the Qore objects are always managed in Qorus, which means that when creating or acquiring a Qore object in Qorus with Python, the object will normally go out of scope after control returns to Qorus, meaning that the Qore destructor is then run, so that Qore's deterministic garbage collector can also be used in Python code.

For example, locks are released when wf.DynamicDataHelper objects are collected after created from Python step code. The same applies to wf.TempDataHelper and wf.SensitiveDataHelper objects.

See also

• UserApi.startCapturingObjectsFromPython()
• UserApi.stopCapturingObjectsFromPython()

Using the Qorus Qore-Language Client API in Python

Qorus's Qore-language client API can be used as in the following example:

import qoreloader
from qore.QorusClientBase import Client
 
qrest = Client.QorusSystemRestHelper(None, True)
print("PID:", qrest.get("system/pid"))

Writing Tests in Python for Qorus Interfaces

To use Qorus (and generally any Qore or Java) APIs in Python, the qoreloader.so module is required. This module can be found in $OMQ_DIR/python/qoreloader.so

See also

• Using the qoreloader Module to Import Java APIs Directly
• Using the qoreloader Module to Import Qore APIs

The following classes can help to write tests written in Python:

• QorusWorkflowTest
• QorusServiceTest
• QorusJobTest

Example Workflow Test

import sys
import qoreloader
from qore.QorusInterfaceTest import QorusPassiveWorkflowTest, ExecSynchronousOrder
 
class MyTest(QorusPassiveWorkflowTest):
    def __init__(self):
        super(QorusPassiveWorkflowTest, self).__init__("SIMPLETEST", "1.0", sys.argv[1:])
        self.addTestCase("wf test", self.test1)
        self.main()
 
    def test1(self):
        self.exec(ExecSynchronousOrder({
            "name": "SIMPLETEST",
            "staticdata": {},
        }))
 
mytest = MyTest()

Example Service Test

import sys
import qoreloader
from qore.QorusInterfaceTest import QorusServiceTest, CallService
 
class MyTest(QorusServiceTest):
    def __init__(self):
        super(QorusServiceTest, self).__init__("http-test", "user", sys.argv[1:])
        self.addTestCase("svc test", self.test1)
        self.main()
 
    def test1(self):
        call = CallService("http-test", "echo", 1)
        self.exec(call)
        self.assertEq(1, call.getResult()[0])
 
mytest = MyTest()

Example Job Test

import sys
import qoreloader
from qore.QorusInterfaceTest import QorusJobTest, RunJob, RunJobResult
from qore.__root__ import OMQ
 
class MyTest(QorusJobTest):
    def __init__(self):
        super(QorusJobTest, self).__init__("test", sys.argv[1:])
        self.addTestCase("job test", self.test1)
        self.main()
 
    def test1(self):
        action = RunJob()
        self.exec(action);
 
        action = RunJobResult(OMQ.StatComplete)
        self.exec(action);
 
mytest = MyTest()