Qore python Module  1.1.4
Qore python Module

python Module Introduction

The python module allows for transparent and dynamic bidirectional importined code from Qore to Python and vice-versa. Qore classes can inherit Python classes and extend them, and Python classes can inherit Qore classes and extend them as well; even Qore abstract classes can be inherited by Python classes and made concrete by implementing the requried concrete methods in Python.

This module also implements support in the same shared object for the qoreloader module for Python so allow native Python code to import Qore APIs; see qoreloader Module for more information.

This module is released under the MIT license (see COPYING.MIT in the source distribution for more information). The module is tagged as such in the module's header (meaning it can be loaded unconditionally regardless of how the Qore library was initialized).

To use the module in a Qore script, use the %requires directive as follows:

%requires python

Helper Qore classes provided by this module:

Class Description
PythonProgram Allows Python code to be embedded and executed in Qore

Examples

Example:
#!/usr/bin/env qore
%new-style
%require-types
%strict-args
%enable-all-warnings
%requires python
PythonProgram::evalExpression("print('hello world!')");
Import Example:
#!/usr/bin/env qore
%new-style
%require-types
%strict-args
%enable-all-warnings
# import the math module into the Qore "Python::math" namespace
%module-cmd(python) import math
# import the JSONEncoder class into the "Python::json" namespace
%module-cmd(python) import json.JSONEncoder
%exec-class Main
public class Main {
constructor() {
JSONEncoder enc();
printf("%s\n", enc.encode(math::sin(1)));
}
}

Importing Python APIs into Qore

To import Python APIs and definitions into a Qore program, the following python-module-specific module command can be used, which also causes the python module to be loaded and initialized if necessary:

  • %module-cmd(python) import module[.symbol|*]
    imports the given Python module into Python and then imports the module or optionally only the given symbol into the Qore program from Python.
Example:
%module-cmd(python) import math.sin
auto sub get(auto arg) {
return math::sin(arg);
}
Example:
Program p(PO_NEW_STYLE);
p.loadModule("python");
p.issueModuleCmd("python", "import math");

Set the Python Module Path

Elements can be added to the Python import module path by using the following module command:

  • %module-cmd(python) add-module-path <path>

The path given is also subjected to environment variable substitutions; the environment variable name must be prefixed with a "$" char and then can be given either surrounded by "()" or "{}" brackets, of if no brackets are used, the environment variable must fit the following pattern: [a-zA-Z0-9_] (alphanumeric + "_").

Example:
%module-cmd(python) add-module-path ${MY_PYTHON_MODS}
%module-cmd(python) import my-python-mod
Example:
Program p(PO_NEW_STYLE);
p.loadModule("python");
p.issueModuleCmd("python", "add-module-path " + get_script_dir());
p.issueModuleCmd("python", "import my-python-module");

Parsing Python Code Immediately

Python code can be parsed immediately using the following module command:

  • %module-cmd(python) parse <filename> <Python source>
Example:
Program p(PO_NEW_STYLE);
string src = "
class PyTest:
def get(self):
return 3
";
p.issueModuleCmd("python", "parse pytest.py " + src);

Exporting Python Classes to Qore

Python classes can be explicitly exported to Qore with the following module command:

  • %module-cmd(python) export-class <class-name>
Example:
Program p(PO_NEW_STYLE);
p.issueModuleCmd("python", "export-class PyTest");
Class cls = Class::forName(p, "PyTest");
object obj = cls.newObject();
printf("get(): %y\n", obj.get());

Exporting Python Functions to Qore

Python functions can be explicitly exported to Qore with the following module command:

  • %module-cmd(python) export-func <func-name>
Example:
Program p(PO_NEW_STYLE);
p.issueModuleCmd("python", "export-func PyTest");
Function f = Function::forName(p, "py_test");
printf("py_test(): %y\n", obj.call());

Importing Qore APIs into Python

Qore APIs and definitions can be imported into a Python program by using the following module command:

  • %module-cmd(python) import-ns <qore-namespace> <python-module-path>
Example:
Program p(PO_NEW_STYLE);
p.issueModuleCmd("python", "import-ns MyQoreNamespace MyPythonModule");

Inheriting Qore Classes in Python

When Python code inherits Qore classes, the Qore constructor must be explicitly called in the Python __init__() method as in the following example.

Example:
import qore.xml
class PyXmlRpcClient(qore.xml.XmlRpcClient):
def __init__(self, opts, do_not_connect):
super(PyXmlRpcClient, self).__init__(opts, do_not_connect)

Creating Python Aliases

The following module command can be used to create an alias in Python code:

  • %module-cmd(python) alias <python-source-path> <python-target-path>

This is often used when importing Qore namespaces to Python to create an alias directly from a child Qore namespace to a Python module or package that is more convenient to access in Python code.

Example:
Program p(PO_NEW_STYLE);
p.issueModuleCmd("python", "import-ns MyQoreNamespace MyPythonModule");
p.issueModuleCmd("python", "alias MyPythonModule.MySubModule1.MySubModule2.MyApi MyApi");

Python Threading and Stack Management

This module implements support for multiple Python interpreters along with multithreading by managing the Python interpreter thread state manually.

Additionally, the Python recursion limit is also set automatically depending on the current thread stack size set in Qore.

This is done to avoid crashes due to stack exhaustion in combination Qore and Python code.

To avoid RecursionError exceptions when mixing Qore and Python in multithreaded code, make sure the thread stack size is set to an appropriate size in Qore (using set_default_thread_stack_size() before the Python interpreter is created, for example); the Python recursion limits are then calculated and managed dynamically according to the thread stack size and thread stack usage at the time of context switches to Python.

qoreloader Module

The Qore python module can also be aliased or symlinked to "qoreloader.so" ("qoreloader.pyd" on Windows) and placed in the Python module path (PYTHONPATH), as the Qore binary module also implement code for the qoreloader Python module in the same shared object, which allows Python programs to dynamically initialize the Qore library from Python programs and then import Qore APIs into Python using an injected qore package.

The qoreloader module implements the following functionality:

  • qore: a special loader package for dynamically importing Qore APIs into Python
  • java: a special loader package for dynamically import Java APIs into Python
  • qoreloader.load_java() : a Python function for loading Java APIs into Qore so they can be imported then into Python
  • qoreloader.issue_module_cmd() : a Python function for issuing Qore module commands, which can be used (among other things) for setting the Java CLASSPATH, for example

Using the qoreloader Module to Import Qore APIs

For example, for Python to import the Qore SqlUtil module, the following code can be used.

Example
import qoreloader
from qore.SqlUtil import Table
t = Table("pgsql:user/pass@db", "table_name")
print(t.getTable().selectRow({"where": {"col1": "value"}}))

When Qore modules are imported, the lowest namespace created by the module is imported as the new Python module under the Python qore package.

See the next section about importing the special qore.__root__ module.

Note
Unlike qore.__root__, using any other submodule name other than __root__ after qore will cause a Qore module to be loaded. If the Qore module cannot be found, an appropriate exception is thrown.

qore.__root__ Module

The special qore package has a special module called __root__ which represents the root namespace of the Qore program object.

Importing qore.__root__ module causes all symbols to be imported into the Python program.

Because Qore namespaces are imported as Python modules, any existing Qore namespace can be imported into a Python program using the qore.__root__ module as in the following examples:

# import the ::Qore::Thread namespace
import qore.__root__.Qore.Thread
# import specific classes from the Qore library
from qore.__root__.Qore.SQL import DatasourcePool, Datasource, SQLStatement
Note
Importing modules using qore.__root__ does not cause any external Qore modules to be loaded; it just imports existing Qore namespaces into the Python program. To load a Qore module explicitly, use a Qore module name after qore.

Using the qoreloader Module to Import Java APIs Directly

The python module implements support for using Qore's jni module to import Java code dynamically into Qore and then into Python using the java package.

To import Java classes into Python, import the java class by prefixing "java." to the package path name.

For example, to import the Java java.util.HashMap class into Python, the following code can be used.

Example
# load the qoreloader API
import qoreloader
# import the Java class into Python
from java.java.util import HashMap
m = new HashMap()
m.put("a", "value")

You can also set the classpath for Java imports as in the following example:

Example
# load the qoreloader API
import qoreloader
# set up the classpath so that the jni module can find the JAR file
qoreloader.issue_module_cmd('jni', 'add-classpath $HOME/jar/qore-jni.jar')
# import the Java class into Python
from java.org.qore.lang.restclient import RestClient
r = RestClient({"url": "https://localhost:8011"})

When Java packages and classes are imported, the lowest package created by the module is imported as the new Python module under the Python java package.

The given APIs are first dynamically imported into Qore, and then the Qore APIs are imported dynamically into Python.

This is the easiest way to import Java APIs into Python.

Importing Java Packages into Python Through Qore

Wildcard imports from Java are not currently directly supported, however you can import the Java package as a Python module and then the classes in the module will be dynamically created when they are referenced. See the following example:

Java Package Import Example
# load the qoreloader API
import qoreloader
# set up the classpath so that the jni module can find the JAR file
qoreloader.issue_module_cmd('jni', 'add-classpath $HOME/jar/qore-jni.jar')
# import the Java package into Python
from java.org.qore.lang import restclient
# the RestClient class is resolved and imported dynamically on reference
r = restclient.RestClient({"url": "https://localhost:8011"})
Note
Direct wildcard imports in the format from java.some.package.path import * are not yet supported, as once the Python module is created, the dynamic attribute lookups in the the global dictionary are not executed at runtime using the wildcard import information, therefore all Java classes to be imported should be either explicitly listed in the import statement, or the entire package should be imported as above; a "*" wildcard import will only work if the required Java classes have already been imported into the Qore program container in advance otherwise.

Importing Java APIs Indirectly into Python Through Qore

The python module implements support for using Qore's jni module to import Java code dynamically into Qore and then into Python using a two-step process.

To import Java code first into Qore, use qoreloader.load_java() with the Java dot path to the class name to load, and then import the Qore class into Python using the qore.__root__ module as in the following example:

Example
# load the qoreloader API
import qoreloader
# set up the classpath so that the jni module can find the JAR file
qoreloader.issue_module_cmd('jni', 'add-classpath $HOME/jar/qore-jni.jar')
# dynamically import the Java code into Qore
qoreloader.load_java("org.qore.lang.restclient.RestClient")
# import the Qore code into Python where it can be used
from qore.__root__.Jni.org.qore.lang.restclient import RestClient

Setting the Java Classpath

To set the Java classpath, call the qoreloader.issue_module_cmd() function with the first argument set to "jni" and the second set to "add-classpath <classpath>" or "add-relative-classpath <classpath>" as needed.

Example
# load the qoreloader API
import qoreloader
# set up the classpath so that the jni module can find the JAR file
qoreloader.issue_module_cmd('jni', 'add-classpath $HOME/jar/qore-jni.jar')
# dynamically import the Java code into Qore
qoreloader.load_java("org.qore.lang.restclient.RestClient")
# import the Qore code into Python where it can be used
from qore.__root__.Jni.org.qore.lang.restclient import RestClient

qoreloader.issue_module_cmd()

qoreloader.issue_module_cmd(module, cmd)
Arguments
  • module: the name of the Qore module where the command will be issued
  • cmd: the full command to be issued

qoreloader.load_java()

qoreloader.load_java(path)
Arguments
  • path: the dot path to the Java class to be loaded into Qore, after which it can be imported into Python under Python module "qore.__root__.Jni"; for example, path = 'javax.jms.Message' would be then imported into Python with "from qore.__root__.Jni.javax.jms import Message".
Example
# load the qoreloader API
import qoreloader
# set up the classpath so that the jni module can find the JAR file
qoreloader.issue_module_cmd('jni', 'add-classpath $HOME/jar/qore-jni.jar')
# dynamically import the Java code into Qore
qoreloader.load_java("org.qore.lang.restclient.RestClient")
# import the Qore code into Python where it can be used
from qore.__root__.Jni.org.qore.lang.restclient import RestClient
Note
Once the Java code has been imported into Qore, it must then be imported into Python; the Python parent module for such Java code is "qore.__root__.Jni"

Type Conversions Between Qore and Python

Type Conversions From Qore To Python

Source Qore Type Target Python Type
binary bytearray
bool bool
date datetime.datetime (absolute date/time values) or datetime.delta (relative date/time values)
float float
hash dict
int int
list list
list string
code (closure or call reference) callable

Type Conversions From Python to Qore

Source Python Type Target Qore Type
bool bool
int int or number if greater than 64-bits
float float
bytes binary
bytearray binary
datetime.datetime date (absolute date/time values)
datetime.delta date (relative date/time values)
dict hash
list list
string list
tuple list
function, method code

All other Python types are converted to a Qore object using a dynamically-created class that wraps the Python type or class.

Python Exception Handling

Python exceptions are mapped to Qore exceptions as follows:

  • err: the fully-qualified Python exception class name (ex: "builtins.SyntaxError", "socket.timeout")
  • desc: the string description, if none, then the exception arguments converted to a string with repr()
  • arg: the Python exception object itself converted to a Qore value

Managing the Lifecycle of Qore objects from Python

Qore's deterministic garbage collection approach and reliance on destructors means that Qore objects created by Python must have their lifecycles managed externally.

Python objects wrapping Qore objects hold only weak references to Qore objects.

Default Qore Object Lifecycle Management

By default, Qore objects are saved in thread-local data, so the lifecycle of the object is automatically limited to the existence of the thread.

The thread-local hash key name used to save the list of objects created is determined by the value of the "_python_save" thread-local key, if set. If no such key is set, then "_python_save" is used instead as the literal key for saving the list of objects.

Explicit Qore Object Lifecycle Management

The lifecycle of Qore objects can be managed explicitly by using the PythonProgram::setSaveObjectCallback() method to set a callback that is called every time a Qore object is created using the Python API.

This callback must take a single object argument as in the following example:

hash<string, object> object_cache;
code callback = sub (object obj) {
# save object in object cache, so it doesn't go out of scope
object_cache{obj.uniqueHash()} = obj;
}
PythonProgram::setSaveObjectCallback(callback);

python Module Release Notes

python Module Version 1.1.4

  • another fix handling subinterpreters (issue 4405)

python Module Version 1.1.3

python Module Version 1.1.2

  • not possible to use the qoreloader module from native Python (issue 4398)
  • not possible to call imported Java or Qore static methods when a normal method exists in the same class with the same name (issue 4397)

python Module Version 1.1.1

  • mark Qore namespaces imported from Python modules so that other languages like Java can use consistent paths with Python symbols (issue 4373)

python Module Version 1.1

python Module Version 1.0.7

  • fixed a bug setting the Qore program context in new Python threads calling imported Qore methods or functions (issue 4330)

python Module Version 1.0.6

  • fixed a bug handling exception propagation between Qore and Python (issue 4329)

python Module Version 1.0.5

  • fixed a deadlock in context switching when Python threads switch to Qore or Java (issue 4328)
  • fixed a crash when Python threads made Qore or Java calls (issue 4327)
  • fixed a regression in recursion depth calculations (issue 4323)

python Module Version 1.0.4

  • fixed calculating the recursion depth in the initial thread so that more Python modules can be loaded in single- threaded programs without changing the stack size (issue 4323)
  • fixed errors import valid Python modules (issue 4322)

python Module Version 1.0.3

  • do not shut down the Python library on exit, as it will cause a crash in exit handlers in Python modules that require the library to be in place (ex: h5py module v 3.3.0) (issue 4290)
  • fixed additional namespace clashes and namespace allocation issues for imported Python classes in Qore (issue 4289)

python Module Version 1.0.2

  • fixed issues handling the case when the python module is initialized with no Program context (issue 4153)
  • fixed a bug handling Python constructors in classes derived from Qore or Java classes where the constructor code accesses Qore or Java class members or methods before initializing the class (issue 4141)

python Module Version 1.0.1

  • fixed a bug where abstract Qore classes could be instantiated in Python (issue 4044)

python Module Version 1.0

  • initial public release