Qore msgpack Module  1.0.0
Qore MessagePack Module

Table of Contents

Contents of this documentation:

Introduction to the MessagePack Module

The msgpack module provides MessagePack serialization and deserialization capabilities to Qore, allowing Qore programs to serialize data more efficiently than with JSON, XML or YAML modules.

This module is released under the MIT license (see LICENSE.txt in the source distribution for more information). The module is tagged as such in the module's header (meaning it can be loaded unconditionally regardless of how the Qore library was initialized).

Example using msgpack_pack and msgpack_unpack functions:

hash example = {"compact": True, "schema": 0};
*binary packed = msgpack_pack(example);
# ---
auto unpacked = msgpack_unpack(packed);

Example using the MsgPack class:

MsgPack mp();
hash example = {"compact": True, "schema": 0};
*binary packed = mp.pack(example);
# ---
auto unpacked = mp.unpack(packed);

Operation modes

The msgpack module supports the following operation modes:

  • simple mode: used for good inter-operability with other software using MessagePack for (de)serialization of data; may lose some precision/information for certain datatypes; this is the default
  • qore mode: aims for maximum compatibility with other Qore applications; does not lose any information

With the functions msgpack_pack() and msgpack_unpack(), operation mode can be passed as their second argument (after the value to be (un)packed), as in the following example:

msgpack_pack(data, MSGPACK_SIMPLE_MODE);
msgpack_unpack(data, MSGPACK_QORE_MODE);

With the objects of class MsgPack, operation mode can be passed as a constructor argument or set later via the setOperationMode method, as in the following example:

MsgPack mp(MSGPACK_QORE_MODE);
mp.setOperationMode(MSGPACK_SIMPLE_MODE);

Operation Mode Constants

Constant Mode Value
MSGPACK_SIMPLE_MODE simple mode 0
MSGPACK_QORE_MODE qore mode 1

Simple Mode

Simple mode is used for good compatibility with other software using MessagePack for (de)serialization of data. And because not all Qore datatypes have their MessagePack counterpart, some values are modified or automatically converted into different types to maintain compatibility. This happens to values of the date, null and number datatypes, and to some values of the string datatype.

Simple mode is the default mode of operation for the packing/unpacking functions and for the MsgPack objects.

The following table shows how Qore values are mapped to MessagePack values during serialization:

Qore Type MessagePack Type Notes
binary Raw Binary
bool Boolean
date (absolute) Timestamp (extension) date is converted into MessagePack Timestamp format
date (relative) Raw String date is converted into ISO-8601 duration format string (PnYnMnDTnHnMnS)
float Float (64bit)
hash Map loses complex type information
int Integer
list Array loses complex type information
MsgPackExtension Extension
nothing Nil
null Nil NULL values are serialized to Nil values, becoming the same as NOTHING values
number Float (64bit) numbers are converted into double (64-bit float) values, thereby possibly losing precision
object not supported
string Raw String all strings are converted to UTF-8 encoding

The following table shows how MessagePack values are unpacked to Qore values:

MessagePack Type Qore Type Notes
Array list
Boolean bool
Extension MsgPackExtension
Float float
Integer (signed) int
Integer (unsigned) int or number converted to int if smaller than 2^63 - 1, and converted to number otherwise
Map hash
Nil nothing
Raw Binary binary
Raw String string assumed to be in UTF-8 encoding
Timestamp (extension) date

Qore Mode

Qore mode aims for maximum compatibility with other Qore applications using this module and maintaining precision of data. Basic MessagePack types are used wherever possible and MessagePack extensions are used for Qore datatypes which don't have any MessagePack counterpart. This happens to values of the date, null and number datatypes, and to some values of the string datatype.

The following table shows how Qore values are mapped to MessagePack values during Qore mode serialization:

Qore Type MessagePack Type Notes
binary Raw Binary
bool Boolean
date (absolute) Extension (Date Extension)
date (relative) Extension (Date Extension)
float Float (64bit)
hash Map loses complex type information
int Integer
list Array loses complex type information
MsgPackExtension Extension
nothing Nil
null Extension (NULL Extension)
number Extension (Number Extension)
object not supported
string (UTF-8) Raw String
string (non-UTF8) Extension (String Extension)

The following table shows how MessagePack values are unpacked to Qore values in Qore mode:

MessagePack Type Qore Type Notes
Array list
Boolean bool
Extension MsgPackExtension
Extension (Date Extension) date
Extension (NULL Extension) null
Extension (Number Extension) number
Extension (String Extension) string
Float float
Integer (signed) int
Integer (unsigned) int or number converted to int if smaller than 2^63 - 1, and converted to number otherwise
Map hash
Nil nothing
Raw Binary binary
Raw String string assumed to be in UTF-8 encoding
Timestamp (extension) date

MessagePack Extensions

For handling MessagePack extensions, MessagePack module uses MsgPackExtension class as a wrapper for extension data. This class holds two pieces of information - extension binary data and extension type (ID). MsgPackExtension objects are returned from the unpacking functions and user can also create these objects themselves and pass them to packing functions.

Extension Types

In Qore operation mode values of the date, null and number datatypes, and non-UTF8 strings get serialized (packed) as MessagePack extensions with the following extension types:

Constant Extension for type Value
MSGPACK_EXT_QORE_NULL null 0
MSGPACK_EXT_QORE_DATE date 1
MSGPACK_EXT_QORE_NUMBER number 2
MSGPACK_EXT_QORE_STRING string 3

Date Extension

Qore date extension type uses MSGPACK_EXT_QORE_DATE extension type ID and is saved in the following format (all data in network byte order - big-endian):

Absolute dates:

1 (absolute) UNIX Epoch time micro-seconds UTC offset in seconds
8-bit int 64-bit int 32-bit int 32-bit int

Relative dates:

0 (not absolute) years months days hours minutes seconds micro-seconds
8-bit int 32-bit int 32-bit int 32-bit int 32-bit int 32-bit int 32-bit int 32-bit int

NULL Extension

Qore null extension type uses MSGPACK_EXT_QORE_NULL extension type ID and has no data (zero-length).

Number Extension

Qore number extension type uses MSGPACK_EXT_QORE_NUMBER extension type ID and is saved in the following format (all data in network byte order - big-endian):

NaN value:

MSGPACK_NUMBER_NAN
8-bit int

Infinity:

MSGPACK_NUMBER_INF
8-bit int

Negative infinity:

MSGPACK_NUMBER_NINF
8-bit int

Normal numbers:

MSGPACK_NUMBER_NORM precision number string representation
8-bit int 32-bit int 1+ chars

Number Sub-type Constants

The following constants are used for distinguishing sub-types of the Number Extension type:

Constant Number type Value
MSGPACK_NUMBER_NAN NaN 0
MSGPACK_NUMBER_INF Positive infinity 1
MSGPACK_NUMBER_NINF Negative infinity 2
MSGPACK_NUMBER_NORM Normal number 3

String Extension

Qore string extension type uses MSGPACK_EXT_QORE_STRING extension type ID and is saved in the following format (all data in network byte order - big-endian):

encoding constant string data
8-bit int 1+ chars

I.e. 1 byte for encoding and the rest for string data.

Encoding Constants

The following constants are used in the String Extension format:

Constant Encoding Value
QE_USASCII US-ASCII 0
QE_UTF8 UTF-8 1
QE_UTF16 UTF-16 2
QE_UTF16BE UTF-16BE 3
QE_UTF16LE UTF-16LE 4
QE_ISO_8859_1 ISO-8859-1 5
QE_ISO_8859_2 ISO-8859-2 6
QE_ISO_8859_3 ISO-8859-3 7
QE_ISO_8859_4 ISO-8859-4 8
QE_ISO_8859_5 ISO-8859-5 9
QE_ISO_8859_6 ISO-8859-6 10
QE_ISO_8859_7 ISO-8859-7 11
QE_ISO_8859_8 ISO-8859-8 12
QE_ISO_8859_9 ISO-8859-9 13
QE_ISO_8859_10 ISO-8859-10 14
QE_ISO_8859_11 ISO-8859-11 15
QE_ISO_8859_13 ISO-8859-13 16
QE_ISO_8859_14 ISO-8859-14 17
QE_ISO_8859_15 ISO-8859-15 18
QE_ISO_8859_16 ISO-8859-16 19
QE_KOI8_R KOI8-R 20
QE_KOI8_U KOI8-U 21
QE_KOI7 KOI7 22

Release Notes

MessagePack Module Version 1.0

  • initial release