Introduction to the DataProvider Module

The DataProvider module provides APIs for hierarchical data structures from arbitrary sources to be described, queried, introspected, and updated. It supports data providers supporting record-based APIs as well as request-reply semantics such as REST schemas or with SOAP messaging, as well as event-based data providers.

The data provider module supports high-performance reading (native or simulated searching, also with advanced search expression support) and writing as well as record creation, upserting, and transaction management for record-based data providers if supported by the underlying data provider implementation as well.

The Qore command-line program qdp provides a user-friendly interface to data provider functionality. It auto-detects DPQL search expressions and also supports forcing DPQL parsing with the -q/–dpql option, for example:

qdp mydb/tables/users search -q '@name == "John"'

qdp Command-Line Usage

The qdp CLI exposes the DataProvider API for inspection and data access. It supports DPQL for search criteria, auto-detects DPQL expressions by default, and can show DPQL syntax help with –dpql-help. For a full DPQL reference, see DPQL - Data Provider Query Language.

Example Searches: qdp mydb/tables/users search '@name == "John"'

qdp mydb/tables/users search '@age between 18 and 65'

qdp mydb/tables/users search '(@active == true) && (@age >= 21)'

qdp mydb/tables/users search '@role in ("admin", "editor")'

Use - or {} as an empty expression to return all records.

This module provides the following primary classes:

Many classes for specific data types are provided as well.

Data Provider Modules

This module uses the "QORE_DATA_PROVIDERS" environment variable to register data provider modules. Each data provider registration module must provide one of the following two public functions.

Data Provider Dynamic Discovery

Implement a public function with the following signature to support dynamic discovery of data providers:

# returns a hash of connection provider factory names to module names

public hash<string, string> sub get_data_provider_map() { ... }

Data Provider Type Dynamic Discovery

Implement a public function with the following signature to support dynamic discovery of data provider types:

# returns a hash of type prefix paths (ex: \c "qore/sftp") to module names

public hash<string, string> sub get_type_provider_map() { ... }

Data provider registration modules declared in the "QORE_DATA_PROVIDERS" environment variable must be separated by the platform-specific PathSep character as in the following examples:

Unix Example

export QORE_DATA_PROVIDERS=MyDataProvider:OtherDataProvider

Windows CMD.EXE Example

set QORE_DATA_PROVIDERS=MyDataProvider;OtherDataProvider

Windows PowerShell Example

$env:QORE_DATA_PROVIDERS="MyDataProvider;OtherDataProvider"

Data Provider Pipelines

Data provider pipelines allow for efficient processing of record or other data in multiple streams and, if desired, in multiple threads.

Pipeline data can be any data type except a list, as list values are interpreted as multiple output values in pipeline procesor objects.

Data Provider Pipeline Bulk Processing

Bulk processing is processing of record data that is in "hash of lists" form, so a single hash, where each key value is a list of values for that key. Records can be formed as by taking each hash key and then using each list value in order for the values of each record. In case a key is assigned a single value instead of a list, it's interpreted as constant value for all records. Note that the first key value for bulk data must be a list of values in order for the bulk data to be properly detected.

Each pipeline processor element must declare if it is compatible with bulk processing by implementing the AbstractDataProcessor::supportsBulkApiImpl() method.

If a processor does not support the bulk API, but bulk data is submitted, then the bulk data will be separately iterated, and each record will be passed separately to the processor with a significant performance penalty when processing large volumes of data.

Debugging Data Providers

Data providers support an opt-in debug info collection mechanism that allows capturing technical details about operations such as HTTP requests, WebSocket messages, and connection events. This is useful for debugging, logging, and monitoring data provider interactions.

Debug Collector Classes

The following classes are provided for debug info collection:

AbstractDebugInfoCollector: Abstract base class - implement your own collector by overriding the collect() method
BufferedDebugInfoCollector: Stores debug info in memory for later retrieval and analysis
ConsoleDebugInfoCollector: Outputs debug info to the console immediately, with optional colorized output

Debug Info Structure

Debug info is captured in the DataProviderDebugInfo hashdecl which includes:

Operation type (request, response, connect, disconnect, send, receive)
Timestamps and duration
URL, HTTP method, and protocol (HTTP/1.1 or HTTP/2)
Request and response headers
Request and response bodies
Status codes and error messages
WebSocket message types and sizes
TLS/SSL information

Using Debug Collection

To enable debug collection, register a debug collector with a data provider:

# Using BufferedDebugInfoCollector for programmatic access
BufferedDebugInfoCollector collector();
provider.setDebugCollector(collector);
 
# Make some requests
auto response = provider.doRequest(request);
 
# Retrieve collected debug info
list<hash<DataProviderDebugInfo>> info = collector.getAll();
foreach hash<DataProviderDebugInfo> debug in (info) {
    printf("%s %s -> %d (%dms)\n", debug.method, debug.url, debug.status_code, debug.duration_ms);
}
 
# Clear the buffer when done
collector.clear();

# Using ConsoleDebugInfoCollector for immediate output
# Verbosity: 1 = basic, 2 = headers, 3 = full with bodies
ConsoleDebugInfoCollector collector(2);
provider.setDebugCollector(collector);
 
# Debug info is printed to console as operations occur
auto response = provider.doRequest(request);
 
# Disable debug collection when done
provider.setDebugCollector();

Custom Debug Collectors

You can create custom debug collectors by inheriting from AbstractDebugInfoCollector:

class MyDebugCollector inherits AbstractDebugInfoCollector {
    collect(hash<DataProviderDebugInfo> info) {
        # Log to a file, send to monitoring system, etc.
        logger.info("Operation: %s %s -> %d", info.method, info.url, info.status_code);
    }
 
    flush() {
        # Called when the data provider is done - flush any buffers
    }
 
    clear() {
        # Called to clear any collected data
    }
}

DPQL - Data Provider Query Language

DPQL (Data Provider Query Language) is a human-readable query language for constructing DataProviderExpression structures. It provides a simple, intuitive syntax for writing search expressions that can be used with data provider search, update, and delete operations.

DPQL Syntax

DPQL supports the following syntax elements:

Field References

Fields are referenced using the @ prefix:

@name - simple field name
@"field with spaces" - quoted field names for special characters
@user.email - dotted paths for nested fields

Comparison Operators

These are standard DPQL operators; individual data providers can expose additional or different operators. DPQL is translated to native server-side queries whenever possible to ensure maximum performance.

== - equals
!= - not equals
< - less than
<= - less than or equal
> - greater than
>= - greater than or equal
between - range comparison (@age between 18 and 65)
in - set membership (@status in ("active", "pending"))
=~ - regex match (@email =~ /example.com$/i)

Logical Operators

These are standard DPQL operators; individual data providers can expose additional or different operators.

&& - logical AND
|| - logical OR
! - logical NOT

Literals

Strings: "hello" or 'world'
Dates: 2026-01-02, 2026-01-02T15:20:11.123+01:00
Binary: <deadbeef>
Lists: (one, two, 3)
Hashes: {a=1, b=two}
Unquoted identifiers are treated as strings: @domain == omq
Numbers: 123, 45.67, -10
Booleans: true, false
Null: null

Function Calls: Custom functions can be used: calculate(@score, 100) > 50

DPQL Examples

Simple Comparison: # Parse a simple equality expression

hash<DataProviderExpression> exp = DataProvider::parseDpqlExpression('@name == "John"');

# Result: {"exp": "==", "args": [{"field": "name"}, "John"]}

Complex Query: # Parse a complex query with multiple conditions

hash<DataProviderExpression> exp = DataProvider::parseDpqlExpression(

'@age > 18 && @status == "active" || @role == "admin"');

Range Query: # Find records where age is between 18 and 65

hash<DataProviderExpression> exp = DataProvider::parseDpqlExpression('@age between 18 and 65');

Set Membership: # Find records with specific status values

hash<DataProviderExpression> exp = DataProvider::parseDpqlExpression(

'@status in ("pending", "active", "review")');

Regex Matching: # Find email addresses from a specific domain (case-insensitive)

hash<DataProviderExpression> exp = DataProvider::parseDpqlExpression('@email =~ /example.com$/i');

Date Literal: # Find records created before a specific date

hash<DataProviderExpression> exp = DataProvider::parseDpqlExpression('@created < 2026-12-01');

Binary Literal: # Match a binary hash field

hash<DataProviderExpression> exp = DataProvider::parseDpqlExpression('@checksum == <deadbeef>');

Negation: # Find non-deleted records

hash<DataProviderExpression> exp = DataProvider::parseDpqlExpression('!(@deleted == true)');

Using with Data Providers: # Search a data provider using DPQL

hash<DataProviderExpression> where = DataProvider::parseDpqlExpression(

'@created > 2024-01-01 && @type == "order"');

AbstractDataProviderRecordIterator i = provider.searchRecords({"where": where});

Round-Trip Conversion

DPQL expressions can be serialized back to strings:

# Parse and serialize an expression
hash<DataProviderExpression> exp = DataProvider::parseDpqlExpression('@name == "John" && @age > 18');
string dpql = DataProvider::serializeDpqlExpression(exp);
# Result: '@name == "John" && @age > 18'

Error Handling and Diagnostics

The parser provides detailed diagnostics for syntax errors:

# Parse with diagnostics
hash<DpqlParseResult> result = DataProvider::parseDpqlExpressionWithInfo('@name ==');
if (!result.success) {
    foreach hash<DpqlDiagnostic> diag in (result.diagnostics) {
        printf("Error at line %d, column %d: %s (code: %s)\n",
            diag.line, diag.column, diag.message, diag.code);
    }
}

Syntax Highlighting

For IDE integration, DPQL provides token information with position tracking:

# Get tokens for syntax highlighting
list<hash<DpqlTokenInfo>> tokens = DataProvider::getDpqlTokens('@name == "John"');
foreach hash<DpqlTokenInfo> tok in (tokens) {
    printf("Token: type=%d value='%s' at line %d, col %d\n",
        tok.type, tok.value, tok.line, tok.column);
}

Token types include:

DPQL_TOK_FIELD (1) - field references
DPQL_TOK_STRING (2) - string literals
DPQL_TOK_NUMBER (3) - numeric literals
DPQL_TOK_BOOLEAN (4) - boolean values
DPQL_TOK_IDENTIFIER (6) - identifiers/keywords
DPQL_TOK_DATE (7) - date literals
DPQL_TOK_BINARY (8) - binary literals
DPQL_TOK_LIST (21) - list literals
DPQL_TOK_HASH (22) - hash literals
DPQL_TOK_OP_EQ (10) through DPQL_TOK_OP_REGEX (19) - operators
DPQL_TOK_REGEX (20) - regex patterns

Auto-Completion

For IDE integration, DPQL provides context-aware completions:

# Get completions at cursor position
hash<string, AbstractDataField> fields = {
    "name": new QoreDataField("name", "Name", StringType),
    "age": new QoreDataField("age", "Age", IntType),
    "email": new QoreDataField("email", "Email", StringType),
};
list<hash<DpqlCompletionItem>> items = DataProvider::getDpqlCompletions('@na', 3, fields);
# Returns completions matching "na", e.g., [{"label": "name", "kind": "field", ...}]

DPQL WebSocket API

The DPQL WebSocket API is provided by the optional DpqlWebSocket module. It exposes language-server services for tokenization, parsing, validation, and completion, and can be used by UI clients for syntax highlighting, diagnostics, and completions in the context of a data provider.

For Creator WebSocket integrations, the action-based DpqlActionSession class provides a transport-agnostic DPQL service with context-aware operations and token output suitable for editor integrations.

%requires HttpServer
%requires DpqlWebSocket
 
DataProvider::DpqlWebSocketHandler handler();
HttpServer server({"port": 8080});
server.addHandler("/ws/dpql", handler);
server.start();

The WebSocket message protocol is defined in:

DpqlWebSocket/DpqlApi.asyncapi.yaml

Expression Validation

Expressions can be validated against available fields:

# Validate expression against fields
hash<string, AbstractDataField> fields = {
    "name": new QoreDataField("name", "Name", StringType),
};
list<hash<DpqlDiagnostic>> errors = DataProvider::validateDpqlExpression(
    '@unknown_field == "test"', fields);
if (errors) {
    printf("Validation error: %s\n", errors[0].message);
    # Output: "Validation error: Unknown field 'unknown_field'"
}

Release Notes

DataProvider v3.3

added methods to return data provider types in the context of option dataprovider_message_support

DataProvider v3.2.2

fixed setting the mandatory flag for hashes with dynamic fields (issue 4992)
fixed returning dynamic options for data-dependent action options (issue 4985)
default values are no longer generated for optional hash types based on their fields (issue 4972)
fixed typos in DataProviderActionCatalog::getTypeForActionOptions() (issue 4971)

DataProvider v3.2.1

improved logic handling locked types in the type cache

DataProvider v3.2

added the SuperSoftStringDataType and SuperSoftStringOrNothingDataType types to support "soft" string types that accept almost any type and convert it to a string

DataProvider v3.1

added an API to retrieve event data in the context of specific option values
fixed handling element allowed values in fields (issue 4951)
fixed handling implicit default values in HashDataType (issue 4941)

DataProvider v3.0

added support for a data provider action catalog (issue 4808)
added support for automatically deregistering factories, applications, and actions if an error occurs during foreign module initialization (issue 4816)

DataProvider v2.7.5

fixed conversion of soft binary, bool, float, int, and number "or nothing" types from an empty string to NOTHING (issue 4836)

DataProvider v2.7.4

added allowed_values to field data info (issue 4762)

DataProvider v2.7.3

added missing base64binary and hexbinary type support (issue 4732)

DataProvider v2.7.2

fixed bugs handling null hash assignments with fields with default values in HashDataType (issue 4715)

DataProvider v2.7.1

added DataProvider::getFactoryObjectWithTemplate() and AbstractDataProviderFactory::createWithTemplate() to allow data providers to be created from factories with template arguments (issue 4624)
AbstractDataProvider::search*() fails with record-based data providers that do not provide a record definition (issue 4624)
AbstractDataProvider::search*() methods fail with an empty where hash argument (issue 4622)

DataProvider v2.7

fixed a bug accepting the element type in SoftListDataType (issue 4603)
added child capability support to data provider summary info (issue 4576)

DataProvider v2.6

added child capability support to data provider info (issue 4572)

DataProvider v2.5

implemented support for the observer pattern / event API (issue 4557)

DataProvider v2.4

allow data providers to be introspected with additional information about children (issue 4543)

DataProvider v2.3

updated to use new generic expression support for searches (issue 4538)
addded support for data provider options to support default values (issue 4532)
addded search_operators to the data provider response to report supported search operators (issue 4528)
fixed a type error in DataProvider::search*() calls with an empty where hash argument (issue 4511)

DataProvider v2.2.1

made it possible to determine the default field type in HashDataType (issue 4491)

DataProvider v2.2

added an attribute to factories to indicate if any children can support the request-response pattern (issue 4486)
added the DTT_ClientOnly type tag constant (issue 4477)

DataProvider v2.1.2

fixed a bug where DataProvider types based on string that accept multiple input types with conversions but are not soft types threw spurious errors at runtime with non-string types (issue 4471)
added the desc key supporting a markdown description to data provider info (issue 4465)
fixed a bug where hash types incorrectly claimed compatibility with incompatible hashes (issue 4463)

DataProvider v2.1.1

fixed a type error iterating data in DefaultBulkRecordInterface (issue 4457)

DataProvider v2.1

added support for tags on data provider types (i.e. AbstractDataProviderType) (issue 4442)
fixed data provider factory info to provide info about the actual data provider factory instead of only provider info (issue 4438)
added "from example" APIs to data provider factories:
- AbstractDataProviderFactory::getProviderFromExample()
- AbstractDataProviderFactory::getExampleProviderRecordOutput() (issue 4430)
added generic search capabilities to data providers with no native search capabilities (issue 4430)

DataProvider v2.0

added a logger to the AbstractDataProvider class to enable logging (issue 4409)
added support for creating, reading, updating, and modifying data providers (issue 4395)

DataProvider v1.3

added DataProviderTypeCache::listParentTypes() to allow a list of parent types to be returned (issue 4385)

DataProvider v1.2.2

added support for starting a transaction with DataProviders that support transaction management (issue 4346)

DataProvider v1.2.1

return the base_type key in input type info in AbstractDataProviderType::getInputInfo() (issue 4311)

DataProvider v1.2

greatly improved dataprovider and type acquisition speed from clients requiring initialization from the environment (issue 4305)

DataProvider v1.1

the DataProvider::DefaultRecordIterator class supports the limit search option

DataProvider v1.0.6

fixed a bug where the data type was not supported correctly as a data provider type (issue 4232)

DataProvider v1.0.5

fixed an issue where it was not possible for pipeline to flush any pending data or to perform reset or purge operations when pipeline processing is complete; for this the AbstractDataProcessor::flush() and AbstractDataProcessor::flushImpl() methods were implemented (issue 4107)

DataProvider v1.0.4

fixed type-handling bugs handling data provider options (issue 4062)

DataProvider v1.0.3

fixed bugs in the default return value for AbstractDataProcessor::getExpectedType() and AbstractDataProcessor::getReturnType() (issue 4043)

DataProvider v1.0.2

fixed a bug where HashDataType would not enforce fields or field types when checking for type and value compatibility (issue 4037)
added new Qore base types to the data type hierarchy
implemented the following classes:
added DUPLICATE-RECORD exception documentation for AbstractDataProvider::createRecord()
updated to allow data provider type attributes to appear as children in the type hierarchy (issue 4015)

DataProvider v1.0.1

implemented callbacks to allow for dynamic elements of request-response data providers (such as URI paths) to be resolved at runtime (issue 3838)
AbstractDataProviderType and AbstractDataField classes made serializable (issue 3809)
implemented the DataProviderTypeCache class to encapsulate reusable type caching logic (issue 3810)

DataProvider v1.0

initial release of the module

Table of Contents