NdjsonDataProvider Module

Table of Contents

• Introduction to the NdjsonDataProvider Module
• Examples
  • Reading NDJSON
  • Writing NDJSON
• Release Notes
  • NdjsonDataProvider v1.0

Introduction to the NdjsonDataProvider Module

The NdjsonDataProvider module provides support for reading and writing NDJSON (Newline Delimited JSON) formatted data, also known as JSON Lines. This format is useful for streaming JSON data where each line contains a valid JSON value.

Features:

• Memory-efficient streaming - processes one line at a time
• Configurable empty line handling
• Configurable error recovery for invalid JSON
• Support for both LF and CRLF line endings
• Maximum line length protection
• UTF-8 encoding support

The following classes are provided by this module:

• NdjsonIterator - iterator for reading NDJSON from strings
• InputStreamNdjsonIterator - iterator for reading NDJSON from streams
• NdjsonWriter - writer for creating NDJSON output
• NdjsonReadDataProvider - data provider for reading NDJSON files
• NdjsonDataProviderFactory - factory for creating NDJSON data providers

Examples

Reading NDJSON

    # Read from string
    string ndjson = '{"name": "Alice", "age": 30}
{"name": "Bob", "age": 25}
{"name": "Charlie", "age": 35}';
 
    NdjsonIterator iter(ndjson);
    while (iter.next()) {
        hash<auto> record = iter.getValue();
        printf("Name: %s, Age: %d\n", record.name, record.age);
    }
 
    # Read from file stream
    FileInputStream fis("data.ndjson");
    InputStreamNdjsonIterator stream_iter(fis);
    map printf("Record: %y\n", $1), stream_iter;

Writing NDJSON

StringOutputStream sos();
NdjsonWriter writer(sos);
writer.write({"name": "Alice", "age": 30});
writer.write({"name": "Bob", "age": 25});
string result = sos.getData();

Release Notes

NdjsonDataProvider v1.0

• initial release of the module