Logger Module

Table of Contents

• Introduction to the Logger Module
  • Scenarios
  • Memory-Based Backpressure
  • v1.1
  • v1.0
  • v0.6
  • v0.5
  • v0.4
  • v0.3
  • v0.2
  • v0.1.1
  • v0.1

Introduction to the Logger Module

The Logger module (aka Log4q) adopts its primary design from the well known log4j library

Other classes can be found in the logger_bin module for performance reasons, such as:

• Logger
• LoggerAppender (abstract)
• LoggerAppenderQueue
• LoggerAppenderWithLayout
• LoggerFilter
• LoggerInterface (abstract)
• LoggerLayout (abstract)
• LoggerLayoutPattern
• LoggerLevel
• LoggerPattern (abstract)

Abstract base classes must be subclassed to implement the desired functionality.

Scenarios

One Thread

The user code will log to a Logger object, and the logging is performed to the appender synchronously.

Example:

Logger l("mylogger", LoggerLevel::LevelInfo);
LoggerAppenderFile laf("myappender", new LoggerLayoutPattern(), "/var/run/log/mylog.log");
l.addAppender(laf);
laf.open();
....
l.info("hello %s #%d", "world", 1);
l.error("the %s is not perfect", "world");

Multiple Threads

The user code will log from multiple threads; the events are pushed to a LoggerAppenderQueue object. The processing is done in a dedicated thread when the events are passed to appenders. The user code logging command is non-blocking as it terminates immediately when the event is pushed in the queue.

Example:

our Logger l("mylogger", LoggerLevel::LevelInfo);
 
sub run() {
    while (!done) {
        ...
        l.info("hello %s #%d", "world", 1);
        l.error("the %s is not perfect", "world");
        ...
    }
}
 
LoggerAppenderFile laf("myappender", new LoggerLayoutPattern(), "/var/run/log/mylog.log");
laf.setQueue(new LoggerAppenderQueue());
l.addAppender(laf);
laf.open();
 
for (int i=0; i<10; i++) {
    background run();
}
 
while (True) {
    laf.getQueue().process();
}

Application Server Running Logging From a Few Sandboxed Program Containers

In this example, the appserver provides a logger API for a few sandboxed programs. The appserver is responsible for the Log4q configuration; i.e. it prepares loggers, appenders, filters, etc. according to configuration and provides the Logger instance to the Program container running the sandboxed code. The sandboxed code will log to this instance; the logging events are processed by the appserver in a dedicated thread which gets the event from a queue and passes it to appenders. Multiple loggers may be configured in a parent/child hierarchy so that a higher logging level (i.e. more event levels) are logged with the logger assigned to the sandbox and fewer (ex: only critical errors) to the global appserver logger.

Example:

LoggerAppenderQueue laq();
LoggerRoot lr("ERROR");
LoggerAppenderFile lar("", new LoggerLayoutPattern(), "/var/run/log/myappserver.log");
lar.setQueue(laq);
lar.open();
lr.addAppender(lar);
 
foreach string pn in ( .... ) {
    Logger l(pn);
    LoggerAppenderFile la(pn, new LoggerLayoutPattern(), "/var/run/log/"+pn+".log");
    la.setQueue(laq);
    la.open();
    l.setParent(lr);
    l.setAdditivity(True);
    l.addAppender(la);
    l.setLevel("DEBUG");
    Program p(PO_NEW_STYLE);
    p.loadModule("Logger");
    p.parse('
        our Logger logger;  # logging API for Program sandbox
        int sub main(string pn) {
            logger.log("INFO", "hello %s #%d", "world", 1);
            ...
            return 0;
        }
    ', pn, WARN_DEFAULT);
    p.setGlobalVarValue("logger", l);
    ...
    background p.callFunction("main", pn);
}
while (True) {
    laq.process(-1);
}

Application Server Running Many Sandboxed Program Containers

This example is basically the same as the previous example, but to avoid I/O bottlenecks in logging, the appserver processing thread gets the event from a queue and passes it to the appender in another worker thread by a submitting the logging action to a ThreadPool. So events targeted to a particular thread may by processed in different threads but nevertheless serially.

Example:

ThreadPool tp();
LoggerAppenderQueueThreadPool laq(tp, 5);
LoggerRoot lr("ERROR");
LoggerAppenderFile lar("", new LoggerLayoutPattern(), "/var/run/log/myappserver.log");
lar.setQueue(laq);
lar.open();
lr.addAppender(lar);
code processing() = sub () {
    while (True) {
        laq.process(-1);
    }
}
# may run in extra thread
background processing();
foreach string pn in ( .... ) {
    Logger l(pn);
    LoggerAppenderFile la(pn, new LoggerLayoutPattern(), "/var/run/log/"+pn+".log");
    la.setQueue(laq);
    la.open();
    l.setParent(lr);
    l.setAdditivity(True);
    l.addAppender(la);
    l.setLevel("DEBUG");
    Program p(PO_NEW_STYLE);
    p.loadModule("Logger");
    p.parse('
        our Logger logger;  # logging API for Program sandbox
        int sub main(string pn) {
            logger.log("INFO", "hello %s #%d", "world", 1);
            ...
            return 0;
        }
    ', pn, WARN_DEFAULT);
    p.setGlobalVarValue("logger", l);
    ...
    background p.callFunction("main", pn);
}
# wait till finished

Memory-Based Backpressure

Starting with version 1.1, the LoggerAppenderQueue class supports memory-based backpressure to prevent unbounded memory growth when log producers outpace consumers.

When using asynchronous logging with queues, if log messages are generated faster than they can be processed, the queue can grow without limit, potentially causing out-of-memory conditions. This is especially problematic when log messages contain large serialized data (such as context hashes formatted with %N).

Enabling Backpressure

Starting with Qore 2.3, backpressure is enabled by default with a 10 MiB limit:

# Create queue with default 10MB limit (recommended)
LoggerAppenderQueue laq();
# or explicitly
LoggerAppenderQueue laq(0);
 
# Create queue with custom 50MB limit
LoggerAppenderQueue laq(50 * 1024 * 1024);
 
# Create queue with no limit (not recommended for production)
LoggerAppenderQueue laq(-1);

Backpressure Behavior

When the queue reaches its memory limit, the push() method behavior depends on the timeout parameter:

• Block indefinitely (timeout = 0, default): The calling thread blocks until space becomes available
• Block with timeout (timeout > 0): Blocks up to the specified milliseconds, then throws QUEUE-TIMEOUT
• Drop immediately (timeout < 0): Returns False immediately if the queue is full

Example: Production Logging with Backpressure

ThreadPool tp();
# Create queue with 100MB limit and 5 worker threads
LoggerAppenderQueueThreadPool laq(tp, 5, 100 * 1024 * 1024);
 
LoggerAppenderFile la("app", new LoggerLayoutPattern(), "/var/log/app.log");
la.setQueue(laq);
la.open();
 
Logger l("app", LoggerLevel::LevelInfo);
l.addAppender(la);
 
# Monitor queue status
if (laq.isFull()) {
    # Queue is at capacity, consider reducing log volume
    stderr.printf("WARNING: Log queue at capacity (%d bytes)\n", laq.currentBytes());
}

Monitoring Methods

The following methods are available for monitoring queue status:

• maxBytes(): Returns the memory limit (-1 if unlimited)
• currentBytes(): Returns current queued bytes
• isFull(): Returns True if at or above the limit

v1.1

• Added memory-based backpressure support to LoggerAppenderQueue to prevent unbounded memory growth when log producers outpace consumers
  • new Qore::Logger::LoggerAppenderQueue::constructor(softint) with configurable memory limit (default: 10 MiB, -1 for unlimited)
  • new push() timeout parameter for backpressure control: block indefinitely, block with timeout, or drop immediately
  • new monitoring methods: maxBytes(), currentBytes(), isFull()
• Added max_bytes parameter to Logger::LoggerAppenderQueueThreadPool "LoggerAppenderQueueThreadPool" constructor

v1.0

• Split classes into the logger_bin module for performance reasons (issue 4884)
• Added a reopen event for rotatable appenders

v0.6

• Fixed a bug where file rotate could result in logging exceptions; removed explicit atomic lock operations and implemented low-level atomic file handling instead (issue 4842)

v0.5

• added LoggerInterface "LoggerInterface" and Logger::LoggerWrapper "LoggerWrapper" classes (issue 4776)

v0.4

• added the StdoutAppender class

v0.3

• fixed a race condition handling log file rotation with active logs (issue 4583)

v0.2

• added support for the %h and %P patterns for hostname and PID, respectively (issue 4179)
• allow file appenders to be reopened (issue 4171)
• enable serialization for LoggerEvent objects as well as for them to be submitted directly to Logger objects (issue 4164)

v0.1.1

• added Logger::Logger::logArgs() "Logger::logArgs()" (issue 3492)

v0.1

• the initial version of the Logger module