TurboVM Bytecode Engine

TurboVM is an optional high-performance normalization engine for liblognorm. It compiles rulebases into bytecode at startup and executes them through a linear virtual machine with SIMD-accelerated parsing primitives. When enabled, it provides significant throughput improvements over the default recursive parser, especially on high-volume log streams.

Overview

The default liblognorm normalization engine (the “recursive walker”) traverses the parse DAG node-by-node for each log message, allocating json-c objects as fields are extracted. TurboVM replaces this with:

  • Bytecode compilation: rulebases are compiled into a compact instruction sequence at load time. Each rule becomes a linear program.

  • Arena allocation: all per-message memory comes from a single pre-allocated arena (~16 KB), fitting in L1 cache. Zero malloc/free per message.

  • SIMD parsing: character scanning, delimiter search, whitespace skipping, and IP address parsing use SSE4.2 or NEON intrinsics when available.

  • Output parity: field values match the standard engine (numeric fields are strings by default, as in the standard parser). The JSON document nests dotted names; see “Known differences” below.

  • Nested JSON: dotted field names (e.g. source.ip) produce nested JSON objects ({"source":{"ip":"..."}}), which is the ECS (Elastic Common Schema) shape.

Building with TurboVM

TurboVM is an optional build feature, disabled by default:

./configure --enable-turbo

The build system automatically detects the CPU architecture and enables the appropriate SIMD instruction set:

  • x86-64: SSE4.2 (Intel Nehalem+, AMD Bulldozer+)

  • ARM64: NEON (all ARMv8-A processors, including Apple M1/M2)

  • Other: scalar fallback (functional but without SIMD acceleration)

No additional dependencies are required.

Using with lognormalizer

The lognormalizer command-line tool supports turbo mode via the -oturbo option:

$ lognormalizer -r rules.rb -e json -oturbo < messages.log

In turbo mode:

  • Normalization uses the TurboVM bytecode engine

  • Output is compact JSON with nested objects for dotted field names

  • Field values match the standard engine (numeric fields are strings by default, or native JSON numbers with format="number")

  • Tags are emitted at the root as tags, the ECS spelling, and only when the matched rule carries any. The standard engine uses a flat event.tags key instead; see “Known differences” below

  • The getline() system call is used for input (more efficient than fgets() for large-scale processing)

If a rulebase cannot be compiled to bytecode (e.g. it uses unsupported parser types), lognormalizer falls back to standard normalization automatically. The same happens per message whenever the bytecode engine declines one.

That fallback is what makes turbo safe to enable, and also what makes a turbo defect hard to see: the standard engine quietly produces the right answer, so comparing output tells you nothing about whether the bytecode engine did any work. -oturbostrict turns the fallback off and reports the failure instead:

$ lognormalizer -r rules.rb -oturbostrict < messages.log

Use it when measuring turbo coverage or writing parity tests; a message reported unparsed under -oturbostrict but parsed under -oturbo is one the bytecode engine declined.

Library API

To enable TurboVM in your application, set the LN_CTXOPT_TURBO option on the normalization context before loading rules:

#include <liblognorm.h>

ln_ctx ctx = ln_initCtx();
ln_setCtxOpts(ctx, LN_CTXOPT_TURBO);
ln_loadSamples(ctx, "/path/to/rules.rb");

After loading, verify that compilation succeeded:

if (ln_turbo_is_available(ctx)) {
    /* bytecode is ready; ln_normalize_to_str() runs it */
}

For direct string output (bypassing json-c entirely):

char *json_str = NULL;
size_t json_len = 0;
int r = ln_normalize_to_str(ctx, msg, msg_len, &json_str, &json_len);
if (r == 0 && json_str) {
    /* json_str contains the normalized JSON string */
    free(json_str);
}

Only ln_normalize_to_str(), ln_turbo_normalize_to_str() and ln_turbo_normalize_raw() execute the bytecode and honour LN_CTXOPT_TURBO_STRICT. ln_normalize() always uses the recursive walker. Comparing those two APIs with turbo enabled compares the walker with itself and hides a turbo decline.

High-performance API (lognorm-turbo.h)

Consumers that want to avoid json-c construction entirely (for example rsyslog’s mmnormalize worker hot path) can use the curated public header lognorm-turbo.h. It is installed alongside liblognorm.h and lognorm-features.h and gated on LOGNORM_TURBO_SUPPORTED:

#include <liblognorm/lognorm-features.h>
#if defined(LOGNORM_TURBO_SUPPORTED)
#include <liblognorm/lognorm-turbo.h>
#endif

The header exposes only opaque types and a function-level contract; the internal turbo*.h headers and the fast-result/snapshot struct layouts are not installed and may change between releases without affecting the ABI. The contract covers:

  • ln_turbo_normalize_raw(): normalize into an opaque, context-owned result (valid until the next normalize call on that context). A truncated result is still returned, with ln_fast_result_is_truncated() set; the string path refuses that result and falls back to the walker.

  • ln_turbo_snapshot_result() / ln_fast_result_snapshot_get() / ln_fast_result_snapshot_free(): retain a result beyond the next call. The snapshot copies used field slots plus the result tail (tags, match info). Unused slots past n_fields are not copied. The allocation still covers a full ln_fast_result_t because ln_fast_result_snapshot_get() returns a pointer to that object.

  • typed accessors ln_fast_result_field_count(), ln_fast_result_get_field(), ln_fast_result_get_field_typed() (preserves LN_FTYPE_* value type and the LN_FFIELD_NESTED flag), ln_fast_result_get_string() / _get_int(), the tag accessors and ln_fast_result_get_rule_id().

Supported Parsers

TurboVM compiles every v2 parser type:

  • Text: word, alpha, string, rest, char-to, char-separated, string-to, op-quoted-string, quoted-string, literal

  • Numeric: number, float, hexnumber

  • Network: ipv4, ipv6, mac48

  • Date/Time: date-rfc3164, date-rfc5424, date-iso, time-24hr, time-12hr, duration, kernel-timestamp

  • Structured: json, cee-syslog, cef, v2-iptables, checkpoint-lea, name-value-list, repeat

  • Special: whitespace, cisco-interface-spec

A construct the compiler cannot express still fails the whole rulebase (compilation is all-or-nothing). Per-message declines fall back to the recursive walker unless LN_CTXOPT_TURBO_STRICT / -oturbostrict is set.

Known differences from the standard engine

The JSON document TurboVM writes is not the one the standard engine writes, by design:

  • Dotted field names nest. %source.ip:word% gives {"source":{"ip":"..."}} under TurboVM and a flat "source.ip" key under the standard engine. This is the ECS-output feature described above, not an accident, but it does mean the two engines produce different documents for any rulebase that uses dotted names.

  • Tags follow from that. TurboVM puts them at the root as "tags", the ECS spelling; the standard engine adds a flat "event.tags" key. Writing "event.tags" in the nested serializer would place the same logical path both inside and outside the event object whenever a rulebase also carries an event.* field.

Consumers of ln_turbo_normalize_raw() and the typed field accessors see none of this: it is the string serializer only.

A few parsers keep a deliberate match with the standard engine:

  • string-to with a single-character delimiter never matches. That is the standard parser’s behaviour (its inner comparison loop cannot run for a delimiter shorter than two bytes) and TurboVM replicates it so a rulebase behaves the same on both engines.

  • json field names follow the same three contracts as the standard parser (see configuration.rst “Special field names”): - is matched and discarded; . inlines object keys at the current context; a real name stores the value as one nested JSON object. Named %field:json% keeps arrays, booleans and nulls as JSON types. The . inline path walks the object into the turbo arena (dotted leaves, capped at LN_FAST_MAX_FIELDS). Nested arrays on that path are kept as JSON arrays (LN_FFIELD_RAW_JSON), not flattened to .0 / .1 keys.

  • The JSON scanner matches libfastjson: trailing whitespace after a value is consumed, single-quoted keys and strings are accepted, and true / false / null are case-insensitive. A named JSON field whose span contains a single quote is re-emitted as RFC JSON so the document stays valid.

Extracted field names (for example a name-value-list key) are JSON-escaped the same way as string values. An unescaped quote in a key would terminate the object syntax.

A rule that binds the same name twice keeps every binding, as libfastjson does. The JSON document lists them newest first (duplicate keys). ln_fast_result_get_string() returns the last, matching json_object_object_get_ex. A RFC JSON parser that last-wins on duplicate keys therefore sees the first-added value, which is what json.loads of the walker document also returns.

Performance Notes

Throughput improvements depend on the rulebase complexity and message format. Typical observations:

  • Simple rulebases (5-10 rules): 2-3x throughput improvement

  • Complex rulebases (50+ rules with alternatives): 5-10x improvement

  • The ln_normalize_to_str() path avoids json-c entirely and provides the highest throughput for applications that consume JSON as strings

TurboVM adds no overhead when disabled (--disable-turbo or default). Compilation is all-or-nothing per rulebase: the compiler walks one shared parse DAG, so a construct it cannot express means the whole rulebase runs on the recursive walker. When that happens the partially emitted program is discarded, ln_turbo_is_available() reports false, and no per-message work is wasted.

Limits

Two fixed-size structures bound what a single message can carry:

  • LN_FAST_MAX_FIELDS (128) fields per result. Rulebases over CSV-shaped sources reach this: a PAN-OS TRAFFIC record is 97 columns before annotations. Exceeding it is not silent: ln_normalize_to_str() refuses the result and falls back to the recursive walker, which has no field limit. Callers of ln_turbo_normalize_raw() get the partial result plus ln_fast_result_is_truncated() and decide for themselves.

  • LN_FAST_MAX_TAGS (16) tags per result, with the same contract.

A third limit is on the rulebase rather than the message. Linear non-terminal chains (one parser, not a terminal) compile in a loop, so a long sequential rule does not consume one C stack frame per field. Recursion is reserved for forks, prefix-terminals, and custom-type / repeat bodies, and is capped at LN_TURBO_MAX_COMPILE_DEPTH (1024). Exceeding it discards the program. Compilation is all-or-nothing, so one such rule puts the whole rulebase on the recursive walker. -oturbostrict and the turbo: compilation failed debug line are the way to notice.

Field names, annotation values and literal texts are not limited by the size of the opcode buffers they normally live in: anything that does not fit in the 60-byte inline slot is interned in the program string pool (OP_LITERAL_EXT for long compacted literals).