Fuzzing and property tests

A generated harness is not a black box. It is a Test::Most file you can read, with a corpus you can extend, random generation you can seed, and an optional property-test layer for invariants that example-based tests cannot reach.

What is in a generated harness

1. The static corpus — every case from cases, yaml_cases, POD examples and the generated boundary list.
2. Boundary cases — three per declared constraint, the outer one expected to die.
3. The fuzz loop — iterations cases (30 by default) built from the declared types, with edge-case pools mixed in.
4. Input validation — each argument spec is re-expressed as a Params::Validate::Strict schema, so a case that should die is checked for dying for the right reason.
5. Output validation — the return value goes through Return::Set, which is what makes "returns an integer" a testable claim.
6. Housekeeping — srand when a seed is set, an alarm-based timeout, and cleanup of files a routine writes into the working directory.

# what a run looks like
$ fuzz-harness-generator -r t/conf/validate.yml
Generated: /tmp/S7Ql2m.t
Result: PASS
Files=1, Tests=118,  0 wallclock secs

If it fails, the TAP output names the case: the parameter, the value, and whether it was expected to live or die. Turn the failing value into an edge_cases entry so it is retested forever, not just until the next random seed.

Turning probes on and off

The config section controls the default probes: NUL bytes, undef, empty values, non-ASCII strings, deduplication, the timeout, and the security-string set. All of them are on by default except the last two, because a routine that accepts undef is unusual — but not rare enough to be a safe assumption.

config:
  test_undef: no
  test_nuls: no
  timeout: 30

Corpora

Random inputs are worth keeping. The coverage-guided fuzzer records the inputs that reached a branch nobody had reached before, plus every input that made the code die, in a corpus file. That corpus can be replayed:

# fuzz as schemas are extracted, keeping the corpus
extract-schemas --fuzz --corpus-dir schemas/corpus lib/My/Module.pm

# turn a corpus into a deterministic regression test
fuzz-harness-generator --replay-corpus schemas/corpus/translate.json \
    -o t/fuzz_replay.t

A replay harness runs the recorded inputs in a fixed order. It is the right thing to commit: no randomness, no coverage tooling, and a failure that reproduces on every machine.

Coverage-guided fuzzing

With Devel::Cover installed, the fuzzer does not just generate — it measures. Each iteration is scored by which branches it executed; an input that adds coverage is kept and becomes a seed for mutation, so later iterations are variations on inputs that already got deep into the code rather than fresh random values.

Two invariants make the minimiser safe to run in CI. Bug inputs are always retained, whether or not they add coverage — they are the most valuable thing the fuzzer produces. And minimisation is idempotent: a second pass finds nothing left to remove.

Without Devel::Cover the fuzzer still runs; it simply has no branch feedback, so it degrades to random generation with corpus retention.

Property-based tests

Example-based tests check the cases you thought of. Property tests check that a statement holds for a thousand generated inputs. Turn them on in the schema:

config:
  properties:
    enable: true
    trials: 1000

Properties come from two places. The generator derives some automatically from each transform's output spec — range bounds, type preservation, definedness, exact values — and you can list more by name, or write them as Perl. See transforms and properties for the full syntax and the list of built-in names.

Generated property tests need Test::LectroTest and Test::LectroTest::Compat. Both are test requirements of the distribution, so make test installs them; a generated file that cannot load them skips its property section with a diagnostic instead of failing.

ok 123 - negative property holds (1000 trials)
ok 124 - positive property holds (1000 trials)

On failure LectroTest shrinks the counterexample to a small one and prints the inputs that broke the property. Those belong in the static corpus.

Properties worth asserting

The properties that pay for themselves are the ones an example-based test cannot express:

• Idempotence — normalising twice changes nothing.
• Round trips — decode(encode($x)) eq $x.
• Order — output of a sort is non-decreasing, and is a permutation of the input.
• Invariants across inputs — a total is never less than any of its parts; a length never grows.

Keeping fuzz runs finite

Fuzzing multiplies: every schema is a harness, every harness is a prove run, and property tests add their trials on top. Two practical rules keep make test quick:

• Keep the aggressive schemas in t/conf/ and the extended-test guard on the runner, so a normal test run does the cheap part and CI does the rest.
• Set iterations low (or 0, with properties only) for a schema whose random generation is slow, and rely on the corpus for depth.