Schema reference

A schema is one file describing one routine. It is read by Config::Abstraction, so YAML, JSON and Perl data files all work — YAML is what everything is tested with. Every other part of the toolkit consumes this same file.

Naming the target

---
module: My::Widget
function: validate

If module is omitted the generator guesses from the file name: My-Widget.conf → My::Widget. If function is omitted it tests run, which is the right default for command modules.

The routine is a function call unless new is present. An empty new: is meaningful — it asks for Module->new() with no arguments and a method call thereafter:

module: My::Digest
function: hexdigest

new:

input

Named parameters

input:
  name:
    type: string
    optional: false
    min: 1
    max: 64
  age:
    type: integer
    optional: true
    min: 0
    max: 130

One unnamed parameter

input:
  type: string
  min: 1
  max: 1000

Positional parameters

When a routine takes a fixed argument order, say so with position. Positions start at 0. A parameter can carry both a name and a position; the name is used for diagnostics.

module: Math::Simple::MinMax
function: max

input:
  left:
    type: number
    position: 0
  right:
    type: number
    position: 1

output:
  type: number

The keyword undef in place of the whole input section means the routine takes no arguments:

input: undef

Types and constraints

Boundary cases

For each declared constraint the harness adds three deterministic cases: just inside the range, exactly on the boundary, and just outside it — the last annotated _STATUS => 'DIES', because a well-written routine should reject it. Booleans get 0/1, true/false and off/on plus values that should be refused. An enum gets one case per allowed value plus one outside.

Semantic generators

Where a plain random string would never be valid, name a generator instead:

input:
  email:
    type: string
    semantic: email
    position: 0

A semantic generator can be combined with matches when you want realistic data that also satisfies a business rule:

input:
  corporate_email:
    type: string
    semantic: email
    matches: '@company\.com$'

output

output:
  type: number
  min: 0

The output schema is checked with Return::Set. It accepts the same type and constraint keys as an input parameter, plus two special forms:

---
module: Scalar::Util
function: blessed

input:
  type: string

output: undef

config

Turn a probe off when the routine legitimately accepts something the default probe assumes it rejects. A string-to-string function that happily accepts undef should set test_undef: no, or the generated test will assert a croak that never comes.

transforms

A transform names a relationship between a subset of the input and a subset of the output. Each one is exercised with its own inputs, and the result is checked against the transform's own output spec. With properties enabled, each transform also becomes a LectroTest property.

---
module: builtin
function: abs

config:
  test_undef: no
  test_empty: no
  test_nuls: no

input:
  number:
    type: number
    position: 0

output:
  type: number
  min: 0

transforms:
  positive:
    input:
      number:
        type: number
        position: 0
        min: 0
    output:
      type: number
      min: 0

  negative:
    input:
      number:
        type: number
        position: 0
        max: 0
    output:
      type: number
      min: 0

  error:
    input: undef
    output:
      _STATUS: DIES

For each transform the generator makes cases from its input spec, calls the routine, checks the return against the transform's output spec, and requires an exact match when the output names a value.

Properties on a transform

Add a properties list to an individual transform to assert an invariant beyond type and range. Several are built in:

Anything the list does not cover can be written as Perl, with $result holding the return value and the input parameters available by name:

transforms:
  normalize:
    input:
      text:
        type: string
    output:
      type: string
    properties:
      - name: single_spaces
        description: No run of two or more spaces remains
        code: $result !~ /  /

      - name: reversible
        description: Length is unchanged
        code: length($result) == length($text)

cases and yaml_cases

A small static corpus maps an expected result to the arguments that produce it. This is the cheapest way to pin down behaviour you already know:

cases:
  ok:
    input: ping
    _STATUS: OK
  error:
    input: ""
    _STATUS: DIES

yaml_cases names a separate YAML file with the same shape, so a large corpus does not clutter the schema:

yaml_cases: t/conf/corpus/greet.yml

edge cases

Random generation will not stumble on the value that broke production. Say it instead. For named parameters, map parameter names to values:

edge_cases:
  name: [ "", "a" x 1024, "\x{263A}" ]
  age: [ -1, 0, 99999999 ]

For a routine with one unnamed parameter, use a list:

edge_case_array:
  - "<script>alert('xss')</script>"
  - "'; DROP TABLE users; --"
  - "\0null\0byte"
  - "emoji😊test"
  - ""
  - " "

During fuzzing there is a 40% chance that a case draws from the array instead of the random generator, so these values are always exercised.

type_edge_cases applies to every parameter of a given type, which is useful when a whole family of routines shares the same weakness:

type_edge_cases:
  string: [ "", " ", "\t", "\n", "\0", "long" x 1024, "\x{1F600}" ]
  number: [ 0, 1.0, -1.0, 1e308, -1e308, 1e-308, "NaN", "Infinity" ]
  integer: [ 0, 1, -1, 2147483647, -2147483648 ]

seed and iterations

seed: 42
iterations: 200

seed makes the generated harness call srand() with that value, so a failing run repeats exactly. iterations defaults to 30 random cases; the boundary and corpus cases are added on top of that, so the total test count is always higher.

accessor

accessor:
  property: ua
  type: getset

Declares the routine an accessor on property, with type one of getter, setter or getset. The generated tests then check the getter/setter contract — that a set is visible to a get, and that the getter with no argument does not change state.