Property-Based Testing: Assert the Law, Not the Example

An example-based test says: for this input, expect that output. It is precise, readable, and limited in a specific way. The examples come from your head, and your head is the same instrument that wrote the bug.

The cases people habitually write are the ones they already reasoned about. The cases that break software are the ones nobody reasoned about: the empty collection, the string with a combining character, the value exactly at the boundary, the negative zero, the input arriving twice, the list already sorted, the timestamp during a leap second.

Key idea: example-based testing samples the input space at points you chose. Since you chose them by thinking, and the bug survived your thinking, the sample is biased away from your bugs by construction.

Property-based testing changes the shape of the claim. Instead of asserting an output for an input, you assert a law that must hold for every input, then hand the search for a violation to a machine that has no preconceptions about which inputs are interesting.

The mechanics are ordinary. You declare the shape of valid inputs, and write an assertion that must hold for all of them.

from hypothesis import given, strategies as st

# Example-based: one case you thought of
def test_roundtrip_example():
    assert decode(encode({"name": "Ada"})) == {"name": "Ada"}

# Property-based: the law, for every dict of this shape
@given(st.dictionaries(st.text(), st.integers()))
def test_roundtrip(payload):
    assert decode(encode(payload)) == payload

The framework, here Hypothesis for Python, generates hundreds of dictionaries: empty ones, ones with empty-string keys, ones with astral-plane characters, ones with keys that differ only by Unicode normalisation, ones with integers beyond 64 bits. Each is fed through the same assertion.

The example test passes forever with a serialiser that mangles empty keys. The property test finds it on the first run, because "a dictionary with an empty-string key" is an unremarkable member of the space it was told to explore, and an exotic thought for a person writing examples.

The idea and the first implementation come from QuickCheck, by Koen Claessen and John Hughes, published at ICFP 2000 and later given that conference's most influential paper award.

The hard part is never the tooling. It is answering "what is true about this function for every input?" Five patterns cover most real cases.

The oracle pattern deserves emphasis because it is the one people forget they can use. When you optimise a function, the previous implementation becomes a free oracle: keep it, and assert the new one agrees with it on everything the generator can produce. The same applies when replacing a library, migrating a schema, or rewriting a service in another language.

Metamorphic properties are the escape hatch when you cannot compute the right answer at all. You may not know what a search should return for a random query, but you know that narrowing a filter must not add results, and that is a checkable law.

Random testing without shrinking is a bad experience. The counterexample it hands you is a random input, which means a 400-element list of arbitrary integers with the failure hidden somewhere inside it.

Shrinking is the search for a smaller input that still fails. When a property is violated, the framework repeatedly simplifies the counterexample, dropping list elements, moving integers toward zero, emptying strings, and re-runs the property on each candidate, keeping any that still fail. It stops when no further simplification reproduces the failure.

Implementations differ in how they do it: QuickCheck takes shrinkers defined per type, while Hypothesis shrinks the underlying random choices that produced the value, an approach called internal shrinking that also inherits shrinking automatically for composed generators. Hedgehog and RapidCheck use integrated shrinking, where generators carry their own shrink trees.

Reading the cycle end to end makes the failure and success modes legible.

The generator draws an input from the declared space. The property runs on it. If it holds, the loop draws again, typically for a fixed number of examples, defaulting to the low hundreds in most frameworks. If the property fails, the framework enters the shrink loop, simplifying while the failure persists, and finally reports the minimal case.

One detail matters operationally: the failing example is usually saved to a local database and replayed first on subsequent runs. That turns a random find into a deterministic regression test automatically, which resolves the obvious objection that random tests are not reproducible. Frameworks also print the seed, so any run can be reproduced exactly.

The cost model is worth internalising too. A property test runs its body hundreds of times, so a property over a function taking 10 ms costs seconds, not milliseconds. That is affordable for pure logic and rarely affordable for anything touching a network, which is the main constraint on where the technique fits.

A property is only as good as the inputs it sees, and default generators produce values that are valid for the type but often meaningless for the domain. A random string is not a valid email address; a random integer pair is not a valid date range.

Frameworks provide composition for this. You build a generator for your domain from primitives, adding constraints and derived fields:

@st.composite
def bookings(draw):
    start = draw(st.datetimes(min_value=datetime(2020, 1, 1)))
    nights = draw(st.integers(min_value=1, max_value=30))
    return Booking(start=start, end=start + timedelta(days=nights))

@given(bookings())
def test_nightly_rate_never_exceeds_total(b):
    assert nightly_rate(b) * b.nights <= total_price(b) + 0.01

Two failure modes to watch for, both of which produce a test that passes while proving nothing:

• Over-constrained generators. Filtering aggressively (.filter(lambda x: x > 0)) discards most draws, and frameworks eventually give up and report too few examples. Construct valid values directly instead of rejecting invalid ones.
• The generator encoding the same assumption as the code. If your generator only produces well-formed input because your parser only handles well-formed input, the property is a tautology dressed as a test.

In practice: when a property finds nothing after several runs, suspect the generator before concluding the code is correct. Print a sample of the drawn values; the answer is usually visible immediately.

Property-based testing is a sharp tool with a specific shape, and pretending it is general purpose is how teams abandon it.

It fits best where a law exists and inputs are cheap to generate: parsers and serialisers, data structures, numeric and date logic, state machines, anything with a slow reference implementation, and any function whose bugs would live in inputs nobody imagines.

It fits badly where the expected output is a matter of judgement rather than law. There is no property that captures "the recommendation is good" or "the copy reads well". It also fits badly where each run is expensive, since hundreds of executions per test rules out live network calls, and where the only property you can state is a restatement of the implementation, which proves nothing beyond that the code equals itself.

Gotcha: property tests complement example tests, they do not replace them. Keep a handful of concrete examples for documentation value: a reader learns more from assert slugify("Hello World") == "hello-world" than from any law, and the law is what protects you from the input you never pictured. Most mature suites carry both, deliberately.

One connection worth drawing: this course's neighbour on formal verification proves properties hold for all inputs by mathematical argument. Property-based testing tests the same kind of claim by sampling. It is enormously cheaper, gives no guarantee, and finds real bugs on a Tuesday afternoon, which is why it is the version most software actually gets.