Comparing Floats, and Testing Code That Uses Them

Exact equality on floats is not wrong in principle. Two doubles either are the same value or they are not, and == answers that correctly.

The problem is that it answers a question people are not asking. What is usually meant is "did these two computations produce the same mathematical quantity", and floating point does not preserve that.

Computing a quantity two algebraically equivalent ways gives two different roundings, and comparing them with == reports a difference that has no mathematical meaning. The previous lessons supply the reasons: rounding at every step, and cancellation that promotes it.

So comparison needs a notion of "close enough", which requires deciding how close and measured how. Neither has a universal answer, and both standard approaches fail in a specific, predictable way.

One case does deserve exact equality: comparing against a value that was assigned rather than computed. If a sentinel was set to 0.0 and never arithmetic'd, x == 0.0 is exactly right.

The first instinct is a fixed threshold: treat values as equal when the difference is below some epsilon.

def close_abs(a, b, tol=1e-9):
    return abs(a - b) < tol

This works over a narrow range of magnitudes and fails outside it, in both directions.

Too strict for large values. Near 1e9, the gap between consecutive doubles is around 1e-7. Two values that differ by a single representable step are already further apart than 1e-9, so close_abs reports them different even though nothing closer exists. At 1e16 no two distinct doubles are ever within 1e-9 of each other, so the function becomes exact equality with extra steps.

Too loose for small values. If the quantities of interest are around 1e-12, then 1e-9 is a thousand times larger than anything being measured, and the comparison returns true for values that differ by orders of magnitude.

The underlying error is treating precision as absolute when floating point provides it relatively. A fixed tolerance is only meaningful if the scale of the values is known and fixed, which for a general utility it is not.

The fix is to scale the tolerance to the values, which is what matches how floating point actually works.

def close_rel(a, b, rel=1e-9):
    return abs(a - b) <= rel * max(abs(a), abs(b))

This is correct across magnitudes, and it has one fatal case.

When the expected value is zero, the tolerance is rel * max(|a|, |b|), and if both are near zero the tolerance is near zero too. Relative comparison against zero degenerates into exact equality.

import math
math.isclose(1e-18, 0.0, rel_tol=1e-9)                   # False
math.isclose(1e-18, 0.0, rel_tol=1e-9, abs_tol=1e-12)    # True

A computed value of 1e-18 where zero was expected is, for almost any purpose, zero. Pure relative comparison calls it different, and no choice of rel_tol fixes it, because the failure is structural rather than a matter of the constant.

This is not an edge case in practice. Anything that should cancel to zero, a residual, a difference of equals, a balanced sum, lands here.

The last branch is what standard library functions implement, and the operator between the two conditions is or, not and.

Python's math.isclose returns true when the difference is within the relative tolerance or within the absolute one. The absolute term is the floor that rescues the near-zero case; the relative term does the work everywhere else.

That is also why abs_tol defaults to zero. A general-purpose default cannot know your scale, so the library refuses to guess, and comparing against zero without setting it will fail. This catches people constantly, and the fix is one argument.

The deeper point is that a tolerance is a claim about the problem, not about floating point. It says how much difference is insignificant for this quantity, and only someone who knows what the quantity represents can supply it.

Which is why copying 1e-9 from another codebase is the wrong move: it encodes that codebase's scale, not yours.

A third approach measures distance in units in the last place: how many representable doubles lie between two values.

import struct

def ulp_distance(a, b):
    def ordered(x):
        n = struct.unpack("<q", struct.pack("<d", x))[0]
        return n if n >= 0 else -(n & 0x7FFFFFFFFFFFFFFF) # map to a monotone int
    return abs(ordered(a) - ordered(b))

def close_ulp(a, b, max_ulps=4):
    return ulp_distance(a, b) <= max_ulps

The trick is that IEEE 754 was designed so that reinterpreting a positive double's bits as an integer gives a monotonically increasing sequence. Adjacent doubles differ by one as integers, so subtracting the reinterpreted values counts representable steps directly.

This automatically adapts to magnitude, since the spacing is baked into the representation. Saying "within 4 ULPs" means "within four representable values", which is meaningful at 1e-30 and at 1e30 alike.

It has the same near-zero weakness as relative comparison, and worse: across the sign boundary the ULP distance between two tiny opposite-signed values is large, so an absolute floor is still needed. ULP comparison is the right tool for testing a numerical routine against a reference, and the wrong one for general application logic.

NaN comparing unequal to itself is documented and still breaks code that never mentions NaN, because it violates assumptions made by sorting and by containers.

Sorting. Comparison sorts require a consistent ordering. With a NaN present, every comparison involving it is false, so the ordering is not consistent, and sort implementations may produce a scrambled result rather than merely misplacing the NaN. The damage is not confined to the NaN's position.

Containers. A NaN placed in a hash set cannot be found again by lookup, since equality fails against itself. Removing it by value fails too, and it remains in the container permanently.

Aggregates. A single NaN propagates through sum, mean, min and max, turning a whole computed result into NaN. That is arguably correct, and it means one bad input destroys an entire aggregate rather than one entry.

The practical defence is to reject NaN at the boundary. Validate on input, before values reach a container or an aggregate, since a NaN detected at the edge is a data quality problem while one detected downstream is a debugging exercise.

Detection is x != x, or the standard library's isnan.

Tests over floating point need a different discipline, because the usual instinct produces tests that are either flaky or vacuous.

Derive the tolerance, do not guess it. If a computation performs about n operations on values of magnitude m, the accumulated error is plausibly around n times machine epsilon times m. That gives a defensible starting point, and it will not be 1e-9 by coincidence.

Never assert exact equality on a computed value. Even if it passes today, a compiler flag, a different CPU, or a library update changes the last bits. Such a test fails for reasons unrelated to correctness.

Test properties rather than values. Many numerical results are hard to pin down exactly but satisfy checkable invariants: a solution substituted back into its equation gives a small residual, a probability distribution sums to one, a rotation preserves length. These are stable under rounding in a way exact outputs are not.

Test the hard inputs deliberately. Near-cancellation, values near overflow and underflow, zero, negative zero, infinities and NaN. Numerical code is usually correct on easy inputs and wrong on the boundary, and randomly generated test data almost never hits it.

Loosen tolerance for parallel results. Non-associativity means a parallel reduction legitimately differs from a serial one, so the test must accommodate that or it will be intermittently red.

The recurring theme is that no comparison is universally correct, because "close enough" is a statement about the problem and not about the arithmetic.

That is the honest summary. Floating point supplies a precision guarantee; what counts as an insignificant difference is supplied by whoever knows what the number means.

One question remains open, and it is the one that determines whether accuracy was ever achievable. Everything so far has concerned how a computation loses precision. The final lesson asks how much precision the problem itself permits, which turns out to be a separate quantity entirely, and one that no choice of algorithm can improve.