AnyLearn
All cursus
Computer Scienceadvanced

Floating Point: Why Your Numbers Are Wrong

Every floating point operation is correctly rounded to about sixteen digits, and results are still wrong in the first digit. This path explains how both are true: the guarantee is per-operation and relative, which means subtracting nearly equal numbers promotes error from insignificant to dominant without introducing any. It covers the representation, the cancellation patterns worth recognising, how to compare and test, and finally how to tell whether an inaccurate answer is your bug or the problem's nature.

0 of 4 lessons complete
Sign in to track progress and earn a certificate.

Lessons, in order

  1. 1
    Computer Science
    How a Float Is Stored, and What That Rules Out
    Start
  2. 2
    Computer Science
    Cancellation: Where the Digits Actually Go
    Start
  3. 3
    Computer Science
    Comparing Floats, and Testing Code That Uses Them
    Start
  4. 4
    Computer Science
    Conditioning and Stability: Whose Fault Is the Error
    Start