All cursus
Computer Scienceadvanced
Floating Point: Why Your Numbers Are Wrong
Every floating point operation is correctly rounded to about sixteen digits, and results are still wrong in the first digit. This path explains how both are true: the guarantee is per-operation and relative, which means subtracting nearly equal numbers promotes error from insignificant to dominant without introducing any. It covers the representation, the cancellation patterns worth recognising, how to compare and test, and finally how to tell whether an inaccurate answer is your bug or the problem's nature.
0 of 4 lessons complete
Sign in to track progress and earn a certificate.

