Why does the computer say 0.1 + 0.2 isn't 0.3?

♣ Newbie

Try writing a third with decimals: 0.3333… However many digits you add, you never get there; something is always missing. There’s nothing odd about the number. It’s just that in decimal, which counts in tenths, hundredths and thousandths, a third has no end.

The computer doesn’t count in tenths: it counts in halves. Halves, quarters, eighths, sixteenths… Each piece is half the one before. And with pieces like that, a tenth has no end either: you get very close, but a sliver is always missing.

Since the computer only has room for a few dozen pieces, it cuts off there and keeps the closest thing. Its 0.1 isn’t 0.1: it’s a hair more. Its 0.2 is also a hair over, and its 0.3 falls a hair short. When you add them, those hairs don’t match up, and out comes 0.30000000000000004.

♦ Intermediate

The computer stores numbers in binary, using only zeros and ones. In decimal, each digit after the point is worth a tenth of the one before: tenths, hundredths… In binary, each digit is worth half the one before: ½, ¼, ⅛, 1/16…

To write 0.1 in binary you check which pieces fit: ½ no, ¼ no, ⅛ no, 1/16 yes… You get 0.0001100110011…, and the group 0011 repeats forever. With nine digits you reach 0.099609375; with more you get closer, but you never finish. It’s the same thing that happens to 1/3 in decimal.

An ordinary computer keeps 53 binary digits and rounds to the nearest number it can write with them. Its 0.1 is exactly 0.1000000000000000055511151231257827021181583404541015625, a little more than 0.1 (Python tutorial).

The same happens with 0.2 and 0.3, each with its own small error. The sum of the first two doesn’t land on the number the computer uses for 0.3, but on the next one: 0.30000000000000004.

♠ Technical

A fraction p/q in lowest terms has a finite expansion in base 2 only if q is a power of 2. 1/10 has a 5 in the denominator, so it repeats: 0.1 = 0.0(0011)₂ = Σₖ≥₁ (2⁻⁴ᵏ + 2⁻⁴ᵏ⁻¹). The geometric series confirms it: (3/32) / (15/16) = 1/10.

With p = 53, fl(0.1) is the nearest fraction J/2ᴺ with a 53-bit J: J = 7205759403792794 and N = 56, which reduces to 3602879701896397 / 2⁵⁵ (Python tutorial). Since 10 × 3602879701896397 = 36028797018963970 and 2⁵⁵ = 36028797018963968:

fl(0.1) − 0.1 = 2 / (10 · 2⁵⁵) = 1 / (5 · 2⁵⁵) ≈ 5.55×10⁻¹⁸

It was rounded up. In 0.1 ⊕ 0.2, the exact sum fl(0.1) + fl(0.2) lands exactly halfway between two 53-bit numbers. The ties-to-even rule sends it to the upper one, which sits one ulp, 2⁻⁵⁴ ≈ 5.55×10⁻¹⁷, above fl(0.3). That’s why 0.1 + 0.2 == 0.3 is false in Python, in JavaScript and in any language that uses this format (own calculation with exact arithmetic; in JavaScript, 0.1 + 0.2 - 0.3 gives 2⁻⁵⁴).

♣ Newbie

In the huge numbers lesson you saw scientific notation: 7×10²⁷ is a 7 followed by 27 zeros. It lets you write huge and tiny numbers with few digits.

The computer does the same, but in binary and always with the same number of digits: about 16 decimal digits. It’s called floating point because the point “floats”: it moves depending on how big or small the number is, while the digits are always the same.

Imagine a ruler with a fixed number of marks. Near zero, the marks are very close together and you can measure small things in great detail. Far from zero, they’re further and further apart. If what you measure falls between two marks, you write down the nearest one: that’s rounding.

With 16 digits, the ruler is very fine: 0.1 is off by less than one part in ten quadrillion. That’s why you almost never notice.

♦ Intermediate

A floating-point number is stored in three parts, like scientific notation but in base 2:

• the sign, + or −;
• the exponent, which says what power of 2 it’s multiplied by;
• the mantissa, which holds the digits.

The usual format is called binary64 (or “double precision”): 64 bits, 1 for the sign, 11 for the exponent and 52 for the mantissa (MDN; NumPy). The mantissa always starts with a 1 that doesn’t need storing, so it has 53 binary digits, about 16 decimal ones. It’s the format of JavaScript numbers, Python floats and Excel. For example, 0.1 = + 1.6 × 2⁻⁴.

Since there are always the same number of digits, the gap between two neighbouring numbers grows with size. Just above 1 it’s 2.2×10⁻¹⁶. Near 10¹⁶ it’s 2, which is why 10¹⁶ + 1 gives 10¹⁶. For the same reason, whole numbers are only exact up to about 9×10¹⁵ (MDN).

Some bit patterns are reserved for zero, infinity and meaningless results, such as the square root of −1 (Microsoft, on Excel).

When the result of an operation falls between two numbers that can be stored, it’s rounded to the nearest one. Each rounding is off by at most half a gap: less than one part in 9×10¹⁵.

♠ Technical

A normal binary64 is x = (−1)ˢ · (1.f)₂ · 2^(E − 1023), with 1 ≤ E ≤ 2046. The values E = 0 and E = 2047 are reserved for zero, subnormals (without the implicit 1), ±∞ and NaN. The parameters p = 53, emin = −1022 and emax = 1023 are those of Goldberg’s table D-1 (1991); the current revision of the standard is IEEE 754-2019.

Between 2ᵉ and 2ᵉ⁺¹ there are 2⁵² equally spaced numbers, ulp(x) = 2^(⌊log₂|x|⌋ − 52) apart. With round to nearest, fl(x) = x(1 + δ) with |δ| ≤ u = 2⁻⁵³, and every basic operation obeys the same bound: x ⊕ y = (x + y)(1 + δ) (Goldberg).

Watch the names. MDN, Python and PyTorch call ε = 2⁻⁵² = ulp(1), the distance from 1 to the next number, epsilon. Goldberg calls the bound on the relative error machine epsilon, which is half that: u = 2⁻⁵³. Integers are exact up to 2⁵³ = 9,007,199,254,740,992; from there on, ulp ≥ 2 (MDN, Number.MAX_SAFE_INTEGER = 2⁵³ − 1).

♣ Newbie

Imagine a slightly odd truck scale: it doesn’t weigh everything again, it adds whatever you put on to the number it already shows and rounds to the kilo. There’s a 10,000-kilo truck on it, and you have a thousand feathers that together weigh one kilo. You add one feather: the scale stays at 10,000. Another: 10,000. All thousand, one by one: 10,000, because each time it rounds to the kilo and one feather alone doesn’t make it. Now put the thousand feathers in a bag and add it all at once: 10,001.

The same happens to the computer. Every sum rounds, and small errors pile up or get lost depending on the order you add in. If you add 0.1 ten times in a row, you don’t get 1: you get 0.9999999999999999.

Most of the time it doesn’t matter, because the error is tiny. But if you compare the result with 1 to decide something, the answer is “they’re not equal”. That’s why, with numbers that have decimals, programmers don’t ask “are they equal?” but “are they very close?”.

♦ Intermediate

Every operation rounds, and the errors pile up. Adding 0.1 in a loop, in JavaScript:

On top of that, the order changes the result. Adding a small number to a very large one may change nothing (10¹⁶ + 1 = 10¹⁶), so (a + b) + c doesn’t always give the same as a + (b + c) (Goldberg, 1991).

Three common remedies:

• Compare with a tolerance instead of with ==. In Python, math.isclose(0.1 + 0.1 + 0.1, 0.3) gives True (Python tutorial).
• Add carefully. math.fsum keeps track of the digits that get lost and rounds only once. And since Python 3.12, sum() compensates for the errors internally: sum([0.1] * 10) == 1.0 gives True on almost every install, while the hand-written loop gives 0.9999999999999999.
• Round when displaying, not when calculating: two decimals for a price.

♠ Technical

In recursive summation, each term xⱼ comes in with a factor (1 + δⱼ), with |δⱼ| < (n − j)u: the error grows with n (Goldberg, 1991, who calls u = 2⁻⁵³ epsilon, as seen in the previous block). Kahan’s compensated summation stores in c what each ⊕ lost and gives it back in the next step. Its error stays at (2u + O(nu²)) Σ|xⱼ|, almost independent of n (Goldberg, theorem 8; Kahan, CACM 8, 40, 1965).

Without associativity, order matters: (10³⁰ ⊕ −10³⁰) ⊕ 1 = 1, but 10³⁰ ⊕ (−10³⁰ ⊕ 1) = 0 (Goldberg). One consequence: splitting a long sum across several processors changes the order, and the result may vary in the last digits.

To compare, math.isclose(a, b) checks |a − b| ≤ max(rel_tol · max(|a|, |b|), abs_tol), with rel_tol = 10⁻⁹ by default: about 9 digits of agreement (Python documentation). sum() has used Neumaier summation since Python 3.12 (gh-100425), and math.fsum rounds only once at the end.

♣ Newbie

Not every number needs 16 digits. A neural network, like the ones behind machine translation, does a huge number of multiplications with millions of numbers. If each number takes up half the space, twice as many fit in memory and the sums can go faster. In exchange, each number is coarser, like a photo with fewer pixels.

With 16 bits, the computer only has about 3 digits. If you add 0.1 over and over, when you reach 256 the sum gets stuck: 0.1 is less than half the gap between two marks on the ruler, and it rounds to nothing. You’ll see it in the widget below.

With money it’s the other way round: we don’t want a single cent of error. For money the advice is not to use these binary numbers with decimals, but to count in cents, with whole numbers, which are stored exactly, or to calculate in decimal, the way people do.

♦ Intermediate

Besides binary64 there are smaller formats:

• float32, 32 bits (1 + 8 + 23): about 7 digits.
• float16, 16 bits (1 + 5 + 10): about 3 digits, and nothing above 65,504.
• bfloat16, 16 bits (1 + 8 + 7): only about 2 digits, but the same range as float32. It was designed at Google for machine learning (Google Cloud blog, 2019).

Neural networks use these small formats a lot: they take half the memory and can go faster. The trick is called mixed precision: do the multiplications in 16 bits and keep a copy of the network’s weights in 32. In float16, a small change can get lost if the number it’s added to is 2048 times larger or more (Micikevicius et al., 2018). It’s the truck scale again.

For money, you do the opposite: decimal arithmetic, which counts in tenths and hundredths like we do, or whole numbers of cents. In decimal, 0.1 + 0.1 + 0.1 − 0.3 gives exactly 0; in binary, 5.55×10⁻¹⁷ (documentation of Python’s decimal module, designed for accounting).

♠ Technical

With p bits of precision and largest exponent emax, ε = 2¹⁻ᵖ and the largest finite value is (2 − 2¹⁻ᵖ) · 2^emax. The bit layouts come from NumPy, PyTorch and Google; the values in the table are my own calculation. For binary16, (2 − 2⁻¹⁰) · 2¹⁵ = 65,504, the same value Micikevicius et al. (2018) give.

w ⊕ Δw = w as soon as |Δw| < ulp(w)/2, and in FP16 that can happen from |w| ≥ 2¹¹|Δw| on (Micikevicius et al.). Their mixed-precision recipe keeps master weights in FP32, multiplies in FP16 and scales the loss so that small gradients don’t drop to zero: in FP16 there is no positive number below 2⁻²⁴, and for their object detector (SSD) a factor of 8 was enough. On Google’s TPUs, each matrix multiplication multiplies in bfloat16 and accumulates in FP32 (Google Cloud, 2019). The widget shows it with 0.1: in float16, from 256 on the ulp is 0.25 and 0.1 < 0.125.

Python’s decimal module is floating point in base 10: 0.1 is stored exactly and 0.1 + 0.1 + 0.1 − 0.3 gives 0. For money, whole numbers of cents also work, exact in binary64 up to 2⁵³.