GuidesWhy Floating-Point Converters Fail (and How Decimal Math Fixes It)
Why Floating-Point Converters Fail (and How Decimal Math Fixes It)
Understand why binary floating-point creates weird conversion residuals, when those errors matter, and how decimal or rational arithmetic keeps unit converters trustworthy.
The classic 0.1 + 0.2 problem meets units
Most CPUs store non-integer numbers in binary floating-point (IEEE 754). Fractions that look simple in decimal—0.1, 0.001, 2.54—often cannot be represented exactly in binary. The stored value is the nearest representable approximation, and arithmetic on approximations accumulates tiny residuals.
Unit conversion multiplies and divides those values repeatedly. Convert inches to millimetres and back, or litres to cubic metres and back, and you may see 2.9999999997 instead of 3. The physical conversion factor was exact; the computer representation was not.
For many displays the difference is invisible after rounding. For finance-adjacent dosing, CNC toolpaths, legal land area, or automated test assertions that expect exact equality, those residuals become real bugs.
Which conversion factors are exact on paper
The international inch is exactly 25.4 mm. The international foot is exactly 0.3048 m. The litre is exactly 0.001 m³. A cubic centimetre is exactly 1 mL in the usual laboratory sense (1 cm³ = 10⁻⁶ m³). These are definitions, not measurements.
If software stores 25.4 as a binary float, it may already be slightly wrong before any conversion runs. Multiplying by another inexact factor compounds the error. Round-trip tests that assert strict equality then fail even when the algorithm is conceptually correct.
Temperature adds offsets: Celsius ↔ Kelvin uses 273.15. That constant is also a decimal value that binary floats only approximate. Affine conversions are especially sensitive because addition of an inexact offset shifts every result.
Where floating-point errors show up in practice
UI converters often hide the issue by rounding to a fixed number of decimals. Users see clean results and never notice the internal noise. Problems appear when another system consumes the raw float: a CAD kernel, a CSV export, or a unit test comparing strings of full precision.
Chained conversions amplify trouble. Going mile → yard → foot → inch → millimetre → metre with binary floats at each step is worse than a single mile → metre conversion with a high-precision factor. Each hop re-rounds the intermediate value.
Squared and cubed quantities make errors worse. Area and volume multiply length scales, so a tiny relative error in length becomes two or three times larger in relative terms for area or volume when you expand (1+ε)ⁿ.
Decimal, rational, and fixed-point alternatives
Decimal libraries (for example decimal.js, Big.js, or language-native decimals) store numbers in base 10. Exact decimal factors like 0.3048 and 0.001 stay exact within the chosen precision. Unit converters that care about correctness often convert through a decimal base unit.
Rational arithmetic stores factors as fractions of integers. If 1 inch = 127/5000 metres exactly (because 25.4 mm = 127/5000 m), multiplication can cancel exactly. Rationals are ideal for pure scale conversions; offsets still need careful handling for temperature.
Fixed-point math scales everything to an integer count of the smallest tick—nanometres, microlitres, or thousandths of an inch. Integer arithmetic is exact until you overflow. It works well for domains with a clear quantum of measurement.
Practical rules for trustworthy converters
Store exact legal factors as decimals or rationals, not as hastily typed binary floats. Convert input → base → output in as few steps as possible. Round only when formatting for humans, and document the rounding mode.
Never test converters with strict equality on binary floats. Compare within an absolute or relative tolerance, or compare decimal string forms at a stated precision. Prefer golden tests that use known exact pairs such as 1 in = 25.4 mm.
If you build a calculator UI, show enough precision to be useful but not so many digits that floating noise looks like signal. A length tool showing twelve decimals after converting through floats is advertising its representation error.
When binary floats are still fine
Rough comparisons, games, data visualisation, and many scientific simulations tolerate float error because the model uncertainty dwarfs representation error. The key is knowing your tolerance. If a millimetre matters, floats need discipline or replacement.
Hardware acceleration and GPU pipelines often force floats. In those cases, keep authoritative conversion in decimal on the CPU or server, and treat GPU floats as a display or intermediate approximation.
Good engineering is matching numeric representation to the decision you will make with the number. Unit conversion for manufacturing, medicine, and legal metrology deserves decimal or rational math; a weather widget converting city temperatures does not. When in doubt, choose the stricter representation—the performance cost of decimals is usually smaller than the cost of a silent round-trip bug.
Related converters
Frequently asked questions
Why does converting back and forth change my number?
Binary floating-point cannot represent many decimal factors exactly. Each multiply or divide adds a tiny rounding error. Round-trip conversion through floats often fails to restore the original bits even when the conversion factors are conceptually exact.
Is double precision enough for unit conversion?
Double precision (float64) is enough for many everyday conversions after display rounding, but it is not a substitute for exact decimal factors when you need bit-stable round trips, legal metrology, or strict automated tests. Prefer decimals or rationals for authoritative converters.
How should I compare converter results in tests?
Compare at a declared precision, use decimal string equality, or assert absolute and relative tolerances. Avoid expecting exact binary equality after several arithmetic steps.