math: increase precision of math.SmallestNonzeroFloat64
The original value was rounded too early, which lead to the surprising behavior that float64(math.SmallestNonzeroFloat64 / 2) wasn't 0. That is, the exact compile-time computation of math.SmallestNonzeroFloat64 / 2 resulted in a value that was rounded up when converting to float64. To address this, added 3 more digits to the mantissa, ending in a 0. While at it, also slightly increased the precision of MaxFloat64 to end in a 0. Computed exact values via https://play.golang.org/p/yt4KTpIx_wP. Added a test to verify expected behavior. In contrast to the other (irrational) constants, expanding these extreme values to more digits is unlikely to be important as they are not going to appear in numeric computations except for tests verifying their correctness (as is the case here). Re-enabled a disabled test in go/types and types2. Updates #44057. Fixes #44058. Change-Id: I8f363155e02331354e929beabe993c8d8de75646 Reviewed-on: https://go-review.googlesource.com/c/go/+/315170 Trust: Robert Griesemer <gri@golang.org> Reviewed-by:Ian Lance Taylor <iant@golang.org>
Loading
Please sign in to comment