toby Well-Known Member Licensed User Longtime User May 5, 2022 #1 I want to store latitude and longitude data as Float data type values. When I moved mouse over flt1 variable, it showed 123.45679 while the Log() yielded another slightly different value. Could someone tell me what's wrong, please? Code: Copy Dim flt1 As Float=123.4567890123456 'hoveover value: 123.45679 (rounded with 5 decimal digits) Log(flt1) 'output: 123.456787109375 TIA
I want to store latitude and longitude data as Float data type values. When I moved mouse over flt1 variable, it showed 123.45679 while the Log() yielded another slightly different value. Could someone tell me what's wrong, please? Code: Copy Dim flt1 As Float=123.4567890123456 'hoveover value: 123.45679 (rounded with 5 decimal digits) Log(flt1) 'output: 123.456787109375 TIA
DonManfred Expert Licensed User Longtime User May 5, 2022 #2 Use Numberformat to format a number with any amount of chars you need. Upvote 0
klaus Expert Licensed User Longtime User May 5, 2022 #4 toby said: Dim flt1 As Float=123.4567890123456 Click to expand... The number of significant digits is bigger than what a Float variable can hold. Therefore use the advice in the two previous posts. Upvote 0
toby said: Dim flt1 As Float=123.4567890123456 Click to expand... The number of significant digits is bigger than what a Float variable can hold. Therefore use the advice in the two previous posts.
E emexes Expert Licensed User Longtime User May 6, 2022 #5 A four-byte single-precision float has a 24 bit mantissa = accurate to within 0.85 metres worst case (at equator, opposite prime meridian ie Greenwich London) An eight-byte double-precision float has an extra 29 bits of mantissa = accurate to within 1.6 nanometres worst case (in Australia aka within a bee's dick) Upvote 0
A four-byte single-precision float has a 24 bit mantissa = accurate to within 0.85 metres worst case (at equator, opposite prime meridian ie Greenwich London) An eight-byte double-precision float has an extra 29 bits of mantissa = accurate to within 1.6 nanometres worst case (in Australia aka within a bee's dick)
