Android Question Problem with Float data type

toby

Well-Known Member
Licensed User
Longtime User
I want to store latitude and longitude data as Float data type values. When I moved mouse over flt1 variable, it showed 123.45679 while the Log() yielded another slightly different value.

Could someone tell me what's wrong, please?

[CODE lang="b4x" title="Problem with float data type" highlight="2"]
Dim flt1 As Float=123.4567890123456
'hoveover value: 123.45679 (rounded with 5 decimal digits)
Log(flt1) 'output: 123.456787109375 [/CODE]

TIA
 
Use Numberformat to format a number with any amount of chars you need.
 
Upvote 0
A four-byte single-precision float has a 24 bit mantissa = accurate to within 0.85 metres worst case (at equator, opposite prime meridian ie Greenwich London)

An eight-byte double-precision float has an extra 29 bits of mantissa = accurate to within 1.6 nanometres worst case (in Australia aka within a bee's dick)
 
Upvote 0
Back
Top