A Alex_197 Well-Known Member Licensed User Longtime User Jan 3, 2023 #1 Hi all I have a table in my database with fields datatype FLOAT. I'm reading values from these fields like this B4X: Copy Dim rs as ResultSet Mytxt.Text=rs.GetDouble("BP1") MyTxt is B4XView If the value in this field is lower than 90 MyTxt.text shows whatever we have in this field. For example 45.3 looks like 45.3 but if it's 96.6 it looks like 96.599999999 Why it's like that? Attachments Screenshot 2023-01-03 at 2.06.48 AM.jpg 66.2 KB · Views: 433
Hi all I have a table in my database with fields datatype FLOAT. I'm reading values from these fields like this B4X: Copy Dim rs as ResultSet Mytxt.Text=rs.GetDouble("BP1") MyTxt is B4XView If the value in this field is lower than 90 MyTxt.text shows whatever we have in this field. For example 45.3 looks like 45.3 but if it's 96.6 it looks like 96.599999999 Why it's like that?
Erel B4X founder Staff member Licensed User Longtime User Jan 4, 2023 #2 Never display non-integer numbers without NumberFormat. Upvote 0
E emexes Expert Licensed User Longtime User Jan 4, 2023 #3 Alex_197 said: If the value in this field is lower than 90 MyTxt.text shows whatever we have in this field. For example 45.3 looks like 45.3 but if it's 96.6 it looks like 96.599999999 Why it's like that? Click to expand... It happens because the 53 bit mantissa (precision/resolution) of (double) FLOAT is log10(2^53) decimal digits ie 15.95 significant digits, and so for randomish values shown to 16 significant digits, some of them are going to be out by +/- 1 least-significant digit column value. If default conversion to decimal was capped at a (safe) 15 significant digits rather than the (optimistic) 16 significant digits, then the decimal representation problem would go away... or, at least, not be visible... although you could still occasionally have a similar issue with such errors accumulating through multiple operations. @Erel's advice is also in that direction - use NumberFormat to limit the significant digits displayed to 15 or fewer (for Double) or to 6 or fewer (for Single Float). Upvote 0
Alex_197 said: If the value in this field is lower than 90 MyTxt.text shows whatever we have in this field. For example 45.3 looks like 45.3 but if it's 96.6 it looks like 96.599999999 Why it's like that? Click to expand... It happens because the 53 bit mantissa (precision/resolution) of (double) FLOAT is log10(2^53) decimal digits ie 15.95 significant digits, and so for randomish values shown to 16 significant digits, some of them are going to be out by +/- 1 least-significant digit column value. If default conversion to decimal was capped at a (safe) 15 significant digits rather than the (optimistic) 16 significant digits, then the decimal representation problem would go away... or, at least, not be visible... although you could still occasionally have a similar issue with such errors accumulating through multiple operations. @Erel's advice is also in that direction - use NumberFormat to limit the significant digits displayed to 15 or fewer (for Double) or to 6 or fewer (for Single Float).
