Discorez Member Licensed User Longtime User Aug 15, 2012 #1 I have a rounded float variable (in debugger I see it as 7.908), and I show result in two labels, but I receive different results: Code: Sub btCalc_Click Dim Rez As Float Rez = Round2(7.907999456789, 3) ' Rez debugger shows as "7.908" lbResult2.Text = Rez ' shows as "7.908" lbResult.Text = Rez & " mV" ' shows as "7.907999456789 mV" End Sub :BangHead: Why? test code is attached Upd. Code: lbResult.Text = NumberFormat(Rez, 0, 3) & " mV" It works correctly, but the situation which I described above is unclear for me... Attachments Float2String.zip Float2String.zip 6.7 KB · Views: 492 Last edited: Aug 15, 2012
I have a rounded float variable (in debugger I see it as 7.908), and I show result in two labels, but I receive different results: Code: Sub btCalc_Click Dim Rez As Float Rez = Round2(7.907999456789, 3) ' Rez debugger shows as "7.908" lbResult2.Text = Rez ' shows as "7.908" lbResult.Text = Rez & " mV" ' shows as "7.907999456789 mV" End Sub :BangHead: Why? test code is attached Upd. Code: lbResult.Text = NumberFormat(Rez, 0, 3) & " mV" It works correctly, but the situation which I described above is unclear for me...
klaus Expert Licensed User Longtime User Aug 15, 2012 #2 If you look at the help Round2 returns a Double. But you define a Float in the routine. This works: Code: Sub btCalc_Click Dim Rez As Double Rez = Round2(7.907999456789,3) lbResult.Text = Rez & " mV" lbResul2.Text = Rez End Sub But I agree it looks a bit strange. Best regards.
If you look at the help Round2 returns a Double. But you define a Float in the routine. This works: Code: Sub btCalc_Click Dim Rez As Double Rez = Round2(7.907999456789,3) lbResult.Text = Rez & " mV" lbResul2.Text = Rez End Sub But I agree it looks a bit strange. Best regards.
Erel B4X founder Staff member Licensed User Longtime User Aug 16, 2012 #3 The best way to format a number is with NumberFormat(2) keywords. What you are seeing is a result of a rounding error (because you used Float instead of Double). In the first case the number is first converted to double and in the second case it doesn't.
The best way to format a number is with NumberFormat(2) keywords. What you are seeing is a result of a rounding error (because you used Float instead of Double). In the first case the number is first converted to double and in the second case it doesn't.
Discorez Member Licensed User Longtime User Aug 16, 2012 #4 Erel, Claus, thanks. I understood this. But why debugger shows value for Rez variable as 7.908 instead 7.907999456789? It should be?
Erel, Claus, thanks. I understood this. But why debugger shows value for Rez variable as 7.908 instead 7.907999456789? It should be?
Erel B4X founder Staff member Licensed User Longtime User Aug 17, 2012 #5 This is related to the way that the number is converted to string by the debugger.