D doncx Active Member Licensed User Longtime User Sep 28, 2016 #1 When I convert a string with several decimal positions to a float, the value gets changed. There must be a simple reason, but it is escaping me. The code below logs two lines 12.34567 (the string value) 12.345669746398926 (the resulting float value) How do I accurately convert a decimal string to a float value? I've tried various formatting methods and continue to get this result. Code: Copy Dim str1 = "12.34567" As String Dim num1 = 0 As Float num1 = CNum(str1) Log(str1) Log(num1) Sub CNum(str2cnvrt) As Float Return str2cnvrt End Sub
When I convert a string with several decimal positions to a float, the value gets changed. There must be a simple reason, but it is escaping me. The code below logs two lines 12.34567 (the string value) 12.345669746398926 (the resulting float value) How do I accurately convert a decimal string to a float value? I've tried various formatting methods and continue to get this result. Code: Copy Dim str1 = "12.34567" As String Dim num1 = 0 As Float num1 = CNum(str1) Log(str1) Log(num1) Sub CNum(str2cnvrt) As Float Return str2cnvrt End Sub
M Mahares Expert Licensed User Longtime User Sep 29, 2016 #2 Code: Copy Dim str1 = "12.34567" As String 'You can use it as is since it is numeric: Log(str1*1.5) 'displays 18.518505 'or you can use this: Dim num1 As Double=str1 Log(num1) 'displays 12.34567 Upvote 0
Code: Copy Dim str1 = "12.34567" As String 'You can use it as is since it is numeric: Log(str1*1.5) 'displays 18.518505 'or you can use this: Dim num1 As Double=str1 Log(num1) 'displays 12.34567
Erel B4X founder Staff member Licensed User Longtime User Sep 29, 2016 #3 A binary number (with limited length) cannot accurately represent 12.34567. You can just ignore this inaccuracy. Always use NumberFormat (or the shortcut) to convert the number to string: Code: Copy Dim num1 As Float = 12.34567 Log($"Num value: $1.5{num1}"$) 'shows at most 5 fraction digits Upvote 0
A binary number (with limited length) cannot accurately represent 12.34567. You can just ignore this inaccuracy. Always use NumberFormat (or the shortcut) to convert the number to string: Code: Copy Dim num1 As Float = 12.34567 Log($"Num value: $1.5{num1}"$) 'shows at most 5 fraction digits