Android Question Best way to write a 2 dimensional array of doubles to file

Kevin L. Johnson

Member
Licensed User
Longtime User
I am looking for the best way to persist (write) a 2 dimensional array of doubles to disk (file). I have looked at the Random Access File .WriteObject(), and .WriteB4XObject() and the b4xSerializator.ConvertObjectToBytes and then use the BytesToFile to persist them to disk but, at this point, I am not sure of which direction to proceed.

The purpose is to generate a protected look-up table, and therefore, I would prefer, the file be binary and read only so as not to be disturbed or corrupted.

Obviously at some later point, the persisted two dimensional array of doubles would then need to be read back from the file to be used as a look-up table.

Any help would be much appreciated.
Thanks
KLJ
 

emexes

Expert
Licensed User
persist (write) a 2 dimensional array of doubles to disk (file)
How big is the array? For smaller arrays, the code and effort and complexity risk of squeezing things down might not be worth the space saved.

Maybe you could write the array to a CSV string, and then compress the string to a file.

a protected look-up table ... not to be disturbed or corrupted
just checking my understanding here: so not encryption to keep hidden? (phew! if correct)
 
Upvote 0

MarkusR

Well-Known Member
Licensed User
Longtime User
maybe you can store data in a type & list
something like array(a,b) is similar a list of type with x,y fields.
B4X:
type mydata(a as double,b as double)
    Dim l As List
   l.Initialize
  
   Dim data As mydata
   data.a = 1
   data.b = 2
  
   l.Add(data)
 
Upvote 0

MarkusR

Well-Known Member
Licensed User
Longtime User
b4j example with B4XSerializator
B4X:
#Region Project Attributes
    #MainFormWidth: 600
    #MainFormHeight: 600
#End Region

Sub Process_Globals
    Private fx As JFX
    Private MainForm As Form
    
    Type mydata(a As Double,b As Double)

End Sub

Sub AppStart (Form1 As Form, Args() As String)
    MainForm = Form1
    'MainForm.RootPane.LoadLayout("Layout1") 'Load the layout file.
    MainForm.Show

    Test
        
End Sub

'Return true to allow the default exceptions handler to handle the uncaught exception.
Sub Application_Error (Error As Exception, StackTrace As String) As Boolean
    Return True
End Sub

Sub Test

    Dim l As List
    l.Initialize
 
    Dim data As mydata
    data.a = 1
    data.b = 2
 
    l.Add(data)
    
    Dim b As B4XSerializator 'jRandomAccessFile Class / Library
    Dim bytes() As Byte = b.ConvertObjectToBytes(l)

    Dim out As OutputStream = File.OpenOutput(File.DirApp,"Test.dat",False)
    
    out.WriteBytes(bytes,0,bytes.Length)
    
    out.Flush
    out.Close
    
End Sub
 
Upvote 0

Kevin L. Johnson

Member
Licensed User
Longtime User
How big is the array? For smaller arrays, the code and effort and complexity risk of squeezing things down might not be worth the space saved.

Maybe you could write the array to a CSV string, and then compress the string to a file.


just checking my understanding here: so not encryption to keep hidden? (phew! if correct)

They array is dimensioned as a 19 by 7 array of doubles (Not that large at all). I was thinking of a CSV string or file too, and that may be the way I go, If I cannot get the RAF.WriteObject method working.

Thanks for the suggestion emexes.
 
Upvote 0

Kevin L. Johnson

Member
Licensed User
Longtime User
maybe you can store data in a type & list
something like array(a,b) is similar a list of type with x,y fields.
B4X:
type mydata(a as double,b as double)
    Dim l As List
   l.Initialize
 
   Dim data As mydata
   data.a = 1
   data.b = 2
 
   l.Add(data)

Interesting ...

Thanks for the suggestion MarkusR
KLJ
 
Upvote 0

Kevin L. Johnson

Member
Licensed User
Longtime User
Here is the code, I have thus far.
In the GenerateLUT subroutine I...
1.) Type the Data in by Rows
2.) Loop through the Row / Pedestal Arrays
3.) Place the elements Row, Cols into a two dimensional array

The missing step is to persist the 2 dimensional array to a file ...

B4X:
Public Sub GenerateLUT()
 
    Dim intRows = 19 As Int
    Dim intCols = 7 As Int
 
    Dim LUT(intRows, intCols) As Double
    
    Dim FrontPed00(7) As Double
    Dim FrontPed01(7) As Double
    Dim FrontPed02(7) As Double
    Dim FrontPed03(7) As Double
    Dim FrontPed04(7) As Double
    Dim FrontPed05(7) As Double
    Dim FrontPed06(7) As Double
    Dim FrontPed07(7) As Double
    Dim FrontPed08(7) As Double
    Dim FrontPed09(7) As Double
    Dim FrontPed10(7) As Double
    Dim FrontPed11(7) As Double
    Dim FrontPed12(7) As Double
    Dim FrontPed13(7) As Double
    Dim FrontPed14(7) As Double
    Dim FrontPed15(7) As Double
    Dim FrontPed16(7) As Double
    Dim FrontPed17(7) As Double
    Dim FrontPed18(7) As Double
 
    'Generate DATA
 
    FrontPed00 = Array As Double ( 0.0,    1.0,    2.0,    3.0,    4.0,    5.0,    6.0)
    FrontPed01 = Array As Double ( 1.0, 152.92, 152.92, 152.92, 152.92, 152.92, 152.92)
    FrontPed02 = Array As Double ( 2.0, 152.08, 152.08, 152.17, 152.17, 152.17, 152.17)
    FrontPed03 = Array As Double ( 3.0, 150.25, 150.25, 150.33, 150.33, 150.42, 150.42)
    FrontPed04 = Array As Double ( 4.0, 148.58, 148.58, 148.67, 148.75, 148.83, 148.92)
    FrontPed05 = Array As Double ( 5.0, 147.00, 147.08, 147.17, 147.25, 147.42, 147.50)
    FrontPed06 = Array As Double ( 6.0, 145.58, 145.75, 145.92, 146.00, 146.08, 146.25)
    FrontPed07 = Array As Double ( 7.0, 144.33, 144.50, 144.67, 144.83, 145.00, 145.08)
    FrontPed08 = Array As Double ( 8.0, 143.17, 143.33, 143.58, 143.75, 144.00, 144.17)
    FrontPed09 = Array As Double ( 9.0, 142.17, 142.42, 142.58, 142.83, 143.08, 143.25)
    FrontPed10 = Array As Double (10.0, 141.33, 141.58, 141.83, 142.08, 142.25, 142.50)
    FrontPed11 = Array As Double (11.0, 140.58, 140.83, 141.17, 141.42, 141.67, 141.92)
    FrontPed12 = Array As Double (12.0, 140.00, 140.25, 140.58, 140.92, 141.17, 141.50)
    FrontPed13 = Array As Double (13.0, 139.50, 139.83, 140.17, 140.50, 140.83, 141.17)
    FrontPed14 = Array As Double (14.0, 139.17, 139.50, 139.92, 140.25, 140.58, 141.00)
    FrontPed15 = Array As Double (15.0, 138.92, 139.33, 139.75, 140.08, 140.50, 140.83)
    FrontPed16 = Array As Double (16.0, 138.83, 139.17, 139.58, 140.00, 140.42, 140.83)
    FrontPed17 = Array As Double (17.0, 138.75, 139.17, 139.58, 140.00, 140.42, 140.92)
    FrontPed18 = Array As Double (18.0, 138.75, 139.17, 139.58, 140.08, 140.50, 140.92)
 
   ' Populate 2 dimensional Array

    Dim intRow As Int
    Dim intCol As Int
    Dim PedArray() As Double
    
    For intRow = 0 To 18
    
        Dim PedArray() As Double
    
        Select intRow
            Case 0:  PedArray = FrontPed00
            Case 1:     PedArray = FrontPed01
            Case 2:     PedArray = FrontPed02
            Case 3:     PedArray = FrontPed03
            Case 4:     PedArray = FrontPed04
            Case 5:     PedArray = FrontPed05
            Case 6:     PedArray = FrontPed06
            Case 7:     PedArray = FrontPed07
            Case 8:     PedArray = FrontPed08
            Case 9:     PedArray = FrontPed09
            Case 10: PedArray = FrontPed10
            Case 11: PedArray = FrontPed11
            Case 12: PedArray = FrontPed12
            Case 13: PedArray = FrontPed13
            Case 14: PedArray = FrontPed14
            Case 15: PedArray = FrontPed15
            Case 16: PedArray = FrontPed16
            Case 17: PedArray = FrontPed17
            Case 18: PedArray = FrontPed18
        End Select
    
        Dim l_strData As String
    
    
        For intCol = 0 To 6
            LUT(intRow, intCol) = PedArray(intCol)
        Next 'intCol
    
    Next 'intRow
 
    'Display the LUT
 
    For intRow = 0 To 18
        Dim l_strData
        l_strData = ""
        For intCol =  0 To 6
            If intCol > 0 Then
                l_strData = l_strData & ", " & TAB
            End If
        
            If intRow = 0 Then
                'Format the whole Top row as integers
                l_strData = l_strData & NumberFormat2(LUT(intRow, intCol), 1, 1, 1, False)
            Else if intCol = 0 Then
                'Format the First Col as an integer
                l_strData = l_strData & NumberFormat2(LUT(intRow, intCol), 1, 0, 0, False)
            Else
                'Format everything else as a double
                l_strData = l_strData & NumberFormat2(LUT(intRow, intCol), 3, 2, 2, False)
            End If
        Next
    
        Log(l_strData)    
    
    Next

      ' Next Step is to persist the 2 dimensional array to file ...
 
Upvote 0

Kevin L. Johnson

Member
Licensed User
Longtime User
B4XSerializator and WriteB4XObject do not support serializing arrays with the exception of arrays of bytes and arrays of objects.

You can use RAF.WriteObject. It is a good option.

Erel ... It is a good option, AND it worked! There are no words ... I had tried to use this technique using RAF.WriteObject and then RAF.ReadObject method once before, but my code returned something about the method not supporting the data type. I did upgrade my B4J afterwords though. Perhaps the upgrade fixed something?

I tell everyone about B4X ...
This product and it's community members are incredible!
Thanks to All ...
 
Upvote 0

wonder

Expert
Licensed User
Longtime User
Are you interested in storing it in JSON format?
 
Upvote 0

emexes

Expert
Licensed User
Is that data fixed and unchanging?

It looks like a lookup table for something like eg battery discharge rate.

Is there a formula that generated that data? It'd be a heck of a lot easier to use that, rather than interpolating from a table.
 
Upvote 0

emexes

Expert
Licensed User
What I was heading towards is something like this, a demo of implementing column 1 of your data as a formula.

You might be concerned that the table data and the calculated data do not precisely match; my thought is that perhaps the table data was derived by experimental measurement, which will have some random error, and which the formula averages out (ironically thus probably making it more accurate).
B4X:
Dim FirstSetting() As Double = Array As Double( _
    152.92, 152.08, 150.25, 148.58, 147, 145.58, 144.33, 143.17, 142.17, 141.33, _
    140.58, 140, 139.5, 139.17, 138.92, 138.83, 138.75, 138.75 _
)

Dim a As Double = -3.3646354072096498E+00    'should probably reduce these to a less-scary number of digits
Dim b As Double = 5.4398256762579855E-01    'eg 0.5439826 would be heaps good enough given that
Dim c As Double = -2.0648854195715742E+00    'original data was 5 significant digits, but...
Dim o As Double = 1.5779972040970864E+02    'why lose both time and precision in reformatting numbers?

For x = 1 To 18
    Dim tabley As Double = FirstSetting(x - 1)
    Dim formulay As Double = a * x + b * Power(x, 1.5) + c * Power(x, -2) + o
    Log(x & "  " & NumberFormat2(tabley, 1, 2, 2, False) & "  " & NumberFormat2(formulay, 1, 2, 2, False))
Next
produces log:
B4X:
Waiting for debugger to connect...
Program started.
1  152.92  152.91
2  152.08  152.09
3  150.25  150.30
4  148.58  148.56
5  147.00  146.98
6  145.58  145.55
7  144.33  144.28
8  143.17  143.16
9  142.17  142.18
10  141.33  141.33
11  140.58  140.62
12  140.00  140.02
13  139.50  139.54
14  139.17  139.18
15  138.92  138.92
16  138.83  138.77
17  138.75  138.72
18  138.75  138.77
 
Upvote 0

emexes

Expert
Licensed User
Completely forgot to point out that the main advantage of using formula rather than table is: what about if you want to know what the value would be for eg x = 1.3 ?
You can interpolate from the table, but that takes extra code and is probably less accurate than using the formula method.
B4X:
Dim x As Double
For x = 14 To 15 Step 0.1
    Dim formulay As Double = a * x + b * Power(x, 1.5) + c * Power(x, -2) + o    'no change to formula
    Log(NumberFormat2(x, 1, 2, 2, False) & "  " & NumberFormat2(formulay, 1, 2, 2, False))
Next
produces log:
B4X:
14.00  139.18
14.10  139.15
14.20  139.12
14.30  139.09
14.40  139.06
14.50  139.04
14.60  139.01
14.70  138.99
14.80  138.97
14.90  138.94
15.00  138.92
 
Last edited:
Upvote 0

Kevin L. Johnson

Member
Licensed User
Longtime User
Completely forgot to point out that the main advantage of using formula rather than table is: what about if you want to know what the value would be for eg x = 1.3 ?
You can interpolate from the table, but that takes extra code and is probably less accurate than using the formula method.
B4X:
Dim x As Double
For x = 14 To 15 Step 0.1
    Dim formulay As Double = a * x + b * Power(x, 1.5) + c * Power(x, -2) + o    'no change to formula
    Log(NumberFormat2(x, 1, 2, 2, False) & "  " & NumberFormat2(formulay, 1, 2, 2, False))
Next
produces log:
B4X:
14.00  139.18
14.10  139.15
14.20  139.12
14.30  139.09
14.40  139.06
14.50  139.04
14.60  139.01
14.70  138.99
14.80  138.97
14.90  138.94
15.00  138.92

Actually emexes,

The original source of the data generation is unknown at the moment and has changed a bit over the years. However, I am busy tracking down the source formulas. Furthermore, we do interpolate the exact numbers using this LUT as the starting point. This is proprietary data that we publish used as a Lookup table for a product that gets installed out in the field.

Thanks
Kevin
 
Upvote 0

Kevin L. Johnson

Member
Licensed User
Longtime User
Is that data fixed and unchanging?

It looks like a lookup table for something like eg battery discharge rate.

Is there a formula that generated that data? It'd be a heck of a lot easier to use that, rather than interpolating from a table.

The data is indeed fixed, although, we have corrected the data over the years in we find an adjustment is warranted.

KLJ
 
Upvote 0

emexes

Expert
Licensed User
The original source of the data generation is unknown at the moment and has changed a bit over the years. However, I am busy tracking down the source formulas.
Lol. That statement brought back mixed memories of similar situations. For one program, I was forever receiving suggestions about how to improve accuracy (aka increase customer engine power numbers ;-) The best example was being given a formula to estimate vehicle drivetrain power loss based on mass and speed (fair enough) which included a sin() term. I thought hmm, that seems odd, and I'm asking all sorts of questions about it to try verify its validity. After a few rounds I finally realised that, above a certain mass, the resistance started to decrease. We all agreed this was unlikely.

Happy days :)
 
Upvote 0
Top