Android Question Best way to write a 2 dimensional array of doubles to file

Discussion in 'Android Questions' started by Kevin L. Johnson, Jul 16, 2019.

  1. Kevin L. Johnson

    Kevin L. Johnson Member Licensed User

    I am looking for the best way to persist (write) a 2 dimensional array of doubles to disk (file). I have looked at the Random Access File .WriteObject(), and .WriteB4XObject() and the b4xSerializator.ConvertObjectToBytes and then use the BytesToFile to persist them to disk but, at this point, I am not sure of which direction to proceed.

    The purpose is to generate a protected look-up table, and therefore, I would prefer, the file be binary and read only so as not to be disturbed or corrupted.

    Obviously at some later point, the persisted two dimensional array of doubles would then need to be read back from the file to be used as a look-up table.

    Any help would be much appreciated.
  2. emexes

    emexes Well-Known Member Licensed User

    How big is the array? For smaller arrays, the code and effort and complexity risk of squeezing things down might not be worth the space saved.

    Maybe you could write the array to a CSV string, and then compress the string to a file.

    just checking my understanding here: so not encryption to keep hidden? (phew! if correct)
  3. Erel

    Erel Administrator Staff Member Licensed User

    B4XSerializator and WriteB4XObject do not support serializing arrays with the exception of arrays of bytes and arrays of objects.

    You can use RAF.WriteObject. It is a good option.
  4. MarkusR

    MarkusR Well-Known Member Licensed User

    maybe you can store data in a type & list
    something like array(a,b) is similar a list of type with x,y fields.
    type mydata(a as double,b as double)
    Dim l As List
    Dim data As mydata
       data.a = 
       data.b = 
    Erel likes this.
  5. MarkusR

    MarkusR Well-Known Member Licensed User

    b4j example with B4XSerializator
    #Region Project Attributes
    #MainFormWidth: 600
    #MainFormHeight: 600
    #End Region

    Sub Process_Globals
    Private fx As JFX
    Private MainForm As Form
    Type mydata(a As Double,b As Double)

    End Sub

    Sub AppStart (Form1 As Form, Args() As String)
        MainForm = Form1
    'MainForm.RootPane.LoadLayout("Layout1") 'Load the layout file.

    End Sub

    'Return true to allow the default exceptions handler to handle the uncaught exception.
    Sub Application_Error (Error As Exception, StackTrace As StringAs Boolean
    Return True
    End Sub

    Sub Test

    Dim l As List
    Dim data As mydata
        data.a = 
        data.b = 
    Dim b As B4XSerializator 'jRandomAccessFile Class / Library
        Dim bytes() As Byte = b.ConvertObjectToBytes(l)

    Dim out As OutputStream = File.OpenOutput(File.DirApp,"Test.dat",False)
    End Sub
  6. Kevin L. Johnson

    Kevin L. Johnson Member Licensed User

    They array is dimensioned as a 19 by 7 array of doubles (Not that large at all). I was thinking of a CSV string or file too, and that may be the way I go, If I cannot get the RAF.WriteObject method working.

    Thanks for the suggestion emexes.
  7. Kevin L. Johnson

    Kevin L. Johnson Member Licensed User

    Interesting ...

    Thanks for the suggestion MarkusR
  8. Kevin L. Johnson

    Kevin L. Johnson Member Licensed User

    Here is the code, I have thus far.
    In the GenerateLUT subroutine I...
    1.) Type the Data in by Rows
    2.) Loop through the Row / Pedestal Arrays
    3.) Place the elements Row, Cols into a two dimensional array

    The missing step is to persist the 2 dimensional array to a file ...

    Public Sub GenerateLUT()
    Dim intRows = 19 As Int
    Dim intCols = 7 As Int
    Dim LUT(intRows, intCols) As Double
    Dim FrontPed00(7As Double
    Dim FrontPed01(7As Double
    Dim FrontPed02(7As Double
    Dim FrontPed03(7As Double
    Dim FrontPed04(7As Double
    Dim FrontPed05(7As Double
    Dim FrontPed06(7As Double
    Dim FrontPed07(7As Double
    Dim FrontPed08(7As Double
    Dim FrontPed09(7As Double
    Dim FrontPed10(7As Double
    Dim FrontPed11(7As Double
    Dim FrontPed12(7As Double
    Dim FrontPed13(7As Double
    Dim FrontPed14(7As Double
    Dim FrontPed15(7As Double
    Dim FrontPed16(7As Double
    Dim FrontPed17(7As Double
    Dim FrontPed18(7As Double
    'Generate DATA
        FrontPed00 = 
    Array As Double ( 0.0,    1.0,    2.0,    3.0,    4.0,    5.0,    6.0)
        FrontPed01 = 
    Array As Double ( 1.0152.92152.92152.92152.92152.92152.92)
        FrontPed02 = 
    Array As Double ( 2.0152.08152.08152.17152.17152.17152.17)
        FrontPed03 = 
    Array As Double ( 3.0150.25150.25150.33150.33150.42150.42)
        FrontPed04 = 
    Array As Double ( 4.0148.58148.58148.67148.75148.83148.92)
        FrontPed05 = 
    Array As Double ( 5.0147.00147.08147.17147.25147.42147.50)
        FrontPed06 = 
    Array As Double ( 6.0145.58145.75145.92146.00146.08146.25)
        FrontPed07 = 
    Array As Double ( 7.0144.33144.50144.67144.83145.00145.08)
        FrontPed08 = 
    Array As Double ( 8.0143.17143.33143.58143.75144.00144.17)
        FrontPed09 = 
    Array As Double ( 9.0142.17142.42142.58142.83143.08143.25)
        FrontPed10 = 
    Array As Double (10.0141.33141.58141.83142.08142.25142.50)
        FrontPed11 = 
    Array As Double (11.0140.58140.83141.17141.42141.67141.92)
        FrontPed12 = 
    Array As Double (12.0140.00140.25140.58140.92141.17141.50)
        FrontPed13 = 
    Array As Double (13.0139.50139.83140.17140.50140.83141.17)
        FrontPed14 = 
    Array As Double (14.0139.17139.50139.92140.25140.58141.00)
        FrontPed15 = 
    Array As Double (15.0138.92139.33139.75140.08140.50140.83)
        FrontPed16 = 
    Array As Double (16.0138.83139.17139.58140.00140.42140.83)
        FrontPed17 = 
    Array As Double (17.0138.75139.17139.58140.00140.42140.92)
        FrontPed18 = 
    Array As Double (18.0138.75139.17139.58140.08140.50140.92)
    ' Populate 2 dimensional Array

    Dim intRow As Int
    Dim intCol As Int
    Dim PedArray() As Double
    For intRow = 0 To 18
    Dim PedArray() As Double
    Select intRow
    Case 0:  PedArray = FrontPed00
    Case 1:     PedArray = FrontPed01
    Case 2:     PedArray = FrontPed02
    Case 3:     PedArray = FrontPed03
    Case 4:     PedArray = FrontPed04
    Case 5:     PedArray = FrontPed05
    Case 6:     PedArray = FrontPed06
    Case 7:     PedArray = FrontPed07
    Case 8:     PedArray = FrontPed08
    Case 9:     PedArray = FrontPed09
    Case 10: PedArray = FrontPed10
    Case 11: PedArray = FrontPed11
    Case 12: PedArray = FrontPed12
    Case 13: PedArray = FrontPed13
    Case 14: PedArray = FrontPed14
    Case 15: PedArray = FrontPed15
    Case 16: PedArray = FrontPed16
    Case 17: PedArray = FrontPed17
    Case 18: PedArray = FrontPed18
    End Select
    Dim l_strData As String
    For intCol = 0 To 6
                LUT(intRow, intCol) = PedArray(intCol)
    Next 'intCol
    Next 'intRow
    'Display the LUT
    For intRow = 0 To 18
    Dim l_strData
            l_strData = 
    For intCol =  0 To 6
    If intCol > 0 Then
                    l_strData = l_strData & 
    ", " & TAB
    End If
    If intRow = 0 Then
    'Format the whole Top row as integers
                    l_strData = l_strData & NumberFormat2(LUT(intRow, intCol), 111False)
    Else if intCol = 0 Then
    'Format the First Col as an integer
                    l_strData = l_strData & NumberFormat2(LUT(intRow, intCol), 100False)
    'Format everything else as a double
                    l_strData = l_strData & NumberFormat2(LUT(intRow, intCol), 322False)
    End If

    ' Next Step is to persist the 2 dimensional array to file ...
  9. Kevin L. Johnson

    Kevin L. Johnson Member Licensed User

    Erel ... It is a good option, AND it worked! There are no words ... I had tried to use this technique using RAF.WriteObject and then RAF.ReadObject method once before, but my code returned something about the method not supporting the data type. I did upgrade my B4J afterwords though. Perhaps the upgrade fixed something?

    I tell everyone about B4X ...
    This product and it's community members are incredible!
    Thanks to All ...
    Erel likes this.
  10. wonder

    wonder Expert Licensed User

    Are you interested in storing it in JSON format?
  11. emexes

    emexes Well-Known Member Licensed User

    Is that data fixed and unchanging?

    It looks like a lookup table for something like eg battery discharge rate.

    Is there a formula that generated that data? It'd be a heck of a lot easier to use that, rather than interpolating from a table.
  12. emexes

    emexes Well-Known Member Licensed User

    What I was heading towards is something like this, a demo of implementing column 1 of your data as a formula.

    You might be concerned that the table data and the calculated data do not precisely match; my thought is that perhaps the table data was derived by experimental measurement, which will have some random error, and which the formula averages out (ironically thus probably making it more accurate).
    Dim FirstSetting() As Double = Array As Double( _
    152.92152.08150.25148.58147145.58144.33143.17142.17141.33, _
    140.58140139.5139.17138.92138.83138.75138.75 _

    Dim a As Double = -3.3646354072096498E+00    'should probably reduce these to a less-scary number of digits
    Dim b As Double = 5.4398256762579855E-01    'eg 0.5439826 would be heaps good enough given that
    Dim c As Double = -2.0648854195715742E+00    'original data was 5 significant digits, but...
    Dim o As Double = 1.5779972040970864E+02    'why lose both time and precision in reformatting numbers?

    For x = 1 To 18
    Dim tabley As Double = FirstSetting(x - 1)
    Dim formulay As Double = a * x + b * Power(x, 1.5) + c * Power(x, -2) + o
    Log(x & "  " & NumberFormat2(tabley, 122False) & "  " & NumberFormat2(formulay, 122False))
    produces log:
    Waiting for debugger to connect...
    Program started.
    1  152.92  152.91
    2  152.08  152.09
    3  150.25  150.30
    4  148.58  148.56
    5  147.00  146.98
    6  145.58  145.55
    7  144.33  144.28
    8  143.17  143.16
    9  142.17  142.18
    10  141.33  141.33
    11  140.58  140.62
    12  140.00  140.02
    13  139.50  139.54
    14  139.17  139.18
    15  138.92  138.92
    16  138.83  138.77
    17  138.75  138.72
    18  138.75  138.77
  13. emexes

    emexes Well-Known Member Licensed User

    Completely forgot to point out that the main advantage of using formula rather than table is: what about if you want to know what the value would be for eg x = 1.3 ?
    You can interpolate from the table, but that takes extra code and is probably less accurate than using the formula method.
    Dim x As Double
    For x = 14 To 15 Step 0.1
    Dim formulay As Double = a * x + b * Power(x, 1.5) + c * Power(x, -2) + o    'no change to formula
        Log(NumberFormat2(x, 122False) & "  " & NumberFormat2(formulay, 122False))
    produces log:
    14.00  139.18
    14.10  139.15
    14.20  139.12
    14.30  139.09
    14.40  139.06
    14.50  139.04
    14.60  139.01
    14.70  138.99
    14.80  138.97
    14.90  138.94
    15.00  138.92
    Last edited: Jul 19, 2019
  14. Kevin L. Johnson

    Kevin L. Johnson Member Licensed User

    Actually emexes,

    The original source of the data generation is unknown at the moment and has changed a bit over the years. However, I am busy tracking down the source formulas. Furthermore, we do interpolate the exact numbers using this LUT as the starting point. This is proprietary data that we publish used as a Lookup table for a product that gets installed out in the field.

  15. Kevin L. Johnson

    Kevin L. Johnson Member Licensed User

    Yes, I have thought that the JSON format would be advantagous for persisting the data.
  16. Kevin L. Johnson

    Kevin L. Johnson Member Licensed User

    The data is indeed fixed, although, we have corrected the data over the years in we find an adjustment is warranted.

  17. emexes

    emexes Well-Known Member Licensed User

    Lol. That statement brought back mixed memories of similar situations. For one program, I was forever receiving suggestions about how to improve accuracy (aka increase customer engine power numbers ;-) The best example was being given a formula to estimate vehicle drivetrain power loss based on mass and speed (fair enough) which included a sin() term. I thought hmm, that seems odd, and I'm asking all sorts of questions about it to try verify its validity. After a few rounds I finally realised that, above a certain mass, the resistance started to decrease. We all agreed this was unlikely.

    Happy days :)
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice