Android Question KeyValueStore & CLV performance?!

[ilan]

hi

i am using the KeyValueStore class in my app to save custom types.

my problem is that when i go through all items in my keyvaluestore list it takes to long.

i am testing it with about 200 items and it takes about 320ms
in my opinion its to much.

i believe if i would go through a list it would take less then 1ms with 200 items in it.

am i doing maybe something wrong?

this is my code:

    Dim startl, endl As Long
    startl = DateTime.DateParse(date)
    endl = DateTime.Add(startl,0,1,0) - 1000  
  
    Log("list size: " & allshift.ListKeys.Size)
          
    For Each str As String In allshift.ListKeys
        Dim newshift As shift = allshift.Get(str)
        If newshift.timein >= startl And newshift.timein <= endl Then
            shiftlist.Add(newshift)
        else if newshift.timein >= DateTime.Add(startl,0,-1,0) And newshift.timein <= DateTime.Add(endl,0,-1,0) Then
            visshiftlist1.Add(newshift)
        else if newshift.timein >= DateTime.Add(startl,0,1,0) And newshift.timein <= DateTime.Add(endl,0,1,0) Then       
            visshiftlist3.Add(newshift)
        End If
    Next
  
    LogColor("finish FOR NEXT LOOP: " & (DateTime.Now-starttime) & " ms",Colors.Blue)
    starttime = DateTime.Now

logs:

*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
Class not found: anywheresoftware.b4a.samples.customlistview.customlistview, trying: www.sagital.mysalarynew.customlistview
Class not found: anywheresoftware.b4a.samples.customlistview.customlistview, trying: www.sagital.mysalarynew.customlistview
list size: 205
finish FOR NEXT LOOP: 319 ms
finish Load CLV: 51 ms
finish CalculateTotalShifts: 2 ms
Panel size is unknown. Layout may not be loaded correctly.
Panel size is unknown. Layout may not be loaded correctly.
** Activity (main) Resume **
list size: 205
finish FOR NEXT LOOP: 335 ms
finish Load CLV: 200 ms
finish CalculateTotalShifts: 3 ms
list size: 205
finish FOR NEXT LOOP: 371 ms
finish Load CLV: 48 ms
finish CalculateTotalShifts: 2 ms
list size: 205
finish FOR NEXT LOOP: 321 ms
finish Load CLV: 179 ms
finish CalculateTotalShifts: 7 ms
list size: 205
finish FOR NEXT LOOP: 372 ms
finish Load CLV: 133 ms
finish CalculateTotalShifts: 2 ms
list size: 205
finish FOR NEXT LOOP: 281 ms
finish Load CLV: 128 ms
finish CalculateTotalShifts: 2 ms
list size: 205
finish FOR NEXT LOOP: 321 ms
finish Load CLV: 76 ms
finish CalculateTotalShifts: 2 ms

thanx, ilan

 

[ilan]

ok i see that if i add all item to an array list it takes much faster so what i will do is everytime i update the keyvaluestore database i will update a global list and load all types from that list

    Dim starttime As Long = DateTime.Now
           
    For Each newshift As shift In globallist ' allshift.ListKeys
'        Dim newshift As shift = allshift.Get(str)
        If newshift.timein >= startl And newshift.timein <= endl Then
            shiftlist.Add(newshift)
        else if newshift.timein >= DateTime.Add(startl,0,-1,0) And newshift.timein <= DateTime.Add(endl,0,-1,0) Then
            visshiftlist1.Add(newshift)
        else if newshift.timein >= DateTime.Add(startl,0,1,0) And newshift.timein <= DateTime.Add(endl,0,1,0) Then        
            visshiftlist3.Add(newshift)
        End If
    Next
   
    LogColor("finish FOR NEXT LOOP: " & (DateTime.Now-starttime) & " ms",Colors.Blue)
    starttime = DateTime.Now

*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
Class not found: anywheresoftware.b4a.samples.customlistview.customlistview, trying: www.sagital.mysalarynew.customlistview
Class not found: anywheresoftware.b4a.samples.customlistview.customlistview, trying: www.sagital.mysalarynew.customlistview
list size: 205
finish FOR NEXT LOOP: 8 ms
finish Load CLV: 51 ms
finish CalculateTotalShifts: 2 ms
Panel size is unknown. Layout may not be loaded correctly.
Panel size is unknown. Layout may not be loaded correctly.
** Activity (main) Resume **
list size: 205
finish FOR NEXT LOOP: 8 ms
finish Load CLV: 229 ms
finish CalculateTotalShifts: 21 ms
list size: 205
finish FOR NEXT LOOP: 8 ms
finish Load CLV: 51 ms
finish CalculateTotalShifts: 3 ms
list size: 205
finish FOR NEXT LOOP: 8 ms
finish Load CLV: 145 ms
finish CalculateTotalShifts: 3 ms
list size: 205
finish FOR NEXT LOOP: 8 ms
finish Load CLV: 81 ms
finish CalculateTotalShifts: 5 ms

8ms vs 320ms!!

 

[ilan]

Now i will only need to see how i can improove the clv updating time.

It also takes to much!

 

[udg]

Hi Ilan,
to cut some ms more you may want to pre-calculate "DateTime.Add(startl,0,-1,0)" and similar in your if-then-else clause inside the for-next loop.
The gain will depend on how many items fall in those ranges following the first one (which you pre.calculated).

 

[ilan]

Hi Ilan,
to cut some ms more you may want to pre-calculate "DateTime.Add(startl,0,-1,0)" and similar in your if-then-else clause inside the for-next loop.
The gain will depend on how many items fall in those ranges following the first one (which you pre.calculated).

Sorry but i am not sure i have understood you correctly.

The time parsing is done before the for next loop. If i put it inside the loop i will parse time 200 times. No need for that. I just add all items in range of 2 dates so i need to parse the time only once

EDIT: oopsss i see now what u mean. I have updated the code today and forgot about that.

Thanx this will save me few ms

 

[ilan]

Hi Ilan,
to cut some ms more you may want to pre-calculate "DateTime.Add(startl,0,-1,0)" and similar in your if-then-else clause inside the for-next loop.
The gain will depend on how many items fall in those ranges following the first one (which you pre.calculated).

updated code:

    Dim starttime As Long = DateTime.Now
           
    For Each newshift As shift In globallist ' allshift.ListKeys
'        Dim newshift As shift = allshift.Get(str)
        If newshift.timein >= startl And newshift.timein <= endl Then
            shiftlist.Add(newshift)
        else if newshift.timein >= startl2 And newshift.timein <= endl2 Then
            visshiftlist1.Add(newshift)
        else if newshift.timein >= startl3 And newshift.timein <= endl3 Then        
            visshiftlist3.Add(newshift)
        End If
    Next
   
    LogColor("finish FOR NEXT LOOP: " & (DateTime.Now-starttime) & " ms",Colors.Blue)
    starttime = DateTime.Now

logs:

list size: 205
finish FOR NEXT LOOP: 0 ms
finish Load CLV: 170 ms
finish CalculateTotalShifts: 2 ms
list size: 205
finish FOR NEXT LOOP: 0 ms
finish Load CLV: 62 ms
finish CalculateTotalShifts: 2 ms
list size: 205
finish FOR NEXT LOOP: 0 ms
finish Load CLV: 202 ms
finish CalculateTotalShifts: 3 ms
list size: 205
finish FOR NEXT LOOP: 0 ms
finish Load CLV: 99 ms
finish CalculateTotalShifts: 2 ms

0ms vs 370ms

as i thought at first post, less then 1ms

thanx @udg for pointing me to my mistake

now i only need to improove CLV update time

 

[ilan]

ok after doing some tests i found another important thing about CLV.

if i create the clv item by code (create each view) its almost 2x faster then loading a layout each time you create the clv item.

loading layout:

*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
Class not found: anywheresoftware.b4a.samples.customlistview.customlistview, trying: www.sagital.mysalarynew.customlistview
Class not found: anywheresoftware.b4a.samples.customlistview.customlistview, trying: www.sagital.mysalarynew.customlistview
list size: 183
finish FOR NEXT LOOP: 3 ms
finish Load CLV: 141 ms
finish CalculateTotalShifts: 3 ms
Panel size is unknown. Layout may not be loaded correctly.
Panel size is unknown. Layout may not be loaded correctly.
** Activity (main) Resume **
list size: 183
finish FOR NEXT LOOP: 5 ms
finish Load CLV: 154 ms
finish CalculateTotalShifts: 3 ms
list size: 183
finish FOR NEXT LOOP: 9 ms
finish Load CLV: 271 ms
finish CalculateTotalShifts: 3 ms
list size: 183
finish FOR NEXT LOOP: 5 ms
finish Load CLV: 149 ms
finish CalculateTotalShifts: 3 ms
list size: 183
finish FOR NEXT LOOP: 9 ms
finish Load CLV: 244 ms
finish CalculateTotalShifts: 2 ms
list size: 183
finish FOR NEXT LOOP: 9 ms
finish Load CLV: 334 ms
finish CalculateTotalShifts: 2 ms

clv items views created by code:

*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
Class not found: anywheresoftware.b4a.samples.customlistview.customlistview, trying: www.sagital.mysalarynew.customlistview
Class not found: anywheresoftware.b4a.samples.customlistview.customlistview, trying: www.sagital.mysalarynew.customlistview
list size: 183
finish FOR NEXT LOOP: 4 ms
finish Load CLV: 119 ms
finish CalculateTotalShifts: 3 ms
Panel size is unknown. Layout may not be loaded correctly.
Panel size is unknown. Layout may not be loaded correctly.
** Activity (main) Resume **
list size: 183
finish FOR NEXT LOOP: 4 ms
finish Load CLV: 100 ms
finish CalculateTotalShifts: 3 ms
list size: 183
finish FOR NEXT LOOP: 5 ms
finish Load CLV: 75 ms
finish CalculateTotalShifts: 3 ms
list size: 183
finish FOR NEXT LOOP: 6 ms
finish Load CLV: 94 ms
finish CalculateTotalShifts: 3 ms
list size: 183
finish FOR NEXT LOOP: 5 ms
finish Load CLV: 100 ms
finish CalculateTotalShifts: 3 ms
list size: 183
finish FOR NEXT LOOP: 12 ms
finish Load CLV: 189 ms
finish CalculateTotalShifts: 3 ms

i know the clv example by erel is done with loading layout but its seems that creating via code is faster.
just a info from me.

 

[LucaMs]

i know the clv example by erel is done with loading layout but its seems that creating via code is faster.

Ok, but how much is more comfortable to create a layout and load it?

Anyway, thanks for the info, can be useful.

 

[ilan]

Ok, but how much is more comfortable to create a layout and load it?

Anyway, thanks for the info, can be useful.

i have modified @Erel's example to demonstrate the difference between creating a clv via code or by loading a layout.

result:

CODE: 563ms LAYOUT: 2027ms (300 items)

so via code its 4x faster!!!!! (tested on galaxy s5)

(please try on release mode)

 

[LucaMs]

so via code its 4x faster!!!!!

Ok, but how much is more comfortable to create a layout and load it?

 

[ilan]

Ok, but how much is more comfortable to create a layout and load it?

You write the code only once and if it will perform 4x faster on every use its something to consider

As you can see in the example its not that much work doing it via code.

 

[LucaMs]

You write the code only once and if it will perform 4x faster on every use its something to consider

As you can see in the example its not that much work doing it via code.

Imagine if the Designer produced the code !!!

 

[ilan]

WOW!!! i switched (back) to Ultimate List View from @Informatix and look at the results:

*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
list size: 286
finish loading list: 4 ms
finish loading ULV: 1 ms
finish calculating shifts: 3 ms
Panel size is unknown. Layout may not be loaded correctly.
Panel size is unknown. Layout may not be loaded correctly.
** Activity (main) Resume **
list size: 286
finish loading list: 4 ms
finish loading ULV: 1 ms
finish calculating shifts: 4 ms
list size: 286
finish loading list: 6 ms
finish loading ULV: 0 ms
finish calculating shifts: 2 ms
list size: 286
finish loading list: 8 ms
finish loading ULV: 1 ms
finish calculating shifts: 5 ms

1ms vs 560ms
this is amazing.

i had some difficulties with ulv but i think i start understand how to use it correctly. some stuff are still unclear to me like how to perform a FullScroll but i will try to get the answer some how anyway ulv is awesome!

EDIT: to perform a fullscroll you can use .JumpTo(...,...) function

 

[Erel]

The numbers I get with your code are:
CLV by code: ~1000ms
CLV with layout file: ~2000ms

Note that I've set the animation duration of CellItem to 0 and removed the AutoScaleAll call.

UltimateListView is optimized for large lists. It only creates the items views when they become visible (similar to ListView and Table class).
CustomListView creates all views. This makes it simpler to work with CustomListView as you can do whatever you like with all the views, as long as the list is short enough for the performance to be reasonable. Note that you can also load more items when the user scrolls the list.

 

[ilan]

The numbers I get with your code are:
CLV by code: ~1000ms
CLV with layout file: ~2000ms

Note that I've set the animation duration of CellItem to 0 and removed the AutoScaleAll call.

UltimateListView is optimized for large lists. It only creates the items views when they become visible (similar to ListView and Table class).
CustomListView creates all views. This makes it simpler to work with CustomListView as you can do whatever you like with all the views, as long as the list is short enough for the performance to be reasonable. Note that you can also load more items when the user scrolls the list.

Yes thats true. CLV is simpler to work with, thats why i switched from ULV to CLV but then i switched back and now i understand how ULV is working.
in my case (working with few 100 items) ULV is still a better choice since loading a list in 1 ms or 1000 ms is a very big difference.

i started this thread first because of the Key Value Store Class and after i modified the Class and am getting much better performance now i wanted to improve the load to list performance and with CLV i could not get the result i wanted.

btw. i found out that when you use 200-300 items in a DB (KVS Class) then going through all of them in a loop will take to much time.
the reason is each time you get a Object you convert it from bytes to Object (ConvertBytesToObject) and this take time when you have 1000 items.
so what i did is i created a public Map that will hold all object and instead of loading each object from the DB i load it from that map.

'KeyValueStore: v2.00
Sub Class_Globals
    Private sql1 As SQL
    Private ser As B4XSerializator
    Public ArrayMap As Map
End Sub

'Initializes the store and sets the store file.
Public Sub Initialize (Dir As String, FileName As String)
    If sql1.IsInitialized Then sql1.Close
#if B4J
    sql1.InitializeSQLite(Dir, FileName, True)
#else
    sql1.Initialize(Dir, FileName, True)
#end if
    CreateTable
    ArrayMap.Initialize
    fillArrayMap
End Sub

Public Sub Put(Key As String, Value As Object)
    sql1.ExecNonQuery2("INSERT OR REPLACE INTO main VALUES(?, ?)", Array As Object(Key, ser.ConvertObjectToBytes(Value)))
    ArrayMap.Put(Key, Value)
End Sub

Public Sub Size As Int
    Return sql1.ExecQuerySingleResult("SELECT count(key) FROM main")
End Sub

Private Sub fillArrayMap
    For Each key As String In ListKeys
        ArrayMap.Put(key,Get(key))
    Next
End Sub

Public Sub getPos(db As KeyValueStore, pos As Int) As Object
    Dim list1 As List
    list1.Initialize
    list1 = ListKeys
    list1.Sort(True)
    Return db.Get(list1.Get(pos))   
End Sub

Public Sub getPosKey(pos As Int) As String
    Dim list1 As List
    list1.Initialize
    list1 = ListKeys
    list1.Sort(True) 'last date show first to save time
    Return list1.Get(pos)
End Sub

Public Sub Get(Key As String) As Object
    Dim rs As ResultSet = sql1.ExecQuery2("SELECT value FROM main WHERE key = ?", Array As String(Key))
    Dim result As Object = Null
    If rs.NextRow Then
        result = ser.ConvertBytesToObject(rs.GetBlob2(0))
    End If
    rs.Close
    Return result
End Sub

Public Sub GetDefault(Key As String, DefaultValue As Object) As Object
    Dim res As Object = Get(Key)
    If res = Null Then Return DefaultValue
    Return res
End Sub

Public Sub PutEncrypted (Key As String, Value As Object, Password As String)
#if B4I
    Dim cipher As Cipher
#else
    Dim cipher As B4XCipher
#end if
    Put(Key, cipher.Encrypt(ser.ConvertObjectToBytes(Value), Password))
End Sub

Public Sub GetEncrypted (Key As String, Password As String) As Object
#if B4I
    Dim cipher As Cipher
#else
    Dim cipher As B4XCipher
#end if
    Dim b() As Byte = Get(Key)
    If b = Null Then Return Null
    Return ser.ConvertBytesToObject(cipher.Decrypt(b, Password))
End Sub

#if not(B4J)
Public Sub PutBitmap(Key As String, Value As Bitmap)
    Dim out As OutputStream
    out.InitializeToBytesArray(0)
    Value.WriteToStream(out, 100, "PNG")
    Put(Key, out.ToBytesArray)
    out.Close
End Sub

Public Sub GetBitmap(Key As String) As Bitmap
    Dim b() As Byte = Get(Key)
    If b = Null Then Return Null
    Dim in As InputStream
    in.InitializeFromBytesArray(b, 0, b.Length)
    Dim bmp As Bitmap
    bmp.Initialize2(in)
    in.Close
    Return bmp
End Sub
#End If

'Removes the key and value mapped to this key.
Public Sub Remove(Key As String)
    sql1.ExecNonQuery2("DELETE FROM main WHERE key = ?", Array As Object(Key))
    ArrayMap.Remove(Key)
End Sub

'Returns a list with all the keys.
Public Sub ListKeys As List
    Dim c As ResultSet = sql1.ExecQuery("SELECT key FROM main")
    Dim res As List
    res.Initialize
    Do While c.NextRow
        res.Add(c.GetString2(0))
    Loop
    c.Close
    Return res
End Sub

'Tests whether a key is available in the store.
Public Sub ContainsKey(Key As String) As Boolean
    Return sql1.ExecQuerySingleResult2("SELECT count(key) FROM main WHERE key = ?", _
        Array As String(Key)) > 0
End Sub

'Deletes all data from the store.
Public Sub DeleteAll
    sql1.ExecNonQuery("DROP TABLE main")
    ArrayMap.Clear
    CreateTable
End Sub


'Closes the store.
Public Sub Close
    sql1.Close
End Sub

'creates the main table (if it does not exist)
Private Sub CreateTable
    sql1.ExecNonQuery("CREATE TABLE IF NOT EXISTS main(key TEXT PRIMARY KEY, value NONE)")
End Sub

i know you will not like it @Erel but maybe you can improve the class so i dont need to load an object from the database and each time convert from bytes to object.
only when you save an object you do the conversation but there is no need to do it when you load an object.

so now when i go through a loop of 300 items i just use the public ArrayMap. its much much faster!!!!

what do you think?

EDIT: i forgot to mention i went from 319 ms to less then 1ms when using ArrayMap instead loading from the DB in a loop of 200 items!

 

[Erel]

Sounds like a good solution. Another possible solution is to store all items in a single Map and save it as a single item. Load the map when the program is started and save it when the program is paused.

 

[ilan]

Sounds like a good solution. Another possible solution is to store all items in a single Map and save it as a single item. Load the map when the program is started and save it when the program is paused.

But i will still need the KVS to save the map that holds custom types correct?

If yes that is what i am doing now. I load on app start everything to the map and get all object from that map. But you sugguest to save the map to KVS on app pause?

This will require me to go on each app pause through a loop of few 100 items. I preffer to save each item in real time because i add each time only 1 item. I never add more then 1 item to the map. Only when loading the ULV i need to go through all items so here its better to use a public map instead of loading from the DB but when adding a new item its better to do it in real time and not go through a loop each time i pause the app.

Or maybe i have understood your suggestion wrong?

EDIT: ok i think i understand what you mean. I should use the KVS to store the map instead to store all objects. The question is how fast will it be to store a single map that hold 300 custom types ? there is only one way to find it out. I will try it, thanx

 

[Erel]

The question is how fast will it be to store a single map that hold 300 custom types ?

As you are only writing it once when the program is paused, it will most probably be fast enough.