Android Question KeyValueStore2 - performance for big lists?

[Dave O]

I'm using KVS2 to save lists of poker hands, where each round of poker has 9 player hands, and each hand has cards and other data, including other small lists.

This all works, but I'm not sure it's going to scale up well. At 650 rounds, that's about probably 10000 - 15000 records, which currently takes about 10 seconds on my Galaxy Note 4.

Two questions:

- Any tips for speeding up this kind of data storage/retrieval? (I'm currently saving/loading each list using a single put/get statement.)

- Any way to show a progress gauge during the kvs.get operation, or is that necessarily a single atomic operation? (I currently have a gauge showing my conversion of the retrieved types back into objects, but that's only a second or two, while the kvs.get is 10-15 seconds by itself.)

Any help appreciated, cheers!

 

[Erel]

Don't save all the data into a single entry. Split it into multiple entries and only get the once you need.

You can modify KVS2 and use the asynchronous methods to retrieve the data and deserialize it. However it will make the flow a bit more complex.

 

[Dave O]

I may be able to do incremental saves instead of a single big save on activity_pause.

I can also review the data model to see which items I could remove.

I'll have to ponder whether I can do an incremental load at app launch.

To avoid the single big list load, I was thinking I could try 2 things:
- save/load the data in some kind of manual loop, so I can show progress (which I can't show now with the single list save/load)
- work directly with SQLite instead of going through the abstraction of KVS. Is this likely to speed up performance much?

Thanks again!

 

[Erel]

work directly with SQLite instead of going through the abstraction of KVS. Is this likely to speed up performance much?

KVS does two things:
1. Serializes the data using B4XSerializator.
2. Writes it into the database.

As long as you are using B4XSerializator to serialize the data, the performance will be identical whether you insert the data or the code in KVS2 does it.

save/load the data in some kind of manual loop, so I can show progress (which I can't show now with the single list save/load)

You should instead use the async methods.

 

[Dave O]

In past apps, I used File.WriteMap and ReadMap to persist data. They seemed fast, though I was storing much less data. I'm guessing that they wouldn't perform as well as KVS for large data sets like I'm working with now?

I'll look into the async methods, thanks.

 

[Erel]

I'm guessing that they wouldn't perform as well as KVS for large data sets like I'm working with now?

No. They will be much slower.

 

[Dave O]

Thanks to all for your help on this.

I've changed from bulk save to incremental save (pretty much instant).

For loading, I now load only the most recent 2 hands (similarly quick), then let the user play while I load the older hands (potentially thousands) in a background thread. Working through that now (I'm new to threads), but it looks promising.

 

[Erel]

For loading, I now load only the most recent 2 hands (similarly quick), then let the user play while I load the older hands (potentially thousands) in a background thread. Working through that now (I'm new to threads), but it looks promising.

As I wrote in the other thread, this is the wrong solution. The SQL library already supports async methods. You should use them.

 

[Dave O]

The SQL library already supports async methods

I thought about that, and may still try it, but I think it would mainly help if I was loading a list of poker hands in one go (i.e. doing a single Get on a list).

What I'm doing instead is saving single hands as they're played (which is fast enough that it doesn't need async).

So, the loading becomes a Get per hand, rather than a single Get for the whole list. At that level, the Get doesn't seen to take long compared to the total time needed to get it and reconstitute it into class instances - making the individual Gets async doesn't seem to be worth it. Which is why I decided to try moving the entire load loop into an async thread.

But I'm new at this, so if I'm missing something important here, I'm very happy to be redirected.

 

[Erel]

If you just try to access the database from multiple threads then it will be corrupted.

Change the query and issue a single query with all the data needed. Make that query an async query.