Android Tutorial [B4X] CloudKVS - synchronized key / value store

Erel

Administrator
Staff member
Licensed User
CloudKVS solves a difficult and common problem. The user needs to work with online data, however as this is a mobile app we cannot assume that the device will always be connected to the remote server.



With CloudKVS the app always works with a local database. If the device can connect to the remote server then the local store will be synchronized with the online store.

The store is implemented as a key/value store. It is similar to a persistent Map collection.
The values are serialized with B4XSerializator.
The following types are supported as values:
Lists, Maps, Strings, primitives (numbers), user defined types and arrays (only arrays of bytes and arrays of objects are supported).
Custom types should be declared in the main module.
Including combinations of these types (a list that holds maps for example).

This is a cross platform solution. The clients can be implemented with B4A, B4i or B4J and the data can be shared between the different platforms.
Note that ClientKVS class source code is exactly the same on all three platforms.

Working with ClientKVS is almost as simple as working with a regular Map.

User field

To allow more flexibility items are grouped by a "user" field.
For example:
B4X:
ckvs.Put("User1", "Key1", 100)
Log(ckvs.Get("User1", "Key1")) '100
Log(ckvs.GetDefault("User2", "Key1", 0)) '0 because User2/Key1 is different than User1/Key1
The synchronization (from the remote store to the local store) is based on the user field.
The SetAutoRefresh method sets the user(s) that will be fetched from the remote store.

For example if we want to auto-synchronize the data of "User1":
B4X:
ckvs.SetAutoRefresh(Array("User1"), 5)
You can pass multiple users in the array. The second parameter is the interval measured in minutes.
This means that the client will check for new data every 5 minutes.
New data in this case means data that was uploaded from other clients.
The NewData event is raised when new data was fetched from the remote server.
Note that auto-refresh is not relevant for local updates. Local updates are uploaded immediately (if possible).

Multiple clients can use the same 'user name'.

Defaults

GetDefaultAndPut:
B4X:
Dim Score As Int = ckvs.GetDefaultAndPut ("User1", "Score", 0)
If there is a User1/Score value in the local store then it will be returned. Otherwise it will return 0 and also put 0 in the store.
This is useful as it allows us later in the program to get the score with: ckvs.Get("User1", "Score").
Defaults put in the store are treated specially. If there is already a non-default value in the remote store then the default value will not overwrite the non-default value. The non-default value will be synchronized to the local store once there is a connection.

Notes & Tips

- CloudKVS is fault tolerant. The local store includes a 'queue' store which holds the changes that were not yet synchronized.
- For performance reasons it is better to use larger values (made of maps or lists) than to use many small items.
- In B4A it is recommended to initialize ClientKVS in the Starter service.
- The B4J server project can run as is. It accepts a single command line argument which is the port number. If you want to run it on a VPS: https://www.b4x.com/android/forum/threads/60378/#content
- In the examples the auto refresh interval is set to 0.1 (6 seconds). In most cases it is better to use larger intervals (1 minute+).
- The keys and user names are case sensitive.
- On older versions of Anrdoid there is a limit of 2mb per field. You will see the following error if you try to put a larger value: java.lang.IllegalStateException: Couldn't read row 0, col 0 from CursorWindow


Projects

The three client projects and the server project are attached.
If you want to add this feature to an existing project then you need to add:
1. CloudKVS and CallSubUtils modules.
2. The two custom types Item and Task to the main module.
3. The following libraries are required: SQL, RandomAccessFile and HttpUtils2.

The server project depends on jBuilderUtils library. The library is attached.

Development Test Server

You can use this link for the ServerUrl during development:
https://www.b4x.com:51041/cloudkvs
Note that the messages are limited to 100k and more importantly the database is deleted every few days. The clients will stop updating after the database is deleted. You can delete the local database (or uninstall and install the app again) for the clients to work again.
Remember to use unique ids for the user value.
 

Attachments

Last edited:

lemonisdead

Well-Known Member
Licensed User
I beg your pardon @Erel : loading the B4A client's project with B4A beta 5.8, I get those errors :
SOLVED : - missing Okhttputils2 : should I install jokhttputils2 (https://www.b4x.com/android/forum/threads/okhttputils2.62105/#content) ? Adding manually httputils2 does not solve
- unknown type resultset in ClientKVS : I have checked and my SQL library is #1.20 (seems to work with 1.30 but I can't find it for B4A)
 
Last edited:

Erel

Administrator
Staff member
Licensed User

b4auser1

Well-Known Member
Licensed User
This is the real B4X strength.
+1000

Erel,

I believe that jBuilderUtils library can be very usefull itself.
I tried to read more about jBuilderUtils library but
https://www.b4x.com/b4j/help/jbuilderutils.html
doesn't contain details - just method declarations.

Could you update documentation.
Thank's in adavance.

Also could you give more details about resolution sync conflicts.
Do you compare data modification time at server with data modification at client to determine which data version is more "fresh" ?
 

Erel

Administrator
Staff member
Licensed User
Please start a new thread for the question about jBuilderUtils (in B4J forum).

Also could you give more details about resolution sync conflicts.
Each item has a time stamp. The time value is set when the item is added to the local store.

The remote store compares the time stamp with the existing item time and updates it if it is newer.
 

Anser

Well-Known Member
Licensed User
Please tolerate my ignorance. It may be a very stupid question, but need to get a clear picture to have a very clear understanding about CloudKVS and in which situation this CloudKVS solution can be utilized.

  1. I understand that this will be helpful where you want your user to work offline and the data will be stored locally on the device and whenever Internet connectivity is available the data from these local device will get synchronized with the said CloudKVS. Right ?
  2. All the headache pertaining to what data is synced and what not, is taken care by B4j CloudKVS Server and B4A/B4i/B4x Client together. Whenever Internet is available, B4J CloudKVS and the B4A/B4i/B4x ClientKVS will talk to each other and will sync the data as per the refresh interval that we choose.
  3. This local data will not get synced to the MySQL or similar databases available on a remote server, instead it will get synced to the B4J CloudKVS server that will be using a Sqlite DB internally. Right ?
  4. Is there any possibility to get this local data synced with MySQL or similar remote Databases utilizing this CloudKVS Server and Client solution ?
  5. I assume that the CLoudKVS has to run on a VPS Server. Right ?
Regards

Anser
 

Roberto P.

Well-Known Member
Licensed User
excellent real-world examples, is the correct way to having integrated systems: Server, Mobiles and Applications. B4X Enterprise ....

I do not want to ask too much, to complete all requirements would also be useful example of a WebApp.

Thanks again
 

wl

Well-Known Member
Licensed User
Excellent work Erel.

However, there is one thing I'm missing... instead of polling the server for updates it would be great if modifications from the server could be pushed by the server and have a local event respond accordingly... Should be able to combine it with the push framework in B4X ?
 

Erel

Administrator
Staff member
Licensed User
Should be able to combine it with the push framework in B4X ?
Yes, you can combine both solutions. Though you will not gain too much by combining them. If you need to add a push feature then add a push feature.

I think that for most uses cases an interval of one minutes (+-) should be good enough. Remember that it is expected to work offline as well.
 

wl

Well-Known Member
Licensed User
Erel,

It would be nice to have both of them combined (out-of-the-box) (also including heartbeat etc...) so that we should not combine them ourselves... :)
 

Erel

Administrator
Staff member
Licensed User
It will not happen as the additional complexity will be higher than the benefit. At least in the general case.
 

johndb

Active Member
Licensed User
This is a very useful framework for cloud data synchronization. I do have a couple of questions:
  1. Is the time stamping "local" or "UTZ (GMT)" based? In other words, will it sync correctly for all devices located in various time zones?
  2. Given a scenario where cloud synchronization will be defined as "subscription" based, is there a way to inhibit all cloud syncing and only use the local database until let's say a flag is set to allow cloud synchronization?
Thanks in advance for your assistance.

John
 
Top