Android Tutorial [B4X] CloudKVS - synchronized key / value store

Discussion in 'Tutorials & Examples' started by Erel, Feb 15, 2016.

  1. Erel

    Erel Administrator Staff Member Licensed User

    CloudKVS solves a difficult and common problem. The user needs to work with online data, however as this is a mobile app we cannot assume that the device will always be connected to the remote server.

    [​IMG]

    With CloudKVS the app always works with a local database. If the device can connect to the remote server then the local store will be synchronized with the online store.

    The store is implemented as a key/value store. It is similar to a persistent Map collection.
    The values are serialized with B4XSerializator.
    The following types are supported as values:
    Lists, Maps, Strings, primitives (numbers), user defined types and arrays (only arrays of bytes and arrays of objects are supported).
    Custom types should be declared in the main module.
    Including combinations of these types (a list that holds maps for example).

    This is a cross platform solution. The clients can be implemented with B4A, B4i or B4J and the data can be shared between the different platforms.
    Note that ClientKVS class source code is exactly the same on all three platforms.

    Working with ClientKVS is almost as simple as working with a regular Map.

    User field

    To allow more flexibility items are grouped by a "user" field.
    For example:
    Code:
    ckvs.Put("User1""Key1"100)
    Log(ckvs.Get("User1""Key1")) '100
    Log(ckvs.GetDefault("User2""Key1"0)) '0 because User2/Key1 is different than User1/Key1
    The synchronization (from the remote store to the local store) is based on the user field.
    The SetAutoRefresh method sets the user(s) that will be fetched from the remote store.

    For example if we want to auto-synchronize the data of "User1":
    Code:
    ckvs.SetAutoRefresh(Array("User1"), 5)
    You can pass multiple users in the array. The second parameter is the interval measured in minutes.
    This means that the client will check for new data every 5 minutes.
    New data in this case means data that was uploaded from other clients.
    The NewData event is raised when new data was fetched from the remote server.
    Note that auto-refresh is not relevant for local updates. Local updates are uploaded immediately (if possible).

    Multiple clients can use the same 'user name'.

    Defaults

    GetDefaultAndPut:
    Code:
    Dim Score As Int = ckvs.GetDefaultAndPut ("User1""Score"0)
    If there is a User1/Score value in the local store then it will be returned. Otherwise it will return 0 and also put 0 in the store.
    This is useful as it allows us later in the program to get the score with: ckvs.Get("User1", "Score").
    Defaults put in the store are treated specially. If there is already a non-default value in the remote store then the default value will not overwrite the non-default value. The non-default value will be synchronized to the local store once there is a connection.

    Notes & Tips

    - CloudKVS is fault tolerant. The local store includes a 'queue' store which holds the changes that were not yet synchronized.
    - For performance reasons it is better to use larger values (made of maps or lists) than to use many small items.
    - In B4A it is recommended to initialize ClientKVS in the Starter service.
    - The B4J server project can run as is. It accepts a single command line argument which is the port number. If you want to run it on a VPS: https://www.b4x.com/android/forum/threads/60378/#content
    - In the examples the auto refresh interval is set to 0.1 (6 seconds). In most cases it is better to use larger intervals (1 minute+).
    - The keys and user names are case sensitive.
    - On older versions of Anrdoid there is a limit of 2mb per field. You will see the following error if you try to put a larger value: java.lang.IllegalStateException: Couldn't read row 0, col 0 from CursorWindow


    Projects

    The three client projects and the server project are attached.
    If you want to add this feature to an existing project then you need to add:
    1. CloudKVS and CallSubUtils modules.
    2. The two custom types Item and Task to the main module.
    3. The following libraries are required: SQL, RandomAccessFile and HttpUtils2.

    The server project depends on jBuilderUtils library. The library is attached.

    Development Test Server

    You can use this link for the ServerUrl during development:
    https://www.b4x.com:51041/cloudkvs
    Note that the messages are limited to 100k and more importantly the database is deleted every few days. The clients will stop updating after the database is deleted. You can delete the local database (or uninstall and install the app again) for the clients to work again.
    Remember to use unique ids for the user value.
     

    Attached Files:

    Last edited: Jul 17, 2016
  2. lemonisdead

    lemonisdead Well-Known Member Licensed User

    Thanks for that "gift" which is a maximum step and will spare us a lot of time. Other editors would have charged for so rich functions.
     
  3. lemonisdead

    lemonisdead Well-Known Member Licensed User

    I beg your pardon @Erel : loading the B4A client's project with B4A beta 5.8, I get those errors :
    SOLVED : - missing Okhttputils2 : should I install jokhttputils2 (https://www.b4x.com/android/forum/threads/okhttputils2.62105/#content) ? Adding manually httputils2 does not solve
    - unknown type resultset in ClientKVS : I have checked and my SQL library is #1.20 (seems to work with 1.30 but I can't find it for B4A)
     
    Last edited: Feb 15, 2016
    int21h likes this.
  4. Erel

    Erel Administrator Staff Member Licensed User

  5. luke2012

    luke2012 Well-Known Member Licensed User

    Great work @Erel! Thank you for this great framework that solve a common problem :)
     
    int21h likes this.
  6. LucaMs

    LucaMs Expert Licensed User

    This is the real B4X strength.
    (I am still studying this :))
     
    int21h likes this.
  7. Erel

    Erel Administrator Staff Member Licensed User

    This is also true about B4XSerializator. The fact that you can easily send objects, including custom types between the different platforms is a great feature. You can see in the code that all the messages sent are first serialized with B4XSerializator.
     
    int21h, gma, Peter Simpson and 4 others like this.
  8. b4auser1

    b4auser1 Well-Known Member Licensed User

    +1000

    Erel,

    I believe that jBuilderUtils library can be very usefull itself.
    I tried to read more about jBuilderUtils library but
    https://www.b4x.com/b4j/help/jbuilderutils.html
    doesn't contain details - just method declarations.

    Could you update documentation.
    Thank's in adavance.

    Also could you give more details about resolution sync conflicts.
    Do you compare data modification time at server with data modification at client to determine which data version is more "fresh" ?
     
    int21h likes this.
  9. Erel

    Erel Administrator Staff Member Licensed User

    Please start a new thread for the question about jBuilderUtils (in B4J forum).

    Each item has a time stamp. The time value is set when the item is added to the local store.

    The remote store compares the time stamp with the existing item time and updates it if it is newer.
     
    int21h likes this.
  10. Erel

    Erel Administrator Staff Member Licensed User

    A development test server is now available. See the first post for more information.
     
    int21h, DonManfred and lemonisdead like this.
  11. Anser

    Anser Well-Known Member Licensed User

    Please tolerate my ignorance. It may be a very stupid question, but need to get a clear picture to have a very clear understanding about CloudKVS and in which situation this CloudKVS solution can be utilized.

    1. I understand that this will be helpful where you want your user to work offline and the data will be stored locally on the device and whenever Internet connectivity is available the data from these local device will get synchronized with the said CloudKVS. Right ?
    2. All the headache pertaining to what data is synced and what not, is taken care by B4j CloudKVS Server and B4A/B4i/B4x Client together. Whenever Internet is available, B4J CloudKVS and the B4A/B4i/B4x ClientKVS will talk to each other and will sync the data as per the refresh interval that we choose.
    3. This local data will not get synced to the MySQL or similar databases available on a remote server, instead it will get synced to the B4J CloudKVS server that will be using a Sqlite DB internally. Right ?
    4. Is there any possibility to get this local data synced with MySQL or similar remote Databases utilizing this CloudKVS Server and Client solution ?
    5. I assume that the CLoudKVS has to run on a VPS Server. Right ?
    Regards

    Anser
     
    int21h likes this.
  12. lemonisdead

    lemonisdead Well-Known Member Licensed User

    Hello,
    1. correct
    2. correct
    3. correct
    4. of course but you will have to create the code to read CloudKVS server's database and sync its data with MySQL
    5. on any device able to run a java program (VPS, Raspberry, own computer at home with access to the net, etc)
     
  13. Erel

    Erel Administrator Staff Member Licensed User

    4. It should be simple to use MySQL instead of SQLite for the server. Go over the DB module and change it to use MySQL.
     
    int21h, Anser and lemonisdead like this.
  14. Roberto P.

    Roberto P. Well-Known Member Licensed User

    excellent real-world examples, is the correct way to having integrated systems: Server, Mobiles and Applications. B4X Enterprise ....

    I do not want to ask too much, to complete all requirements would also be useful example of a WebApp.

    Thanks again
     
    int21h likes this.
  15. wl

    wl Well-Known Member Licensed User

    Excellent work Erel.

    However, there is one thing I'm missing... instead of polling the server for updates it would be great if modifications from the server could be pushed by the server and have a local event respond accordingly... Should be able to combine it with the push framework in B4X ?
     
    int21h, pesquera and Roberto P. like this.
  16. Erel

    Erel Administrator Staff Member Licensed User

    Yes, you can combine both solutions. Though you will not gain too much by combining them. If you need to add a push feature then add a push feature.

    I think that for most uses cases an interval of one minutes (+-) should be good enough. Remember that it is expected to work offline as well.
     
    int21h and luke2012 like this.
  17. wl

    wl Well-Known Member Licensed User

    Erel,

    It would be nice to have both of them combined (out-of-the-box) (also including heartbeat etc...) so that we should not combine them ourselves... :)
     
    int21h likes this.
  18. Erel

    Erel Administrator Staff Member Licensed User

    It will not happen as the additional complexity will be higher than the benefit. At least in the general case.
     
    int21h likes this.
  19. wl

    wl Well-Known Member Licensed User

    Hi Erel .. it would save me (and possibly some others) a lot of time :)
     
    int21h likes this.
  20. johndb

    johndb Active Member Licensed User

    This is a very useful framework for cloud data synchronization. I do have a couple of questions:
    1. Is the time stamping "local" or "UTZ (GMT)" based? In other words, will it sync correctly for all devices located in various time zones?
    2. Given a scenario where cloud synchronization will be defined as "subscription" based, is there a way to inhibit all cloud syncing and only use the local database until let's say a flag is set to allow cloud synchronization?
    Thanks in advance for your assistance.

    John
     
    int21h, fredo and pesquera like this.
Loading...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice