Android Question Question about HTTPUtils2

[androb]

I have been able to build a quite complicated application with b4a, but am starting to have all kinds of weird problems.

The app is for recording data. Using httputils2, I connect to my php webservice which queries mysql and returns results as JSON, the b4a app stores this into a sqllite db on the device. This all seems to work fine.

Users change data and then click "submit", which creates a JSON string from the sqllite db and uses PostString to send it to the webservice.

99% of the time this works flawlessly. However after running the app sitewide with 25 users, I have noticed an obscure issue, which I am trying to isolate.

The issue is that the data gets submitted twice. Sometimes 2 minutes apart, sometimes 10 minutes afterwards. I know this because mysql timestamps when the webservices receives the posted data. In both cases it is the exact same dataset, and never more than submitted twice. This sounds like a logic error. But I only call the PostString a single time.

There is only a single place the data gets submitted, and this is only called a single place in the b4a app when the user clicks the button. I know users are not clicking the button multiple times.

I am wondering if this could be caused by network issues. Is there anything built into httputils2 that will try and resend data? Could the httpjob be being submitted twice? My app is not programmed to retry, only show an error if it fails.

Could there be a conflict/bug because I run consecutive httpjobs to fetch and post json data?

I am finding a whole range of odd, inconsistent issues, which are almost impossible to reproduce. Often it is errors of views not being initialised (but not always). There is just no logic to the problems. I am finding debugging on android intensely frustrating.

 

[LucaMs]

To know if the problem can be httputils2 you will have to wait for the response of Erel.

However, if I have not misunderstood, in the last part of your post you say:
"Often it is errors of views not being initialised"
and this certainly can not depend on httputils2.

You are certain that your users do not press twice the button for submit; have you put a Log in the event routine? Have you disabled the button?

 

[Erel]

Assuming that it is not a programming error then it might be a network issue. HttpClient does try to resend the data if there is a network error. However it should happen more or less immediately after it failed.

If this is really the case then you can change req.InitializePost2 to req.InitializePost to prevent this (InputStream.InitializeFromBytesArray will help you to create the inputstream).

 

[nwhitfield]

You could also try handling this in the server side code; I do that for one of my apps and websites, mostly because I started noticing that people were creating duplicate messages when using the site on iPad, where for some reason (perhaps the user reloading pages, or the browser doing it when they switch apps) the same info was being POSTed twice. So, now when I receive data for a message, I create an md5sum of it.

I then check to see if there's a message already existing with the same sender, recipient and md5sum, within a specific time period; for web browsers, I silently discard duplicates, and for calls via the API, I send back a 'duplicate message' response.

It doesn't fix the underlying problem, but it does ensure you don't end up with duplicates in the database.

 

[KMatle]

To be honest. That combination of "weird" things looks like an issue of the app + webserver. If there are delays of 10 mins and/or inconsistent states of the tables, there must be something wrong in the logic. Httputils2 is "too simple" to cause such effects. This sounds like a synchronisation problem between SQLLite & your web services.

I would log anything (for a time) with timestamps. So you are getting closer to the problem. Check if you can set columns to "unique" to prevent double entries. Store additional data in it (usernames, etc.) to get behind it.

And: Do not trust any user (log when the button is pressed).

 

[androb]

Thank you for the replies.

I took the chance to add some code to my webservice to see how many actual requests are made from the client, and found it is sending the POST multiple times.

According to the api logging timestamps, these are some seconds apart, so unlikely it is a user mashing the button that is causing it.

After the submit is successfull the screen is closed, so even button mashing wouldn't explain the minutes delays in some cases.

05/09/2014 14:20.27  SELECT * FROM submissions WHERE session_id='7067cb22e16a49502dd12eb2812e58565409a6dd7aa700.06607183'

05/09/2014 14:20.27  checking if submission exists (7067cb22e16a49502dd12eb2812e58565409a6dd7aa700.06607183) result - rc=0
05/09/2014 14:20.28  #### submit_insert:INSERT INTO submissions (staff_id,group_name,group_id,period,day_of_week,subject_id,group_type,date,dt_submitted,session_id,schema_id) VALUES (22,'6SC',183,'P4',5,5,'form','2014-9-5',now(),'7067cb22e16a49502dd12eb2812e58565409a6dd7aa700.06607183',0)

05/09/2014 14:20.52  SC**** SUBMIT called:
05/09/2014 14:20.52  SELECT * FROM submissions WHERE session_id='7067cb22e16a49502dd12eb2812e58565409a6dd7aa700.06607183'
05/09/2014 14:20.52  checking if submission exists (7067cb22e16a49502dd12eb2812e58565409a6dd7aa700.06607183) result - rc=1
05/09/2014 14:20.52  ############## ALERT - DUPLICATE session ID posted (is_edit=no) - device_id:R32D30446WP, settings={
  
05/09/2014 14:23.39  SELECT * FROM submissions WHERE session_id='7067cb22e16a49502dd12eb2812e58565409a6dd7aa700.06607183'
05/09/2014 14:23.39  checking if submission exists (7067cb22e16a49502dd12eb2812e58565409a6dd7aa700.06607183) result - rc=1
05/09/2014 14:23.39  ############## ALERT - DUPLICATE session ID posted (is_edit=no) - device_id:R32D30446WP, settings={
  
05/09/2014 14:25.58  SELECT * FROM submissions WHERE session_id='7067cb22e16a49502dd12eb2812e58565409a6dd7aa700.06607183'
05/09/2014 14:25.58  checking if submission exists (7067cb22e16a49502dd12eb2812e58565409a6dd7aa700.06607183) result - rc=1
05/09/2014 14:25.58  ############## ALERT - DUPLICATE session ID posted (is_edit=no) - device_id:R32D30446WP, settings={

However, I can see the whole POST processing returns the data to the client within 5 seconds at maximum. So I cannot see that lag is an issue.

Are these multiple POSTS expected behaviour?

I call the function with the POST like this:

    gen.Initialize(DBUtils.ExecuteJSON(SQL1, "SELECT * from dataset", Null, 0,Array As String(DBUtils.DB_INTEGER,DBUtils.DB_INTEGER, DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_TEXT,DBUtils.DB_INTEGER,DBUtils.DB_TEXT,DBUtils.DB_TEXT,DBUtils.DB_INTEGER,DBUtils.DB_TEXT,DBUtils.DB_INTEGER, DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER, DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER, DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER, DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,    DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER, DBUtils.DB_INTEGER,DBUtils.DB_TEXT,DBUtils.DB_TEXT,DBUtils.DB_TEXT,DBUtils.DB_TEXT, DBUtils.DB_TEXT,DBUtils.DB_TEXT,DBUtils.DB_INTEGER,DBUtils.DB_INTEGER)))
      
  
      JSONString_dataset = gen.ToPrettyString(4)
    'Msgbox(JSONString, "")
  
    '--- extract settings from localdb
    Dim gen As JSONGenerator 'Requires a reference to the JSON library.
    gen.Initialize(DBUtils.ExecuteJSON(SQL1, "SELECT * from settings", Null, 0, Array As String(DBUtils.DB_TEXT,DBUtils.DB_TEXT)))
    JSONString_settings = gen.ToPrettyString(4)
      
      json_submit.Initialize("json_submit", Me)
    ProgressDialogShow("-- Submitting Data --")
  
    log_entry("post_data: JSONString_dataset: " & JSONString_settings)
  
    json_submit.PostString(server_path & "fuseapi.php","action=submit&device_id="&GetDeviceId&"&dataset=" & JSONString_dataset & "&settings=" & JSONString_settings)
      
    ' after successful post, we show message and then clear localdb data

At other times I use a JSON request via httputils to retrieve information to populate a DB on the device. However, for those, I use this code:

request_str = server_path & "fuseapi.php?action=setup&edit_data&session_id="&r_session_key&"&group_type="&r_group_type&"&group_id=" & r_group_id
Log(request_str)


json_download.Initialize("json_download", Me)
json_download.Download(request_str)

I have never experienced multiple requests on those requests.. So maybe the fault lies somehow with PostString?

The post function is called by a user selecting from the android menu (the three dots) which incidentlly, seems to dissaper from the top of the screen and move to the bottom, or sometimes both (this is on nexus 10). Then I showprogressdialog when the POST is sent, and parse the response in JobDone, which hides the progress.

I was hoping that ShowProgress might block the UI, but it it either httputils doing it (which seems unlikely as the problem happens about 1 in 25 times a user submits data). Or perhaps the menu item is being sent multiple "clicks" which triggers multiple requests?

 

[KMatle]

No. The UI isn't locked then.

Can't you test that thing? (I mean check out how one can send multiple posts f.e. by clicking an menue or a button?).

I generally lock all buttons, timers before sending/posting data to prevent such situations.

 

[androb]

I think I have tracked it down.

The android menu button always shows (with the submit option) on my other layout that isn't the data entry screen, so some users are clicking it by mistake. Maybe they think the data still needs to be submitted .

I have removed the android menu item and replaced with a quick action menu, which means I can only have it showing on the data entry screen -although the three dot menu still shows for some reason, even tho it has no entries.