Bug? Broken parsing of json when updating b4xlibs

[Sandman]

Sorry, I couldn't figure out a more descriptive headline.

I have a project that uses a several b4xlibs. One - and only one - of these handle communication to my server backend, using json.

Somewhat often, when I update one of the b4xlibs, the json parsing breaks and causes an exception:

org.json.JSONException: Unterminated object at character 2702 of {
...snipped json...

The json is perfect, I have verified this numerous times. And I also sniff my own development traffic to ensure the quality. And yet the parser crashes because it thinks the last character is missing from the json, making it invalid.

Every time this happens, I can quickly solve it by doing a Clean Project in my main project, the one that imports the b4xlibs and is actually run.

It's a known issue for me now, and quickly solved each time it happens. Still enough to warrant a post to the forum, I thought.

(Related: Oh boy, how I would love a #RefreshLibrariesBeforeCompile and also #CleanProjectBeforeCompile so I wouldn't have to do it manually. Each. Time. I. Compile.)

 

[Erel]

More information please. Where does this error happen? At what stage?
Can you post the full error message?

 

[Sandman]

Ok, now it happened again. When trying to narrow down the problem to give a better description I realized that I hadn't tried rerunning the main app over and over, without changes between runs. (My workflow is that I edit a project that is compiled to a b4xlib, then I go to the main app, refresh libs and run it. Rinse and repeat.)

So I tried running the main app, killing it via IDE, rerunning it. Just to see what would happen. Turns out that my initial report was flawed and I probably made a bad assumption when I connected it to the libraries.

I ran the app 20 times with no changes between the runs. 15 of them worked just fine, but five times the json parser crashed due to it not seeing the last character of the json string (making it consider the object unterminated). (*)


For completeness I did this test on a couple of other ways also. Here are all tests:

B4A Debug 20 runs: 15 successful runs, 5 json exceptions
B4A Release 20 runs: 14 successful runs, 6 json exceptions
B4J Debug 20 runs: 20 successful runs, 0 errors of any kind

So no difference between debug/release on B4A, and no errors at all in B4J.

At this point I realized I could check the generated source to see if there were any changes. So I cleaned the project in B4A and compiled a debug version. Worked fine, so I made a copy of the Objects directory. Then I cleaned/ran a couple of times until I got a fail and made a second copy of the Objects directory.

Then I compared the two directories using Meld and found that there are changes in two source files:
(And also a couple of binary files, list available upon request)

• httputils2service.java
• httputils2service_subs_0.java

The change seems to always be that the first digit in the big number is different between the two versions. Here is an example:

anywheresoftware.b4a.keywords.Common.LogImpl("319857414","HttpUtils2Service: job completed multiple times - "+BA.NumberToString(_taskid),0);

anywheresoftware.b4a.keywords.Common.LogImpl("519857414","HttpUtils2Service: job completed multiple times - "+BA.NumberToString(_taskid),0);

And another example:

httputils2service.mostCurrent.__c.runVoidMethod ("LogImpl","319857414",RemoteObject.concat(RemoteObject.createImmutable("HttpUtils2Service: job completed multiple times - "),_taskid),0);

httputils2service.mostCurrent.__c.runVoidMethod ("LogImpl","519857414",RemoteObject.concat(RemoteObject.createImmutable("HttpUtils2Service: job completed multiple times - "),_taskid),0);

Does this mean anything? Does it help figuring out what the problem is?



As for your request: There isn't really a fuller error message to post, except that it contains lots of internal, unrelated log messages and the truncated json message. Let me know if you want the full json and I'll send it by email to you.

And here's the sub where it all fails:

' This will parse the response according to structure 1
Private Sub ParseOne(inData As String) As Map

    If SYS.DevelopmentMode Then SYS.libHELPERS.DevelopmentDumper("ParseOne1", inData)

    Dim response_map As Map = CreateMap()
    Dim key As String = "response"

    Try
        
        ' Put the response in a map
        Dim parser As JSONParser
        parser.Initialize(inData)
        Dim main_map As Map = parser.NextObject
        
        If main_map.ContainsKey(key) Then
            response_map = main_map.Get(key)
        Else
            SYS.libLOG.Add(libID, SYS.libLOG.LEVEL_DEBUG, $"ParseOne: '${key}' not found"$)
        End If
        success = PARSE_OK
        
    Catch

        SYS.libLOG.Add(libID, SYS.libLOG.LEVEL_ERROR, $"ParseOne: ${LastException.Message}"$)
        success = PARSE_ERROR
        Log(LastException.Message)

    End Try

    Return response_map
    
End Sub

* This result was so weird that I actually went into my parser and network code to verify that I hadn't added a chaos monkey there during testing to make the parsing fail now and then. Nope, nothing like it. So it seems the json parser considers my json bad about 25% of the time.

 

[Erel]

The number in the LogImpl line is an encoded reference to the B4J line. It will not have any effect on the program behavior.

 

[Sandman]

The number in the LogImpl line is an encoded reference to the B4J line. It will not have any effect on the program behavior.

You mean the B4A line, right?

Just curious: is it meant to change between compilations if the source is unchanged?

 

[Erel]

You mean the B4A line, right?

Yes, it is the same feature in B4A, B4i and B4J.

Just curious: is it meant to change between compilations if the source is unchanged?

It can change every compilation.

 

[agraham]

Maybe you have already thought of/done this but to narrow it down how about putting a Log of the length on inData and maybe also the last character value immediately before calling NextObject.

 

[Sandman]

Maybe you have already thought of/done this but to narrow it down how about putting a Log of the length on inData and maybe also the last character value immediately before calling NextObject.

That's a good idea, I'll do that - thanks!

In B4J I also added logging that saved the json for each and every call to my api, and the json was always perfect. (Which the network sniffer already told me, but...) But then again, I'm not seeing these issues on B4J so no surprise there. I'll do your suggestion and depending on that I might start saving the json responses on Android too.

At some point I will also start moving the project over to B4i, which also might provide some information on this, I suppose.