B4J Question Error handling large data with AsyncStreamsText

[bdunkleysmith]

I have a client which consumes XML data sent from a server. On connection the server sends a large complete package of data and then periodically sends smaller incremental packages of data.

I use the AsyncStreamsText class to acquire the data:

Sub AStreams_NewData (Buffer() As Byte)
    Dim msg As String
    msg = BytesToString(Buffer, 0, Buffer.Length, "UTF8")
    Log (msg)
    WriteToFile(msg)
    FullGameRxd(msg)
End Sub

and then the xml2map class so I can use the JSON parser to extract the required data:

Sub FullGameRxd(HTTPText As String)
    Dim xm As Xml2Map
    xm.Initialize
    ParsedData = xm.Parse(HTTPText)        ' Parse HTTP text (xml) via Xml2Map
    Dim jg As JSONGenerator
    jg.Initialize(ParsedData)            ' Generate JSON from map
    Dim parser As JSONParser
    parser.Initialize(jg.ToPrettyString(4))        ' Parse JSON to extract all values
    Dim parser As JSONParser
    parser.Initialize(jg.ToPrettyString(4))
    Dim root As Map = parser.NextObject

While this method is working well for the small incremental packets of data, the following error is thrown by the initial large package of data:

[Fatal Error] :2:8153: XML document structures must start and end within the same entity.
xml2map._parse2 (java line: 173)
org.xml.sax.SAXParseException; lineNumber: 2; columnNumber: 8153; XML document structures must start and end within the same entity.
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
at anywheresoftware.b4a.objects.SaxParser.parse(SaxParser.java:80)
at anywheresoftware.b4a.objects.SaxParser.Parse(SaxParser.java:73)
at b4j.example.xml2map._parse2(xml2map.java:173)
at b4j.example.xml2map._parse(xml2map.java:161)
at b4j.example.main._fullgamerxd(main.java:622)
at b4j.example.main._astreams_newdata(main.java:338)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:90)
at anywheresoftware.b4a.BA$2.run(BA.java:229)
at com.sun.javafx.application.PlatformImpl.lambda$null$173(PlatformImpl.java:295)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.application.PlatformImpl.lambda$runLater$174(PlatformImpl.java:294)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$null$148(WinApplication.java:191)
at java.lang.Thread.run(Thread.java:745)

Writing the acquired data to a file and examining it in Notepad++ it appears that after each 8193 characters in the file there is a LF character which is no doubt the cause of the subsequent processing error. As the incremental packages are always less than that, typically 2000 to 3000 characters, there is no problem processing that received data.

So is it likely that the server, over which I have no control, is embedding the unwanted LF characters and if so, what's the easiest method to strip the unwanted (approximately 18) LF characters out of the data?

Of do I need to do something in the AsyncStreamsText or xml2map class to be able to handle the superfluous LF characters?

Thanks,

Bryon

 

[bdunkleysmith]

Well I thought I was using the AsyncStreamsText class, but now I see that while I have the AsyncStreamsText module loaded, my code is using the AsyncStreams!

So I have now implemented data gathering using the AsyncStreamsText class:

Sub Process_Globals
    Private ast As AsyncStreamsText
    Dim TCPSocket As Socket
    Dim TCPAddress As String
    Dim TCPPort As Int
    Dim TCPDelay As Int
    .
    .
End Sub

Sub btnConnect_MouseClicked (EventData As MouseEvent)
        TCPAddress = txtTCPServerIP.text
        TCPPort = 7676
        TCPDelay = 5000
        TCPSocket.Initialize("TCPSocket")
        TCPSocket.Connect(TCPAddress , TCPPort, TCPDelay)
End Sub

Sub TCPSocket_Connected (Successful As Boolean)
    If Successful Then
        ast.Initialize(Me, "astreamtext", TCPSocket.InputStream, TCPSocket.OutputStream)
        btnConnect.text = "Connected"
    Else
        Log("Error: " & LastException)
    End If
End Sub

Sub astreamtext_NewText(Text As String)
    Log (Text)
    WriteToFile(Text)
    FullGameRxd(Text)
End Sub

Sub FullGameRxd(HTTPText As String)
    Dim xm As Xml2Map
    xm.Initialize
    ParsedData = xm.Parse(HTTPText)        ' Parse HTTP text (xml) via Xml2Map
    Dim jg As JSONGenerator
    jg.Initialize(ParsedData)            ' Generate JSON from map
    Log(jg.ToPrettyString(4))
    Dim parser As JSONParser
    parser.Initialize(jg.ToPrettyString(4))        ' Parse JSON to extract all values
    Dim root As Map = parser.NextObject

and the superfluous LF characters are no longer in the data, however there is no LF or CR at the end of the XML string and so now the following error is thrown:

[Fatal Error] :1:149643: The processing instruction target matching "[xX][mM][lL]" is not allowed.
xml2map._parse2 (java line: 173)
org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 149643; The processing instruction target matching "[xX][mM][lL]" is not allowed.
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
at anywheresoftware.b4a.objects.SaxParser.parse(SaxParser.java:80)
at anywheresoftware.b4a.objects.SaxParser.Parse(SaxParser.java:73)
at b4j.example.xml2map._parse2(xml2map.java:173)
at b4j.example.xml2map._parse(xml2map.java:161)
at b4j.example.main._amendedactionrxd(main.java:114)
at b4j.example.main._astreamtext_newtext(main.java:318)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:90)
at anywheresoftware.b4a.keywords.Common.CallSub4(Common.java:480)
at anywheresoftware.b4a.keywords.Common.access$0(Common.java:460)
at anywheresoftware.b4a.keywords.Common$CallSubDelayedHelper.run(Common.java:534)
at com.sun.javafx.application.PlatformImpl.lambda$null$173(PlatformImpl.java:295)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.application.PlatformImpl.lambda$runLater$174(PlatformImpl.java:294)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)

because the end of the initial xml data package can't be distinguished from the start of the following incremental data packages, eg.:

x="171" y="264" itp="N" end="L"/></team></scorechart></NewAction></LiveStats><?xml version="1.0" encoding="utf-8"?>LF

So the error is thrown after the "xml" is hit by the parser when I'd expect there to be an EOL character of some sort after </LiveStats> which is the end of the XML data.

What am I doing wrong?

Thanks,

Bryon

 

[Erel]

The server doesn't return each message on a separate line. This means that AsyncStreamsText is not very useful here. You will need to collect the data with StringBuilder and check each time whether it contain "<?xml" and later a "</LiveStats" (or the next "<?xml")
The next step is to use substring to get the correct part that includes the message.

 

[bdunkleysmith]

Thanks for pointing me in the right direction Erel.

I'm not sure how technically correct my approach is, but here's what I've done which seems to do the job:

Sub AStreams_NewData (Buffer() As Byte)
    Dim msg As String
    msg = BytesToString(Buffer, 0, Buffer.Length, "UTF8")
    If js.InString(msg, "</LiveStats>") > 0 Then
        sb.Append(js.Left(msg, js.InString(msg, "</LiveStats>") + 12))
        Dim remainderTxt = js.Right(msg, msg.Length - (js.InString(msg, "</LiveStats>") + 12)) As String
        Dim tmpTxt = sb.ToString As String
        sb.Initialize
        sb.Append(remainderTxt)
        ProcessData(tmpTxt)
    Else
        sb.Append(msg)
    End If
End Sub

Thanks again,

Bryon