B4J Question [Server][Solved] Stream JSON to HTTP server response

bdunkleysmith

bdunkleysmith

Active Member
Licensed User
Longtime User
I am trying to replicate the behavior of a server which responds to a http call with an ongoing JSON response and which via the assistance gained in this post, I have developed a client to consume that JSON data.

Within my server main module JSON is generated periodically using:

B4X: 
    m = CreateMap("type":"clock","clock":gameClock,"shotClock":shotClock)
    jg.Initialize(m)

My current handler code is:

B4X: 
Sub Handle(req As ServletRequest, resp As ServletResponse)
    resp.ContentType = "application/json"
    resp.Write($"{"type":"connection","status":"CONNECTED"}"$)

    resp.Outputstream ??????????
    
End Sub

The initial JSON response is used by the client to confirm it has a connection to the server and then as the JSON is updated periodically in the main module, I require it to be "piped" to the handler response stream without requiring a new request.

But I am struggling to understand how I create a stream from the JSON (jg) and pipe it to the response (resp.Outputstream) each time the JSON is updated.

Examples of similar server functionality I can find on the Forum use websockets, however I want to replicate the other server functionality using a HTTP call and so I appreciate any pointers in the right direction.
 
Sort by date Sort by votes
bdunkleysmith

bdunkleysmith

Active Member
Licensed User
Longtime User
This is what I expect to see from the response in my browser:

JSON: 
{"type":"connection","status":"CONNECTED"}{"shotClock":"24","clock":"9:56","type":"clock"}{"shotClock":"23","clock":"9:55","type":"clock"}{"shotClock":"22","clock":"9:54","type":"clock"}{"shotClock":"21","clock":"9:53","type":"clock"}{"shotClock":"20","clock":"9:52","type":"clock"}

and each time the clock data is updated, another dataset ({"shotClock":"ss","clock":"m:ss","type":"clock"}) is appended.

My problem is not in generating the JSON string, but appending it to the handler outputstream.
 
Upvote 0
aeric

aeric

Expert
Licensed User
Longtime User
Can you use a list and append a new map into the list? You will need to modify the B4A client to read the json as List like the following format.

JSON: 
[
    {"type":"connection","status":"CONNECTED"},
    {"shotClock":"24","clock":"9:56","type":"clock"},
    {"shotClock":"23","clock":"9:55","type":"clock"},
    {"shotClock":"22","clock":"9:54","type":"clock"},
    {"shotClock":"21","clock":"9:53","type":"clock"},
    {"shotClock":"20","clock":"9:52","type":"clock"}
]
B4X: 
Dim L as List
L.Initialize

Dim C As Map = CreateMap("type":"connection","status":"CONNECTED")
L.Add(C)
' For Loop to add new map of data
Dim M As Map = CreateMap("type":"clock","clock":gameClock,"shotClock":shotClock)
L.Add(M)
' End For Loop

Dim jg As JSONGenerator    
jg.Initialize2(L)
resp.ContentType = "application/json"
resp.Write(jg.ToString)
 
Upvote 0
bdunkleysmith

bdunkleysmith

Active Member
Licensed User
Longtime User
Thanks for your input @aeric, but I do not want to append each dataset and then resend the whole dataset. I am just trying to replicate the way an existing server (over which I have no control) responds with each subsequent dataset sent alone so I can use the client I have already developed.
 
Upvote 0
bdunkleysmith

bdunkleysmith

Active Member
Licensed User
Longtime User
Sorry @aeric if I have confused you, but no, this server I'm developing is not consuming a response from another server. Any reference to another server is to explain that I am trying to replicate the way another server, over which I have no control, responds to a HTTP call.

The server which is the subject of this question has an established data source which is used to create the JSON describing each successive dataset - that's not a problem for me.

My question is how can can I "pipe" that JSON each time it is periodically created to the outputstream of the handler response once the initial HTTP call from the client has triggered the handler.

This thread provides some background to the nature of the server response to which I am trying to replicate.
 
Upvote 0
OliverA

OliverA

Expert
Licensed User
Longtime User
Just use resp.Write in a loop? And use StartMessageLoop in Handle.

B4X: 
Sub Handle(req As ServletRequest, resp As ServletResponse)
    resp.ContentType = "application/json"
    resp.Write($"{"type":"connection","status":"CONNECTED"}"$)

    WriteOutput (resp)
    StartMessageLoop
    
End Sub

Sub WriteOutput(resp as ServletResponse)
   
    While True
        'create your json here
        resp.Write(TheJsonData)
        Sleep(1000) ' Let's take a short break
     Loop
     StopMessageLoop ' Call this whenever you are done and want to break the connection with the client
End Sub
Note: Could contain typos, but should convey the overall idea
Note2: It's HTTP, so connection breaks may occur
 
Upvote 0
bdunkleysmith

bdunkleysmith

Active Member
Licensed User
Longtime User
Thanks for your suggestion @OliverA.

It's still not working, but I'll flesh out your overall idea tomorrow when a clear head after some sleep might help.

I think I need to do more reading/research into the program flow of non-UI applications and the use of StartMessageLoop. I already have a StartMessageLoop in the AppStart sub and so from your comment I need another in the handler too.
 
Upvote 0
bdunkleysmith

bdunkleysmith

Active Member
Licensed User
Longtime User
The code to generate the JSON as the scoreboard clock data updates in Main is:
Main: 
Sub clk_Tick
    m = CreateMap("type":"clock","clock":gameClock,"shotClock":shotClock)
    jg.Initialize(m)
    update = True
End Sub

Based on @OliverA's suggestion I tried this code:
Handler: 
Sub Handle(req As ServletRequest, resp As ServletResponse)
'    Main.id = req.GetParameter("id")
    resp.ContentType = "application/json"
    resp.Write($"{"type":"connection","status":"CONNECTED"}"$)

    WriteOutput(resp)
    
    StartMessageLoop   

End Sub

Sub WriteOutput(resp As ServletResponse)
    Do While True
        'create your json here
        If Main.update = True Then           
            resp.Write(Main.jg.ToString)
            Log("Writing . . . " & Main.jg.ToString)
            Main.update = False
        End If
        Log("About to sleep . . .")
        Sleep(300) ' Let's take a short break
        Log("Looping . . .")
    Loop
'    StopMessageLoop ' Call this whenever you are done and want to break the connection with the client
End Sub

The resulting log when the handler is triggered by the HTTP call is:
Log: 
Server started
Update DuckDNS: OK
Writing . . . {"shotClock":"18","clock":"9:50","type":"clock"}
About to sleep . . .
Looping . . .
About to sleep . . .
Looping . . .
About to sleep . . .
Looping . . .
About to sleep . . .
Looping . . .
Writing . . . {"shotClock":"17","clock":"9:49","type":"clock"}
About to sleep . . .
Looping . . .
About to sleep . . .
Looping . . .
About to sleep . . .
Looping . . .
Writing . . . {"shotClock":"16","clock":"9:48","type":"clock"}
About to sleep . . .
Looping . . .
About to sleep . . .
Looping . . .
About to sleep . . .
Looping . . .
Writing . . . {"shotClock":"15","clock":"9:47","type":"clock"}
About to sleep . . .
Looping . . .
About to sleep . . .
Looping . . .
About to sleep . . .

However resp is not being returned from the WriteOutput sub to the Handle sub because the response is not seen in the browser making the request.

So I tried this code:
Handler: 
Sub Handle(req As ServletRequest, resp As ServletResponse)
    resp.ContentType = "application/json"
    resp.Write($"{"type":"connection","status":"CONNECTED"}"$)

    Do While True
        If Main.update = True Then
            resp.Write(Main.jg.ToString)
            Log("Writing . . . " & Main.jg.ToString)
            Main.update = False
        End If
        Log("About to sleep . . .")
        Sleep(300)
        Log("Looping . . .")
    Loop
    
    StartMessageLoop    
End Sub

and the resulting log when the handler is triggered by the HTTP call is:
Log: 
Server started
Update DuckDNS: OK
Writing . . . {"shotClock":"13","clock":"9:45","type":"clock"}
About to sleep . . .

I see
Response: 
{"type":"connection","status":"CONNECTED"}{"shotClock":"13","clock":"9:45","type":"clock"}
in the browser making the request and so the initial JSON and first clock dataset is sent, but it seems to be hanging at
B4X: 
Sleep(300)
and so perhaps Sleep(x) is not allowable in a Handle sub.

More head scratching to be done, but my gut feeling is that I need to create a stream for resp.Outputsream to consume so I can just feed each subsequent JSON as it becomes available rather than a looping approach.
 
Upvote 0
bdunkleysmith

bdunkleysmith

Active Member
Licensed User
Longtime User
@Erel I have taken on board your pointer to use Wait For in the server handler, but I'm still encountering a "road block".

As indicated above, the code to generate the JSON as the scoreboard clock data updates in Main is:
Main: 
Sub clk_Tick
    m = CreateMap("type":"clock","clock":gameClock,"shotClock":shotClock)
    jg.Initialize(m)
    update = True
    Log("clk_Tick : " & jg.ToString)
End Sub

I have added a timer to the handler code so I can use the Timer_Tick event in conjunction with Wait For (because I can't see another suitable event):
Handler: 
Sub Handle(req As ServletRequest, resp As ServletResponse)
    resp.ContentType = "application/json"
    resp.Write($"{"type":"connection","status":"CONNECTED"}"$)
    
    StartTimer

    Do While True
        If Main.update = True Then           
            resp.Write(Main.jg.ToString)
            Log("Writing . . . " & Main.jg.ToString)
            Main.update = False
        End If
        Log("Waiting . . .")
        Wait For trg_Tick
        Log("Looping . . .")
    Loop
    
    StartMessageLoop   
End Sub

Sub StartTimer
    Public trg As Timer
    trg.Initialize("trg", 300)
    trg.enabled = True
    Log("Timer started . . ")
End Sub

Sub trg_Tick
    Log("trg_Tick")
End Sub

However while as per the log below, the timer (trg) is started, the relevant timer event (trg_Tick) does not appear to fire and so the loop just sits waiting for it:
Log: 
Server started
Update DuckDNS: OK
clk_Tick : {"shotClock":"24","clock":"9:56","type":"clock"}
clk_Tick : {"shotClock":"23","clock":"9:55","type":"clock"}
clk_Tick : {"shotClock":"22","clock":"9:54","type":"clock"}
clk_Tick : {"shotClock":"21","clock":"9:53","type":"clock"}
clk_Tick : {"shotClock":"20","clock":"9:52","type":"clock"}
clk_Tick : {"shotClock":"19","clock":"9:51","type":"clock"}
clk_Tick : {"shotClock":"18","clock":"9:50","type":"clock"}
clk_Tick : {"shotClock":"17","clock":"9:49","type":"clock"}
clk_Tick : {"shotClock":"16","clock":"9:48","type":"clock"}
Request received by handler
Timer started . .
Writing . . . {"shotClock":"16","clock":"9:48","type":"clock"}
Waiting . . .
clk_Tick : {"shotClock":"15","clock":"9:47","type":"clock"}
clk_Tick : {"shotClock":"14","clock":"9:46","type":"clock"}
clk_Tick : {"shotClock":"13","clock":"9:45","type":"clock"}
clk_Tick : {"shotClock":"12","clock":"9:44","type":"clock"}
clk_Tick : {"shotClock":"11","clock":"9:43","type":"clock"}
clk_Tick : {"shotClock":"10","clock":"9:42","type":"clock"}
clk_Tick : {"shotClock":"9","clock":"9:41","type":"clock"}
clk_Tick : {"shotClock":"8","clock":"9:40","type":"clock"}
clk_Tick : {"shotClock":"7","clock":"9:39","type":"clock"}

Moving the "Do while True" loop to a separate sub and calling that from the handle sub results in different issue:
Handler: 
Sub WriteOutput(resp As ServletResponse)
    Do While True
        If Main.update = True Then
            resp.Write(Main.jg.ToString)
            Log("Writing . . . " & Main.jg.ToString)
            Main.update = False
        End If
        Log("Waiting . . .")
        Wait For trg_Tick
        Log("Looping . . .")
    Loop
End Sub

As per the log below, it seems timer event (trg_Tick) is firing because we see "Waiting . . ." and then "Looping . . .", but while it says "Writing . . . {"shotClock":"14","clock":"9:46","type":"clock"}", that is not actually sent as a response, I assume because the sub is not exited.
Log: 
Server started
Update DuckDNS: OK
clk_Tick : {"shotClock":"24","clock":"9:56","type":"clock"}
clk_Tick : {"shotClock":"23","clock":"9:55","type":"clock"}
clk_Tick : {"shotClock":"22","clock":"9:54","type":"clock"}
clk_Tick : {"shotClock":"21","clock":"9:53","type":"clock"}
clk_Tick : {"shotClock":"20","clock":"9:52","type":"clock"}
clk_Tick : {"shotClock":"19","clock":"9:51","type":"clock"}
clk_Tick : {"shotClock":"18","clock":"9:50","type":"clock"}
clk_Tick : {"shotClock":"17","clock":"9:49","type":"clock"}
clk_Tick : {"shotClock":"16","clock":"9:48","type":"clock"}
clk_Tick : {"shotClock":"15","clock":"9:47","type":"clock"}
clk_Tick : {"shotClock":"14","clock":"9:46","type":"clock"}
Request received by handler
Timer started . .
Writing . . . {"shotClock":"14","clock":"9:46","type":"clock"}
Waiting . . .
Looping . . .
Waiting . . .
clk_Tick : {"shotClock":"13","clock":"9:45","type":"clock"}
Looping . . .
Writing . . . {"shotClock":"13","clock":"9:45","type":"clock"}
Waiting . . .
Looping . . .
Waiting . . .
Looping . . .
Waiting . . .
clk_Tick : {"shotClock":"12","clock":"9:44","type":"clock"}
Looping . . .
Writing . . . {"shotClock":"12","clock":"9:44","type":"clock"}
Waiting . . .
Looping . . .
Waiting . . .
Looping . . .
Waiting . . .
clk_Tick : {"shotClock":"11","clock":"9:43","type":"clock"}
Looping . . .
Writing . . . {"shotClock":"11","clock":"9:43","type":"clock"}
Waiting . . .
Looping . . .
Waiting . . .
Looping . . .

Often when I reach these "road blocks" I find I'm using an incorrect method to achieve the desired functionality and perhaps that's again the case, but any assistance will be gratefully received.
 
Upvote 0
bdunkleysmith

bdunkleysmith

Active Member
Licensed User
Longtime User
I think I am defeated!

In this post it says
The Handle sub cannot be a resumable sub itself.
Click to expand...

Therefore I now realise that the actual Handle sub of a server handler cannot be a resumable sub and so I can see no way to have a loop within the Handle sub so it pauses to gather the updated data from Main to then be written to the response:
Handle: 
resp.Write(Main.jg.ToString)
All examples I have found which move such writing of the response out of the Handle sub to a separate sub involve just a single write to the response and then Handle completes on return from the sub. They do not allow ongoing/periodic writes to the response.

It now seems impossible to replicate the server functionality I require in that the server response to a HTTP call is an initial JSON response to which further JSON data is sent as updates occur in server (scoreboard) data.
 
Upvote 0
OliverA

OliverA

Expert
Licensed User
Longtime User
Use this
B4X: 
Sub Flush(resp As ServletResponse)
    Dim jo As JavaObject
    jo = resp
    jo = jo.RunMethod("getWriter", Null)
    jo.RunMethod("flush", Null)
End Sub
Call Flush after every resp.Write that you want to deliver to the client
B4X: 
resp.Write("Some string data")
Flush(resp)
Resources used:
1) B4J's Git repository: https://github.com/AnywhereSoftware...nywheresoftware/b4j/object/JServlet.java#L359
2) Java documentation on ServletResponse and PrintWriter:

Plus some experience in from the old days of needing to flush your data before a file is saved/a printer prints
 
Upvote 0
bdunkleysmith

bdunkleysmith

Active Member
Licensed User
Longtime User
@Erel I understand a Websocket solution may be trivial, but I wanted to create a server which responds with JSON, as does the server I described in this Consuming streaming data from HTTP call thread and with your great assistance I was able to create a client to consume its ongoing response when you created a custom class detailled in this Consuming http streams tutorial where you said:
There are some cases where the server keeps the connection open and streams data to the client.
Click to expand...

This is the server behaviour I was trying to replicate so I would be able to just point my existing client to my own server in lieu of the other existing server by replicating its performance.

Thank you @OliverA for your additional input, including references. I will study that to see if I can use that in a solution.
 
Upvote 0
Erel

Erel

B4X founder
Staff member
Licensed User
Longtime User
bdunkleysmith said:
This is the server behaviour I was trying to replicate so I would be able to just point my existing client to my own server in lieu of the other existing server by replicating its performance.
Click to expand...
WebSockets is an extension to the http protocol that was designed exactly for such use cases where you want to break the request - response pattern. There is really no reason to "fight" http and make it act like a limited WebSocket.
 
Upvote 0
bdunkleysmith

bdunkleysmith

Active Member
Licensed User
Longtime User
Thank you @OliverA for sharing your experience from the "old days". It has enabled me to remove the road block!

For completeness in case there's interest from others, this is the handler code I finished up with which incorporates the all important "flush":
Handler: 
Handler class
Sub Class_Globals
   
End Sub

'Initializes the object. You can add parameters to this method if needed.
Public Sub Initialize

End Sub

Sub Handle(req As ServletRequest, resp As ServletResponse)
    Log("Request received by handler")
    resp.ContentType = "application/json"
    resp.Write($"{"type":"connection","status":"CONNECTED"}"$ & Chr(10))  
    Flush(resp)
    StartTimer
    WriteOutput(resp)
    StartMessageLoop  
End Sub

Sub WriteOutput(resp As ServletResponse)
    Do While True
        If Main.update = True Then
            resp.Write(Main.jg.ToString & Chr(10))
            Flush(resp)
            Log("Writing . . . " & Main.jg.ToString)
            Main.update = False
        End If
        wait for trg_Tick
    Loop
End Sub

Sub StartTimer
    Public trg As Timer
    trg.Initialize("trg", 100)
    trg.enabled = True
    Log("Timer started . . ")
End Sub

Sub Flush(resp As ServletResponse)
    Dim jo As JavaObject
    jo = resp
    jo = jo.RunMethod("getWriter", Null)
    jo.RunMethod("flush", Null)
End Sub

Here is a video in which the first part shows how the data is is received in a browser and the second part shows my scoreboard client with the game and shot clock displays updating as the data is received.

I am very happy that this has enabled me to break the local network constraint of my scoreboard client so it will work over the internet, thus allowing remote production of live sport video streams incorporating an overlay of the scoreboard.

But indeed it has been a "fight" @Erel using http and so I may look at a (more robust) WebSocket solution, even though that would require redevelopment of the communication component of my client.

Thanks to all who provided input, once again proving how great this community is.
 
Last edited:
Upvote 0
You must log in or register to reply here.

Similar Threads

aeric
  • Article
Share My Creation [Project Template] [Web] API Server 2
2 3
Replies
54
Views
11K
Xela
Xela
Erel
  • Article
B4A Library [B4X] FTP Server implemented with Socket and AsyncStreams
2 3
Replies
43
Views
39K
OliverA
OliverA
Erel
  • Locked
  • Article
Android Tutorial [B4X] Network + AsyncStreams + B4XSerializator
2
Replies
21
Views
60K
Erel
Erel
Erel
  • Article
B4J Tutorial [Server] Online Json Tree Example
Replies
16
Views
20K
Phayao
Phayao
Erel
  • Locked
  • Article
B4J Tutorial [Server] Building web servers with B4J
2 3 4
Replies
75
Views
90K
Erel
Erel
Top