B4J Question [Server][Solved] Stream JSON to HTTP server response

bdunkleysmith

bdunkleysmith

Active Member
Licensed User
Longtime User
I am trying to replicate the behavior of a server which responds to a http call with an ongoing JSON response and which via the assistance gained in this post, I have developed a client to consume that JSON data.

Within my server main module JSON is generated periodically using:

B4X: 
    m = CreateMap("type":"clock","clock":gameClock,"shotClock":shotClock)
    jg.Initialize(m)

My current handler code is:

B4X: 
Sub Handle(req As ServletRequest, resp As ServletResponse)
    resp.ContentType = "application/json"
    resp.Write($"{"type":"connection","status":"CONNECTED"}"$)

    resp.Outputstream ??????????
    
End Sub

The initial JSON response is used by the client to confirm it has a connection to the server and then as the JSON is updated periodically in the main module, I require it to be "piped" to the handler response stream without requiring a new request.

But I am struggling to understand how I create a stream from the JSON (jg) and pipe it to the response (resp.Outputstream) each time the JSON is updated.

Examples of similar server functionality I can find on the Forum use websockets, however I want to replicate the other server functionality using a HTTP call and so I appreciate any pointers in the right direction.
 
Sort by date Sort by votes
Do you want to write the map as json string then return as server response?
 
Last edited:
Upvote 0
This is what I expect to see from the response in my browser:

JSON: 
{"type":"connection","status":"CONNECTED"}{"shotClock":"24","clock":"9:56","type":"clock"}{"shotClock":"23","clock":"9:55","type":"clock"}{"shotClock":"22","clock":"9:54","type":"clock"}{"shotClock":"21","clock":"9:53","type":"clock"}{"shotClock":"20","clock":"9:52","type":"clock"}

and each time the clock data is updated, another dataset ({"shotClock":"ss","clock":"m:ss","type":"clock"}) is appended.

My problem is not in generating the JSON string, but appending it to the handler outputstream.
 
Upvote 0
Can you use a list and append a new map into the list? You will need to modify the B4A client to read the json as List like the following format.

JSON: 
[
    {"type":"connection","status":"CONNECTED"},
    {"shotClock":"24","clock":"9:56","type":"clock"},
    {"shotClock":"23","clock":"9:55","type":"clock"},
    {"shotClock":"22","clock":"9:54","type":"clock"},
    {"shotClock":"21","clock":"9:53","type":"clock"},
    {"shotClock":"20","clock":"9:52","type":"clock"}
]
B4X: 
Dim L as List
L.Initialize

Dim C As Map = CreateMap("type":"connection","status":"CONNECTED")
L.Add(C)
' For Loop to add new map of data
Dim M As Map = CreateMap("type":"clock","clock":gameClock,"shotClock":shotClock)
L.Add(M)
' End For Loop

Dim jg As JSONGenerator    
jg.Initialize2(L)
resp.ContentType = "application/json"
resp.Write(jg.ToString)
 
Upvote 0
Thanks for your input @aeric, but I do not want to append each dataset and then resend the whole dataset. I am just trying to replicate the way an existing server (over which I have no control) responds with each subsequent dataset sent alone so I can use the client I have already developed.
 
Upvote 0
You mean you have a server that read the response from another server and then you return a new response to another client?
 
Upvote 0
bdunkleysmith said:
Examples of similar server functionality I can find on the Forum use websockets, however I want to replicate the other server functionality using a HTTP call
Click to expand...
I think I haven’t seen any sample in this forum that make use of websocket and http at the same time.
 
Upvote 0
Sorry @aeric if I have confused you, but no, this server I'm developing is not consuming a response from another server. Any reference to another server is to explain that I am trying to replicate the way another server, over which I have no control, responds to a HTTP call.

The server which is the subject of this question has an established data source which is used to create the JSON describing each successive dataset - that's not a problem for me.

My question is how can can I "pipe" that JSON each time it is periodically created to the outputstream of the handler response once the initial HTTP call from the client has triggered the handler.

This thread provides some background to the nature of the server response to which I am trying to replicate.
 
Upvote 0
Just use resp.Write in a loop? And use StartMessageLoop in Handle.

Code: 
Sub Handle(req As ServletRequest, resp As ServletResponse)
    resp.ContentType = "application/json"
    resp.Write($"{"type":"connection","status":"CONNECTED"}"$)

    WriteOutput (resp)
    StartMessageLoop
    
End Sub

Sub WriteOutput(resp as ServletResponse)
   
    While True
        'create your json here
        resp.Write(TheJsonData)
        Sleep(1000) ' Let's take a short break
     Loop
     StopMessageLoop ' Call this whenever you are done and want to break the connection with the client
End Sub
Note: Could contain typos, but should convey the overall idea
Note2: It's HTTP, so connection breaks may occur
 
Upvote 0
Thanks for your suggestion @OliverA.

It's still not working, but I'll flesh out your overall idea tomorrow when a clear head after some sleep might help.

I think I need to do more reading/research into the program flow of non-UI applications and the use of StartMessageLoop. I already have a StartMessageLoop in the AppStart sub and so from your comment I need another in the handler too.
 
Upvote 0
The code to generate the JSON as the scoreboard clock data updates in Main is:
[CODE lang="b4x" title="Main"]Sub clk_Tick
m = CreateMap("type":"clock","clock":gameClock,"shotClock":shotClock)
jg.Initialize(m)
update = True
End Sub[/CODE]

Based on @OliverA's suggestion I tried this code:
[CODE lang="b4x" title="Handler"]Sub Handle(req As ServletRequest, resp As ServletResponse)
' Main.id = req.GetParameter("id")
resp.ContentType = "application/json"
resp.Write($"{"type":"connection","status":"CONNECTED"}"$)

WriteOutput(resp)

StartMessageLoop

End Sub

Sub WriteOutput(resp As ServletResponse)
Do While True
'create your json here
If Main.update = True Then
resp.Write(Main.jg.ToString)
Log("Writing . . . " & Main.jg.ToString)
Main.update = False
End If
Log("About to sleep . . .")
Sleep(300) ' Let's take a short break
Log("Looping . . .")
Loop
' StopMessageLoop ' Call this whenever you are done and want to break the connection with the client
End Sub[/CODE]

The resulting log when the handler is triggered by the HTTP call is:
[CODE lang="b4x" title="Log"]Server started
Update DuckDNS: OK
Writing . . . {"shotClock":"18","clock":"9:50","type":"clock"}
About to sleep . . .
Looping . . .
About to sleep . . .
Looping . . .
About to sleep . . .
Looping . . .
About to sleep . . .
Looping . . .
Writing . . . {"shotClock":"17","clock":"9:49","type":"clock"}
About to sleep . . .
Looping . . .
About to sleep . . .
Looping . . .
About to sleep . . .
Looping . . .
Writing . . . {"shotClock":"16","clock":"9:48","type":"clock"}
About to sleep . . .
Looping . . .
About to sleep . . .
Looping . . .
About to sleep . . .
Looping . . .
Writing . . . {"shotClock":"15","clock":"9:47","type":"clock"}
About to sleep . . .
Looping . . .
About to sleep . . .
Looping . . .
About to sleep . . .[/CODE]

However resp is not being returned from the WriteOutput sub to the Handle sub because the response is not seen in the browser making the request.

So I tried this code:
[CODE lang="b4x" title="Handler"]
Sub Handle(req As ServletRequest, resp As ServletResponse)
resp.ContentType = "application/json"
resp.Write($"{"type":"connection","status":"CONNECTED"}"$)

Do While True
If Main.update = True Then
resp.Write(Main.jg.ToString)
Log("Writing . . . " & Main.jg.ToString)
Main.update = False
End If
Log("About to sleep . . .")
Sleep(300)
Log("Looping . . .")
Loop

StartMessageLoop
End Sub
[/CODE]

and the resulting log when the handler is triggered by the HTTP call is:
[CODE lang="b4x" title="Log"]Server started
Update DuckDNS: OK
Writing . . . {"shotClock":"13","clock":"9:45","type":"clock"}
About to sleep . . .[/CODE]

I see
[CODE lang="b4x" title="Response"]{"type":"connection","status":"CONNECTED"}{"shotClock":"13","clock":"9:45","type":"clock"}[/CODE]
in the browser making the request and so the initial JSON and first clock dataset is sent, but it seems to be hanging at
B4X: 
Sleep(300)
and so perhaps Sleep(x) is not allowable in a Handle sub.

More head scratching to be done, but my gut feeling is that I need to create a stream for resp.Outputsream to consume so I can just feed each subsequent JSON as it becomes available rather than a looping approach.
 
Upvote 0
@Erel I have taken on board your pointer to use Wait For in the server handler, but I'm still encountering a "road block".

As indicated above, the code to generate the JSON as the scoreboard clock data updates in Main is:
[CODE lang="b4x" title="Main"]Sub clk_Tick
m = CreateMap("type":"clock","clock":gameClock,"shotClock":shotClock)
jg.Initialize(m)
update = True
Log("clk_Tick : " & jg.ToString)
End Sub[/CODE]

I have added a timer to the handler code so I can use the Timer_Tick event in conjunction with Wait For (because I can't see another suitable event):
[CODE lang="b4x" title="Handler"]Sub Handle(req As ServletRequest, resp As ServletResponse)
resp.ContentType = "application/json"
resp.Write($"{"type":"connection","status":"CONNECTED"}"$)

StartTimer

Do While True
If Main.update = True Then
resp.Write(Main.jg.ToString)
Log("Writing . . . " & Main.jg.ToString)
Main.update = False
End If
Log("Waiting . . .")
Wait For trg_Tick
Log("Looping . . .")
Loop

StartMessageLoop
End Sub

Sub StartTimer
Public trg As Timer
trg.Initialize("trg", 300)
trg.enabled = True
Log("Timer started . . ")
End Sub

Sub trg_Tick
Log("trg_Tick")
End Sub[/CODE]

However while as per the log below, the timer (trg) is started, the relevant timer event (trg_Tick) does not appear to fire and so the loop just sits waiting for it:[CODE lang="b4x" title="Log"]Server started
Update DuckDNS: OK
clk_Tick : {"shotClock":"24","clock":"9:56","type":"clock"}
clk_Tick : {"shotClock":"23","clock":"9:55","type":"clock"}
clk_Tick : {"shotClock":"22","clock":"9:54","type":"clock"}
clk_Tick : {"shotClock":"21","clock":"9:53","type":"clock"}
clk_Tick : {"shotClock":"20","clock":"9:52","type":"clock"}
clk_Tick : {"shotClock":"19","clock":"9:51","type":"clock"}
clk_Tick : {"shotClock":"18","clock":"9:50","type":"clock"}
clk_Tick : {"shotClock":"17","clock":"9:49","type":"clock"}
clk_Tick : {"shotClock":"16","clock":"9:48","type":"clock"}
Request received by handler
Timer started . .
Writing . . . {"shotClock":"16","clock":"9:48","type":"clock"}
Waiting . . .
clk_Tick : {"shotClock":"15","clock":"9:47","type":"clock"}
clk_Tick : {"shotClock":"14","clock":"9:46","type":"clock"}
clk_Tick : {"shotClock":"13","clock":"9:45","type":"clock"}
clk_Tick : {"shotClock":"12","clock":"9:44","type":"clock"}
clk_Tick : {"shotClock":"11","clock":"9:43","type":"clock"}
clk_Tick : {"shotClock":"10","clock":"9:42","type":"clock"}
clk_Tick : {"shotClock":"9","clock":"9:41","type":"clock"}
clk_Tick : {"shotClock":"8","clock":"9:40","type":"clock"}
clk_Tick : {"shotClock":"7","clock":"9:39","type":"clock"}[/CODE]

Moving the "Do while True" loop to a separate sub and calling that from the handle sub results in different issue:[CODE lang="b4x" title="Handler"]Sub WriteOutput(resp As ServletResponse)
Do While True
If Main.update = True Then
resp.Write(Main.jg.ToString)
Log("Writing . . . " & Main.jg.ToString)
Main.update = False
End If
Log("Waiting . . .")
Wait For trg_Tick
Log("Looping . . .")
Loop
End Sub[/CODE]

As per the log below, it seems timer event (trg_Tick) is firing because we see "Waiting . . ." and then "Looping . . .", but while it says "Writing . . . {"shotClock":"14","clock":"9:46","type":"clock"}", that is not actually sent as a response, I assume because the sub is not exited.[CODE lang="b4x" title="Log"]Server started
Update DuckDNS: OK
clk_Tick : {"shotClock":"24","clock":"9:56","type":"clock"}
clk_Tick : {"shotClock":"23","clock":"9:55","type":"clock"}
clk_Tick : {"shotClock":"22","clock":"9:54","type":"clock"}
clk_Tick : {"shotClock":"21","clock":"9:53","type":"clock"}
clk_Tick : {"shotClock":"20","clock":"9:52","type":"clock"}
clk_Tick : {"shotClock":"19","clock":"9:51","type":"clock"}
clk_Tick : {"shotClock":"18","clock":"9:50","type":"clock"}
clk_Tick : {"shotClock":"17","clock":"9:49","type":"clock"}
clk_Tick : {"shotClock":"16","clock":"9:48","type":"clock"}
clk_Tick : {"shotClock":"15","clock":"9:47","type":"clock"}
clk_Tick : {"shotClock":"14","clock":"9:46","type":"clock"}
Request received by handler
Timer started . .
Writing . . . {"shotClock":"14","clock":"9:46","type":"clock"}
Waiting . . .
Looping . . .
Waiting . . .
clk_Tick : {"shotClock":"13","clock":"9:45","type":"clock"}
Looping . . .
Writing . . . {"shotClock":"13","clock":"9:45","type":"clock"}
Waiting . . .
Looping . . .
Waiting . . .
Looping . . .
Waiting . . .
clk_Tick : {"shotClock":"12","clock":"9:44","type":"clock"}
Looping . . .
Writing . . . {"shotClock":"12","clock":"9:44","type":"clock"}
Waiting . . .
Looping . . .
Waiting . . .
Looping . . .
Waiting . . .
clk_Tick : {"shotClock":"11","clock":"9:43","type":"clock"}
Looping . . .
Writing . . . {"shotClock":"11","clock":"9:43","type":"clock"}
Waiting . . .
Looping . . .
Waiting . . .
Looping . . .[/CODE]

Often when I reach these "road blocks" I find I'm using an incorrect method to achieve the desired functionality and perhaps that's again the case, but any assistance will be gratefully received.
 
Upvote 0
I think I am defeated!

In this post it says
The Handle sub cannot be a resumable sub itself.
Click to expand...

Therefore I now realise that the actual Handle sub of a server handler cannot be a resumable sub and so I can see no way to have a loop within the Handle sub so it pauses to gather the updated data from Main to then be written to the response:[CODE lang="b4x" title="Handle"]resp.Write(Main.jg.ToString)[/CODE] All examples I have found which move such writing of the response out of the Handle sub to a separate sub involve just a single write to the response and then Handle completes on return from the sub. They do not allow ongoing/periodic writes to the response.

It now seems impossible to replicate the server functionality I require in that the server response to a HTTP call is an initial JSON response to which further JSON data is sent as updates occur in server (scoreboard) data.
 
Upvote 0
Use this
Code: 
Sub Flush(resp As ServletResponse)
    Dim jo As JavaObject
    jo = resp
    jo = jo.RunMethod("getWriter", Null)
    jo.RunMethod("flush", Null)
End Sub
Call Flush after every resp.Write that you want to deliver to the client
Code: 
resp.Write("Some string data")
Flush(resp)
Resources used:
1) B4J's Git repository: https://github.com/AnywhereSoftware...nywheresoftware/b4j/object/JServlet.java#L359
2) Java documentation on ServletResponse and PrintWriter:

Plus some experience in from the old days of needing to flush your data before a file is saved/a printer prints
 
Upvote 0
@Erel I understand a Websocket solution may be trivial, but I wanted to create a server which responds with JSON, as does the server I described in this Consuming streaming data from HTTP call thread and with your great assistance I was able to create a client to consume its ongoing response when you created a custom class detailled in this Consuming http streams tutorial where you said:
There are some cases where the server keeps the connection open and streams data to the client.
Click to expand...

This is the server behaviour I was trying to replicate so I would be able to just point my existing client to my own server in lieu of the other existing server by replicating its performance.

Thank you @OliverA for your additional input, including references. I will study that to see if I can use that in a solution.
 
Upvote 0
bdunkleysmith said:
This is the server behaviour I was trying to replicate so I would be able to just point my existing client to my own server in lieu of the other existing server by replicating its performance.
Click to expand...
WebSockets is an extension to the http protocol that was designed exactly for such use cases where you want to break the request - response pattern. There is really no reason to "fight" http and make it act like a limited WebSocket.
 
Upvote 0
Thank you @OliverA for sharing your experience from the "old days". It has enabled me to remove the road block!

For completeness in case there's interest from others, this is the handler code I finished up with which incorporates the all important "flush":[CODE lang="b4x" title="Handler"]Handler class
Sub Class_Globals

End Sub

'Initializes the object. You can add parameters to this method if needed.
Public Sub Initialize

End Sub

Sub Handle(req As ServletRequest, resp As ServletResponse)
Log("Request received by handler")
resp.ContentType = "application/json"
resp.Write($"{"type":"connection","status":"CONNECTED"}"$ & Chr(10))
Flush(resp)
StartTimer
WriteOutput(resp)
StartMessageLoop
End Sub

Sub WriteOutput(resp As ServletResponse)
Do While True
If Main.update = True Then
resp.Write(Main.jg.ToString & Chr(10))
Flush(resp)
Log("Writing . . . " & Main.jg.ToString)
Main.update = False
End If
wait for trg_Tick
Loop
End Sub

Sub StartTimer
Public trg As Timer
trg.Initialize("trg", 100)
trg.enabled = True
Log("Timer started . . ")
End Sub

Sub Flush(resp As ServletResponse)
Dim jo As JavaObject
jo = resp
jo = jo.RunMethod("getWriter", Null)
jo.RunMethod("flush", Null)
End Sub[/CODE]

Here is a video in which the first part shows how the data is is received in a browser and the second part shows my scoreboard client with the game and shot clock displays updating as the data is received.

I am very happy that this has enabled me to break the local network constraint of my scoreboard client so it will work over the internet, thus allowing remote production of live sport video streams incorporating an overlay of the scoreboard.

But indeed it has been a "fight" @Erel using http and so I may look at a (more robust) WebSocket solution, even though that would require redevelopment of the communication component of my client.

Thanks to all who provided input, once again proving how great this community is.
 
Last edited:
Upvote 0
You must log in or register to reply here.

Similar Threads

J
  • Question Question
Android Question error QR scanner with R8
Replies
1
Views
45
Erel
Erel
Erel
Server software is being upgraded
Replies
3
Views
113
LucaMs
LucaMs
A
eMedNY
Replies
0
Views
99
Alex_197
A
Daestrum
When tech works for you. (minimum effort)
Replies
0
Views
131
Daestrum
Daestrum
Robert Valentino
  • Question Question
Android Question #R8 and B4XLib crashing method not found in: okhttp3.Request$Builder
Replies
5
Views
184
Erel
Erel
Back
Top