Android Question B4A HttpJob Post JSON Request without result (specific problem)

[bitben]

Hello,
my problem is i get an empy json string. I want to get data of Tradingview.com. I find and read out all nessesary infos for a post request (i think nothing overseen) with network analysing tool of firefox. URL and post data in following source code. B.t.w. with my way is only a single request possible, but i want to get the values (e.g.) every second, is it possible? I tried different ways, i tried with 'Sub JobDone' and 'wait for (job)', tried 1 and 2 HttpJobs, tried json as direct string and with JSONGenerator and i set header, but nothing works. I get no cookie data, is it maybe important in headers..? Whole Project (small) uploaded.
Tried same project with python, same empty json string.

(Unused subs here deleted for short view)

Sub Process_Globals
    'These global variables will be declared once when the application starts.
    'These variables can be accessed from all modules.
    Dim start,check As HttpJob
End Sub

Sub Service_Start (StartingIntent As Intent)
    Service.StopAutomaticForeground 'Starter service can start in the foreground state in some edge cases.
    startReq
End Sub

Sub startReq()
    start.Initialize("",Me)
    start.Download("https://tradingview.com/symbols/USOIL/technicals/")
    start.GetRequest.SetHeader("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
    start.GetRequest.SetContentEncoding("gzip, deflate, br")
    start.GetRequest.SetHeader("Accept-Language", "de,en-US;q=0.7,en;q=0.3")
    start.GetRequest.SetHeader("Cache-Control","no-cache")
    start.GetRequest.SetHeader("Connection","keep-alive")
    start.GetRequest.SetHeader("DNT","1")
    start.GetRequest.SetHeader("Host", "tradingview.com")
    start.GetRequest.SetHeader("Upgrade-Insecure-Requests", "1")
    start.GetRequest.SetHeader("Pragma", "no-cache")
    start.GetRequest.SetHeader("User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0")
    
    wait for (start) JobDone(job As HttpJob)
    If job.success Then
        getData()
    Else
        Log("Error:")
        Log(job.ErrorMessage)
    End If
    job.release
End Sub

Sub getData()
    Dim fx As String = "FX:USOIL"
                
    Dim vals As Map
    vals.Initialize
                
    Dim symbols As Map
    symbols.Initialize
                
    Dim tickers As List
    tickers.Initialize
    tickers.Add(fx)
                
    Dim query As Map
    query.Initialize
                
    Dim types As List
    types.Initialize
                
    query.put("types",types)
                
    Dim columns As List
    columns.Initialize
    columns.AddAll(Array As String("Recommend.Other|5","Recommend.All|5","Recommend.MA|5"))
                
    symbols.Put("tickers",tickers)
    symbols.Put("query",query)
                
    vals.Put("symbols", symbols)
    vals.Put("columns",columns)
                
    Dim k As JSONGenerator
    k.Initialize(vals)
    Log(k.ToString)
                
'    "{" & Chr(34) & "symbols" & Chr(34) & ":{" & Chr(34) & "tickers" & Chr(34) & ":[" & Chr(34) & fx & Chr(34) & "]," & Chr(34) & "query" & Chr(34) & ":{" & Chr(34) & "types" & Chr(34) & ":[]}}," & Chr(34) & "columns" & Chr(34) & ":[" & Chr(34) & "Recommend.Other|15" & Chr(34) & "," & Chr(34) & "Recommend.All|15" & Chr(34) & "," & Chr(34) & "Recommend.MA|15" & Chr(34) & "]}"
    check.Initialize("", Me)
'    check.PostMultipart("https://scanner.tradingview.com/forex/scan", vals, Null)
    check.PostString("https://scanner.tradingview.com/forex/scan", k.tostring)
    check.GetRequest.SetHeader("Accept","*/*")
    check.GetRequest.SetContentEncoding("gzip, deflate, br")
    check.GetRequest.SetHeader("Accept-Language", "de,en-US;q=0.7,en;q=0.3")
    check.GetRequest.SetHeader("Cache-Control","no-cache")
    check.GetRequest.SetHeader("Connection","keep-alive")
    check.GetRequest.SetContentType("application/x-www-form-urlencoded")
    check.GetRequest.SetHeader("DNT","1")
    check.GetRequest.SetHeader("Host", "scanner.tradingview.com")
    check.GetRequest.SetHeader("Origin", "https://tradingview.com")
    check.GetRequest.SetHeader("Pragma", "no-cache")
    check.GetRequest.SetHeader("TE", "Trailers")
    check.GetRequest.SetHeader("User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0")
    
    wait for (check) JobDone(job As HttpJob)
    If job.Success Then
        Log(job.GetString)
    Else
        Log("Error:")
        Log(job.ErrorMessage)
    End If
    
    job.release
End Sub

The JSON String is

{"symbols":{"tickers":["FX:USOIL"],"query":{"types":[]}},"columns":["Recommend.Other|5","Recommend.All|5","Recommend.MA|5"]}

same as original.

The response should be like

{ "data":[{"s":"FX:USOIL","d":[-0.09090909,-0.44545455,-0.8]}],"totalCount":1}

but response is

{ "data":[],"totalCount":0}

Did i oversee something important or is there a much easier way?

Thanks for replies and your time.

 

[drgottjr]

2 comments:
if you're sending json and you want the server to know it's json (and the server is actually expecting json), you should
SetContentType("application/json"), not what you did. see if a different content type gets you a different result.

the site seems to be set up for websockets, which would work perfectly with your envisioned once-a-second plan. for that, you need one of the
websocket scripts floating around here. okhttputils is standard tcp. i suppose, technically, it would run as quickly as it could, but the whole tcp handshake and tear-down is expensive; i think you'd have some problems. see if you can find one of the websocket clients here. there is a library (2.11). i've used it in the last month to test with a local ws server. it worked fine.

if you go to the website with your desktop browser and switch to the developer's page, you'll see clear evidence of ws activity on the network. i know that's the way yahoo's stock quotations page works these days

 

[bitben]

Thanks for reply.
I forgot to say i testet it with 'application/json', same thing.
Ok i'll search around for websocket and test it and reply results.

 

[bitben]

Ok i have no idea how to start. All websocket examples seems to be p2p and the developer's page shows nothing more for me.

Can you give an attempt please with my target site, how can i get the html source over 'your' websocket idea, do i need the source with your idea or how do i stream the websocket/website data to my application
There is also a javascript file listed as initiator

https://tradingview.com/static/bundles/category.f39b143b846d4ec3bda9.js

maybe i have to run this.. with your websocket idea?

Sorry, but when connection/getting data is done the rest should be no problem.

 

[OliverA]

You’re posting JSON, yet your header tells the server that you are posting application/x-www-form-urlencoded. I think this is a case of garbage in/garbage out

 

[bitben]

You’re posting JSON, yet your header tells the server that you are posting application/x-www-form-urlencoded. I think this is a case of garbage in/garbage out

It appears in header request information, as you see in post 3 i tested it with 'application/json', but didn't work or what should i change additionally?

 

[OliverA]

Look at their api reference (google it). Request post data is URL encoded and responses are json. I wrote a map to URL encoded string for just such situations: https://www.b4x.com/android/forum/threads/b4x-urlencodemap.118626/#content

 

[bitben]

Thank you, i will try with api reference and your sub if needed. The big bang is i never worked with such api reference infos, i could do all over get/post and website html source before. Is there for every website such api references?

 

[drgottjr]

just for a little clarity: i'm not saying that your request will work with websockets. it will involve a different type of request. websockets would be used for quick, repetitive simple requests.

if you're sure okhttputils2 poststring is the correct the way to get that json response you show, then we need to find out why it's not working. forget websockets for the moment. there is no poststring in websockets, ok?

 

[OliverA]

Is there for every website such api references?

If they offer a service, then usually yes

 

[bitben]

just for a little clarity: i'm not saying that your request will work with websockets. it will involve a different type of request. websockets would be used for quick, repetitive simple requests.

if you're sure okhttputils2 poststring is the correct the way to get that json response you show, then we need to find out why it's not working. forget websockets for the moment. there is no poststring in websockets, ok?

ok, i am not sure, but this was the way it worked before, so i wonder why i can't get no data. I did the same, collecting needed data, on plus500 an all worked fine. They offer a widget, but it provides only the general pointer data (middle one). Can someone try out my project maybe?

If they offer a service, then usually yes

On google/twitter was a post, they don't offer data, they offer charts, so when i understand it right i have to get the data directly from my url in source code, would be the easiest way. I will try out your sub tomorrow with current project.

I read something with 'webviewextras', is there a way to open the website and get data directly from it with injecting js?

 

[drgottjr]

here's the deal: there is no api for tradingview.com. they repeat this often. you need a key.
your first sub, the call to tradingview.com is a complete waste of time. it gets you nothing except the web page. i'm angry that i wasted time trying to figure out what its purpose was.
the second sub (getdata) is a standalone. it takes no parameters (you never use the one you assigned). you can just run it without running startreq first!
the json you formulate is wrong. that is to say, the server expects more. the actual json it expects is very long.
the url you use is wrong. it's https://scanner.tradingview.com/cfd/scan, not https://scanner.tradingview.com/forex/scan.
when i use the url which (i think) is correct, i get { "data":[{"s":"FX:USOIL","d":[-0.09090909,-0.44545455,-0.8]}],"totalCount":1}
this is not quite what you're expecting, but it's certainly closer than what you're seeing. if you fed the server the full json it's expecting, you would probably get the price (unless the lack of a key overrides that. there are numerous checks for a valid authentication key).
complete disclosure, here is the full json the server uses internally to get the price:

{"symbols":{"tickers":["FX:USOIL"],"query":{"types":[]}},"columns":["Recommend.Other","Recommend.All","Recommend.MA","RSI","RSI[1]","Stoch.K","Stoch.D","Stoch.K[1]","Stoch.D[1]","CCI20","CCI20[1]","ADX","ADX+DI","ADX-DI","ADX+DI[1]","ADX-DI[1]","AO","AO[1]","Mom","Mom[1]","MACD.macd","MACD.signal","Rec.Stoch.RSI","Stoch.RSI.K","Rec.WR","W.R","Rec.BBPower","BBPower","Rec.UO","UO","EMA5","close","SMA5","EMA10","SMA10","EMA20","SMA20","EMA30","SMA30","EMA50","SMA50","EMA100","SMA100","EMA200","SMA200","Rec.Ichimoku","Ichimoku.BLine","Rec.VWMA","VWMA","Rec.HullMA9","HullMA9","Pivot.M.Classic.S3","Pivot.M.Classic.S2","Pivot.M.Classic.S1","Pivot.M.Classic.Middle","Pivot.M.Classic.R1","Pivot.M.Classic.R2","Pivot.M.Classic.R3","Pivot.M.Fibonacci.S3","Pivot.M.Fibonacci.S2","Pivot.M.Fibonacci.S1","Pivot.M.Fibonacci.Middle","Pivot.M.Fibonacci.R1","Pivot.M.Fibonacci.R2","Pivot.M.Fibonacci.R3","Pivot.M.Camarilla.S3","Pivot.M.Camarilla.S2","Pivot.M.Camarilla.S1","Pivot.M.Camarilla.Middle","Pivot.M.Camarilla.R1","Pivot.M.Camarilla.R2","Pivot.M.Camarilla.R3","Pivot.M.Woodie.S3","Pivot.M.Woodie.S2","Pivot.M.Woodie.S1","Pivot.M.Woodie.Middle","Pivot.M.Woodie.R1","Pivot.M.Woodie.R2","Pivot.M.Woodie.R3","Pivot.M.Demark.S1","Pivot.M.Demark.Middle","Pivot.M.Demark.R1"]}

how much of that you actually need to use (if not all), i can't say. you can start expanding on your original poststring until you find the magic element.

and as far as websockets is concerned, this is the call tradingview uses internally:
wss://data.tradingview.com/socket.io/websocket?from=symbols%2FUSOIL%2Ftechnicals%2F&date=2020_06_26-13_48

a final note: you're right regarding application/x-www-form-urlencoded and application/json. i get the same partial response regardless of which content-type i use (in this case).

second final note: there's got to be somebody else that can supply the price of oil.

 

[bitben]

The first function was to get cookie data, so sorryy for wasting time ?
The json data i posted is all i need, the rest of it i don't need at the moment, i could extend the json as needed later.

i looked up the url

the url you use is wrong. it's https://scanner.tradingview.com/cfd/scan, not https://scanner.tradingview.com/forex/scan.

and you are right, in post data there is yours, i don't know how i got my url ? so angry about that... now it worked for me, all is fine.
many thanks

 

[drgottjr]

angry, happy - all good.
just fyi, it is possible to find the price from the tradingview.com technicals site. it and other "invisible" data appears as a text span: <span>38.165</span>.
you've obviously spent more time trying to track this down than i have (or intend to), but i'm surprised you didn't mention it before. it might (just might) be possible to extract this with webviewextras from a "hidden" webview. i mention this in case you find your call to scanner.tradingview.com shut down.

and you should really look into websockets if you're doing to try to turn your app into a streaming price feed.

 

[bitben]

and you should really look into websockets if you're doing to try to turn your app into a streaming price feed.

i have no experience with websockets. Could you give an example? I found the websocket example, but how i 'stream' the feed? Very interesting. And how i know what commands to send on what the server react? It is like telnet..?

 

[drgottjr]

technically, i suppose "stream" is poor choice of words. first we had http which was/is a request for something, an answer and a disconnect. then we had ajax, which tried to reduce the overhead of an http request. now, websockets keeps the connection open, and you send and receive "messages". once the connection has been established, it (more or less) stays connected, like a phone conversation. but within the limitations of messages.

there are 2 problems for you: 1) learning websockets (not a problem at all) and 2) finding a server that will respond to you.
i think there are some "test" ws servers out there. you send them "hello", they echo your message back. you'll have to look around.

as for websocket examples, we have them here. there are also several very easy tutorials online. see what you can find and open a new thread if you get in trouble. if i spot it and it looks like you're getting nowhere, i'll dig out my test app. it's not handy right this minute. you don't use okhttutils2; it's a slightly different protocol, but it's easy. and if you can get access to that trading site's ws server, you'll be streaming. access may be denied without a token, and i don't think you can just "browse" to a ws server.

 

[bitben]

ok thanks, 3 last questions:
1.

and as far as websockets is concerned, this is the call tradingview uses internally:
wss://data.tradingview.com/socket.io/websocket?from=symbols%2FUSOIL%2Ftechnicals%2F&date=2020_06_26-13_48

how did you get it?

2.
so i have to know which messages i have to send to get the right answer on external webservers, this is with testing or standard commands..? The websocket exmples i found are mainly with own B4J server, so have their own events etc.

3.
In which case are websockets better than httputils?

 

[drgottjr]

1) it's seen in the developer's tab on my desktop browser (chrome) firefox (and, presumably, the others) have a similar feature. in looking at the network traffic, the ws call jumps out (as does its reply). but you can't just plug it in to a browser; you need a websocket client.
2) there is a simple messaging standard to follow. when i say look online, i don't just mean here. just google websockets tutorial. the whole thing is very simple. even the ws server i run is simple.
3) httputils is designed for http(s), not websockets. to use my phone call analogy from before: websockets is like having an open connection to a server, http is like texting (or leaving a voice mail). the former is open all the time, the latter dials, waits for an answer, does something, hangs up, dials again, waits for an answer, hangs up, etc, etc. to get something like an ever-changing discrete datum without all the overhead, websockets (or similar) is better. to get a web page and its ancillaries, you want http. httputils is a way to mimic some of what a browser does in cases where you are not actually interested in displaying a web page. (eg, you can upload a file without actually using a browser and filling in a form. or download some data (eg, json)) but it's not meant for messaging.

ps - you need to start a new different thread for websockets
pps - access is denied on the wss call shown above. not sure whether it's due to a missing token or because the call is not coming from tradingview itself. i just wanted to see if it would go through with a websocket client. seems not to, at least not without further examination