B4J Question [SOLVED] Download rendered HTML

[jroriz]

I'm trying to download the html rendered by this page: https://casino.sportingbet.com/pt-br/livecasino/c/liveroulette


I'm particularly interested in roulette results.

My first attempt was as follows (unsuccessful):

Dim jj As HttpJob, tmp As String
    jj.Initialize("", Me)

    jj.Download("https://casino.sportingbet.com/pt-br/livecasino/c/liveroulette")

    Wait For (jj) JobDone(jj As HttpJob)
   
    If jj.Success Then
        Log(DateTime.Time(DateTime.Now))
        tmp = jj.GetString
       
        File.WriteString("c:\temp", "roleta.txt", tmp)
        jj.Release

        Log("salvo")      
    Else
        LogError(jj.ErrorMessage)
    End If

    jj.Release

Can anybody help me?

 

[drgottjr]

i assume you downloaded the webpage and looked at it,yes?
i assume you did not find any numbers that matched results you were looking for, yes?

this means the results are added dynamically to the webpage after your
browser has loaded the page. in other words, the results are not part of
the webpage.

it may be possible - and not too difficult - to see how the server adds the
results to the page. but you need to look at the site with your browser's
developer options. it is a tab that lets you look at how the page is created.
it involves more than simply the html source. it will involve javascript.

i tried to look at the link you provided. 1) it is incorrect. i had to look for
roleta results. 2) it requires a password, so i could not enter.

 

[xulihang]

You may need to use selenium: https://www.b4x.com/android/forum/t...create-any-robot-with-browser.104344/#content

 

[jroriz]

You may need to use selenium: https://www.b4x.com/android/forum/t...create-any-robot-with-browser.104344/#content

All links are giving me "forbidden"...

 

[jroriz]

i assume you downloaded the webpage and looked at it,yes?
i assume you did not find any numbers that matched results you were looking for, yes?

this means the results are added dynamically to the webpage after your
browser has loaded the page. in other words, the results are not part of
the webpage.

it may be possible - and not too difficult - to see how the server adds the
results to the page. but you need to look at the site with your browser's
developer options. it is a tab that lets you look at how the page is created.
it involves more than simply the html source. it will involve javascript.

i tried to look at the link you provided. 1) it is incorrect. i had to look for
roleta results. 2) it requires a password, so i could not enter.

 

[jroriz]

i assume you downloaded the webpage and looked at it,yes?
i assume you did not find any numbers that matched results you were looking for, yes?

this means the results are added dynamically to the webpage after your
browser has loaded the page. in other words, the results are not part of
the webpage.

YES - YES -YES

The link I gave was wrong.
Now the link in post #1 is correct. No login required.

 

[drgottjr]

this is the image when loaded. no numbers. they add the numbers after the page has been rendered by the browser. i'm looking to see when they do that...

i think this is where they get the results: https://casino.sportingbet.com/pt-br/games/api/LiveCasino/GetData?isSitecoreInfoRequired=false
try that url in your desktop browser. it's a json string, you can see the results from various games. the site downloads the results periodically.
if you know how to parse json, you can figure it out. ( see second image )

code, more or less, like this:

    Dim json As JSONParser
    json.Initialize( text )
    
    Dim root As Map = json.NextObject
    Dim data As List = root.Get( "liveCasinoApiDataItems" )
    Log("items: " & data.Size)
    
    Dim i As Int
    For i = 0 To data.Size - 1
        Log("looking at item: " & i)
        Dim record As Map = data.Get(i)
        Log( record.Get("gameType"))
        Log( record.Get("tableName"))
        Dim results As List = record.Get("results")
        Log("results: " & results)
        Log("----------------------------------")
    Next

it's a long list...

 

[jroriz]

this is the image when loaded. no numbers. they add the numbers after the page has been rendered by the browser. i'm looking to see when they do that...

i think this is where they get the results: https://casino.sportingbet.com/pt-br/games/api/LiveCasino/GetData?isSitecoreInfoRequired=false
try that url in your desktop browser. it's a json string, you can see the results from various games. the site downloads the results periodically.
if you know how to parse json, you can figure it out. ( see second image )

code, more or less, like this:

    Dim json As JSONParser
    json.Initialize( text )
   
    Dim root As Map = json.NextObject
    Dim data As List = root.Get( "liveCasinoApiDataItems" )
    Log("items: " & data.Size)
   
    Dim i As Int
    For i = 0 To data.Size - 1
        Log("looking at item: " & i)
        Dim record As Map = data.Get(i)
        Log( record.Get("gameType"))
        Log( record.Get("tableName"))
        Dim results As List = record.Get("results")
        Log("results: " & results)
        Log("----------------------------------")
    Next

it's a long list...

Guys, honestly, I'm impressed with this forum.
I don't have a single post of mine that hasn't been resolved in MINUTES. Often in more than one way.
I won't say that I don't know how to thank you because I'm an active and licensed member of the forum, and I've already contributed more than once by donating to those who helped me here, and I've even made a modest contribution to "Share my creation".
Congratulations and thanks to the whole community.

 

[jroriz]

this is the image when loaded. no numbers. they add the numbers after the page has been rendered by the browser. i'm looking to see when they do that...

i think this is where they get the results: https://casino.sportingbet.com/pt-br/games/api/LiveCasino/GetData?isSitecoreInfoRequired=false
try that url in your desktop browser. it's a json string, you can see the results from various games. the site downloads the results periodically.
if you know how to parse json, you can figure it out. ( see second image )

code, more or less, like this:

    Dim json As JSONParser
    json.Initialize( text )
   
    Dim root As Map = json.NextObject
    Dim data As List = root.Get( "liveCasinoApiDataItems" )
    Log("items: " & data.Size)
   
    Dim i As Int
    For i = 0 To data.Size - 1
        Log("looking at item: " & i)
        Dim record As Map = data.Get(i)
        Log( record.Get("gameType"))
        Log( record.Get("tableName"))
        Dim results As List = record.Get("results")
        Log("results: " & results)
        Log("----------------------------------")
    Next

it's a long list...

i think this is where they get the results: https://casino.sportingbet.com/pt-br/games/api/LiveCasino/GetData?isSitecoreInfoRequired=false

How do You figured out how did they get the results?

 

[drgottjr]

i looked at the source first. all dynamic content. not helpful. i looked at browser's developer tab for anything that looked like a dynamic download by server to obtain results in real time. looked for pattern of some sort. saw what i wanted after a couple of tries (download and look at payload). saw giant json with "results". figured that was it. sometimes not so easy.

very little static content in web pages these days. all dynamic content after basic page (with layout) is loaded. i saw this site used websockets so that told me there was going to be a lot of realtime activity (ie, same thing over and over). i just looked for some link that kept repeating itself.

 

[jroriz]

i looked at the source first. all dynamic content. not helpful. i looked at browser's developer tab for anything that looked like a dynamic download by server to obtain results in real time. looked for pattern of some sort. saw what i wanted after a couple of tries (download and look at payload). saw giant json with "results". figured that was it. sometimes not so easy.

very little static content in web pages these days. all dynamic content after basic page (with layout) is loaded. i saw this site used websockets so that told me there was going to be a lot of realtime activity (ie, same thing over and over). i just looked for some link that kept repeating itself.