Android Question How to get "hidden" data from a web page

LucaMs

Expert
Licensed User
Longtime User
I can "analize"* (using Firefox or Chrome tools) the "data table" you'll see in the web page at this link:
https://www.lottomaticaitalia.it/it/prodotti/lotto

but when I download its souce code... nothing!

Is there a way to do it?

The table:
1584337269417.png



* 83 Firefox utils
1584337547401.png
 

sirjo66

Well-Known Member
Licensed User
Longtime User
All data from "estrazione del lotto" are protected !!
As you can see at line #3 there is:
HTML:
<html ng-app="LottomaticaItalia">
So your HTML page are using LottomaticaItalia application
Then, from line #5526 to line #5546 there is your table.
Take a look at
HTML:
<tr ng-repeat="ext in extracionTable" ng-init="outerLast = $last; addBlu = ext.ruota === simbolotti.ruota" ng-class="addBlu && ! outerLast && 'blu'" ng-cloak>
The ng-repeat tag make a "for each" loop in extracionTable (that is where you can find datas)
 
Upvote 0

sirjo66

Well-Known Member
Licensed User
Longtime User
Thanks, but if I download the page (httpjob download) the table (or at least its content) is "missing", there isn't.

yes, I already see it !! (per questo ti ho detto che sono protetti)

What you want to do is very simple in other language, I realize it in 5 minutes, using VB.NET and a WebBrowser object.

Here is a screenshot:
Immagine.png


And here is source code (wb is a WebBrowser object):
B4X:
Public Class Form1

    Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        wb.InitIE11()
        Me.WindowState = FormWindowState.Maximized
    End Sub

    Private Sub Form1_Shown(sender As Object, e As System.EventArgs) Handles Me.Shown

        wb.Navigate("https://www.lottomaticaitalia.it/it/prodotti/lotto")
        wb.Wait()

        Dim trs = wb.Document.Body.SelectElement("tbody").SelectElements("tr")
        Dim result = ""
        For Each tr As HtmlElement In trs
            Dim tds = tr.SelectElements("td")
            For Each td As HtmlElement In tds
                result &= td.InnerText & " "
            Next
            result &= Environment.NewLine
        Next

        MsgBox(result)

        Me.Dispose()

    End Sub
End Class
 
Upvote 0

sirjo66

Well-Known Member
Licensed User
Longtime User
Immagine.png


from where can I download it ??

[EDIT] Found it !!!
 
Last edited:
Upvote 0

Erel

B4X founder
Staff member
Licensed User
Longtime User
The reason that it is missing it that the browser makes additional calls from the JavaScript code.
You can indeed use WebViewExtra to get the full html code after all requests and then parse it.

Another option which requires some investigation is to directly download the json data.

B4J example:
B4X:
Sub AppStart (Args() As String)
    DownloadTable ("20200314")
    StartMessageLoop
End Sub

Private Sub DownloadTable (DateString As String)
    Dim jg As JSONGenerator
    jg.Initialize(CreateMap("data": DateString))
    Dim j As HttpJob
    j.Initialize("", Me)
    j.PostString("https://www.lottomaticaitalia.it/gdl/estrazioni-e-vincite/estrazioni-del-lotto.json", jg.ToString)
    j.GetRequest.SetContentType("application/json")
    Wait For (j) JobDone(j As HttpJob)
    If j.Success Then
        Dim parser As JSONParser
        parser.Initialize(j.GetString)
        Dim m As Map = parser.NextObject
        jg.Initialize(m) 'just to better understand the data
        Log(jg.ToPrettyString(4))        
    End If
    j.Release
End Sub

Output:
B4X:
Waiting for debugger to connect...
Program started.
{
    "simbolotti": {
        "simbolotti": [
            14,
            42,
            4,
            45,
            1
        ],
        "ruota": "FI"
    },
    "numeriVincenti": [
        4,
        7,
        11,
        15,
        18,
        20,
        25,
        26,
        31,
        32,
        45,
        47,
        54,
        56,
        58,
        60,
        66,
        76,
        83,
        87
    ],
    "esito": "OK",
    "messaggio": null,
    "data": 1584140400000,
    "estrazione": [
        {
            "ruota": "BA",
            "ruotaExtended": "BARI",
            "numeri": [
                83,
                7,
                32,
                65,
                79
            ]
        },
        {
            "ruota": "CA",
            "ruotaExtended": "CAGLIARI",
            "numeri": [
                18,
                45,
                76,
                77,
                9
            ]
        },
        {
            "ruota": "FI",
            "ruotaExtended": "FIRENZE",
            "numeri": [
                31,
                15,
                77,
                13,
                37
            ]
        },
        {
            "ruota": "GE",
            "ruotaExtended": "GENOVA",
            "numeri": [
                11,
                4,
                22,
                45,
                23
            ]
        },
        {
            "ruota": "MI",
            "ruotaExtended": "MILANO",
            "numeri": [
                25,
                66,
                76,
                16,
                33
            ]
        },
        {
            "ruota": "NA",
            "ruotaExtended": "NAPOLI",
            "numeri": [
                54,
                58,
                60,
                30,
                47
            ]
        },
        {
            "ruota": "PA",
            "ruotaExtended": "PALERMO",
            "numeri": [
                20,
                26,
                58,
                57,
                29
            ]
        },
        {
            "ruota": "RM",
            "ruotaExtended": "ROMA",
            "numeri": [
                20,
                56,
                34,
                82,
                70
            ]
        },
        {
            "ruota": "TO",
            "ruotaExtended": "TORINO",
            "numeri": [
                87,
                60,
                18,
                51,
                12
            ]
        },
        {
            "ruota": "VE",
            "ruotaExtended": "VENEZIA",
            "numeri": [
                32,
                47,
                66,
                21,
                68
            ]
        },
        {
            "ruota": "RN",
            "ruotaExtended": "NAZIONALE",
            "numeri": [
                62,
                8,
                84,
                75,
                41
            ]
        }
    ],
    "numeroSpeciale": 83
}
 
Upvote 0

aeric

Expert
Licensed User
Longtime User
I assume it is common that client side JavaScript like AngularJS need to communicate through API in an interval to retrieve data maybe through Ajax so the page refresh itself with the latest data. So resources like json also part of the files that need to be downloaded to display the whole page content.
 
Upvote 0

Erel

B4X founder
Staff member
Licensed User
Longtime User
Did you (Erel) assume that a json file (that file) was downloaded or did you see something in the source of the page that got you there?
I looked over the network requests and the json one called me. I didn't find it, it found me.
 
Upvote 0
Top