B4J Question jOkHttpUtils2 getstring stops download after 3 dots (spread operator)

[gezueb]

Hi All, I try to read a local html website said utils and parse some data. However, the job.Getstring truncates everything after (including) the first occurrence of three dots (...).
Size of website is some 250'000 characters, the string is only 1100 chars.

Sub getLocalVartaData As ResumableSub

    Dim job As HttpJob
    job.Initialize("",Me)
    job.Download("http://192.168.1.18/home.html")
    Wait For (job) JobDone(job As HttpJob)
    If job.Success Then
        Dim result As String
        result = job.GetString
        Log("String-Length: "&result.Length)
        Log(result)
    End If
    
    job.Release
End Sub

 

[gezueb]

OK, here comes the code. However, dont spend too much time on it. I run several html validators and they report many errors. I have come to the conclusion that this page is crap.

 

[Daestrum]

If you get %C3 and %9C as single characters - you probably need to read the string as windows-1252 as opposed to utf-8

windows-1252 %C3
utf-8 %C3%83

windows-1252 %9C
utf-8 %C5%93

 

[gezueb]

Tried

result = job.GetString2("windows-1252")

no change however?

 

[DonManfred]

I have come to the conclusion that this page is crap.

The page is reloading data with javascript. As the javascript is minified it is hard to read.

Not all data shown in a browser are inside the html. DATA are probably missing.

 

[aeric]

No content is truncated.

Sub DownloadPage
    Dim job As HttpJob
    job.Initialize("", Me)
    job.Download("http://127.0.0.1/varta/home.html")
    Wait For (job) JobDone(job As HttpJob)
    If job.Success Then
        Log("Response Content Length=" & job.GetString.Length)
        Dim result As String = job.GetString
    End If
    job.Release
    File.WriteString(File.DirApp, "output.txt", result)
    ' Verify content length
    Dim FileContent As String = File.ReadString(File.DirApp, "output.txt")
    Log("File Content Length=" & FileContent.Length)
End Sub

Logs:

Waiting for debugger to connect...
Program started.
Response Content Length=269601
File Content Length=269601

 

[aeric]

This also prove that the content after ... is not truncated and can be read.

Sub DownloadPage
    Dim job As HttpJob
    job.Initialize("", Me)
    job.Download("http://127.0.0.1/varta/home.html")
    Wait For (job) JobDone(job As HttpJob)
    If job.Success Then
        Dim result As String = job.GetString
    End If
    job.Release
    
    Dim occurence As Int
    Dim start As Int = -3
    Dim found As Int
    Do Until found = -1
        found = result.IndexOf2("...", start+3)
        If found > -1 Then
            Log(found & "->" & result.SubString2(found, found+30)) ' Log 30 characters after ...
            occurence = occurence + 1
            start = found           
        End If
    Loop
    Log("occurence=" & occurence)
End Sub

Logs:

Waiting for debugger to connect...
Program started.
5494->...c){for(const b of c)if(a=a[
6203->...e){}const b={};new Map;b.is
6286->...a){return this.logModule(nu
6319->...a)};b.logModule=function(a,
6349->...c){return d("log",a,...c)};
6372->...c)};b.logInfo=function(...a
6398->...a){return this.logModuleInf
6435->...a)};b.logModuleInfo=functio
6469->...c){return d("info",a,...c)}
6493->...c)};b.logWarning=function(.
6522->...a){return this.logModuleInf
6559->...a)};b.logModuleWarning=func
6596->...c){return d("warning",a,...
6623->...c)};b.logError=function(...
6650->...a){this.logModuleError(null
6683->...a)};b.logModuleError=functi
6718->...c){return d("error",a,...c)
6743->...c)};b.proxyObject=function(
occurence=18

 

[aeric]

By the way, I have replaced the string

http://192.168.1.18/

to

http://127.0.0.1/varta/

in home.html
to load into my local server.

 

[gezueb]

Ok, I see that the page works. Thats good news. But how do I get http://192.168.1.18/ programmatically into http://127.0.0.1/varta/ ?

 

[aeric]

Ok, I see that the page works. Thats good news. But how do I get http://192.168.1.18/ programmatically into http://127.0.0.1/varta/ ?

I am not sure what you are trying to do is correct and whether my mod gives what you are looking for.
I use Laragon development server. Just drop the files into a folder name varta inside www folder and refresh the server.

 

[gezueb]

Ok, sorry I have to be more precise. I can access the page on my local network on 192.168.1.18. (or any other local url defined by my router) But I cannot change the contents, its read only. I guess the problem might be file naming and file structure. I wonder how I can access the files from the my embedded server without copying and zipping manually.

 

[aeric]

I am not sure you are familiar with server, HTML and JavaScript. The content of the HTML or DOM can be modified by JavaScripts.
I am also not sure what device it is hosted the web pages. I think you have to access the File Manager inside the server using FTP tool.

 

[gezueb]

I have no other access to the server than the unmodified local url. No FTP. If I must change the html code, I will have to copy the contents by opening the webpage with a browser and store it from there and then possibly edit. That's how I made the zip file. But I cannot use this procedure periodically (say every 15 min). Thats why I am looking for a simpler way with a B4J (or eventually a B4A) app.

 

[teddybear]

Hi All, I try to read a local html website said utils and parse some data. However, the job.Getstring truncates everything after (including) the first occurrence of three dots (...).
Size of website is some 250'000 characters, the string is only 1100 chars.

As DonManfred said at #24

page is reloading data with javascript.

The page is dynamic and you can‘t entirely download it using jOkHttpUtils2.
You may use webview to load the url and then parse html code.

​

 

[aminoacid]

I am not sure you are familiar with server, HTML and JavaScript. The content of the HTML or DOM can be modified by JavaScripts.
I am also not sure what device it is hosted the web pages. I think you have to access the File Manager inside the server using FTP tool.

I see what you are saying, @aeric. If the file "home.html" that GetString is reading from his device server contains javascript that produces the HTML that he is seeing in his browser, then the size of the string read from the server is definitely going to be different (and perhaps shorter) that the size of the HTML code displayed in the browser.

@gezueb, I think what you should do is use @aeric 's code in post#25 and save the string to a file. Then look at the javascript in the file to see what it's doing.

 

[aminoacid]

The page is dynamic and you can‘t entirely download it using jOkHttpUtils2.
You may use webview to load the url and then parse html code.

​

Exactly!

 

[gezueb]

Thanks, I will try to learn webview and its functions