HTTPUtils problem

[raistlin74]

Hi all,
I'm having some problems getting the HTML for this page :
http://it.wikipedia.org/wiki/Episodi_di_Big_Bang_Theory_(quinta_stagione)

Until some days ago, everything works without problem, but since yesterday (or few days ago) I receive in the HTML string a lot of strange chars.

I think is related to a unicode problem, but I cannot find a solution.

Can someone help?

The code I use is very simple :

HttpUtils.CallbackActivity = "view"
HttpUtils.CallbackJobDoneSub = "JobDone"
strSearch = "http://it.wikipedia.org/wiki/Episodi_di_Big_Bang_Theory_(quinta_stagione)"
HttpUtils.Download("Job3", strSearch)

And in the JobDone sub :

If HttpUtils.IsSuccess(strSearch) Then
strHTML = HttpUtils.GetString(strSearch)
strSplitted = Regex.Split(strSplit, strHTML)

Here if I look in the strHTML I find a lots of strange chars and not the HTML.

Thanks in advance,
Fabio

 

[Erel]

This page is encoded with UTF8. So the default GetString should work.
Try to save the response to a file and then compare it to the page you see in the browser.

 

[raistlin74]

Hi Erel,
thanks for the answer.

When I open the page using the Android browser, it say me that the page is Latin-1 and not UTF-8.

Also, if I open the page in a normal web browser, in upper right corner, the page shows me an alert.

maybe it is not UTF-8.

Thanks,
Fabio

 

[Erel]

You should find the encoding. If it is not UTF8 then you should use TextReader with HttpUtils.GetInputStream.

 

[raistlin74]

Hi Erel,
thanks again.

Now the problem is become a mistery.

The page, I linked in first topic, from yesterday works with UTF-8.
But another page that was UTF-8 now is not recognized anymore -.-

 

[raistlin74]

Again here

I made an adb pull of the file generated from the HTTPUtils.

In attach the file.
Can someone help me to understand which encoding is it?
I'm trying with Notepad++ but cannot find it.

The original web page is :
http://it.wikipedia.org/wiki/Episodi_di_Supernatural_(prima_stagione)

Thanks in advance,
Fabio

 

[thedesolatesoul]

It is encoded with UTF-8 (but not UTF-8 without BOM)
but it doesnt even look like a text file, it is all corrupted or something
If you want to pull data from wikipedia you can try using their API that sends an xml file. MediaWiki API

 

[raistlin74]

Unfortunately the wikipedia API doesn't provide me all the informations I need

 

[thedesolatesoul]

Have you tried other pages from wikipedia/non-wikipedia?

 

[raistlin74]

Yes. Other pages, also from wikipedia, works without problems

Inviato dal mio GT-N7000 usando Tapatalk

 

[raistlin74]

Forgot to say that the file has been got using GetInputStream and not GetString.

 

[raistlin74]

Found this link :
java.util.Scanner and Wikipedia - Stack Overflow

He has the same problem.

How can I convert this code in b4a?

Thanks
Fabio

 

[raistlin74]

Ok.

I've modified the HttpUtils procedure to get the ContentEncoding :

Sub hc_ResponseSuccess (Response As HttpResponse, TaskId As Int)
Response.GetAsynchronously("response", File.OpenOutput(TempFolder, TaskId, False), _
True, TaskId)
Try
strEncoding = Response.ContentEncoding
Catch
End Try
End Sub

And for the page that gives me error, it shows "gzip" as ContentEncodig lol.
I didn't know that wikipedia could send gzip format.

Now the problem is : how to handle gzip encoding?

thanks again,
Fabio

 

[raistlin74]

Great, solved with the previous add to HttpUtilsService and :


If HttpUtilsService.strEncoding = "gzip" Then
in = compStream.WrapInputStream(HttpUtils.GetInputStream(strSearch), "gzip")
End If

and everything works

Erel, maybe you can change the HttpUtils library to handle the encoding parameter, just to see which is the correct encoding of the page

 

[thedesolatesoul]

wow good work and nice debugging
what is wikipedia sending gzip for that is beyond me?
are you sure your user-agent is set correctly.
anyway glad you got it working and i learnt something new