Android Question How do I download a web page and get the HTML/text

[wwregistry]

I used HttpUtil2 to get the pages HTML text but they are blocked by the site - Error: Forbidden? But when using the phones browser and typing in the address from the keyboard the page comes up fine.

So in basic4android how do i use the phone browser to get the HTML text.

This is the code that generates the Error
'=========================================================HttpUtils2
Sub Activity_Create(FirstTime As Boolean)

Page.Initialize("Page", Me)

Page.Download("http://www.411.com/search/FindNearby?utf8=✓&street=11406+boulder+ln&where=78726")

End Sub

Sub JobDone (Job As HttpJob)
Log("JobName = " & Job.JobName & ", Success = " & Job.Success)
If Job.Success = True Then
Select Job.JobName
Case "Page"
Dim w as String
w = Job.getString
Log(w)
End Select
Else
Log("Error: " & Job.ErrorMessage) '<<<< this is where the error message is generated
End If

Job.Release

End Sub
'=========================================================HttpUtils2

HOWEVER

Typing http://www.411.com/search/FindNearby?utf8=✓&street=11406+boulder+ln&where=78726 into the phone browser works fine.

 

[NJDude]

That site might be checking for headers (like what browser is being used for example), without the correct headers any request gets rejected.

 

[wwregistry]

Yes NJDude, i agree the site is checking the headers.

So if i

Dim ph As PhoneIntents
StartActivity(ph.OpenBrowser(http://www.411.com/search/FindNearby?utf8=✓&street=11406+boulder+ln&where=78726))

This works and brings up the browser and page.
But what I want is the HTML text so I can parse it preferably without having to view the page.

 

[NJDude]

Then add the User-agent to your request, that might be the ticket.

 

[wwregistry]

I am not sure about user agents?

But it seems to me that
If android will pull down the web page and interpret the HTML code to display it on the screen?
I should be able to access the same code <<-- that is what i want to do

 

[DonManfred]

See this post. And use this to find out the user-agent your browser is using...
Example.

 

[wwregistry]

pardon my ignorance
but this code seems to be excessively complicated to acquire the HTML text residing in the phone by virtue of StartActivity(ph.OpenBrowser(.... above

 

[DonManfred]

i just wanted to show an example which is using headers...

the maincode is

Dim job_GetInfo As HttpJob

  job_GetInfo.Initialize("job_GetInfo", Me)
  job_GetInfo.PostString(Post_URL, Post_Data)
  job_GetInfo.GetRequest.SetHeader("Sign", API_Signed)
  job_GetInfo.GetRequest.SetHeader("Key", API_Key)

  'Stuff that appears unnecessary but since others use it.....
  job_GetInfo.GetRequest.SetHeader("User-Agent","Mozilla/4.0 (compatible; Cryptsy API B4A client)")
  job_GetInfo.GetRequest.SetContentType("application/x-www-form-urlencoded")

Please note that ALL GetRequest-calls are AFTER PostString (Download or whatever)

 

[wwregistry]

to make this work i need a user name and password?

 

[wwregistry]

i ran this code and got the usual error

Sub Activity_Create(FirstTime As Boolean)
Dim Post_URL, Post_Data, API_Signed, API_Key As String
MakeURL: Post_URL = s
job_GetInfo.Initialize("job_GetInfo", Me)
job_GetInfo.PostString(Post_URL, Post_Data)
job_GetInfo.GetRequest.SetHeader("Sign", API_Signed)
job_GetInfo.GetRequest.SetHeader("Key", API_Key)

'Stuff that appears unnecessary but since others use it.....
job_GetInfo.GetRequest.SetHeader("User-Agent","Mozilla/4.0 (compatible; Cryptsy API B4A client)")
job_GetInfo.GetRequest.SetContentType("application/x-www-form-urlencoded")


job_GetInfo.Download(s)

 

[DonManfred]

I did not say that you should use this code unchanged!

You was searching for an example of how to use a useragent

job_GetInfo.GetRequest.SetHeader("User-Agent","Mozilla/4.0 (compatible; Cryptsy API B4A client)")

THATS the line to set one.

I´m out here now...

 

[wwregistry]

Thank you for your assistance, but without understanding what you are trying to get at here i am clueless?
What confuses me is why after downloading the page using
StartActivity(ph.OpenBrowser(http://www.411.com/search/FindNearby?utf8=✓&street=11406 boulder ln&where=78726))
we can't just snatch out out of cache?

 

[NJDude]

I don't use the HTTPUtils library, but the attached example works, you can adapt it to your needs.

 

[wwregistry]

That is amazing. It works great! You must be a very smart person. I had tried that HTTP code too but did not (still do not LOL) understand the complexities of grabbing a web page. Thank you again.

P.S. I sent you a little something.

 

[NJDude]

There's nothing really complex about that, when you use the HTTP lib you send a request and you get a reply, which you can capture in a file, like in this example.

Thanks for the little something, I appreciate it.