HttpUtils2 can't seem to download long lists

kanaida · Dec 5, 2012

I've got a list of product url's. If I pass a few (like less than 15) it works. If I add all of them (about 500) to the queue for download (creating a local cache, then It just stops after a few images.

B4X:

Dim StyleList As List = File.ReadList(File.DirInternal,"styles.txt")
If File.Exists(File.DirInternal,"styles") = False Then
      File.MakeDir(File.DirInternal,"styles")
   End If
   For Each p As String In StyleList
   Dim ImgPath As String = File.Combine(File.DirInternal,"styles")
   Dim ImgFile As String = p & ".jpg"
      If File.Exists(ImgPath,ImgFile) = False Then
         Dim job1 As HttpJob
          job1.Initialize(p, Me)
         job1.Download("http://www.somewhere.com/images/1137/Styles/" & p.Replace(" ","%20") & ".jpg")
         Else
         
         Dim b As Bitmap = LoadBitmap(ImgPath,ImgFile)
         lv_Products.AddTwoLinesAndBitmap(p,"Collection 1", b)
      End If

   Next

I'm pretty sure that's the android limit, but I was under the assumtion that the HttpUtils was meant to be used to get around that the proper way. Normally i'd just start a different thread and do a loop synchronously inside to get all the files, but I don't exactly see a synchronous method to get files via http...

Erel · Dec 5, 2012

Do you see any messages in the logs?

kanaida · Dec 6, 2012

Yes, TONS of starting service messages, and a few http errors from missing images (a few were expected by design). Also downloaded about 3 images that were there, properly.

I'm on a Nexus 10 on android 4.2.1

I didn't get a result for most of the requests.

kanaida · Dec 6, 2012

B4X:

Installing file.
PackageAdded: package:kanaida.madison.online
** Activity (main) Create, isFirst = true **
** Activity (main) Resume **
** Activity (main) Pause, UserClosed = false **
** Activity (editorder) Create, isFirst = true **
** Activity (editorder) Resume **
** Service (httputils2service) Create **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
** Service (httputils2service) Start **
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">

<HTML><HEAD><TITLE>The page cannot be found</TITLE>

<META HTTP-EQUIV="Content-Type" Content="text/html; charset=Windows-1252">

<STYLE type="text/css">

  BODY { font: 8pt/12pt verdana }

  H1 { font: 13pt/15pt verdana }

  H2 { font: 8pt/12pt verdana }

  A:link { color: red }

  A:visited { color: maroon }

</STYLE>

</HEAD><BODY><TABLE width=500 border=0 cellspacing=10><TR><TD>



<h1>The page cannot be found</h1>

The page you are looking for might have been removed, had its name changed, or is temporarily unavailable.

<hr>

<p>Please try the following:</p>

<ul>

<li>Make sure that the Web site address displayed in the address bar of your browser is spelled and formatted correctly.</li>

<li>If you reached this page by clicking a link, contact

 the Web site administrator to alert them that the link is incorrectly formatted.

</li>

<li>Click the <a href="javascript:history.back(1)">Back</a> button to try another link.</li>

</ul>

<h2>HTTP Error 404 - File or directory not found.<br>Internet Information Services (IIS)</h2>

<hr>

<p>Technical Information (for support personnel)</p>

<ul>

<li>Go to <a href="http://go.microsoft.com/fwlink/?linkid=8180">Microsoft Product Support Services</a> and perform a title search for the words <b>HTTP</b> and <b>404</b>.</li>

<li>Open <b>IIS Help</b>, which is accessible in IIS Manager (inetmgr),

 and search for topics titled <b>Web Site Setup</b>, <b>Common Administrative Tasks</b>, and <b>About Custom Error Messages</b>.</li>

</ul>



</TD></TR></TABLE></BODY></HTML>

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">

<HTML><HEAD><TITLE>The page cannot be found</TITLE>

<META HTTP-EQUIV="Content-Type" Content="text/html; charset=Windows-1252">

<STYLE type="text/css">

  BODY { font: 8pt/12pt verdana }

  H1 { font: 13pt/15pt verdana }

  H2 { font: 8pt/12pt verdana }

  A:link { color: red }

  A:visited { color: maroon }

</STYLE>

</HEAD><BODY><TABLE width=500 border=0 cellspacing=10><TR><TD>



<h1>The page cannot be found</h1>

The page you are looking for might have been removed, had its name changed, or is temporarily unavailable.

<hr>

<p>Please try the following:</p>

<ul>

<li>Make sure that the Web site address displayed in the address bar of your browser is spelled and formatted correctly.</li>

<li>If you reached this page by clicking a link, contact

 the Web site administrator to alert them that the link is incorrectly formatted.

</li>

<li>Click the <a href="javascript:history.back(1)">Back</a> button to try another link.</li>

</ul>

<h2>HTTP Error 404 - File or directory not found.<br>Internet Information Services (IIS)</h2>

<hr>

<p>Technical Information (for support personnel)</p>

<ul>

<li>Go to <a href="http://go.microsoft.com/fwlink/?linkid=8180">Microsoft Product Support Services</a> and perform a title search for the words <b>HTTP</b> and <b>404</b>.</li>

<li>Open <b>IIS Help</b>, which is accessible in IIS Manager (inetmgr),

 and search for topics titled <b>Web Site Setup</b>, <b>Common Administrative Tasks</b>, and <b>About Custom Error Messages</b>.</li>

</ul>



</TD></TR></TABLE></BODY></HTML>

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">

<HTML><HEAD><TITLE>The page cannot be found</TITLE>

<META HTTP-EQUIV="Content-Type" Content="text/html; charset=Windows-1252">

<STYLE type="text/css">

  BODY { font: 8pt/12pt verdana }

  H1 { font: 13pt/15pt verdana }

  H2 { font: 8pt/12pt verdana }

  A:link { color: red }

  A:visited { color: maroon }

</STYLE>

</HEAD><BODY><TABLE width=500 border=0 cellspacing=10><TR><TD>



<h1>The page cannot be found</h1>

The page you are looking for might have been removed, had its name changed, or is temporarily unavailable.

<hr>

<p>Please try the following:</p>

<ul>

<li>Make sure that the Web site address displayed in the address bar of your browser is spelled and formatted correctly.</li>

<li>If you reached this page by clicking a link, contact

 the Web site administrator to alert them that the link is incorrectly formatted.

</li>

<li>Click the <a href="javascript:history.back(1)">Back</a> button to try another link.</li>

</ul>

<h2>HTTP Error 404 - File or directory not found.<br>Internet Information Services (IIS)</h2>

<hr>

<p>Technical Information (for support personnel)</p>

<ul>

<li>Go to <a href="http://go.microsoft.com/fwlink/?linkid=8180">Microsoft Product Support Services</a> and perform a title search for the words <b>HTTP</b> and <b>404</b>.</li>

<li>Open <b>IIS Help</b>, which is accessible in IIS Manager (inetmgr),

 and search for topics titled <b>Web Site Setup</b>, <b>Common Administrative Tasks</b>, and <b>About Custom Error Messages</b>.</li>

</ul>



</TD></TR></TABLE></BODY></HTML>

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">

<HTML><HEAD><TITLE>The page cannot be found</TITLE>

<META HTTP-EQUIV="Content-Type" Content="text/html; charset=Windows-1252">

<STYLE type="text/css">

  BODY { font: 8pt/12pt verdana }

  H1 { font: 13pt/15pt verdana }

  H2 { font: 8pt/12pt verdana }

  A:link { color: red }

  A:visited { color: maroon }

</STYLE>

</HEAD><BODY><TABLE width=500 border=0 cellspacing=10><TR><TD>



<h1>The page cannot be found</h1>

The page you are looking for might have been removed, had its name changed, or is temporarily unavailable.

<hr>

<p>Please try the following:</p>

<ul>

<li>Make sure that the Web site address displayed in the address bar of your browser is spelled and formatted correctly.</li>

<li>If you reached this page by clicking a link, contact

 the Web site administrator to alert them that the link is incorrectly formatted.

</li>

<li>Click the <a href="javascript:history.back(1)">Back</a> button to try another link.</li>

</ul>

<h2>HTTP Error 404 - File or directory not found.<br>Internet Information Services (IIS)</h2>

<hr>

<p>Technical Information (for support personnel)</p>

<ul>

<li>Go to <a href="http://go.microsoft.com/fwlink/?linkid=8180">Microsoft Product Support Services</a> and perform a title search for the words <b>HTTP</b> and <b>404</b>.</li>

<li>Open <b>IIS Help</b>, which is accessible in IIS Manager (inetmgr),

 and search for topics titled <b>Web Site Setup</b>, <b>Common Administrative Tasks</b>, and <b>About Custom Error Messages</b>.</li>

</ul>



</TD></TR></TABLE></BODY></HTML>

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">

<HTML><HEAD><TITLE>The page cannot be found</TITLE>

<META HTTP-EQUIV="Content-Type" Content="text/html; charset=Windows-1252">

<STYLE type="text/css">

  BODY { font: 8pt/12pt verdana }

  H1 { font: 13pt/15pt verdana }

  H2 { font: 8pt/12pt verdana }

  A:link { color: red }

  A:visited { color: maroon }

</STYLE>

</HEAD><BODY><TABLE width=500 border=0 cellspacing=10><TR><TD>



<h1>The page cannot be found</h1>

The page you are looking for might have been removed, had its name changed, or is temporarily unavailable.

<hr>

<p>Please try the following:</p>

<ul>

<li>Make sure that the Web site address displayed in the address bar of your browser is spelled and formatted correctly.</li>

<li>If you reached this page by clicking a link, contact

 the Web site administrator to alert them that the link is incorrectly formatted.

</li>

<li>Click the <a href="javascript:history.back(1)">Back</a> button to try another link.</li>

</ul>

<h2>HTTP Error 404 - File or directory not found.<br>Internet Information Services (IIS)</h2>

<hr>

<p>Technical Information (for support personnel)</p>

<ul>

<li>Go to <a href="http://go.microsoft.com/fwlink/?linkid=8180">Microsoft Product Support Services</a> and perform a title search for the words <b>HTTP</b> and <b>404</b>.</li>

<li>Open <b>IIS Help</b>, which is accessible in IIS Manager (inetmgr),

 and search for topics titled <b>Web Site Setup</b>, <b>Common Administrative Tasks</b>, and <b>About Custom Error Messages</b>.</li>

</ul>



</TD></TR></TABLE></BODY></HTML>

JobName = 1155 2001, Success = false
Error: Not Found
JobName = 1093 0088, Success = false
Error: Not Found
JobName = 1093 0092, Success = false
Error: Not Found
JobName = 1170 0042, Success = false
Error: Not Found
JobName = 1173 1026, Success = false
Error: Not Found

Erel · Dec 6, 2012

You might be breaking your server. It is not recommended to send many requests at once to the same server.
The server might protect itself by holding some of the requests.

You should instead download a few links and then whenever a job completes submit a new one.

kanaida · Dec 6, 2012

I'll try saving the list of images in a global variable and iterating through them as each finishes. I'll see if I can update your httpUtils2 to include a queue mechanism that limits active requests to X amount. It'd be nice never having to worry about that again

Out of curiosity, I looked at my server just to see what it had to say.
It's IIS 6.0 on Server 2003 x32, on a 10Mbit Up/Down pipe.
connections limited to 2,000,000,000 (lol, no kidding.)

I was asking for about 200 Images that are 172x172 at the biggest. (10-30Kb each)

kanaida · Dec 7, 2012

Thanks, that worked. Loading one after the other... My only problem now though, is that there's about 500 images. I'm running into memory constraints.

I see that the google market gets around this by loading stuff as needed. How do you know what's visible in the Listview though? I don't even see a Scolling() event?

I'd imagine then I could do Image = Null for stuff that's out of view, and Load it from disk if i have a copy in my cache or download it when it comes into view.

Erel · Dec 7, 2012

Please start a new thread for this question as it is not related to this thread.

HttpUtils2 can't seem to download long lists

kanaida

Active Member

Erel

B4X founder

kanaida

Active Member

kanaida

Active Member

Erel

B4X founder

kanaida

Active Member

kanaida

Active Member

Erel

B4X founder