Android Question [Solved] HttpJob error with unexpected end of stream

[max123]

Hi all,

I started on my PC an http server to serve a directory.
In Windows prompt I write this command: (for convenience I use python command line)

PS C:\Users\Massimo\Downloads\three.js-master> python -m http.server

and python reply this:

Serving HTTP on :: port 8000 (http://[::]:8000/) ...

started to serving a three.js-master directory on localhost:8000 as default port.

I just used python command line to test, but next I will use B4J with http server.

Now I want download a file from B4A with OkHttpUtils2 and HttpJob.

Because I will test on Android Emulator, initially I had troubles because I've used 'localhost' or 127.0.0.1 in my requests,
and this cannot work because it refer to the same android device and not to a localhost of host machine.

I works a lot with networks, but I learned this on this thread thanks to @aeric tip.
https://www.b4x.com/android/forum/threads/httpjob-not-connecting-to-localhost.145737/#post-989752
Now that I remember I had the same issue on VirtualBox in past.

As expected this don't worked because different loopbacks and python server on host machine always refused to connect.

Now, searching on the web, I found a tip to use 10.0.2.2 as IP loopback to connect the emulator to the host machine.
This works now, and the emulator can connect to the python server started on the host machine.

Here a code:

Private Sub Button1_Click
    DownloadAndSaveFile ("http://10.0.2.2:8000/examples/models/gcode/benchy.gcode")
End Sub

Sub DownloadAndSaveFile (Link As String)
    Dim j As HttpJob
    j.Initialize("", Me)
    j.Download(Link)
 
'    j.GetRequest.SetHeader("Access-Control-Allow-Origin", "true")
'    j.GetRequest.SetHeader ("ContentType", "text/plain")
    j.GetRequest.SetHeader ("ContentType", "application/octet-stream")
'    j.GetRequest.SetHeader ("Host", "exampleproject.com")
'    j.GetRequest.SetHeader ("Connection", "close")
 
    Wait For (j) JobDone(j As HttpJob)
 
    Dim map As Map = j.Response.GetHeaders
    For i = 0 To map.Size - 1
        Log("HEADER: " & map.GetKeyAt(i) & ": " & map.GetValueAt(i))
    Next
 
    Log(" ")
    Log("Success: " & j.Success)
    Log("StatusCode: " & j.Response.StatusCode)
    Log("ErrorResponse: " & j.Response.ErrorResponse)
    Log(" ")
    
    If j.Success Then
        Dim out As OutputStream = File.OpenOutput(File.DirInternal, "benchy.gcode", False)
        File.Copy2(j.GetInputStream, out)
        out.Close '<------ very important
        Log("DONE")
    Else
        Log("ERROR: " & j.ErrorMessage)
    End If
    j.Release
 
    For Each f As String In File.ListFiles(File.DirInternal)
        Log("File: " & f)
    Next
End Sub

Now it can connect to the server and I see the log on the command prompt that show served file:

::ffff:127.0.0.1 - - [24/May/2024 20:56:28] "GET /examples/models/gcode/benchy.gcode HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [24/May/2024 21:22:17] "GET /examples/models/gcode/benchy.gcode HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [24/May/2024 21:24:16] "GET /examples/models/gcode/benchy.gcode HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [24/May/2024 21:25:22] "GET /examples/models/gcode/benchy.gcode HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [25/May/2024 09:55:52] "GET /examples/models/gcode/benchy.gcode HTTP/1.1" 200 -

And this is the B4A log, here the http code is 200, it even have content length:

** Activity (main) Pause, UserClosed = false **
** Activity (main) Create (first time) **
** Activity (main) Resume **
*** Receiver (httputils2service) Receive (first time) ***
HEADER: content-length: [1940664]
HEADER: content-type: [application/octet-stream]
HEADER: date: [Sat, 25 May 2024 11:14:24 GMT]
HEADER: last-modified: [Fri, 26 Apr 2024 07:52:13 GMT]
HEADER: server: [SimpleHTTP/0.6 Python/3.11.5]

Success: false
StatusCode: 200
ErrorResponse:

ERROR: java.net.ProtocolException: unexpected end of stream
File: virtual_assets

The problem now is that the HttpJob fails with this error:

ERROR: java.net.ProtocolException: unexpected end of stream

I've tried to manage headers but the result not changed.
Note that I placed in the manifest this:

' 28 - Non-ssl (non-https) communication is not permitted by default.
' It can be enabled in B4A v9+ by adding this line to the manifest editor:
CreateResourceFromFile(Macro, Core.NetworkClearText)

Please, can someone help me to know what happen here and because this error appear ?

Thanks

 

[aeric]

Android Emulator

Don't use emulator.

 

[max123]

Emulator is only for debug. Not tried with real device because I need to test on host pc to point the server, and this serve to me because I have to do the same in other codes. I still work on threejs library and probably it need a server to get files served over http server, so I need on development process.

 

[aeric]

I tried with real device, it works with my modified code but getting the same error as yours with emulator.
I think python http server has issue with okhttp in android.

 

[max123]

I tried with real device, it works with my modified code but getting the same error as yours with emulator.
I think python http server has issue with okhttp in android.

Thanks, from browser I can access files.
Seem a problem with headers. I have returned back from a server the right file ContentLength that is same as pointed file size.

I'm not forced to use python http server, I just use it for tests. May I can start a server with same B4A app and access it.
Please can you post your modified code ? So my actual code never worked on real device, you had to modify it....

I still using all this as just a test, because I'm working on a threejs library and all distribution files should be served from http server, not file:///.

 

[aeric]

I just test with a simple B4J server where I put the example file inside www folder. The B4A code also very simple.

Note: Content of benchy.gcode has been truncated to minimize the file size.

 

[max123]

Hi @aeric, your example works well with B4A and B4J server.
Now I want to do the same thing directly on B4A and it won't work.

Please, did you know if B4A HttpServer can serve static files like jServer do ?
Hi don't have to do nothing of special, just serving static files.

What I want to do is to unzip the theejs distribution from File.DirAssets to File.DirInternal, then use HttpServer to serve the root of the distribution,
just inside unzipped threejs-master folder.
Because threejs distribution have subfolders and files and the folder structure is manteined, when any html file in /examples subfolder is launched (by the browser, not by double click), the threejs library is already studied to get import files and any other file via http server, so it just do a request and the server serve the requested files inside a distribution.

Now I tried to acces with HttpServer that way:

    Server.Initialize("Server")
'    Server.TempFolder = File.Combine(File.DirInternal, "www")
    Server.Start(8000)
    Log("Server started")
 
    Private Sub Server_HandleRequest (Request As ServletRequest, Response As ServletResponse)
    Log("Request Client IP address: " & Request.RemoteAddress)
    Log("Request URI: " & Request.RequestURI)

    If Request.Method = "GET" Then
        Log("GET")
        Dim Map As Map = GetParameterNamesMap(Request)
        For i = 0 To Map.Size - 1
            Log("Key: " & Map.GetKeyAt(i))
            Log("Value: " & Map.GetValueAt(i))
        Next
    Else If Request.Method = "POST" Then
        Log("POST")
    End If
End Sub

Private Sub GetParameterNamesMap(Request As ServletRequest) As Map
    Dim ServletRequestWrapper As Reflector

    ServletRequestWrapper.Target = Request
  
    Dim req As JavaObject = ServletRequestWrapper.GetField("req")
    Dim ParameterMap As Map = req.RunMethod("getParameterMap",Null)
    Return ParameterMap
End Sub

And then I used HttpJob to try download a file.

Here my first problem, I put the same www folder you provided in the B4J server example, in Objects folder of B4A, but when I execute the app, the folder disappear, it is deleted
So may I have to change approach and directly unzip it to DirInternal from FileAssets ?

The HttpJob return True Success, I see the server header on the log, so server replies, but contentLength of benchy.gcode is zero bytes.

** Activity (main) Create (first time) **
2024-06-01 09:57:45.181:INFO::jetty-7.x.y-SNAPSHOT
2024-06-01 09:57:45.215:INFO::started o.e.j.s.ServletContextHandler{/,null}
2024-06-01 09:57:45.230:INFO::Started [email protected]:8000 STARTING
org.eclipse.jetty.server.Server@369b0d8
Server started
** Activity (main) Resume **
Start download...
*** Receiver (httputils2service) Receive (first time) ***

Request.GetUploadedFile:
Request Client IP address: 127.0.0.1
Request URI: /examples/models/gcode/benchy.gcode
GET
HEADER: content-length: [0]
HEADER: server: [Jetty(7.x.y-SNAPSHOT)]

Success: true
StatusCode: 200
ErrorResponse:
ContentType:
ContentLength: 0
ContentEncoding:

ERROR:

Listing files in DirInternal ...
File: virtual_assets
File: benchy.gcode
File: www

Please, can you help me get it working ?
I've attached my last not working B4A test project.

Many Thanks

 

[aeric]

I have no experience with HttpServer library. Can you put the file as zip inside assets directory and copy to internal directory then unzip it?

 

[max123]

Yes, I can do.... but did you know because if I put your www folder inside B4A Object folder it is deleted ?
It is normal ?
On B4J this don't happen.
So is not possible place files here and access ? I enforced to use DirAssets ?

Here my full code, for now I just tried to access from Object folder www like B4J:

Sub Process_Globals
    Private Server As HttpServer
End Sub

Sub Globals
    'These global variables will be redeclared each time the activity is created.
    Private Button1 As Button
End Sub

Sub Activity_Create(FirstTime As Boolean)
    Activity.LoadLayout("Layout")
 
'    If File.Exists(File.DirInternal, "www") = False Then
'        File.MakeDir(File.DirInternal, "www")
'    
'    Else
'
'    End If

    Server.Initialize("Server")
'    Server.TempFolder = File.Combine(File.DirInternal, "www")
    Server.Start(8000)
    Log("Server started")
End Sub

Sub Activity_Resume

End Sub

Sub Activity_Pause (UserClosed As Boolean)

End Sub

Private Sub Server_HandleRequest (Request As ServletRequest, Response As ServletResponse)
    Log("Request Client IP address: " & Request.RemoteAddress)
    Log("Request URI: " & Request.RequestURI)

    If Request.Method = "GET" Then
        Log("GET")
        Dim Map As Map = GetParameterNamesMap(Request)
        For i = 0 To Map.Size - 1
            Log("Key: " & Map.GetKeyAt(i))
            Log("Value: " & Map.GetValueAt(i))
        Next
    Else If Request.Method = "POST" Then
        Log("POST")
    End If
End Sub

Private Sub GetParameterNamesMap(Request As ServletRequest) As Map
    Dim ServletRequestWrapper As Reflector

    ServletRequestWrapper.Target = Request
   
    Dim req As JavaObject = ServletRequestWrapper.GetField("req")
    Dim ParameterMap As Map = req.RunMethod("getParameterMap",Null)
    Return ParameterMap
End Sub

Private Sub Button1_Click

'    DownloadAndSaveFile ("http://10.0.2.2:8000/examples/models/gcode/benchy.gcode")
'    DownloadAndSaveFile ("http://localhost:8000/examples/models/gcode/benchy.gcode")
    DownloadAndSaveFile ("http://127.0.0.1:8000/examples/models/gcode/benchy.gcode")
'    DownloadAndSaveFile ("http://192.168.178.51:8000/examples/models/gcode/benchy.gcode") ' OK for B4J jServer
 
End Sub

Sub DownloadAndSaveFile (Url As String)
'    File.Delete(File.DirInternal, "benchy.gcode")
'    Log("DELETED")
 
    Try
        Log("Start download...")
        Dim j As HttpJob
        j.Initialize("", Me)
        j.Download(Url)
 
'    j.GetRequest.SetHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0")
'    j.GetRequest.SetHeader("Access-Control-Allow-Origin", "true")
'    j.GetRequest.SetHeader ("ContentType", "text/plain")
'    j.GetRequest.SetHeader ("ContentType", "application/octet-stream")
'    j.GetRequest.SetHeader ("Host", "exampleproject.com")
'    j.GetRequest.SetHeader ("Connection", "close")
 
        Wait For (j) JobDone(j As HttpJob)
 
        Dim map As Map = j.Response.GetHeaders
        For i = 0 To map.Size - 1
            Log("HEADER: " & map.GetKeyAt(i) & ": " & map.GetValueAt(i))
        Next
        Log(" ")
        Log("Success: " & j.Success)
        Log("StatusCode: " & j.Response.StatusCode)
        Log("ErrorResponse: " & j.Response.ErrorResponse)
        Log("ContentType: " & j.Response.ContentType)
        Log("ContentLength: " & j.Response.ContentLength)
        Log("ContentEncoding: " & j.Response.ContentEncoding)
        Log(" ")
     
        If j.Success And j.Response.ContentLength > 0 Then
            Dim out As OutputStream = File.OpenOutput(File.DirInternal, "benchy.gcode", False)  '"benchy.gcode"
            File.Copy2(j.GetInputStream, out)
            out.Close ' <------ Very important
     
            Log("File saved!    Size: " & File.Size(File.DirInternal, "benchy.gcode") & " Bytes")
         
            Dim s As String = File.ReadString(File.DirInternal, "benchy.gcode")
'            Log(s.SubString2(0, 200)) ' Just log some content
            Log(s)
            Log("DONE")
        Else
            Log("ERROR: " & j.ErrorMessage)
        End If
    Catch
        Log(LastException.Message)
    End Try
    j.Release
 
    Log(" ") : Log("Listing files in DirInternal ...")
    For Each f As String In File.ListFiles(File.DirInternal)
        Log("File: " & f)
    Next
End Sub

 

[aeric]

did you know because if I put your www folder inside B4A Object folder it is deleted ?

Yes, as far as I know, Objects folder is always a temporary folder to store the Java class files and resources during compilation and debug.
You should not put any files inside this folder.
There is exception for B4J.

 

[max123]

I will try to unzip it from assets, so mantain the folder structure, just I test with benchy.gcode for now.

 

[aeric]

I attach here my attempt.

Http Server hosting a page with download link

Just a simple example with a link to a file stored in File.DirInternal. Required Archiver library. This is a response to this thread.

www.b4x.com

 

[max123]

Many thanks @aeric , you are faster than me
My attemp will not yet works.

Here is my last part of code. Now the www folder is on DirAssets, but the server always reply with zero contentLength, no file, but the file is here...

    If File.Exists(File.DirInternal, "www") = False Then
        Log("Not exist")
        File.CopyAsync(File.DirAssets, "www.zip", File.DirInternal, "www.zip")
   
        Wait For (File.CopyAsync(File.DirAssets, "www.zip", File.DirInternal, "www.zip")) Complete (Success As Boolean)
        Log("Success: " & Success)
   
        If Success Then
            Arc.AsyncUnZip(File.DirInternal, "www.zip", File.DirInternal, "Arc")
            Wait For Arc_UnZipDone(CompletedWithoutError As Boolean, NbOfFiles As Int)
            If CompletedWithoutError Then
                Log("Successfully unzipped " & NbOfFiles & " files")
            Else
                Log("Something went wrong while unziping")
                Return
            End If
        Else
            Log("Something went wrong while copy Zip file to DirInternal")
        End If
   
'        File.MakeDir(File.DirInternal, "www")
    Else
      Log("www folder already exists")
        Wait For (File.ListFilesAsync(File.Combine(File.DirInternal, "www"))) Complete (Success As Boolean, Files As List)
        If Success Then
            ListFolder (File.Combine(File.DirInternal, "www"))
        Else
            Log("List not success")
        End If
    End If

    Server.Initialize("Server")
    Server.TempFolder = File.Combine(File.DirInternal, "")
    Server.Start(8000)
    Log("Server started")

Sub ListFolder (folder As String)
    For Each f As String In File.ListFiles(folder)
        If File.IsDirectory(folder, f) Then
            Log(" DIR: " & File.Combine(folder, f) & "   " & File.Size(folder, f) & " Bytes")
            ListFolder(File.Combine(folder, f))
            Continue
        End If
        Log("FILE: " & File.Combine(folder, f) & "   " & File.Size(folder, f) & " Bytes")
    Next
End Sub

Here the log:

** Activity (main) Create (first time) **
www folder already exists
** Activity (main) Resume **
DIR: /data/user/0/b4a.webviewload/files/www/examples 4096 Bytes
DIR: /data/user/0/b4a.webviewload/files/www/examples/models 4096 Bytes
DIR: /data/user/0/b4a.webviewload/files/www/examples/models/gcode 4096 Bytes
FILE: /data/user/0/b4a.webviewload/files/www/examples/models/gcode/benchy.gcode 1598 Bytes
2024-06-01 13:11:52.015:INFO::jetty-7.x.y-SNAPSHOT
2024-06-01 13:11:52.028:INFO::started o.e.j.s.ServletContextHandler{/,null}
2024-06-01 13:11:52.041:INFO::Started [email protected]:8000 STARTING
org.eclipse.jetty.server.Server@a586a2
Server started
Start download...
*** Receiver (httputils2service) Receive (first time) ***

Request.GetUploadedFile:
Request Client IP address: 127.0.0.1
Request URI: /examples/models/gcode/benchy.gcode
GET
HEADER: content-length: [0]
HEADER: server: [Jetty(7.x.y-SNAPSHOT)]

Success: true
StatusCode: 200
ErrorResponse:
ContentType:
ContentLength: 0
ContentEncoding:

ERROR:

Listing files in DirInternal ...
File: virtual_assets
File: benchy.gcode
File: www.zip
File: www

Now I will try your code. Many thanks for your help. Appreciated.
So, you have created something like a code snippet from that. So other user can get it. +1
Max

 

[aeric]

I actually don't understand what you are trying to do.
You create a server app and has a button to download a file from itself.
Isn't this weird?

 

[aeric]

What I understand is a http server is to serve request from another client.
I have no idea what is the point to create a single (server + client) app that download a resource from itself.

Not sure this is similar to what you want. It is for B4J server so not sure it will work in B4A.

[server] jOkHttpUtils2 - server version

OkHttpUtils2 methods are designed to be called on the main thread. This is the case in all platforms and configurations, except of B4J server solutions. Starting from jOkHttpUtils2 v2.95 it is possible to make it thread safe. This is done by following these steps: 1. Add the SERVER conditional...

www.b4x.com

 

[max123]

Your example works even on Emulator, just put http://1.0.2.16:5566 in the emulator browser.
It probably even works with 'localhost' and with 127.0.0.1 if the browser is launched on emulator, in this case I use Chrome.
I don't know if this can be accessed even on the host pc browser.

Apart these irrilevant, but may useful infos on emulator, I will study your example that is not too mutch different than mine, apart that you intelligently used a Starter service to ensure it is not paused and you handled better the requests by cheking files types and more.

I actually tried it only on emulator, Chrome download a benchy.gcode when I click on the link, and it is placed as any downloaded files in /Downloads.

To test with it I will try to attack to your example an HttpJob to download a file, but just for test... it should be act as browser download.

Your example should be similar to that I have to do with threejs library, finally there is a threejs.zip, (instead of www.zip) zip it to DirInternal, then inside /examples subfolder there are a tons of html files, on that there are imports and file loaders for models, textures etc.. that when called just need to be served from a webserves.

I will try to experiment it.

Many Thanks

 

[aeric]

I clean up your project and add the www.zip file. It works and getting file size > 0B.

2024-06-01 22:19:40.038:INFO::jetty-7.x.y-SNAPSHOT
2024-06-01 22:19:40.065:INFO::started o.e.j.s.ServletContextHandler{/,null}
2024-06-01 22:19:40.090:INFO::Started [email protected]:8000 STARTING
org.eclipse.jetty.server.Server@10a6d33
Server started
Ip address: 192.168.50.91
** Activity (main) Resume **
ready
DELETED
Start download...
*** Receiver (httputils2service) Receive (first time) ***
Client: 192.168.50.91
/examples/models/gcode/benchy.gcode
examples/models/gcode/benchy.gcode
/data/user/0/b4a.webviewload/files/www/examples/models/gcode/benchy.gcode
true

File saved! Size: 1598 Bytes

;FLAVOR:Marlin
;TIME:1799
;Filament used: 3.38094m
;Layer height: 0.4

 

[max123]

Great.
You are so faster than me.... probably I'm mutch older than you
I will try it in zero time.
Thanks

 

[aeric]

Now the www folder is on DirAssets, but the server always reply with zero contentLength

You need to know File.Size doesn't work with files inside assets folder.

 

[max123]

Now the www folder is on DirAssets, but the server always reply with zero contentLength

Sorry, my mistake, I had to write DirInternal, I just unzipped the www.zip file from DirAssets to DirInternal, then tried to serve this folder.