B4J Question PostMultipart - Handle on B4J Server

[Harris]

I have larger mp4 files to upload from devices to my server.
I have been using "PostFile", which does work but seems to take considerable time when uploading (say) a 20 meg file (over cell or wifi).

Now I am trying to test speed difference with PostMultipart - get nothing but failure.
( Failed Postfile: 20190306_112004.mp4 msg: java.net.SocketException: sendto failed: EPIPE (Broken pipe) )

I see many example to send the file - but nothing on how to receive it on the server... (EXCEPT for PHP examples - Thanks Don...).

B4A code:

Sub SendPostFile(dir As String, filename As String)
    Dim st, en As Long = 0
    CallSubDelayed(Main,"SetLargeFile") ' show a label that upload has started
    Sleep(500) ' need delay to make label show

    st = DateTime.Now

    Dim j As HttpJob
    j.Initialize( "", Me)
    ' Note: this works - with much time to complete...
    '    j.PostFile(Starter.srvlinkins & "?type=file&name=" & su.EncodeUrl( filename, "UTF8") , dir, filename)

' Now - try and substitute Multipart...

    Dim fd As MultipartFileData
    fd.Initialize
    fd.KeyName = "file"
    fd.Dir = dir
    fd.FileName = filename
    fd.ContentType = "application/octet-stream"
    j.PostMultipart(Starter.srvlinkins,  CreateMap("name": filename), Array(fd))


    Wait For (j) JobDone(j As HttpJob)
    If j.Success Then
    Log("Current link: " & Starter.link& "  Dir: "&dir&"   File: "&filename)
    Log(" Success PostFile: "&j.GetString)
    ToastMessageShow("Large File Was Sent: "&CRLF&filename, True) '  &" msg: "&j.ErrorMessage,False)

    Else
    LogColor(" Failed Postfile: "&filename  &" msg: "&j.ErrorMessage,Colors.Red)
    ToastMessageShow(" Failed Postfile: "&filename  &" msg: "&j.ErrorMessage,False)
    End If
    j.Release
           
    en = DateTime.Now

    Log(" it took this time to send: "&(en-st))
    CallSubDelayed(Main,"StopMsgLabel")   ' hide label when upload has finished
   
End Sub

B4J Server side:

Select reqType
    Case "file"
           
            Dim name As String = req.GetParameter("name")
            Dim fd As String =  File.DirApp &"/"&Main.inspectFolder
            File.MakeDir(File.DirApp , Main.inspectFolder)
            Dim out As OutputStream = File.OpenOutput( fd, name, False)
           
            File.Copy2(In, out)
            out.Close
            Log("Postbyte Received : " & name & ", size=" & File.Size(fd, name))
            resp.Write(" postbyte saved this file: "&name)
           
End Select

ABM uses PostMultiPart - which I have used well, yet he handles the processing, for the most part...

Sub Handle(req As ServletRequest, resp As ServletResponse)  

    Dim data As Map  
    Dim fileName As String
    Dim tmpFileName As String
    Try

        data = req.GetMultipartData( downloadfolder,  MaxSize )
             
     Catch
            Log(" Download Error: "&LastException.Message)
     End Try      

     Dim filePart As Part = data.Get("upl")  
               
     If filePart.IsInitialized Then
            fileName =     filePart.SubmittedFilename
            tmpFileName = filePart.TempFile  
     End If          
       

' internally handle this type of upload request with - ABM.HandleUpload
       
        If ABM.HandleUpload(  downloadfolder, tmpFileName, fileName) Then
            CallSubDelayed3(callback, "Page_FileUploaded", fileName, True)
        End If

End Sub

Thanks

 

[OliverA]

I don’t think switching to multi part will help. Unless you are using a super slow server, I’m going to say that the bottle neck is the upload speed of the devices that are sending you these files. What is the average upload speed for the devices in question? For example, at my house, cell service is lousy. I’m getting 4Mbits down and 1Mbits up. At that upload speed, with everything going smoothly, it’ll take at least 200 seconds to upload a 20Mbyte file. Rough calculation: 20000000bytes / ((1000000bits / (8bits/byte))*.8)

The *.8 is for the overhead of the packets that transport your data. It’s a rough, from the hip estimate that I was told about in my networking days.

 

[Harris]

it’ll take at least 200 seconds to upload a 20Mbyte file

That is what I am seeing from my vantage point. I am connected to the high-speed fiber connection over a wifi extender that is 30 feet away, having signal to go thru the metal walls of my RV.

When I move next to the extender (2 feet away), same file upload is reduced to around 30 seconds.

The biggest issue is to reduce the size of these mp4 files - in the first place. I contacted @moster67 about his compression lib.
He took a 76 meg file of mine (mp4 - with 30 seconds of data) and reduced it to 5 meg (aggressive) and 15 meg (moderate).
The output of both was more than acceptable. The only difference I could detect was a slight bit of (white) pixelation of the blue sky in the stream. The audio was unchanged. These reduced files play well on the ABMVideo viewer (on my server site) and on the device.
I shall go this route since it makes sense all-round... (less data to upload - less data to store on server - less data to playback).

What is strange however:
I use the ExoPlayer (Erel's Simple Example) to "Play" the uploaded video on the Android device (again - 20 meg).
It takes far less time (1/4) to play it from the server than it did to upload the same file to it!!!
(sources.Add(player1.CreateUriSource(link&vidfile)) '"/samplevideo1_mediumCRF28.mp4")))

What's up with that? It is the same size to read back?

This is what started me on this rant...

Thanks @OliverA

 

[OliverA]

Did you notice my phone’s Speedtest? 4 down and 1 up. So my phone can download 4 times faster than it can upload. So a file may take 200 seconds to upload, but only 50 to donload. Of course, the server must have good upload speeds (most do).

 

[Harris]

OK, that makes sense. Doing connection speed tests do show a big difference between up and down...
So noted, and understood...

 

[Harris]

However insightful, this banter did not answer the principal question - how to receive (multipart) on the server (without php)?
Is it possible?

Thanks

 

[DonManfred]

Is it possible?

Don´t know; never tried.

But i guess you should get the hole Payload using the req.Inputstream. Get the requests Inputstream before you use any req.Get methods.
Write the payload to a file. Check the filecontent. It may be an mix of Data and some additional lines .

Guess you need to parse the payload by yourself.

 

[Erel]

Is it possible?

Yes:

Dim data As Map = req.GetMultipartData(Main.TempDir, 100000) 'max 100,000 bytes
For Each key As String In data.Keys
 Dim p As Part = data.Get(key)
 Log(key & ": " & p)
Next

 

[Harris]

Yes

In my server handler, where do I slip your jewel in to handle it?
My handler processes many things - based on it's req.GetParameter("type") - (examples found on forum.)

Sub Handle( req As ServletRequest, resp As ServletResponse)

    If req.Method <> "POST" Then
        resp.SendError(500, "method not supported.")
        Return
    End If

    'we need to call req.InputStream before calling GetParameter.
    'Otherwise the stream will be read internally (as the parameter might be in the post body).

    Dim In As InputStream = req.InputStream
    
    Dim reqType As String = req.GetParameter("type")   
' will this work with PostMultipart - or we need separate handler????
' with PostMultipart - we don't explicitly set a "type"
    
    If reqType = "" Then
        resp.SendError(500, "Missing type parameter.")
        Return
    End If

    Select reqType

        Case "getmissed"    ' full example - all others truncated - but work in same fashion
            Dim tr As TextReader
            tr.Initialize(In)
            
            Log("Received getmissed message: Size "&In.BytesAvailable)
            
            Dim JSON As JSONParser
            Dim l1 As List
            l1.Initialize
            
            Dim Map1, Map2 As Map
            JSON.Initialize(tr.ReadAll) 'Read the text from a file.
        
            l1 = JSON.NextArray
            
            Log(" how many elements in l1: "&l1.Size)
            Dim tok As String = ""
            
            Map1.Initialize
            Map1 = l1.Get(0)
            tok = Map1.Get("token")
            If tok.Length > 20 Then
                CallSubDelayed2(Main,"SendtoToken",tok)
                resp.Write("OK")
            Else
                resp.Write("Invalid Token")
            End If   


        Case "subscription"
            
            Log("Received newsubs message: Size "&In.BytesAvailable)
            ' ....  do more stuff
            resp.Write("OK")


        Case "newsubs"
            
            Log("Received newsubs message: Size "&In.BytesAvailable)
            '  do more stuff...
            resp.Write("subscriptions handled...)
            
            
        Case "register"

            Log("Received text message: Size "&In.BytesAvailable)
            ' do more stuff.....
            tresp = UpdateUsers(Map1)
            resp.Write(tresp)

        Case "text"
            
            Log("Received text message: Size "&In.BytesAvailable)
            ' do much more stuff....
            resp.Write(postkey)
            
        Case "file"  ' POSTFILE or POSTBYTE
            
            Dim name As String = req.GetParameter("name")
            Dim fd As String =  File.DirApp &"/"&Main.inspectFolder
            File.MakeDir(File.DirApp , Main.inspectFolder)
            Dim out As OutputStream = File.OpenOutput( fd, name, False)
            
            File.Copy2(In, out)
            out.Close
            Log("Postbyte Received : " & name & ", size=" & File.Size(fd, name))
            resp.Write(" postbyte saved this file: "&name)
            
    End Select

End Sub

Thanks all...

 

[Erel]

It might be simpler to split it into different handlers.

I'm not sure that you can call GetParameter before calling GetMultipartData (try it).

 

[Harris]

Closer....

Handler looks at the req.ContentType to determine if multipart request...

    Dim conttype As String = req.ContentType
    If conttype.Contains("multipart/form-data")   Then  '= "application/octet-stream"
         Log("Multipartpost file: "&conttype)
        Dim name As String = req.GetParameter("name")
        Dim fd As String =  File.DirApp &"/"&Main.inspectFolder
        File.MakeDir(File.DirApp , Main.inspectFolder)
    
        Dim data As Map = req.GetMultipartData(fd, 100000000 * 1000) 'max 10000000000 bytes
        For Each key As String In data.Keys
            Dim p As Part = data.Get(key)
            Log(key & ": " & p)
        Next
        Return
    Else
        Log("NOT Multipartpost file: "&conttype)
    End If
    
'continue on and process as other types....

The file gets sent and resides on the server, but with a multipartxxxxxxxxxxx file name.

Here are the results of testing with multipart and PostFile using the asme large file (22.5 meg).
Seems MP file sends are a bit faster than PostFile...

1rst multipart time
it took this time to send: 104088  (23.5 meg size)

Multipartpost file: multipart/form-data; boundary=---------------------------1461124740692
' the log() output...   The filename (fn) is there - but it doesn't change the result file???
file: (MultiPart) Part{n=file,fn=20190306_112004.mp4,ct=application/octet-stream,s=24070233,t=true,f=/var/www/www/ccwatch/in/MultiPart5362183876696345125}
name: (MultiPart) Part{n=name,fn=null,ct=null,s=19,t=true,f=null}

2nd multipart time
it took this time to send: 134664  (23.5 meg size)
Multipartpost file: multipart/form-data; boundary=---------------------------1461124740692
 was sent
file: (MultiPart) Part{n=file,fn=20190306_112004.mp4,ct=application/octet-stream,s=24070233,t=true,f=/var/www/www/ccwatch/in/MultiPart1037759947691331369}
name: (MultiPart) Part{n=name,fn=null,ct=null,s=19,t=true,f=null}

3rd multipart time
 it took this time to send: 119002
Multipartpost file: multipart/form-data; boundary=---------------------------1461124740692
fn: (MultiPart) Part{n=fn,fn=null,ct=null,s=19,t=true,f=null}
filepart: (MultiPart) Part{n=filepart,fn=20190306_112004.mp4,ct=application/octet-stream,s=24070233,t=true,f=/var/www/www/ccwatch/in/MultiPart319909230147824828}
namepart: (MultiPart) Part{n=namepart,fn=null,ct=null,s=19,t=true,f=null}

4th multipart time
 it took this time to send: 104500
Multipartpost file: multipart/form-data; boundary=---------------------------1461124740692
fn: (MultiPart) Part{n=fn,fn=null,ct=null,s=19,t=true,f=null}
filepart: (MultiPart) Part{n=filepart,fn=20190306_112004.mp4,ct=application/octet-stream,s=24070233,t=true,f=/var/www/www/ccwatch/in/MultiPart7488826129733347884}
namepart: (MultiPart) Part{n=namepart,fn=null,ct=null,s=19,t=true,f=null}

5th multipart time
 it took this time to send: 93304
(same as above)

6th multipart time
 it took this time to send: 90867



********************************************************* POST File Tests


1rst with POSTFILE:
it took this time to send: 210011

2nd with POSTFILE:
it took this time to send: 121526

3rd with POSTFILE:
 it took this time to send: 124315

4th with POSTFILE:
 it took this time to send: 173794

5th with POSTFILE:
 it took this time to send: 132917

6th with POSTFILE:
 it took this time to send: 128354

 

[Erel]

Seems MP file sends are a bit faster than PostFile...

This is not possible. You are doing something wrong.

 

[Harris]

This is not possible. You are doing something wrong.

Yea - I wish I knew... Yet the logs state time spent in each process - either MP or PF.

The file gets sent and resides on the server, but with a multipartxxxxxxxxxxx file name.

I will gladly abandon this exercise in higher learning if this can't be (simply) resolved.
You, I, and everyone else have better things to do than chasing our tails - to no avail.

@ erel - Like and I stop right here... Save this issue for another day.