Uploading large files with POST multipart/form-data

[thedesolatesoul]

This is something I have been struggling to understand for a while.

The code I have been using to upload files is:

Sub PrepareFileToUpload (link As String, Dir As String, FileName As String) As HttpRequest
   Dim req As HttpRequest
   req.InitializePost2(HttpUtils.EncodeUrl(link), ("file=" & HttpUtils.EncodeUrl(FileName)).GetBytes("UTF8"))
   req.SetContentType("application/x-www-form-urlencoded")
   OAuth.Sign(req)
   Dim authHeader As String
   authHeader = OAuth.GetAuthorizationHeaderValue(req)
   'write the file
   Dim stream As OutputStream

   Dim size_i As Int 
   stream.InitializeToBytesArray(20)
   
   Dim EOL As String
   EOL = Chr(13) & Chr(10) 'CRLF constant matches Android end of line character which is chr(10).
   Dim b() As Byte
   b = ("--" & boundary & EOL & "Content-Disposition: form-data; name=" _ 
      & QUOTE & "file" & QUOTE & "; filename=" & QUOTE & FileName & QUOTE _
      & EOL & "Content-Type: " & "application/octet-stream" & EOL & _ 
      "Content-Transfer-Encoding: binary" & EOL & EOL).GetBytes("UTF8")
   stream.WriteBytes(b, 0, b.Length)
   Dim in As InputStream
   in = File.OpenInput(Dir, FileName)
   File.Copy2(in, stream) 'read the file and write it to the stream
   
   b = EOL.GetBytes("UTF8")
   stream.WriteBytes(b, 0, b.Length)
   b = (EOL & "--" & boundary & "--" & EOL).GetBytes("UTF8")
   stream.WriteBytes(b, 0, b.Length)
   b = stream.ToBytesArray

   Dim in As InputStream 
   in.InitializeFromBytesArray(b,0,b.Length)
   UploadLength = in.BytesAvailable 
   PostInputStream.Initialize(in)

   Dim req As HttpRequest
   'req.InitializePost2(link, b)
   
   req.InitializePost(HttpUtils.EncodeUrl(link), PostInputStream,b.Length)
   req.SetContentType("multipart/form-data; boundary=" & boundary)
   req.SetContentEncoding("UTF8")
   req.SetHeader("Authorization", authHeader)
   Return req
End Sub

When uploading relatively small files, the upload works fine for me, mainly because:

b = stream.ToBytesArray

is a small value. But any file over 12-13 MB causes memory overflow exceptions.

This means in order to upload large files I must use a (buffered) stream and not load the whole file in to a byte array.
However I struggle to use a an input stream to create the POST request, because I need to add some more form data (filename etc).
So what I do not understand is how to create a form request with both the form-data and binary file.
As you can see I am writing everything out to an OutputStream and then gettings its bytes which will give me Memory overflow errors. Alternatively, I can write the Output stream to file, but that is heavy on the sdcard. I am trying to find a way where I can direct the outputstream to the inputstream without resorting to the intermediate.
If this doesnt work, then I will need to find an alternative way to upload files maybe by wrapping the Http Multipart Request object?

 

[Erel]

Do you have control over the server side? If yes then you can use a simpler format.

I don't see any problem with temporary saving the stream in the sdcard.

 

[thedesolatesoul]

No, I have no control on the server side. The servers are the dropbox servers and they expect

We require a multipart upload (multipart/form-data), and the filename parameter of this field should be set to the desired destination filename. While signing this request for OAuth, the file parameter should be set to the destination filename, and then switched to the file contents when preparing the multipart request.

Some files to be uploaded can be upto 150MB (I think this is the Dropbox upload limit).
I guess for now this is the better solution to save and remove them provided, the user does not have a full sdcard (or maybe he doesnt even have an sdcard!).

Is there another way to join the header with a stream?

 

[Erel]

Your only option is to use a file. Otherwise you will need to hold the entire data in memory (which is more limited).

 

[thedesolatesoul]

Okay, thanks Erel!

 

[thedesolatesoul]

Another question!
Do you think I should use DirDefaultExternal or should I use DirInternalCache?

 

[Erel]

If the sdcard is available and writable then you should prefer to use it. You can also check the available space before starting (though 15mb is not too large).

 

[thedesolatesoul]

It is actually 150MB not 15MB. But still some people are complaining about increasing the upload size even more. Anyway that day is far!
I will have to implement it this way.
Thanks a lot Erel

 

[vb1992]

So that solves your question?

"So what I do not understand is how to create a form request with both the form-data and binary file."

I would think for speed reasons, you still would want to use this method for files under 10MB?

 

[thedesolatesoul]

So that solves your question?

"So what I do not understand is how to create a form request with both the form-data and binary file."

I would think for speed reasons, you still would want to use this method for files under 10MB?

Yes.
But I am also going to experiment on what happens if I dont send the header at all. (By looking inside the Dropbox sdk, they dont do this at all).

 

[Roger Garstang]

I ran into this issue a lot even in other languages. I developed my own web server in PowerBASIC that runs in the notification/task tray in Windows. You really have to do a lot of digging in HTTP RFCs to fully understand everything, but it is fairly simple once you get beyond the weird notation used in the RFCs. I've wrote a lot of applications in Windows and in .Net on Win Mobile too that interface directly with a PHP script sending form data and file data like pictures just over a straight TCP communication to port 80.

It is possible to send both in one go too as I do it all the time. Multi-Part data has divisions of what I'd call fences and each "fence" contains a header describing the data to follow then at the end of all the pieces and fences is a final "fence" marking the end. If you get a tool like Wireshark you can monitor your HTTP traffic and see all the headers and data sent to get an idea of how forms on a web page send. It really helps to see it. Viewing a Raw email file is helpful too as most emails are also multi-part.

I luck out on Win Mobile in that they usually are sending a file so the stream reader can parse it a byte/block at a time. Even on Windows though I ran into issues trying to load a large file into a string/byte array and sending it. Most languages don't have support for a low enough level of TCP/IP to supply them a block at a time, so you end up having to give an entire stream/byte array. HTTP does have a Chunked Data mode for sending, but most docs I read on it seem to indicate server side and not coming from browser/application side. There has to be a way to do it better though since Browsers have no issue sending large amounts of data.

 

[Roger Garstang]

I don't see any problem with temporary saving the stream in the sdcard.

Internal memory in devices appears to be different or of a better quality than most SD Cards. Even the Commercial/Industrial Grades wear out quick when used for temporary files. The worse thing to put on an SD Card is a Database too. We had some Commercial Grade SD Cards in some Win Mobile devices that wore out in 1-2 months with a database on it. Temp files would put some similar wear...especially if the controller on the card has no wear leveling. You may not see it as much storing small files on 8, 16, or 32GB cards since they have room to use other areas with leveling, but they all wear out eventually.

When we moved the database to internal we had no issues and the devices have been going 3-4 years now. (Internal is less than 1GB too and shows no signs of wear/errors) I disagree with a lot of the rooters/flashers over at XDA and various other boards too in how they Flash ROMs. They wear out the same or worse than SD Cards. Early devices didn't have any leveling and wore out in less than 5-10 updates. Newer ones are slightly longer, but all those claiming to Flash to a stock ROM then update every time have no clue and it isn't accomplishing anything but wearing out the chip.

 

[thedesolatesoul]

Okay, so here is the code I tried:

Sub PrepareFileToUpload (link As String, Dir As String, FileName As String) As HttpRequest
   Dim req As HttpRequest
   req.InitializePost2(HttpUtils.EncodeUrl(link), ("file=" & HttpUtils.EncodeUrl(FileName)).GetBytes("UTF8"))
   req.SetContentType("application/x-www-form-urlencoded")
   OAuth.Sign(req)
   Dim authHeader As String
   authHeader = OAuth.GetAuthorizationHeaderValue(req)
   'write the file
   
   Dim in As InputStream
   Dim filesize As Int 
   Dim stream As OutputStream

   in = File.OpenInput(Dir, FileName)
   filesize = in.BytesAvailable 

   If filesize < 4096 Then
      stream.InitializeToBytesArray(20)
   Else
      'Copy!       'Check sdcard space       'Check writable
      stream = File.OpenOutput(File.DirDefaultExternal,"temp",False)
   End If
   
   Dim EOL As String
   EOL = Chr(13) & Chr(10) 'CRLF constant matches Android end of line character which is chr(10).
   Dim b() As Byte
   b = ("--" & boundary & EOL & "Content-Disposition: form-data; name=" _ 
      & QUOTE & "file" & QUOTE & "; filename=" & QUOTE & FileName & QUOTE _
      & EOL & "Content-Type: " & "application/octet-stream" & EOL & _ 
      "Content-Transfer-Encoding: binary" & EOL & EOL).GetBytes("UTF8")
   stream.WriteBytes(b, 0, b.Length)
   File.Copy2(in, stream) 'read the file and write it to the stream
   
   b = EOL.GetBytes("UTF8")
   stream.WriteBytes(b, 0, b.Length)
   b = (EOL & "--" & boundary & "--" & EOL).GetBytes("UTF8")
   stream.WriteBytes(b, 0, b.Length)

   Dim in As InputStream 
   If filesize < 4096 Then
      b = stream.ToBytesArray
      in.InitializeFromBytesArray(b,0,b.Length)   
   Else
      stream.Flush 
      stream.Close 
      in = File.OpenInput(File.DirDefaultExternal,"temp")
   End If

   UploadLength = in.BytesAvailable 
   PostInputStream.Initialize(in)

   Dim req As HttpRequest
   'req.InitializePost2(link, b)
   
   'req.InitializePost(HttpUtils.EncodeUrl(link), PostInputStream,b.Length)
   req.InitializePost(HttpUtils.EncodeUrl(link), PostInputStream,in.BytesAvailable )
   req.SetContentType("multipart/form-data; boundary=" & boundary)
   req.SetContentEncoding("UTF8")
   req.SetHeader("Authorization", authHeader)
   Return req
End Sub

So no more straight up crashes now.
Files < 4MB are quick.
I tried a 100MB file, but copying is too slow. My device twice gave me: Application is taking too long to respond, Wait, Close. If I wait the file starts getting uploaded fine.
So I dont see this file copy as a viable solution. Even assuming 4MB/sec sdcard copy speed (which is optimistic and not considering other overhead, read/write at same time), this means 25 seconds in which the device is copying and the UI is unresponsive.
I looked at the Dropbox SDK and they are using '/files_put' which is simpler than POSTing the file because the body is just the file and any params are part of the url. However this method is only available in API ver 1 (I am using ver 0).

 

[Erel]

In order to avoid the ANR dialog you should copy the file yourself (not with File.Copy2) and add DoEvents every x iterations (start with x = 100).

 

[thedesolatesoul]

Thanks for the cunning idea Erel. :sign0188: Sometimes I wonder why I cant think of this things myself?
Works great.

I'll post the code here for reference:

Sub PrepareFileToUpload (link As String, Dir As String, FileName As String) As HttpRequest
   Dim req As HttpRequest
   req.InitializePost2(HttpUtils.EncodeUrl(link), ("file=" & HttpUtils.EncodeUrl(FileName)).GetBytes("UTF8"))
   req.SetContentType("application/x-www-form-urlencoded")
   OAuth.Sign(req)
   Dim authHeader As String
   authHeader = OAuth.GetAuthorizationHeaderValue(req)
   'write the file
   
   Dim in As InputStream
   Dim filesize As Int 
   Dim stream As OutputStream

   in = File.OpenInput(Dir, FileName)
   filesize = in.BytesAvailable 

   If filesize < 4096 Then
      stream.InitializeToBytesArray(20)
   Else
      'Copy!       'Check sdcard space       'Check writable
      stream = File.OpenOutput(File.DirDefaultExternal,"temp",False)
   End If
   
   Dim EOL As String
   EOL = Chr(13) & Chr(10) 'CRLF constant matches Android end of line character which is chr(10).
   Dim b() As Byte
   b = ("--" & boundary & EOL & "Content-Disposition: form-data; name=" _ 
      & QUOTE & "file" & QUOTE & "; filename=" & QUOTE & FileName & QUOTE _
      & EOL & "Content-Type: " & "application/octet-stream" & EOL & _ 
      "Content-Transfer-Encoding: binary" & EOL & EOL).GetBytes("UTF8")
   stream.WriteBytes(b, 0, b.Length)

   'Manually copy stream to keep responsiveness
   Dim offset As Int 
   offset = 0 
   Dim buffer(2048) As Byte 
   Dim refreshInterval As Int 
   refreshInterval  = 0
   Dim rb As Int
   Do While in.BytesAvailable > 0
       rb = in.ReadBytes(buffer,0,buffer.Length )
       offset = offset + rb
       stream.WriteBytes(buffer,0,rb)
       refreshInterval = refreshInterval + 1
       If refreshInterval = 100 Then
          DoEvents
         refreshInterval = 0
       End If
   Loop
   
   b = EOL.GetBytes("UTF8")
   stream.WriteBytes(b, 0, b.Length)
   b = (EOL & "--" & boundary & "--" & EOL).GetBytes("UTF8")
   stream.WriteBytes(b, 0, b.Length)

   Dim in As InputStream 
   If filesize < 4096 Then
      b = stream.ToBytesArray
      in.InitializeFromBytesArray(b,0,b.Length)   
   Else
      stream.Flush 
      stream.Close 
      in = File.OpenInput(File.DirDefaultExternal,"temp")
   End If

   UploadLength = in.BytesAvailable 
   PostInputStream.Initialize(in)

   Dim req As HttpRequest
   req.InitializePost(HttpUtils.EncodeUrl(link), PostInputStream,in.BytesAvailable )
   req.SetContentType("multipart/form-data; boundary=" & boundary)
   req.SetContentEncoding("UTF8")
   req.SetHeader("Authorization", authHeader)
   Return req
End Sub