B4J Question Async XMLHttpRequest & Multiple File Upload

[tchart]

So to workaround multiple file uploads to jServer I'm using XMLHttpRequest to sequentially upload the files before I submit the form.

I dont know if this is a bug but the first image comes through and the part is an image. For the remaining parts the IsFile attribute is set to False - even though they have been uploaded successfully and are on the server.

Log output below.

Is this a bug?

multipart/form-data; boundary=----WebKitFormBoundaryeES4YOrY1HBx7oRJ
1
-----------------------------------------
Key = e9a31f7d49d34c7194f0af56741591c6
Value = Part{n=e9a31f7d49d34c7194f0af56741591c6,fn=thefly.png,ct=image/png,s=228272,t=true,f=C:\Dropbox\MYAPPS~1\B4J\TheFly\Objects\www\uploads\MultiPart1877381194291074993}
Is a file? true
thefly.png
C:\Dropbox\My Apps\B4J\TheFly\Objects\www\uploads\MultiPart1877381194291074993

multipart/form-data; boundary=----WebKitFormBoundaryBwXtOpvqB7ju4wnF
1
-----------------------------------------
Key = e9a31f7d49d34c7194f0af56741591c6
Value = Part{n=e9a31f7d49d34c7194f0af56741591c6,fn=thefly_480.png,ct=image/png,s=17431,t=true,f=null}
Is a file? false
thefly_480.png
C:\Dropbox\My Apps\B4J\TheFly\Objects\www\uploads\MultiPart3213932765403010606

multipart/form-data; boundary=----WebKitFormBoundaryCK2say2P2m9Kgl0c
1
-----------------------------------------
Key = e9a31f7d49d34c7194f0af56741591c6
Value = Part{n=e9a31f7d49d34c7194f0af56741591c6,fn=thefly_240.png,ct=image/png,s=8213,t=true,f=null}
Is a file? false
thefly_240.png
C:\Dropbox\My Apps\B4J\TheFly\Objects\www\uploads\MultiPart6296547987014848106

 

[Erel]

Can you post the html + JS code?

 

[tchart]

Here is the Javascript;

function fileinfo()
{
    var files= document.getElementById("files").files;
   
    document.getElementById("upload_queue").value = document.getElementById("upload_queue").value + files.length;
    
     //get the input and UL list
    var input = document.getElementById('files');
    var list = document.getElementById('fileList');
    var id = document.getElementById("id").value;

    //for every file...
    for (var x = 0; x < input.files.length; x++)
    {
        console.info("Uploading " + x);   
        uploadImage(id, x);
    }
}

function uploadImage(id, x)
{
    var icon = document.getElementById("upload_icon");
    icon.innerHTML = '<i class="fa fa-spinner  fa-pulse fa-2x fa-fw" style="color:black;"></i>';
       
    var fd = new FormData();
    fd.append(id, document.getElementById('files').files[x]);
    var xhttp = new XMLHttpRequest();
   
    xhttp.onreadystatechange = function(){
        if (this.readyState == 4 && this.status == 200)
        {
            document.getElementById("upload_queue").value = document.getElementById("upload_queue").value - 1;
           
            if (document.getElementById("upload_queue").value == 0)
            {
                icon.innerHTML = '<i class="fa fa-upload fa-2x fa-fw"></i>';
                document.getElementById("files").value = "";
            }
        }
    };

    xhttp.open("POST", "./uploadimage", true);
    xhttp.send(fd);
}

And the HTML for the file input;

<input type="file" name="files[]" id="files" multiple accept="image/*" onchange="fileinfo()">

The idea is that when a file or files are dropped onto the file input they are uploaded.

It is working BTW its just not all parts are identified as files even though they are. I have tried in different browsers etc.

Thanks

 

[tchart]

@Erel, looks like it is related to Single vs Multi threaded.

If I set SingleThreadHandler to False for the handler the uploads all come through with IsFile = True.

srvr.AddHandler("/fly/uploadimage","fly_uploadimage",False)

 

[Erel]

You should set SingleThreadHandler to False by default.

 

[tchart]

@Erel I am still having issues with uploads.

I've now discovered that PNG images aren't coming through as files. They are uploading successfully but IsFile reports false.

Below is the output from multiple PNG file uploads

multipart/form-data; boundary=----WebKitFormBoundaryN8nssGocFbYfKW9k
Parts Size = 1
Key = 3ca7246c203b44d2aa8c6afc982cfa13
Value = Part{n=3ca7246c203b44d2aa8c6afc982cfa13,fn=esri_desktop_assoc_desktop.png,ct=image/png,s=20425,t=true,f=null}
IsFile = false
SubmittedFilename = esri_desktop_assoc_desktop.png
TempFile = C:\Dropbox\My Apps\B4J\TheFly\Objects\www\uploads\MultiPart3776371683559043439
multipart/form-data; boundary=----WebKitFormBoundary6huy4YDjYhRXQMKX
Parts Size = 1
Key = 3ca7246c203b44d2aa8c6afc982cfa13
Value = Part{n=3ca7246c203b44d2aa8c6afc982cfa13,fn=esri_ent_prof_geodata.png,ct=image/png,s=23211,t=true,f=null}
IsFile = false
SubmittedFilename = esri_ent_prof_geodata.png
TempFile = C:\Dropbox\My Apps\B4J\TheFly\Objects\www\uploads\MultiPart4593836225573073040
multipart/form-data; boundary=----WebKitFormBoundaryw4RrCXJIieNDyBvu
Parts Size = 1
Key = 3ca7246c203b44d2aa8c6afc982cfa13
Value = Part{n=3ca7246c203b44d2aa8c6afc982cfa13,fn=esri_ent_assoc_design.png,ct=image/png,s=21445,t=true,f=null}
IsFile = false
SubmittedFilename = esri_ent_assoc_design.png
TempFile = C:\Dropbox\My Apps\B4J\TheFly\Objects\www\uploads\MultiPart2700667014731149317
multipart/form-data; boundary=----WebKitFormBoundaryN7AeIDf83B9CJV8g
Parts Size = 1
Key = 3ca7246c203b44d2aa8c6afc982cfa13
Value = Part{n=3ca7246c203b44d2aa8c6afc982cfa13,fn=esri_ent_assoc_admin.png,ct=image/png,s=21402,t=true,f=null}
IsFile = false
SubmittedFilename = esri_ent_assoc_admin.png
TempFile = C:\Dropbox\My Apps\B4J\TheFly\Objects\www\uploads\MultiPart5201029391604071029

And the same with several JPG file uploads

multipart/form-data; boundary=----WebKitFormBoundary4ILBU5Zswst4AxCv
Parts Size = 1
Key = 0af5af3afb2b4ad39af1f94a02043438
Value = Part{n=0af5af3afb2b4ad39af1f94a02043438,fn=ope_contact.jpg,ct=image/jpeg,s=992294,t=true,f=C:\Dropbox\MYAPPS~1\B4J\TheFly\Objects\www\uploads\MultiPart6760806282716988841}
IsFile = true
SubmittedFilename = ope_contact.jpg
TempFile = C:\Dropbox\My Apps\B4J\TheFly\Objects\www\uploads\MultiPart6760806282716988841
multipart/form-data; boundary=----WebKitFormBoundaryGFY7lxvOPc0p5Bs9
Parts Size = 1
Key = 0af5af3afb2b4ad39af1f94a02043438
Value = Part{n=0af5af3afb2b4ad39af1f94a02043438,fn=ope_fire.jpg,ct=image/jpeg,s=1455816,t=true,f=C:\Dropbox\MYAPPS~1\B4J\TheFly\Objects\www\uploads\MultiPart4089420198963929015}
IsFile = true
SubmittedFilename = ope_fire.jpg
TempFile = C:\Dropbox\My Apps\B4J\TheFly\Objects\www\uploads\MultiPart4089420198963929015
multipart/form-data; boundary=----WebKitFormBoundary02L43hcjKAMyegSA
Parts Size = 1
Key = 0af5af3afb2b4ad39af1f94a02043438
Value = Part{n=0af5af3afb2b4ad39af1f94a02043438,fn=togaf.jpg,ct=image/jpeg,s=63828,t=true,f=null}
IsFile = false
SubmittedFilename = togaf.jpg
TempFile = C:\Dropbox\My Apps\B4J\TheFly\Objects\www\uploads\MultiPart8382138533331396287
multipart/form-data; boundary=----WebKitFormBoundary5c0b7plmq4VwUBgY
Parts Size = 1
Key = 0af5af3afb2b4ad39af1f94a02043438
Value = Part{n=0af5af3afb2b4ad39af1f94a02043438,fn=ope_water.jpg,ct=image/jpeg,s=2470058,t=true,f=C:\Dropbox\MYAPPS~1\B4J\TheFly\Objects\www\uploads\MultiPart3339917051457730116}
IsFile = true
SubmittedFilename = ope_water.jpg
TempFile = C:\Dropbox\My Apps\B4J\TheFly\Objects\www\uploads\MultiPart3339917051457730116
multipart/form-data; boundary=----WebKitFormBoundarygowJa1URO2WvrLpo
Parts Size = 1
Key = 0af5af3afb2b4ad39af1f94a02043438
Value = Part{n=0af5af3afb2b4ad39af1f94a02043438,fn=ope_header.jpg,ct=image/jpeg,s=1072630,t=true,f=C:\Dropbox\MYAPPS~1\B4J\TheFly\Objects\www\uploads\MultiPart2337972868974684584}
IsFile = true
SubmittedFilename = ope_header.jpg
TempFile = C:\Dropbox\My Apps\B4J\TheFly\Objects\www\uploads\MultiPart2337972868974684584
multipart/form-data; boundary=----WebKitFormBoundaryMoAuAVHfzivAwMHV
Parts Size = 1
Key = 0af5af3afb2b4ad39af1f94a02043438
Value = Part{n=0af5af3afb2b4ad39af1f94a02043438,fn=ope_earth.jpg,ct=image/jpeg,s=2067818,t=true,f=C:\Dropbox\MYAPPS~1\B4J\TheFly\Objects\www\uploads\MultiPart2965418035943998002}
IsFile = true
SubmittedFilename = ope_earth.jpg
TempFile = C:\Dropbox\My Apps\B4J\TheFly\Objects\www\uploads\MultiPart2965418035943998002

Whats odd is that one of the JPG files always says False to IsFile. The others report true.

 

[Erel]

What happens if you ignore the value of IsFile and try to access the file?

 

[tchart]

What happens if you ignore the value of IsFile and try to access the file?

It works fine Erel.

Im doing this as a work around for now.

                Dim IsFile As Boolean = False
               
                Try
                    Dim FileName As String = ThisPart.SubmittedFilename.ToLowerCase()                   
                    If FileName.EndsWith(".jpg") Or FileName.EndsWith(".jpeg") Or FileName.EndsWith(".png") Then IsFile = True
                Catch
                    Log(LastException)
                End Try
               
                If IsFile = True Then
                   'Handle image...

 

[OliverA]

Im doing this as a work around for now.

Just out of curiosity, what was the original code for setting/checking isFile?

 

[tchart]

Just out of curiosity, what was the original code for setting/checking isFile?

@OliverA its part of the jServer library that comes with b4j.

 

[OliverA]

Looking at the source of the jServer library (https://github.com/AnywhereSoftware...src/anywheresoftware/b4j/object/JServlet.java), it's interesting that for isFile we just have:

public boolean getIsFile() {
    return getObject().getFile() != null;
}

But for TempFile we have:

public String getTempFile() throws IOException {
    if (getObject().getFile() == null) {
        try {
            Method m = MultiPart.class.getDeclaredMethod("createFile");
            m.setAccessible(true);
            m.invoke(getObject());
            m = MultiPart.class.getDeclaredMethod("close");
            m.setAccessible(true);
            m.invoke(getObject());
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }
    return getObject().getFile().getCanonicalPath();
}

What happens if you call TempFile before isFile?

 

[OliverA]

Ok, I've reproduced you issues with the isFile. From a really quick analysis, isFile is false on my system when the file size is just under 84K (a 84K file produced true, a 75K and anything below that size produces false on my system). It does not matter if it is a JPG, PNG, text file, PDF, etc.

What happens if you call TempFile before isFile?

isFile returns true all the time, no matter the file size.

So as it stands, isFile does not really represent if a part of a multi-part is a file part, but an indicator if a temporary file has been created. It looks like some sort of optimization(?) is happening where files smaller than 84K do not have a temporary file automatically generated unless it is requested(?) or an inquiry (such as TempFile) is made pertaining to the temporary file (besides isFile). I have no clue if this is a bug or just the way isFile (which is a wrapper for getFile() method of org.eclipse.jetty.util.MultiPartInputStreamParser.MultiPart). The documentation for that method states:

• getFile
  public File getFile()
  Get the file
  Returns:
  the file, if any, the data has been written to.

So, isFile just returns true if a file has been created for the data, but that may not happen/have happened yet (alluded by the "if any"). That may explain the "contortions" that the getTempFile method (in the jServer library linked in the previous post) is going through in order to get the temporary file name (before calling getObject().getFile().getCanonicalPath()) and explains why isFile returns true consistently AFTER TempFile has been called (the temp file is created). In other words, isFile, by iteself, is NOT a proper indicator for checking if a multipart is a file. Whew. Hope this makes sense.

 

[Erel]

It does look like a bug. Not sure whether it is in jServer or the underlying library. I will check it.

 

[OliverA]

It does look like a bug. Not sure whether it is in jServer or the underlying library. I will check it.

I think it is a semantic issue, not a bug. It looks like Jetty has an internal buffer that it will use on smaller multi-form parts (instead of writing them out to a file). I also tested this on a textarea field a temporary file was created for a large textarea (98K). This tells me that getFile() is not an indicator of the submitted information being a file, but that it indicates that the submitted information was large enough to be written to a file (instead of held in a memory buffer).

Clue: http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/plain/jetty-util/src/test/java/org/eclipse/jetty/util/MultiPartInputStreamTest.java

    private void testMulti(String filename) throws IOException, ServletException, InterruptedException
    {
        MultipartConfigElement config = new MultipartConfigElement(_dirname, 1024, 3072, 50);
        MultiPartInputStreamParser mpis = new MultiPartInputStreamParser(new ByteArrayInputStream(createMultipartRequestString(filename).getBytes()),
                _contentType,
                config,
                _tmpDir);
        mpis.setDeleteOnExit(true);
        Collection<Part> parts = mpis.getParts();
        assertThat(parts.size(), is(2));
        Part field1 = mpis.getPart("field1");  //field 1 too small to go into tmp file, should be in internal buffer

(see name of method and comments for last line shown)

Clue: http://book2s.com/java/api/org/ecli...treamparser/multipartinputstreamparser-4.html

File tmpfile = ((MultiPartInputStreamParser.MultiPart) stuff)
            .getFile();
assertThat(tmpfile, notNullValue()); // longer than 100 bytes, should already be a tmp file

For files, if isFile() returns false, the getValue() method should have the file content. The problem with that is that getValue is cast to a string in the jServer library. An alternative for now is to call the TempFile() method (forcing the creation of a file). For the future, the getBytes() method may need to be exposed/wrapped.

In conclusion, isFile() (since it is based on getFile()) can only tell you if the content of the multi-part has been copied to a temporary file. If it returns false, it only means that the information is still in a memory buffer. isFile() should have no meaning in regards to the type of content that is contained in a multi-part.

 

[tchart]

Thanks @OliverA and @Erel