B4J Question How to define HttpSession.Id in jRDC2 client?

[Diceman]

I am using jRDC2 and it is working fine. I am trying to write a stress test program in a single B4J client that simulates 100 different users accessing the jRDC2 server. The only problem is the server gets the 100 different requests and they all have the same HTTPSession.Id.

Is there a way for the client to override the builtin session id like "node01s01isq9met0v1xc6zb51r2njy9" so it sends requests to the server using different session ids?

This would only be done for this stress testing app. I could assign simple session id's like "001", "002" etc, or generate them with the proper syntax "node ....". It doesn't really matter.

Is this possible?

TIA

 

[OliverA]

I wrote a test client for some jRDC2 testing in B4J non-gui mode. I then had a batch file start as many sessions of that client for me. I'm attaching source for both the client and the batch file and hopefully it gives some inspiration. Please note that this client was adapted for a heavily modified jRDC2 server that I was using to test various SQL pooling options (H2, HSQLDB, Hikari, Tomcat, Vibur, and 3CP0) and pooling/JDBC properties. I was also testing various DB engines (H2, Apachy Derby, SQLite, HSQLDB, and MySQL). I'll throw in the config.properties file so you can find the SQL statement that was used for testing. It's not a canned project for you, but it may help you (hopefully) in your endeavors.

 

[Diceman]

I wrote a test client for some jRDC2 testing in B4J non-gui mode. I then had a batch file start as many sessions of that client for me. I'm attaching source for both the client and the batch file and hopefully it gives some inspiration. Please note that this client was adapted for a heavily modified jRDC2 server that I was using to test various SQL pooling options (H2, HSQLDB, Hikari, Tomcat, Vibur, and 3CP0) and pooling/JDBC properties. I was also testing various DB engines (H2, Apachy Derby, SQLite, HSQLDB, and MySQL). I'll throw in the config.properties file so you can find the SQL statement that was used for testing. It's not a canned project for you, but it may help you (hopefully) in your endeavors.

Thanks for the files. I will dissect it tomorrow to see what I can scavenge.
BTW, have you published the results of your benchmark? Like which pooling option was superior and which was the worst? I noticed you did not test PostgreSQL which I am using on the server.

 

[OliverA]

One of the things I was curious about when I build this jRDC2 server is to see if SQLite is really that bad when it comes to concurrency. Well, it's not and it actually beat the pants of all the other solutions. When it came to pooling, I really did not see huge differences in the various pooling options. The pool size seemed to be optimal between 8-10. Anything above that did not improve anything (in my limited testing). The nice thing about H2, Apache Derby, SQLite and HSQLDB is that they are embedded DB's that really need no administrating. SQLite is fast, memory efficient, but type (SQL types) poor. H2 (successor to HSQLDB?) is very type rich, is not much slower, but is a memory hog compared to SQLite. MySQL requires a separate server daemon (locally or remote) but is actually relatively memory efficient (in comparison to H2). Pooling is definitely required when using MySQL on a remote server (with pool sizes between 8-10 being optimal). I'll dig for some data tomorrow and post some of it. Please note that the data is very specific to my testing and your test environment may discover something else. I could post the jRDC2 server, but it's a tad rough. It's functional, but not 100% finished. You can see the options that were allowed to be tweaked in the config file that I posted above.

 

[Diceman]

One of the things I was curious about when I build this jRDC2 server is to see if SQLite is really that bad when it comes to concurrency. Well, it's not and it actually beat the pants of all the other solutions. When it came to pooling, I really did not see huge differences in the various pooling options. The pool size seemed to be optimal between 8-10. Anything above that did not improve anything (in my limited testing). The nice thing about H2, Apache Derby, SQLite and HSQLDB is that they are embedded DB's that really need no administrating. SQLite is fast, memory efficient, but type (SQL types) poor. H2 (successor to HSQLDB?) is very type rich, is not much slower, but is a memory hog compared to SQLite. MySQL requires a separate server daemon (locally or remote) but is actually relatively memory efficient (in comparison to H2). Pooling is definitely required when using MySQL on a remote server (with pool sizes between 8-10 being optimal). I'll dig for some data tomorrow and post some of it. Please note that the data is very specific to my testing and your test environment may discover something else. I could post the jRDC2 server, but it's a tad rough. It's functional, but not 100% finished. You can see the options that were allowed to be tweaked in the config file that I posted above.

If the database (Sqlite, PostgreSQL, etc.) is on the same machine as the web server, I doubt you will see much speed difference between Sqlite and the client server database. (BTW, putting the db on the same machine as the web server is a security risk if you are storing sensitive data.) If the transaction duration is kept at a minimum and there are very few writes to the db, then SQLite will be faster because it has much less overhead than a C/S db. The problem with Sqlite is it uses database locking so if you have 10 or more clients updating any table in the database that takes longer than 100 ms each, you will have locking contention. The clients will be sitting and waiting for the db lock to be released because one slow client is writing to the db. If the database is on a separate computer, then C/S databases will be more efficient at returning only a subset of data over the network. C/S databases also have stored procedures & triggers so a lot of the "intelligence" can be offloaded to the C/S db where it runs faster instead of running it on the client. I would use Sqlite on a web server if 95% of the operations are Reads, and there are less than 100 queries/second. Anything more than that and I would use a C/S database from the outset because I don't want to be rewriting my app for C/S after it has gone live. Just my 2 cents.

Thanks for the testlauncher. I ran it today and it has given me a few ideas on how I can implement it on my existing benchmark pgm.

 

[Diceman]

I wrote a test client for some jRDC2 testing in B4J non-gui mode. I then had a batch file start as many sessions of that client for me. I'm attaching source for both the client and the batch file and hopefully it gives some inspiration. Please note that this client was adapted for a heavily modified jRDC2 server that I was using to test various SQL pooling options (H2, HSQLDB, Hikari, Tomcat, Vibur, and 3CP0) and pooling/JDBC properties. I was also testing various DB engines (H2, Apachy Derby, SQLite, HSQLDB, and MySQL). I'll throw in the config.properties file so you can find the SQL statement that was used for testing. It's not a canned project for you, but it may help you (hopefully) in your endeavors.

Oliver,

I was able to run your test program just fine without any errors. I converted my test app into a command line B4J app and used a similar batch file. It works fine if I run it with only 1 or 2 clients but if I try 5 or more clients I start getting errors on the client. Some of the errors complain about file sharing which is odd because the client does not access any text files or config files. It only accesses the jRDC2 server. Even with a lot of Try/Catch statements I can't find the code that throws the exception.

The jRDC2 server runs fine without any errors.
If I run a single client in a DOS window it runs fine. As soon as I start running 3 or more clients I start getting exceptions in some of the client windows.

On the client (client is compiled as Release and run as Admin on Win10) I will see errors either right away or after a few iterations, like:

(FileNotFoundException) java.io.FileNotFoundException: C:\Users\User1\AppData\Local\Temp\465 (The process cannot access the file because it is being used by another process)
ERROR: StartMessageLoop aborted!!!! Exception: java.io.FileNotFoundException: C:\Users\User1\AppData\Local\Temp\465 (The process cannot access the file because it is being used by another process)

I can reduce the number of times the error above occurs by adding a "TimeOut 3" inside the batch file to delay loading the next benchmark app by 3 seconds. But the error can still appear later after a few iterations, only less frequently around 10% of the time.

Now if I run 10 clients, 8 of them will complete without errors and two of them will produce this error after several iterations:

java.lang.RuntimeException: java.util.zip.ZipException: invalid distance too far back
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.readType(B4XSerializator.java:297)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.readObject(B4XSerializator.java:357)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.ReadObject(B4XSerializator.java:112)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.ConvertBytesToObject(B4XSerializator.java:82)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator$2.call(B4XSerializator.java:93)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator$2.call(B4XSerializator.java:1)
        at anywheresoftware.b4a.BA$4.run(BA.java:272)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.zip.ZipException: invalid distance too far back
        at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
        at java.io.DataInputStream.readFully(DataInputStream.java:195)
        at java.io.DataInputStream.readFully(DataInputStream.java:169)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.readObject(B4XSerializator.java:327)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.readMap(B4XSerializator.java:231)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.readObject(B4XSerializator.java:347)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.readMap(B4XSerializator.java:231)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.readType(B4XSerializator.java:275)
        ... 11 more
[HandleJobAsync] Error reading response: (ZipException) java.util.zip.ZipException: invalid distance too far back

Same with this error. If I run 10 clients, 8 of them will complete without errors and two of them will produce this error after several iterations:

[GetQuestions] ERROR: java.lang.NullPointerException
java.lang.RuntimeException: java.lang.RuntimeException: Unsupported type: 52
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.readType(B4XSerializator.java:297)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.readObject(B4XSerializator.java:357)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.ReadObject(B4XSerializator.java:112)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.ConvertBytesToObject(B4XSerializator.java:82)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator$2.call(B4XSerializator.java:93)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator$2.call(B4XSerializator.java:1)
        at anywheresoftware.b4a.BA$4.run(BA.java:272)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Unsupported type: 52
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.readObject(B4XSerializator.java:359)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.readMap(B4XSerializator.java:231)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.readObject(B4XSerializator.java:347)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.readMap(B4XSerializator.java:231)
        at anywheresoftware.b4a.randomaccessfile.B4XSerializator.readType(B4XSerializator.java:275)
        ... 11 more
[HandleJobAsync] Error reading response: (RuntimeException) java.lang.RuntimeException: Unsupported type: 52

Everything works fine if I run only 1 or 2 clients at the same time. There are never any errors on the jRDC2 server.

Have you experienced this problem before? Any idea what it is complaining about?
TIA

 

[OliverA]

java.io.FileNotFoundException: C:\Users\User1\AppData\Local\Temp\465

I had the same issues with this as you. My guess was that so many http jobs were running that one was assigned the same random file name as another. So once one job was finished, it deleted the file, bombing out the other http job. I was going to make a question post out of this one, but never got to it. It only happened when I had lots of http jobs running (therefore my guess).

java.lang.RuntimeException: Unsupported type: 52

That's a new one for me. Looks like that's thrown by B4XSerializator as it encounters a type it does not understand. May be worth a new thread too. For both cases (this and above) would need to create some minimal system that can reproduce the issues (else it will be hard for anyone else to look at it).

 

[Diceman]

Thanks for getting back to me so quickly. I thought the problem was my code.
I would have thought the random job number would have been a GUID. I use them all the time in my code to reduce collisions between client and server.

I will try and simplify it and create a test program for Erel.