B4J Question JPOI and JAVA HEAP ERROR

[EnriqueGonzalez]

hello!

there is something that is bugging me, the JPOI library is quite powerfull but it takes a lot of Memory to run.

XLSX file: 700MB memory for a 2 MB file with 70 000 ROWS * 6 COLUMNS.
XLS file: 300MB memory for a 8 MB with same row and columns.


i am pretty sure that my code is not the problem as the error (JAVA HEAP ERROR) occurs in this line:

Detail.InitializeExisting(reportspath,"Detail.xlsx","")

it simple does not finish.

i had to add this line to my project #VirtualMachineArgs: -Xms1024m -Xmx1024m and right now i do not find any problem adding it because it is very very fast, it will iterate all the cells in 7 seconds or less despite that Erel used it for 3.7GB text file.

but in some point in time i want to use this library with some workbooks spanning 120MB file size, it will just simply break.

is this behavour intended? and if that is so, any suggestion?

THANKS!

 

[Roycefer]

On a more general note, check out VisualVM. It comes packaged with NetBeans but you can get a stand alone version here: https://visualvm.java.net/

It's a great tool for monitoring memory and cpu usage and thread usage (and a bunch of other performance characteristics) of your Java programs. You can even see what Java objects are consuming your Java heap space and their behavior over time. It might be worth it to check out the behavior of your Excel app with VisualVM to see how it behaves over time. Java's garbage collector isn't deterministic so it definitely helps to have a tool like VisualVM to see how memory management works in non-trivial cases.

 

[EnriqueGonzalez]

Nice tool thanks...
to see the memory it was using i used the Task Manager.

i particulary noted (maybe) that JPOI opens workbooks on different threads.

 

[Roycefer]

The Task Manager is a very different tool. It gives you the Private Working Set (by default) of a process. But Java divides its memory usage up into the Java heap, the native heap and non-heap and each of them behave differently. Task Manager won't give you any information about that. VisualVM does. VisualVM will also tell you what objects are taking up memory. For example, array of ints: 20 MB, Threads: 33 KB, etc... To see these stats in VisualVM, click on the Sampler tab and press the Memory button.

 

[EnriqueGonzalez]

oh... thanks! that is very useful to know, what a shame that until now i learn about that tool.

one of the perks to begin with Java from B4J i suppose.

 

[Roycefer]

Yeah, VisualVM has tons of great statistics to monitor the performance of your Java apps. You can even monitor them remotely, I think. You should take a look at the tutorials on that website for more info on how to use VisualVM.

If you want to monitor the performance of a Java application from within that application, you can use my jAWTRobot library. It exposes tons of JVM and System level performance statistics.

 

[Mashiane]

Hey, I was about to ask the same question on this heap then, then it hit me, 'Did you search first?', then I remembered that each time someone asks a question that has been asked before and has a solution, someone would say, 'use search, it works!' ha ha ha...

Imma gonna try your solution perhaps it will help. Eish, I have 477 excel files to process and merge data.

Ta!

 

[DonManfred]

This is an 4 Years old thread. You should always create a NEW thread for your questions.

 

[Mashiane]

This is an 4 Years old thread. You should always create a NEW thread for your questions.

Lol. You're funny.. ha ha ha

 

[DonManfred]

Lol. You're funny.. ha ha ha

LOL, TRUE! Welcome on my ignorelist.

https://www.b4x.com/android/forum/pages/results/?query=you+should+never+post

 

[Peter Simpson]

Lol. You're funny.. ha ha ha

Manfred is 100% correct, you should always create a new post, even if it's the same as an old post.

If a certain person would have told you that who is always telling users to create a new thread, you would not have responded like that.

 

[Mashiane]

i had to add this line to my project #VirtualMachineArgs: -Xms1024m -Xmx1024m

Guys, I was not asking a question, I was commenting. Yes I wanted to ask a question and then I found this thread. In the first post the same person who is asking is proposing a solution, so I was just making a comment on a question. So I have gone ahead and tried the proposed solution. The proposed solution for me is working.

Thank you.

 

[EnriqueGonzalez]

Guys, I was not asking a question, I was commenting. Yes I wanted to ask a question and then I found this thread. In the first post the same person who is asking is proposing a solution, so I was just making a comment on a question. So I have gone ahead and tried the proposed solution. The proposed solution for me is working.

Thank you.

Yeah, i got it that way!