B4J Question Java Heap error and InputStreamToBytes

udg

Expert
Licensed User
Longtime User
Hi all,

I wrote a quick and dirty sub to check hash signatures for downloaded packages.
It worked well against 500/600MB iso images, but failed with the following error trace when I attempted to calculate the hash from the 1.4GB Ubuntu 16.04.1 iso image.

main._checksha1 (java line: 84)
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at anywheresoftware.b4a.objects.streams.File.Copy2(File.java:357)
at anywheresoftware.b4a.keywords.Bit.InputStreamToBytes(Bit.java:94)
at b4j.example.main._checksha1(main.java:84)
at b4j.example.main._appstart(main.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:93)
at anywheresoftware.b4a.BA.raiseEvent(BA.java:84)
at b4j.example.main.start(main.java:38)
at com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$162(LauncherImpl.java:863)
at com.sun.javafx.application.LauncherImpl$$Lambda$55/1964646456.run(Unknown Source)
at com.sun.javafx.application.PlatformImpl.lambda$runAndWait$175(PlatformImpl.java:326)
at com.sun.javafx.application.PlatformImpl$$Lambda$48/1567581361.run(Unknown Source)
at com.sun.javafx.application.PlatformImpl.lambda$null$173(PlatformImpl.java:295)
at com.sun.javafx.application.PlatformImpl$$Lambda$50/842889876.run(Unknown Source)

I'm aware of the "-Xmx" java option but I'd like to know if there's something we can do in code to overcome the problem. I mean, something like instructing the chain InputStream-InputStreamToBytes-GetMessageDigest to work on chunks of data.

TIA

udg
 

udg

Expert
Licensed User
Longtime User
Hi Erel,

do you mean something like:
B4X:
Dim InputStream1 AsInputStream
InputStream1 = File.OpenInput(FPath, FName)
Dim OutputStream1 AsOutputStream
OutputStream1.InitializeToBytesArray(1024)
File.Copy2(InputStream1, OutputStream1)
Dim Buffer() As Byte 
Buffer = OutputStream1.ToBytesArray

'then passing Buffer to MessageDigest
md.GetMessageDigest(Buffer,HashMethod)

Or are you suggesting to study sha-1/md5.. algorythms and then write from the ground up code able to read in and operate on chunks of data to keep the memory footage at a minimum?

udg
 
Upvote 0

Erel

B4X founder
Staff member
Licensed User
Longtime User
B4X:
Sub AppStart (Args() As String)
   Dim md As JavaObject
   md = md.InitializeStatic("java.security.MessageDigest").RunMethod("getInstance", Array("MD5"))
   Dim in As InputStream = File.OpenInput("C:\Users\H\Documents", "backup.pst") '4gb file
   Dim buffer(100000) As Byte
   Do While True
     Dim count As Int = in.ReadBytes(buffer, 0, buffer.Length)
     If count = -1 Then Exit
     md.RunMethod("update", Array(buffer, 0, count))
   Loop
   Dim md5() As Byte = md.RunMethod("digest", Null)
   Dim bc As ByteConverter
   Log(bc.HexFromBytes(md5))
   in.Close
End Sub
 
Upvote 0

udg

Expert
Licensed User
Longtime User
Thanks Erel.
I realized how to read chunks of data but failed to think to by-pass the Encryption lib.
 
Upvote 0
Top