B4J Question Java Heap error and InputStreamToBytes

[udg]

Hi all,

I wrote a quick and dirty sub to check hash signatures for downloaded packages.
It worked well against 500/600MB iso images, but failed with the following error trace when I attempted to calculate the hash from the 1.4GB Ubuntu 16.04.1 iso image.

Spoiler: Heap Out of Memory error

main._checksha1 (java line: 84)
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at anywheresoftware.b4a.objects.streams.File.Copy2(File.java:357)
at anywheresoftware.b4a.keywords.Bit.InputStreamToBytes(Bit.java:94)
at b4j.example.main._checksha1(main.java:84)
at b4j.example.main._appstart(main.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:93)
at anywheresoftware.b4a.BA.raiseEvent(BA.java:84)
at b4j.example.main.start(main.java:38)
at com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$162(LauncherImpl.java:863)
at com.sun.javafx.application.LauncherImpl$$Lambda$55/1964646456.run(Unknown Source)
at com.sun.javafx.application.PlatformImpl.lambda$runAndWait$175(PlatformImpl.java:326)
at com.sun.javafx.application.PlatformImpl$$Lambda$48/1567581361.run(Unknown Source)
at com.sun.javafx.application.PlatformImpl.lambda$null$173(PlatformImpl.java:295)
at com.sun.javafx.application.PlatformImpl$$Lambda$50/842889876.run(Unknown Source)

I'm aware of the "-Xmx" java option but I'd like to know if there's something we can do in code to overcome the problem. I mean, something like instructing the chain InputStream-InputStreamToBytes-GetMessageDigest to work on chunks of data.

TIA

udg

 

[Erel]

Bit.InputStreamToBytes reads the complete file and returns an array of bytes. You shouldn't use it with very large files. You should instead read data directly from the stream and calculate the hash.

 

[udg]

Hi Erel,

do you mean something like:

Dim InputStream1 AsInputStream
InputStream1 = File.OpenInput(FPath, FName)
Dim OutputStream1 AsOutputStream
OutputStream1.InitializeToBytesArray(1024)
File.Copy2(InputStream1, OutputStream1)
Dim Buffer() As Byte 
Buffer = OutputStream1.ToBytesArray

'then passing Buffer to MessageDigest
md.GetMessageDigest(Buffer,HashMethod)

Or are you suggesting to study sha-1/md5.. algorythms and then write from the ground up code able to read in and operate on chunks of data to keep the memory footage at a minimum?

udg

 

[Erel]

Sub AppStart (Args() As String)
   Dim md As JavaObject
   md = md.InitializeStatic("java.security.MessageDigest").RunMethod("getInstance", Array("MD5"))
   Dim in As InputStream = File.OpenInput("C:\Users\H\Documents", "backup.pst") '4gb file
   Dim buffer(100000) As Byte
   Do While True
     Dim count As Int = in.ReadBytes(buffer, 0, buffer.Length)
     If count = -1 Then Exit
     md.RunMethod("update", Array(buffer, 0, count))
   Loop
   Dim md5() As Byte = md.RunMethod("digest", Null)
   Dim bc As ByteConverter
   Log(bc.HexFromBytes(md5))
   in.Close
End Sub

 

[udg]

Thanks Erel.
I realized how to read chunks of data but failed to think to by-pass the Encryption lib.