B4J Question Write image data from buffer to a bitmap

[bdunkleysmith]

My question is how to write image data held in a buffer to a bitmap for subsequent display in imageview or further manipulation?

With reference to the library I'm working on in this post, I am successfully capturing video frames sent via NDI with the data of each frame stored in a ByteBuffer:

Private bytebuffer As JavaObject
bytebuffer.InitializeStatic("java.nio.ByteBuffer")
bytebuffer = bytebuffer.RunMethod("allocateDirect",Array(videoFrame.XResolution * videoFrame.YResolution * 4))

So bytebuffer contains the data of each pixel in four bytes representing RGBA. I'm satisfied the data is being captured and stored in bytebuffer because I can read out the data for each pixel using:

Log("Video data received (" & videoFrame.XResolution & "x" & videoFrame.YResolution & ", " & (videoFrame.FrameRateN / videoFrame.FrameRateD) & ", " & videoFrame.FormatType.toString & ", " & videoFrame.FourCCType.toString & ")")
For i = 0 To bytebuffer.RunMethod("capacity",Null) - 1 Step 4
      Dim r, g, b, a As Byte
      Log((i/4) & " | " & bytebuffer.RunMethod("get", Array(i)) & " | " & bytebuffer.RunMethod("get", Array(i+1)) & " | " & bytebuffer.RunMethod("get", Array(i+2)) & " | " & bytebuffer.RunMethod("get", Array(i+3)))
Next

to show:

Video data received (1920x1080, 50, PROGRESSIVE, RGBA)
0 | 122 | 72 | 125 | 27
1 | 122 | 72 | 125 | 27
2 | 122 | 72 | 125 | 27
3 | 123 | 72 | 125 | 27
4 | 123 | 73 | 125 | 28
5 | 122 | 73 | 125 | 27
6 | 122 | 73 | 125 | 27
7 | 122 | 73 | 125 | 26
8 | 122 | 73 | 127 | 26
9 | 122 | 73 | 127 | 26
10 | 122 | 73 | 126 | 26
11 | 122 | 74 | 126 | 26
12 | 122 | 72 | 126 | 27
13 | 122 | 72 | 126 | 27
14 | 122 | 72 | 126 | 27
15 | 122 | 72 | 126 | 27
16 | 121 | 72 | 125 | 27
17 | 121 | 72 | 125 | 27
18 | 121 | 72 | 125 | 27
19 | 121 | 72 | 125 | 27
20 | 121 | 71 | 125 | 28
21 | 121 | 71 | 125 | 28
22 | 121 | 71 | 125 | 28
23 | 121 | 71 | 125 | 28
24 | 120 | 71 | 124 | 28

Any guidance would be appreciated.

 

[DonManfred]

My question is how to write image data held in a buffer to a bitmap for subsequent display in imageview or further manipulation?

Public Sub ImageToBytes(Image As B4XBitmap) As Byte()
   Dim out As OutputStream
   out.InitializeToBytesArray(0)
   Image.WriteToStream(out, 100, "JPEG")
   out.Close
   Return out.ToBytesArray
End Sub

Public Sub BytesToImage(bytes() As Byte) As B4XBitmap
   Dim In As InputStream
   In.InitializeFromBytesArray(bytes, 0, bytes.Length)
#if B4A or B4i
   Dim bmp As Bitmap
   bmp.Initialize2(In)
#else
   Dim bmp As Image
   bmp.Initialize2(In)
#end if
   Return bmp
End Sub

I can´t help you with the decoding though...

 

[bdunkleysmith]

Thanks @DonManfred however this is throwing an error like some other approaches I've tried.

Based on your code I tried:

Dim In As InputStream
In.InitializeFromBytesArray(bytebuffer, 0, bytebuffer.RunMethod("capacity",Null))
Dim bmp As Image
bmp.Initialize2(In)
ivMain.SetImage(bmp)

but it throws the error:

main._btnrecvframes_click (java line: 196)
java.lang.ClassCastException: java.nio.DirectByteBuffer cannot be cast to [B
at b4j.example.main._btnrecvframes_click(main.java:196)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:91)
at anywheresoftware.b4a.BA$1.run(BA.java:216)
at com.sun.javafx.application.PlatformImpl.lambda$null$172(PlatformImpl.java:295)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.application.PlatformImpl.lambda$runLater$173(PlatformImpl.java:294)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$null$147(WinApplication.java:177)
at java.lang.Thread.run(Thread.java:748)

It seems that a ByteBuffer cannot be cast to a byte array.

Perhaps I have to use the BytesBuilder class to iteratively build a bytes array from the data within the ByteBuffer and use that in the In.InitializeFromBytesArray statement.

 

[DonManfred]

bytebuffer can not be cast to a bytearray

i found on stackoverflow

ByteBuffer bb =..

byte[] b = new byte[bb.remaining()]
bb.get(b, 0, b.length);

 

[Daestrum]

You could try

If bytebuffer.RunMethod("hasArray",null) Then
     Dim bytearray() As Byte = bytebuffer.RunMethod("array",Null)
...' use bytearray for image etc
End If

 

[bdunkleysmith]

Thanks @DonManfred and @Daestrum

I need to investigate further because I'm not skilled enough to be able to convert @DonManfred's code to B4J and

bytebuffer.RunMethod("hasArray",null)

is returning False even though I can iterate through bytebuffer as stated above with

For i = 0 To bytebuffer.RunMethod("capacity",Null) - 1 Step 4
      Log((i/4) & " | " & bytebuffer.RunMethod("get", Array(i)) & " | " & bytebuffer.RunMethod("get", Array(i+1)) & " | " & bytebuffer.RunMethod("get", Array(i+2)) & " | " & bytebuffer.RunMethod("get", Array(i+3)))
Next

and I can see there is data which appears valid because it varies as I change the video test pattern sent via NDI.

I'll continue investigating and report back, but now it's time for bed . . . .

 

[Erel]

If you can change the format to BGRA premultiplied then you can use BitmapCreator to convert the raw data for you.

You can also see how getBitmap is implemented and modify it to work with other formats.

 

[bdunkleysmith]

Thanks @Erel for the pointer to BitmapCreator

As I understand it, by default the NDI receiver passes through the frame color format of the source, eg. UYVY, YV12, NV12, I420, BGRA, BGRX, RGBA, RGBX or UYVA, but the color format passed to you when a frame is received can be set by specifying the receiver color option.

At the moment I am having trouble setting that option because of the way that option is presented in the wrap of the original NewTek NDI SDK. I'll investigate that further and probably make a separate post if I can't overcome my current difficulty.

However if I select a .png image in the NewTek Test Pattern Generator, it is received as BGRA and so I used the following code to successfully display an image send over the network via NDI:

Sub Process_Globals

    Private bytebuffer, WriteableImage As JavaObject
    Private PixelFormat As JavaObject
    
End Sub

Sub btnRecvFrames_Click

    ' Initialise bytebuffer and load video frame data from NDI
    bytebuffer.InitializeStatic("java.nio.ByteBuffer")
    bytebuffer = bytebuffer.RunMethod("allocateDirect",Array(videoFrame.XResolution * videoFrame.YResolution * 4))
    bytebuffer = videoFrame.Data

    ' Create byte array and iteratively load data read from bytebuffer
    Dim bytearray(bytebuffer.RunMethod("capacity",Null)) As Byte
    For i = 0 To bytebuffer.RunMethod("capacity",Null) - 1
        bytearray(i) = bytebuffer.RunMethod("get", Null)
    Next

    ' Code from BitmapCreator to convert the bytes array (buffer) to a bitmap
    If WriteableImage.IsInitialized = False Then
        WriteableImage.InitializeNewInstance("javafx.scene.image.WritableImage", Array(videoFrame.XResolution, videoFrame.YResolution))
    End If
    Dim writer As JavaObject = WriteableImage.RunMethod("getPixelWriter", Null)
    If PixelFormat.IsInitialized = False Then
        PixelFormat.InitializeStatic("javafx.scene.image.PixelFormat")
    End If
    writer.RunMethod("setPixels", Array(0, 0, videoFrame.XResolution, videoFrame.YResolution, PixelFormat.RunMethod("getByteBgraPreInstance", Null), bytearray, 0, videoFrame.XResolution * 4))
    ivMain.SetImage(WriteableImage)

    ' Allow UI to update with image
    Sleep(0)                   

End Sub

There's much to do to get it to a stable application in the form I want, but the image below at least demonstrates proof of concept.

Thanks again to this great community for the support.