B4J Question NDI SDK

[bdunkleysmith]

NewTek's innovative Network Device Interface (NDI) is a high performance standard that allows anyone to use real time, ultra low latency video on existing IP video networks.

I have a desktop application which already uses ImageView to display images and MediaView to display videos, but would like to add a "view" to display an NDI stream.

NewTek provide an SDK which supports Windows®, Linux®, and macOS™ and examples are provided for C#, C++ and VB.net.

My understanding from NDI SDK Documentation.pdf contained within the SDK is that I would have to use the NDI-Find library to identify all NDI streams on the network, selecting the desired stream to consume it using the NDI-Receive library.

I'm looking for guidance from this community as to whether the resources of the SDK can be used to create a desktop application to find and then display an NDI source using B4J. Or is it not possible or at least very difficult.

Incidentally, based on one of the VB.Net examples I've created my own NDI source and no doubt could achieve what I want in VB.Net, but I want to include the NDI display as an extension to an existing B4J application.


Just to give the community the flavour of the SDK library the following code will create an NDI-Find instance, and then list the current available sources. It uses NDIlib_find_wait_for_sources to sleep until new sources are found on the network and, when they are seen, it will call NDIlib_find_get_current_sources to get the current list of sources.

// Create the descriptor of the object to create
NDIlib_find_create_t find_create;
find_create.show_local_sources = true;
find_create.p_groups = nullptr;
// Create the instance
NDIlib_find_instance_t pFind = NDIlib_find_create_v2(&find_create);
if (!pFind) /* Error */;
while(true) // You would not loop forever of course !
{ // Wait up till 5 seconds to check for new sources to be added or removed
if (!NDIlib_find_wait_for_sources(pNDI_find, 5000))
{ // No new sources added !
printf("No change to the sources found.\n");
}
else
{ // Get the updated list of sources
uint32_t no_sources = 0;
const NDIlib_source_t* p_sources = NDIlib_find_get_current_sources(pNDI_find, &no_sources);
// Display all the sources.
printf("Network sources (%u found).\n", no_sources);
for (uint32_t i = 0; i < no_sources; i++)
printf("%u. %s\n", i + 1, p_sources[i].p_ndi_name);
}
}
// Destroy the finder when you’re all done finding things
NDIlib_find_destroy(pFind);

 

[moster67]

Never heard of it before but it looks indeed like an interesting platform.

Maybe I did not check properly but I could not see an SDK for Java which you most likely will need if you want to use it with B4J.

However, I found this Github project which provides Java bindings:
https://github.com/WalkerKnapp/devolay

I am not sure but perhaps you can use NDI with B4J if you wrap the Java bindings mentioned above? Maybe you can use JavaObject or inline Java?
If you are unable to do this yourself you can always make a post about it in the job-offer section of the B4X forum and maybe somebody can help.

Good luck.

 

[bdunkleysmith]

Thanks @moster67 You are correct that NewTek don't provide an SDK for Java. An alternative approach would be to port all of my BJ4 code to VB.net so I can use that SDK, but my existing B4J application into which I want to bring the NDI stream is over 1800 lines of code and so would be a big job. Plus I find B4J so much easier to use!

I have done extensive searching, but didn't come up with that Github project which you found and it looks like a possible path, although I'm getting way out on my comfort (competency) zone. However I'm always keen to expand my mind by extending my horizons and so I'll explore it further.

 

[bdunkleysmith]

Thanks to the reference provided by @moster67 I have made significant progress and have produced a B4J library based on the Java source files located in https://github.com/WalkerKnapp/devolay

As a proof of concept I have been able to use the devolay equivalent of NDI-FIND to find / identify NDI stream on a network and the devolay equivalent of NDI-SEND to send a series of blank frames at 50FPS as an NDI stream over a network.

Once I have satisfied myself that the library is working OK I will release it for use by others with an interest in using B4J to create NDI based applications.

However I've struck a roadblock in converting the devolay Java examples to B4J. In particular the SendVideoExample declares the frame data as a ByteBuffer:

        final int width = 1920;
        final int height = 1080;
        // BGRX has a pixel depth of 4
        final ByteBuffer data = ByteBuffer.allocateDirect(width * height * 4);

        // Create a video frame
        DevolayVideoFrame videoFrame = new DevolayVideoFrame();
        videoFrame.setResolution(width, height);
        videoFrame.setFourCCType(DevolayFrameFourCCType.BGRX);
        videoFrame.setData(data);
        videoFrame.setFrameRate(60, 1);

which I presume is accessed via the statement:

import java.nio.ByteBuffer;

For reference below is the full code for the SendVideoExample.

What type can I use in lieu of ByteBuffer or alernatively how can I import this type?

package com.walker.devolayexamples;

import com.walker.devolay.Devolay;
import com.walker.devolay.DevolayFrameFourCCType;
import com.walker.devolay.DevolaySender;
import com.walker.devolay.DevolayVideoFrame;

import java.nio.ByteBuffer;

/**
 * Adapted from NDIlib_Send_Video.cpp
 */
public class SendVideoExample {
    public static void main(String[] args) {
        Devolay.loadLibraries();

        // Create the sender using the default settings, other than setting a name for the source.
        DevolaySender sender = new DevolaySender("Devolay Example Video");

        final int width = 1920;
        final int height = 1080;
        // BGRX has a pixel depth of 4
        final ByteBuffer data = ByteBuffer.allocateDirect(width * height * 4);

        // Create a video frame
        DevolayVideoFrame videoFrame = new DevolayVideoFrame();
        videoFrame.setResolution(width, height);
        videoFrame.setFourCCType(DevolayFrameFourCCType.BGRX);
        videoFrame.setData(data);
        videoFrame.setFrameRate(60, 1);

        int frameCounter = 0;
        long fpsPeriod = System.currentTimeMillis();

        // Run for one minute
        long startTime = System.currentTimeMillis();
        while(System.currentTimeMillis() - startTime < 1000 * 60) {

            //Fill in the buffer for one frame.
            fillFrame(width, height, frameCounter, data);

            // Submit the frame. This is clocked by default, so it will be submitted at <= 60 fps.
            sender.sendVideoFrame(videoFrame);

            // Give an FPS message every 30 frames submitted
            if(frameCounter % 30 == 29) {
                long timeSpent = System.currentTimeMillis() - fpsPeriod;
                System.out.println("Sent 30 frames. Average FPS: " + 30f / (timeSpent / 1000f));
                fpsPeriod = System.currentTimeMillis();
            }

            frameCounter++;
        }

        // Destroy the references to each. Not necessary, but can free up the memory faster than Java's GC by itself
        videoFrame.close();
        sender.close();
    }

    private static void fillFrame(int width, int height, int frameCounter, ByteBuffer data) {
        data.position(0);
        double frameOffset = Math.sin(frameCounter / 120d);
        for(int i = 0; i < width * height; i++) {
            double xCoord = i % width;
            double yCoord = i / (double)width;

            double convertedX = xCoord/width;
            double convertedY = yCoord/height;

            double xWithFrameOffset = convertedX + frameOffset;
            double xWithScreenOffset = xWithFrameOffset - 1;
            double yWithScreenOffset = convertedY + 1;

            double squaredX = xWithFrameOffset * xWithFrameOffset;
            double offsetSquaredX = xWithScreenOffset * xWithScreenOffset;
            double squaredY = convertedY * convertedY;
            double offsetSquaredY = yWithScreenOffset * yWithScreenOffset;

            byte r = (byte) (Math.min(255 * Math.sqrt(squaredX + squaredY), 255));
            byte g = (byte) (Math.min(255 * Math.sqrt(offsetSquaredX + squaredY), 255));
            byte b = (byte) (Math.min(255 * Math.sqrt(squaredX + offsetSquaredY), 255));

            data.put(b).put(g).put(r).put((byte)255);
        }
        data.flip();
    }
}

 

[Daestrum]

Dim bytebuffer As JavaObject
….
 bytebuffer.InitializeStatic("java.nio.ByteBuffer")
 bytebuffer = bytebuffer.RunMethod("allocateDirect",Array(100 * 100 * 4))
 Log(bytebuffer.RunMethod("capacity",Null)) ' 40_000 bytes

 

[bdunkleysmith]

Thank you @Daestrum I keep forgetting how powerful that JavaObject is in exposing otherwise hidden methods. I have implemented your code successfully.

However I have a problem with the library I have created.

Using the Java source files located in https://github.com/WalkerKnapp/devolay with the Simple Library Compiler (SLC) I generated devolay.xml, however I had to add tags for the short names of the classes, methods and properties. I then placed devolay.xml and the already compiled devolay.jar into my additional libraries folder.

The problem I have is that the methods exposed, as listed in devolay.xml are not complete and inconsistent with the source (Java) files. for example for the DevolayVideoFrame class, while setFrameRate and setResolution are exposed correctly, Data is shown as a property rather than the methods setData and getData.

Am I using the SLC tool incorrectly, are the source files incompatible with the SLC parser or what else may be causing incorrect generation of devolay.xml?

This seems relevant to my "problem": https://www.b4x.com/android/forum/threads/doclet-not-reporting-the-right-method-name.16805/

Is there an alternative way to generate the required .xml file?

For example below is DevolayVideoFrame.java:

package com.walker.devolay;

import java.nio.ByteBuffer;
import java.util.concurrent.atomic.AtomicReference;

public class DevolayVideoFrame implements AutoCloseable {

    final long structPointer;

    // set when a buffer is allocated by a receiver that later needs to be freed w/ that receiver.
    AtomicReference<DevolayReceiver> allocatedBufferSource = new AtomicReference<>();

    public DevolayVideoFrame() {
        // TODO: Implement this forced reference more effectively
        Devolay.loadLibraries();

        this.structPointer = createNewVideoFrameDefaultSettings();
    }

    public void setResolution(int width, int height) {
        setXRes(structPointer, width);
        setYRes(structPointer, height);
    }
    public int getXResolution() {
        return getXRes(structPointer);
    }
    public int getYResolution() {
        return getYRes(structPointer);
    }

    public void setFourCCType(DevolayFrameFourCCType type) {
        setFourCCType(structPointer, type.id);
    }
    public DevolayFrameFourCCType getFourCCType() {
        return DevolayFrameFourCCType.valueOf(getFourCCType(structPointer));
    }

    public void setFrameRate(int numerator, int denominator) {
        setFrameRateN(structPointer, numerator);
        setFrameRateD(structPointer, denominator);
    }
    public int getFrameRateN() {
        return getFrameRateN(structPointer);
    }
    public int getFrameRateD() {
        return getFrameRateD(structPointer);
    }

    public void setAspectRatio(float aspectRatio) {
        setPictureAspectRatio(structPointer, aspectRatio);
    }
    public float getAspectRatio() {
        return getPictureAspectRatio(structPointer);
    }

    public void setFormatType(DevolayFrameFormatType type) {
        setFrameFormatType(structPointer, type.id);
    }
    public DevolayFrameFormatType getFormatType() {
        return DevolayFrameFormatType.valueOf(getFrameFormatType(structPointer));
    }

    public void setTimecode(long timecode) {
        setTimecode(structPointer, timecode);
    }
    public long getTimecode() {
        return getTimecode(structPointer);
    }

    public void setLineStride(int lineStride) {
        setLineStride(structPointer, lineStride);
    }
    public int getLineStride() {
        return getLineStride(structPointer);
    }

    public void setMetadata(String metadata) {
        setMetadata(structPointer, metadata);
    }
    public String getMetadata() {
        return getMetadata(structPointer);
    }

    public void setTimestamp(int timestamp) {
        setTimestamp(structPointer, timestamp);
    }
    public int getTimestamp() {
        return getTimestamp(structPointer);
    }

    public void setData(ByteBuffer buffer) {
        if(allocatedBufferSource.get() != null) {
            allocatedBufferSource.getAndSet(null).freeVideo(this);
        }
        setData(structPointer, buffer);
    }
    public ByteBuffer getData() {
        return getData(structPointer);
    }

    @Override
    public void close() {
        if(allocatedBufferSource.get() != null) {
            allocatedBufferSource.getAndSet(null).freeVideo(this);
        }
        // TODO: Auto-clean resources.
        destroyVideoFrame(structPointer);
    }

    // Native Functions

    private static native long createNewVideoFrameDefaultSettings();
    private static native void destroyVideoFrame(long pointer);

    private static native void setXRes(long pointer, int xRes);
    private static native void setYRes(long pointer, int yRes);
    private static native int getXRes(long pointer);
    private static native int getYRes(long pointer);

    private static native void setFourCCType(long pointer, int fourCCType);
    private static native int getFourCCType(long pointer);

    private static native void setFrameRateN(long pointer, int frameRateN);
    private static native void setFrameRateD(long pointer, int frameRateD);
    private static native int getFrameRateN(long pointer);
    private static native int getFrameRateD(long pointer);

    private static native void setPictureAspectRatio(long pointer, float aspectRatio);
    private static native float getPictureAspectRatio(long pointer);

    private static native void setFrameFormatType(long pointer, int frameFormatType);
    private static native int getFrameFormatType(long pointer);

    private static native void setTimecode(long pointer, long timecode);
    private static native long getTimecode(long pointer);

    private static native void setLineStride(long pointer, int lineStride);
    private static native int getLineStride(long pointer);

    private static native void setMetadata(long pointer, String metadata);
    private static native String getMetadata(long pointer);

    private static native void setTimestamp(long pointer, int timestamp);
    private static native int getTimestamp(long pointer);

    /**
     *
     * @param pointer
     * @param buffer MUST BE A DIRECT BUFFER, ALLOCATED USING BUFFER#ALLOCATEDIRECT()
     */
    private static native void setData(long pointer, ByteBuffer buffer);
    private static native ByteBuffer getData(long pointer);
}

Here can be seen the associated Javadoc showing the correct methods, but here is the documentation generated using the Basic4Android (B4A) library and moduletool reference generator from devolay.xml.

 

[Daestrum]

Getters and Setters are generally converted to properties in the xml. It should make no difference as the IDE will only allow you to read a method that only has getXXX, and conversely only let you write to one which has a setXXX method.Ones that have get & set are read/write from the ide code.

 

[bdunkleysmith]

OK @Daestrum I now understand, but it would be nice if leaving the Getters and Setters as is was an option in the parser so for the less experienced coders like me, the methods correlated with the original library.

My next issue involved ENums and I provide this information for the interest of othetres encountering a similar issue.

Nine ENums are defined in the DevolayFrameFourCCType class, however the .xml doesn't expose them. Firstly I used the ENumClass by @stevel05 in this code:

'    Dim EPos As ENumClass
    EPos.Initialize("com.walker.devolay.DevolayFrameFourCCType")
    Dim a As Int = 0
    For a = 0 To EPos.ValueStrings.Length - 1
        Log(EPos.ValueStrings(a))
    Next

which reveals the expected 9 ENums:

UYVY
YV12
NV12
I420
BGRA
BGRX
RGBA
RGBX
UYVA

So I replicated the equivalent code from an example in https://github.com/WalkerKnapp/devolay on which I'm basing my development:

videoFrame.setFourCCType(DevolayFrameFourCCType.BGRX);

with:

videoFrame.FourCCType = EPos.ValueOf("BGRX")

Now I am trying to find a workaround for the equivalent code in the example which creates a videosender instance and names it:

        // Create the sender using the default settings, other than setting a name for the source.
        DevolaySender sender = new DevolaySender("Devolay Example Video");

DevolaySender is a class, with DevolaySender() and DevolaySender(SourceName as string) are constructors.

Currently I just declare it:

    Private SenderVideo As DevolaySender

because I don't know how to access the constructors,which seems to work OK and then:

    Dim s As String = SenderVideo.Source.SourceName
    Log(s)

logs the source name as the generic "java".

I just can't figure out how to name the source!

 

[bdunkleysmith]

In some example code which uses the Java library I'm trying to wrap for use in B4J, the NDI stream receiver is initialised with default settings by:

DevolayReceiver receiver = new DevolayReceiver();

In my B4J code I create a receiver instance like this:

Dim Receiver As DevolayReceiver

However I wish to set some receiver parameters, particularly the ColorFormat, by using the expanded version:

DevolayReceiver receiver = new DevolayReceiver(DevolaySource source, DevolayReceiver.ColorFormat colorFormat, int receiveBandwidth, boolean allowVideoFields, java.lang.String name)

I cannot figure out how to expand my simple:

Dim Receiver As DevolayReceiver

to enable creation of a receiver instance with all options specified.

The DevolayReceiver.java source java file (attached) commences:

package com.walker.devolay;

public class DevolayReceiver implements AutoCloseable {
    // Receive only metadata
    public static final int RECEIVE_BANDWIDTH_METADATA_ONLY = -10;
    // Receive only audio and metadata
    public static final int RECEIVE_BANDWIDTH_AUDIO_ONLY = 10;
    // Receive metadata, audio, and video at a lower bandwidth and resolution
    public static final int RECEIVE_BANDWIDTH_LOWEST = 0;
    // Receive metadata, audio, and video at full resolution
    public static final int RECEIVE_BANDWIDTH_HIGHEST = 100;

    public enum ColorFormat {
        /**
         * When there is an alpha channel, BGRA, otherwise BGRX
         */
        BGRX_BGRA(0),
        /**
         * When there is an alpha channel, BGRA, otherwise UYVY
         */
        UYVY_BGRA(1),
        /**
         * When there is an alpha channel, RGBA, otherwise BGRX
         */
        RGBX_RGBA(2),
        /**
         * When there is an alpha channel, RGBA, otherwise UYVY
         */
        UYVY_RGBA(3),
        /**
         * Use the fastest available color format for the incoming video signal.
         * Different platforms may vary in what format this will choose.
         *
         * When using this format, allow_video_fields is true, and a source supplies fields, individual fields will always be delivered.
         *
         * For most video sources on most platforms, the following will be used:
         *      No alpha channel: UYVY
         *      Alpha channel: UYVA
         */
        FASTEST(100),
        /**
         * Use the format that is closest to native for the incoming codec for the best quality.
         * Allows for receiving on 16bpp color from many sources
         *
         * When using this format, allow_video_fields is true, and a source supplies fields, individual fields will always be delivered.
         *
         * For most video sources on most platforms, the following will be used:
         *      No alpha channel: P216 or UYVY
         *      Alpha channel: PA16 or UYVA
         */
        BEST(101);

        private int id;

        ColorFormat(int id) {
            this.id = id;
        }
    }

    /**
     * Holds the reference to the NDIlib_send_instance_t object
     */
    private final long ndilibRecievePointer;

    public DevolayReceiver(DevolaySource source, ColorFormat colorFormat, int receiveBandwidth, boolean allowVideoFields, String name) {
        // TODO: Implement this forced reference more effectively
        Devolay.loadLibraries();

        this.ndilibRecievePointer = receiveCreate(source.structPointer, colorFormat.id, receiveBandwidth, allowVideoFields, name);
    }

    public DevolayReceiver(DevolaySource source) {
        this(source, ColorFormat.UYVY_BGRA, RECEIVE_BANDWIDTH_HIGHEST, true, null);
    }

    public DevolayReceiver() {
        // TODO: Implement this forced reference more effectively
        Devolay.loadLibraries();

        this.ndilibRecievePointer = receiveCreateDefaultSettings();
    }

and finishes with:

  // Native methods
    private static native long receiveCreate(long pSource, int colorFormat, int receiveBandwidth, boolean allowVideoFields, String name);
    private static native long receiveCreateDefaultSettings();
    private static native void receiveDestroy(long structPointer);

    private static native void receiveConnect(long structPointer, long pSource);

    private static native int receiveCaptureV2(long structPointer, long pVideoFrame, long pAudioFrame, long pMetadataFrame, int timeout);
    private static native void freeVideoV2(long structPointer, long pVideoFrame);
    private static native void freeAudioV2(long structPointer, long pAudioFrame);
    private static native void freeMetadata(long structPointer, long pMetadata);
    private static native boolean receiveSendMetadata(long structPointer, long pMetadataFrame);

    private static native void receiveGetPerformance(long structPointer, long pTotalPerformance, long pDroppedPerformance);

    private static native void receiveClearConnectionMetadata(long structPointer);
    private static native void receiveAddConnectionMetadata(long structPointer, long pMetadataFrame);

    public static native int receiveGetNoConnections(long structPointer);
    public static native String receiveGetWebControl(long structPointer);
}

Any suggestions on how I can do it with the compiled .jar as is, or what changes do I need to make to DevolayReceiver.java to achieve my objective?

 

[bdunkleysmith]

Well this seems to be the answer, at least in part:

Dim JORx As JavaObject
JORx.InitializeNewInstance("com.walker.devolay.DevolayReceiver", Array(Finder.CurrentSources(sourceNo)))
Dim Receiver As DevolayReceiver = JORx

because the in the Constructor DevolayReceiver(DevolaySource source) in DevolayReceiver.java the default ColorFormat has been set to what I want, ie. ColorFormat.BGRX_BGRA

I just need to get the syntax correct in regard to specifying colorFormat as ColorFormat so I can use the fully expanded Constructor DevolayReceiver(DevolaySource source, DevolayReceiver.ColorFormat colorFormat, int receiveBandwidth, boolean allowVideoFields, java.lang.String name)

The same approach solves another issue I mentioned at the end of post #8 where I was trying to name the NDI source:

Dim JOTx As JavaObject
JOTx.InitializeNewInstance("com.walker.devolay.DevolaySender", Array("BDS Send"))
Dim SenderVideo As DevolaySender = JOTx