B4J Question IP-camera reading frames

[peacemaker]

Hi, All

I'm trying to work with jOpenCV lib for custom object set recognition\identification.
But i have found that only reading the video stream and showing the frames by jOpenCV lib (without any processing) consumes lots of CPU time:

1. FullHD video reading without showing on the screen: 25% of CPU (AMD Ryzen5. Intel i7 was better, lower)
2. FullHD video reading with showing on the screen: ~35%
3. FullHD video reading with showing on the screen, and with some recognition processing: 100% of CPU with interval about 800...1200 ms between frames (iterations)
4. VGA video (640 x 480) reading with showing on the screen: 10%
5. VGA video (640 x 480) reading with showing on the screen and with some recognition processing: ~70 - 80% of CPU with interval about 200 ms between frames (iterations)

Only the latest 5th variant looks acceptable for usage.
The threading lib is ... hard for me to try to use here, especially when the debugging in the Debug mode is required.

So, i guess, maybe for the better speed - is there any lib to just read the frames from a video stream of IP-camera ? To read frames by the separate app (in the separate CPU thread) and save as just JPG pictures. And the main app with jOpenCV is just reading these pictures (and deletes) to process in its separate thread.

Any idea about such lib to read the IP-camera stream ?

 

[Magma]

If mjpeg... you can use the following..

https://www.b4x.com/android/forum/threads/b4x-class-mjpeg-decoder.73702/#content

or you can use: ffmpeg (with command line) need some tries...

 

[peacemaker]

Ooops, i have found the lib https://www.b4x.com/android/forum/threads/b4jvlcj-embed-vlc-mediaplayer-in-your-program-app.77098/ that works with VLC player.
VLC itself consumes almost zero of CPU, but 15% of GPU (FullHD local video file showing).
And this VLC lib wrapper in the demo B4J-app uses 20% of CPU and 25% of GPU (the same FullHD local video file showing).

And if to try "rtsp://" stream from an IP-camera - it consumes just 10% of CPU, will try to use.

 

[Magma]

OK good luck... please tell us your results... to know + will help all here !

 

[peacemaker]

but.... it's unclear the main - how to grab the frame...

https://videolan.videolan.me/vlc/gr...layer.html#ga612605f2e5c638d9f4ed59021d714bf0 - but no idea how to use it in B4J

The following limitations affect performance:

• Hardware video decoding acceleration will either be disabled completely, or require (relatively slow) copy from video/DSP memory to main memory.
• Sub-pictures (subtitles, on-screen display, etc.) must be blent into the main picture by the CPU instead of the GPU.
• Depending on the video format, pixel format conversion, picture scaling, cropping and/or picture re-orientation, must be performed by the CPU instead of the GPU.

This VLC warning gives idea that no much performance can be expected , in whole, comparing to ready jOpenCV lib

 

[Magma]

Hmmm
I have a simple idea... and you can use (i think) and a simple webview (no need 3rd party libs or tools)... you can capture as screenshot every some ms... with a timer...

Sub timer1_tick
timer1.Enabled=False
    Dim Rect1 As JavaObject
    Rect1.InitializeNewInstance("java.awt.Rectangle",Array(0,0,1920,1080))  'you can give only the rectangle you want !

    Dim Jo As JavaObject = Me
    Dim bbi() As Byte
    bbi=Jo.RunMethod("capturess",Array(Rect1))
timer1.Enabled=true
end sub

#if java

import java.awt.AWTException;
import java.awt.Dimension;
import java.awt.Rectangle;
import java.awt.Robot;
import java.awt.Toolkit;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.util.Objects;
import java.io.*;
import javax.imageio.ImageIO;
import org.apache.commons.codec.binary.Base64;
import com.github.zakgof:webp4j:0.0.2;

   public static byte[] capturess(final Rectangle rect) throws AWTException,
         IOException {
    Robot robot = new Robot();
    BufferedImage screenFullImage = robot.createScreenCapture(rect);
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
      ImageIO.write(screenFullImage, "jpg", baos);
      byte[] bytes = baos.toByteArray();
    //String bytesBase64 = Base64.encodeBase64String(bytes);
    return bytes;
   }

    // convert BufferedImage to byte[]
    public static byte[] toByteArray(BufferedImage bi, String format)
        throws IOException {

        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        ImageIO.write(bi, format, baos);
        byte[] bytes = baos.toByteArray();
        return bytes;

    }

#End If

 

[peacemaker]

a simple webview

web-browser clients do not support RTSP schema

 

[Magma]

web-browser clients do not support RTSP schema

[yet] ---> you have absolute right (for RTSP)

then you can use vlc lib and capture or use a middle tool web-page like youtube or something like that www.ipcamlive.com...


check here:
https://stackoverflow.com/questions/2245040/how-can-i-display-an-rtsp-video-stream-in-a-web-page
activex plugins (i hate them but here may be is a solution)

 

[xulihang]

About how to capture frames, check out the barcode reader example using vlcj: https://www.b4x.com/android/forum/t...e-qr-code-scanning-library.130728/post-822967

 

[peacemaker]

barcode reader example using vlcj

Yes, thanks for pointing, it works:

Spoiler: Code

private Sub getMp As JavaObject
    Dim jo As JavaObject=vlc.player
    Dim mp As JavaObject=jo.RunMethod("getMp",Null)
    Return mp
End Sub

Sub tim_Tick
    Dim bufferedImage As JavaObject = getMp.RunMethod("getSnapshot",Null)
    If bufferedImage.IsInitialized Then
        Dim SwingFXUtils As JavaObject
        SwingFXUtils.InitializeStatic("javafx.embed.swing.SwingFXUtils")
        Dim image As B4XBitmap = SwingFXUtils.RunMethod("toFXImage",Array(bufferedImage, Null))
        iv.SetImage(image)
    End If
End Sub

But the result is that ... this consumes the same 32...35% of CPU for FullHD (local video file) Anyway. The same as jOpenCV directly.

 

[Magma]

so pitty... i remember Linus... use a special card to do that... "things"

 

[peacemaker]

When i have all in a single B4J app with such high load of IP-cam stream and processing by jOpenCV - the PC is very irresponsible, hungs every second or more. Processing algorithm is hard (or unclear for me how) to make fast and smooth by interface at the same time.

But if:
1) one B4J app (grabber) using jOpenCV lib is just grabbing IP-cam's stream into ... a single JPG file-frame.
2) second B4J app (neuro-analyzer) using jOpenCV lib also is just waiting for this single JPG file-frame and process it.

each app can work on separate CPU core, and overall PC responsibility ... will it be better ?

 

[Magma]

...I think that the biggest problem is the analyzing/processing/identification... "check the youtube video..."

Yes you can possible with the taskmanager of windows change affinity/cpu.. but i am not sure that this will solve the problem... at last/the end is the same cpu...

Sometimes... cloud can be helpful... a vps with many cpus that will be used only for processing... and just wait the result at your local pc...

Or you can resize image or change compression / crop the image at specific region (taking only the face somehow) and play with your hardware...

 

[peacemaker]

I know that i have to adjust the recognition parameters and all this.
But anyway the PC must be non-hunging, responsible under this app - this is the main task.

 

[peacemaker]

Yes, app for just grabbing from IP-camera by OpenCV is OK, 5-7 times per second via LAN for such code:

Sub Process_Globals

    Dim mCore As OCVCore    'must be first declared !
    Dim mImgProc As OCVImgproc        ' Image processing class
    Dim mVideoCapture As OCVVideoCapture
    Dim mHighGui As OCVHighGui
    Dim OCV As OCVHelper
    Dim imcodecs As OCVImgcodecs
    Dim vio As OCVVideoio
    Dim vcMat As OCVMat    'current frame grabbed
    Public doProcessStream, isProcessingStream As Boolean
'-------------------------------------------------------------------
End Sub


Sub processStream(streamPath As String)
    If streamPath = "" Then
        mVideoCapture.Initialize2(0)    'default web-cam
    Else
        mVideoCapture.Initialize1(streamPath)    ' -1: capture from video file or URL
    End If
 
    If mVideoCapture.isOpened Then
        others.AddToLog("Frame size: " & mVideoCapture.get(vio.CAP_PROP_FRAME_WIDTH) & " x " & mVideoCapture.get(vio.CAP_PROP_FRAME_HEIGHT))
    Else
        others.AddToLog("Video stream capture trouble...")
        Sleep(0)
    End If
 
    doProcessStream = True
    mVideoCapture.set(vio.CV_CAP_PROP_BUFFERSIZE, 3) 'small frame buffer
 
    Do While doProcessStream
        Captured = False    'reset
        isProcessingStream = True
            For i = 0 To 3
                mVideoCapture.grab    'clear the buffer, skip 3 frames
            Next
            Captured = mVideoCapture.retrieve1(vcMat)    ' get the latest fresh frame only

        If Captured Then
            Dim img As Image = OpenCVutils.MatToBitmap(vcMat)
            others.Save_Picture_JPG(GrabbingFolder, GrabbingFile, img)
            others.AddToLog("Saved")
            Sleep(0)
        Else
            Log("Capture error: " & streamPath)
            isProcessingStream = False
            doProcessStream = False    'interrupt processing
            mVideoCapture.release
            Run_NN    'restart
            Return
        End If
    Loop
 
    mVideoCapture.release
    isProcessingStream = False
    Log("CAM released")
End Sub

Sub Run_NN
    doProcessStream = False    'interrupt processing
    CallSub2(Me, "processStream", SourceURL)
End Sub

Frame was 1280 x 720, and CPU is used just for 3...5% !!!
So, need to make some recognition of the grabbed JPG frame, but in a second app.

 

[Magma]

...
            For i = 0 To 3
                mVideoCapture.grab    'clear the buffer, skip 3 frames
            Next
    ...

can i ask something ?

that isn't grabbing 3 times ???? - if you grab out of loop (without for..next)... wouldn;t be faster ? ...2% of cpu ?

 

[Magma]

So, need to make some recognition of the grabbed JPG frame, but in a second app.

First of all crop - don;t forget to crop at the window-region + you want check for recognition !

0) resize the picture to smaller (???)
1) crop at Window-region
2) Grayscale the jpg (make it 8bit too) - sometimes colors not needed...
3) compress if possible nobody needs an 100% quality image... 70~80% works for all... try

 

[peacemaker]

@Magma, yes, i use all these points.
Trouble is to make:
1) responsible PC under all these neuro-apps
2) recognitions: to reach 2-3 times per second at least, but...
3) but many examples are to be recognized - it makes all very hard

 

[Magma]

...also...

are you using "...CallSubDelayed" ?
inside loops "Sleep(0)" command?

hmmm...

 

[peacemaker]

that isn't grabbing 3 times ???? - if you grab out of loop (without for..next)... wouldn;t be faster ? ...2% of cpu ?

jOpenCV with IP-cameras

Hi, All After long trying and googling i have found that OpenCV with IP-cameras must work by special way. Regular way is to use "mVideoCapture.read(vcMat)", as recommended by the lib author (again ton of Thanks to him !), it's OK for stable stream of USB-camera: Video: VideoCapture class...

www.b4x.com